Method and system for generating and verifying program formal specification based on large language model

Abstract

The application discloses a program formalization specification generation and verification method and system based on a large language model, relates to the field of program code formalization verification, and has the powerful computing capacity of the large language model, can efficiently generate a large amount of formalization specifications in a short time, and significantly improves the efficiency of formalization specification generation. Meanwhile, the generated formalization specifications are verified by combining a static verification method and a fuzzy test method, the specifications that fail in static verification are divided into static verification error specifications and timeout specifications, the fuzzy test is further performed on the timeout specifications, the problem of insufficient formalization specification generation caused by error feedback of the to-be-verified specifications due to the capacity limitation of a single verification method is reduced, and the success rate of formalization specification verification is improved.

Description

Technical Field

[0001] This invention relates to the field of formal verification of program code, and more specifically, to a method and system for generating and verifying formal program specifications based on a large language model. Background Technology

[0002] Formal verification of a program refers to proving the correctness and security of a computer program through a series of rigorous and systematic methods. Formal verification requires the construction of formal specifications to accurately describe the program's behavior and functionality. Currently, the construction of formal specifications mainly relies on program analysis methods such as abstract interpretation, inductive logical reasoning, symbolic execution, or data-driven methods for specification derivation and extraction. These methods require extensive professional knowledge and significant human effort, and are typically used for numerical programs, specific programs, or loop invariants. When dealing with multifunctional programs, such as those involving different data types (e.g., arrays, pointers) and code structures (e.g., loops, function calls), these methods often fall short.

[0003] After constructing formal specifications, their correctness, consistency, and implementability need to be verified. Currently, most formal specification verifications use a single static verification engine. However, a single static verification engine is prone to timeouts when verifying complex logic code, leading to false positives for correct specifications, and it is also prone to missing false positives for incorrect specifications. Summary of the Invention

[0004] The purpose of this invention is to improve the efficiency of generating formal specifications for programs, while also improving the quality of formal specification verification.

[0005] To achieve the above objectives, this invention provides a method for generating and verifying formal program specifications based on a large language model, the method comprising the following steps:

[0006] Step 1: Construct a program call graph based on the initial program code and expected behavior;

[0007] Step 2: Generate a prompt query statement for any node in the program call graph;

[0008] Step 3: Based on the suggested query statement, use the large language model to generate the corresponding node's specification to be verified;

[0009] Step 4: Insert the specification to be verified into the initial program code to obtain the program code to be verified. Perform static verification on the program code to be verified. If the static verification passes, output the program code to be verified as the correct program code containing the specification; if the static verification fails, proceed to step 5.

[0010] Step 5: Analyze the reasons for the static verification failure. If the reason for the static verification failure is that the specification to be verified only contains static verification error specifications that conflict with the initial program code, but does not contain timeout specifications that cause the static verification to time out, then delete the static verification error specifications in the program code to be verified, and return to Step 3 to update the specification to be verified until the output of correct program code containing the specifications or the number of updates to the specification to be verified reaches the preset limit.

[0011] If the static verification fails because the specification to be verified only contains the timeout specification and not the static verification error specification, then proceed to step 6.

[0012] Step 6: Perform fuzz testing on the program code to be verified. If the fuzz test passes, output the program code to be verified as the correct program code containing the specification.

[0013] If the fuzz test fails, the fuzz test error specification that caused the fuzz test to fail is deleted from the program code to be verified, and the process returns to step 3 to update the specification to be verified until the correct program code containing the specification is output or the number of updates to the specification to be verified reaches a preset limit.

[0014] The large language model refers to a deep learning model trained on a large amount of text data, encompassing various programming languages ​​and styles. This enables the large language model to recognize and understand various code structures and patterns, thereby generating high-quality formal specifications. This invention utilizes the large language model to generate corresponding node specifications based on the suggested query statement, achieving the acquisition of a large number of specifications in a short time. Compared to manual methods, this significantly improves the efficiency of specification generation. Furthermore, for complex, multifunctional programs, the large language model can automatically generate corresponding specifications, compensating for the shortcomings of traditional program analysis techniques in this regard. In addition, this invention employs a combination of static verification and fuzz testing to verify the specifications. Specifically, static verification refers to scanning and analyzing code using a static verification engine without executing the program code. Static verification can discover potential errors and vulnerabilities in the code. Static verification is performed first; if it passes, the specification is considered correct, and the correct program code containing the specification is directly output. If static validation fails, analyze the cause: if it's due to a static validation error, delete the error, iterate and update the specification to be validated using a large language model based on the previous specification, and then validate again; if it's due to a timeout, ignore the timeout specification and perform fuzzing directly. This is because timeout specifications may contain some correct specifications, and simply deleting them as errors would result in insufficient formal specification of the final program, leading to validation failure. Fuzzing, as a supplement to static validation, can identify and delete fuzzing errors in the timeout specification, ensuring the correctness of the final specification.

[0015] This invention combines static verification and fuzz testing, which to some extent reduces the problem of insufficient formal specification generation caused by the limitations of a single verification method leading to erroneous feedback on the specification to be verified, or the problem of missing reports of erroneous specifications, thereby improving the success rate of formal specification verification.

[0016] Furthermore, in step 5, if the static verification fails because the specification to be verified contains both the static verification error specification and the timeout specification, then step 7 is executed.

[0017] Step 7: Delete the static verification error specification from the program code to be verified to obtain timeout program code containing the timeout specification;

[0018] Perform a fuzz test on the timeout program code. If the fuzz test passes, return to step 3 to update the specification to be verified until the correct program code containing the specification is output or the number of updates to the specification to be verified reaches a preset limit.

[0019] If the fuzz test fails, delete the fuzz test error specification that caused the fuzz test to fail, and return to step 3 to update the specification to be verified until the correct program code containing the specification is output or the number of times the specification to be verified is generated reaches the preset limit.

[0020] In some cases, the specification to be verified that causes static verification to fail may contain both the static verification error specification and the timeout specification. In this situation, the static verification error specification is first deleted, while the timeout specification is left in the initial program for subsequent fuzzing. Instead of updating the specification to be verified, the updated specification is inserted into the initial program to continue the static verification-fuzzing process. This is because developers can use the fuzzing results of the timeout program code to selectively adjust the instructions input to the large language model to generate a more accurate specification, reducing the number of iterations and improving verification efficiency.

[0021] Furthermore, in step 1, the construction of the program call graph based on the initial program code and expected behavior includes:

[0022] The expected behavior is inserted into the initial program code as a code comment to form program code with expected behavior comments;

[0023] The program code annotated with the expected behavior is parsed, the function containing the expected behavior is selected as the entry point, the abstract syntax tree of the source code is traversed, user-defined functions are identified, the call relationship between user-defined functions is established, and loop structures are identified.

[0024] A program call graph is constructed based on user-defined functions, the call relationships between user-defined functions, and the loop structure.

[0025] The expected behavior refers to the behavior exhibited by the program during runtime according to the functions and logic specified in its design. This invention inserts the expected behavior into the initial program code. In subsequent verification of the formal specifications, the correctness of the specifications can be verified by comparing the expected behavior in the comments with the actual execution behavior of the code. This invention uses functions containing the expected behavior as entry points to construct a program call graph, which makes it easier to trace and analyze the program's structure and execution flow, facilitating developers to accurately construct the suggested query statements.

[0026] Furthermore, in step 2, the prompt query statement includes a query statement and several prompt phrases;

[0027] The query statement includes the code captured by the corresponding node, the existing specifications of the corresponding node, and the specifications of the direct child nodes of the corresponding node;

[0028] The prompt phrase includes sample code and the corresponding sample specification.

[0029] If a corresponding node does not have an existing specification, or if the corresponding node does not have direct child node specifications, the query statement will indicate that the existing specifications for the corresponding node or the direct child node specifications for the corresponding node are blank. The query statement is used to provide the large language model with the code information and related examples for that node, enabling the large language model to generate the specification to be verified more accurately.

[0030] This invention parses and decomposes the initial program code through a program call graph, and generates a prompt query statement for each node in the program call graph. This allows the large language model to focus on the small program segment corresponding to the node to generate specifications, thereby reducing the complexity of specification generation and improving the quality and efficiency of specification generation.

[0031] Furthermore, the prompt phrase is obtained in the following way:

[0032] Step 201: Collect known program code;

[0033] Step 202: Annotate the known program code to obtain a training set;

[0034] Step 203: Construct a first code structure recognition model, and train the first code structure recognition model based on the training set to obtain a second code structure recognition model;

[0035] Step 204: Input the code captured by the corresponding node into the second code structure recognition model, and output the first similar program code with the highest structural similarity to the code captured by the corresponding node, as well as the corresponding similarity value;

[0036] Step 205: Determine whether the first similar program code passes formal verification. If it passes, determine whether the similarity value between the first similar program code and the code captured by the corresponding node exceeds the first threshold.

[0037] If the first threshold is exceeded, the first similar program code is used as the example code, and the formal specification corresponding to the first similar program code is used as the example specification.

[0038] If the first similar program code fails formal verification, then proceed to step 206;

[0039] Step 206: Input the first similar program code into the second code structure recognition model, and output the second similar program code with the highest structural similarity to the first similar program code, as well as the corresponding similarity value;

[0040] Determine whether the second similar program code passes formal verification. If it passes, determine whether the similarity value between the second similar program code and the first similar code exceeds a second threshold. If it exceeds the second threshold, use the second similar program code as the example code and the formal specification corresponding to the second similar program code as the example specification.

[0041] If the second similar program code fails formal verification, then step 206 is repeated for the second similar program code until the code structure recognition model outputs code that passes formal verification and whose structural similarity with the code captured by the corresponding node exceeds the corresponding threshold as the example code.

[0042] The higher the structural similarity between the example code and the initial code, the higher the probability that the large language model will output the correct specification to be verified. However, for a given initial code, manually finding a highly similar and verified code sample is often quite difficult. This invention uses a network model to automatically search, which can effectively improve search efficiency. However, in actual development, many known programs can run normally without formal verification. Therefore, the first similar program code with the highest structural similarity to the code captured by the corresponding node, output by the second code structure recognition model, may not have an existing formal specification. In this case, the first similar program code is input into the second code structure recognition model again to obtain the second similar program code with the highest similarity to the first similar program code. Then, it is determined whether the second similar program code passes formal verification, that is, whether a formal specification exists. By repeating this process, the code with the highest structural similarity to the code captured by the corresponding node is finally obtained as the example code, and its corresponding specification is used as the example specification.

[0043] Furthermore, the annotation of the verified existing program code includes: annotating the structural and functional features of the verified existing program code;

[0044] The structural features include functions, classes, modules, and control flow;

[0045] The functional features include functional description, business logic, exception handling, and performance.

[0046] In addition to annotating structural features, this invention also annotates functional features, which helps the second code structure recognition model to better parse the code structure and helps it to more accurately identify and understand the code.

[0047] Furthermore, in step 2, generating a prompt query statement for any node in the program call graph includes:

[0048] Starting from the leaf node of the program call graph, traverse backwards to the root node, generating the prompt query statement for each node.

[0049] In a program call graph, leaf nodes typically represent specific function calls or operations, and they do not have further sub-calls. Root nodes are usually the program's entry point or key modules. By traversing backward from the leaf nodes to the root node, it's possible to ensure that all necessary paths are covered during the traversal, facilitating a more comprehensive understanding of the program's logic and flow.

[0050] Furthermore, in step 3, generating the corresponding node's verification specification based on the suggested query statement using a large language model includes:

[0051] Input the prompt query statement and explanation statement into the large language model, and output the specification to be verified.

[0052] The explanatory statements are used to guide the large language model, helping it to better understand the problems that need to be addressed, thereby improving the accuracy of the output specifications to be verified.

[0053] This invention also provides a system for generating and verifying formal program specifications based on a large language model, the system comprising:

[0054] The program analysis unit is used to construct a program call graph based on the initial program code and expected behavior;

[0055] The prompt query unit is used to generate prompt query statements for any node in the program call graph;

[0056] The specification generation unit is used to generate the specification to be verified for the corresponding node based on the prompt query statement using a large language model;

[0057] A verification unit is used to verify the specification to be verified.

[0058] Specifically, the verification of the specification to be verified includes:

[0059] The specification to be verified is inserted into the initial program code to obtain the program code to be verified. Static verification is performed on the program code to be verified. If the static verification passes, the program code to be verified is output as the correct program code containing the specification.

[0060] If the static verification fails, the reason for the failure is analyzed. If the reason for the failure is that the specification to be verified only contains static verification error specifications that conflict with the initial program code, but does not contain timeout specifications that cause the static verification to time out, the static verification error specifications are deleted from the program code to be verified, and the specification generation unit is used to update the specification to be verified until the correct program code containing the specification is output or the number of updates to the specification to be verified reaches a preset limit.

[0061] If the static verification fails because the specification to be verified only contains the timeout specification and not the static verification error specification, then the program code to be verified is subjected to fuzz testing. If the fuzz test passes, the program code to be verified is output as the correct program code containing the specification.

[0062] If the fuzz test fails, the fuzz test error specification that caused the fuzz test to fail is deleted from the program code to be verified, and the specification to be verified is updated using the specification generation unit until the correct program code containing the specification is output or the number of updates to the specification to be verified reaches a preset limit.

[0063] One or more technical solutions provided by this invention have at least the following technical effects or advantages:

[0064] 1. This invention proposes a method for generating and verifying formal specifications for programs based on a large language model. This method relies on the powerful computing capabilities of the large model to efficiently generate a large number of formal specifications in a short time, significantly improving the efficiency of formal specification generation. At the same time, the large language model can easily cope with complex and multifunctional programs, making up for the shortcomings of traditional program analysis methods in effectively constructing formal specifications when facing such complex programs.

[0065] 2. This invention parses and decomposes the initial program code through a program call graph, and generates a prompt query statement for each node of the program call graph. This allows the large language model to focus on the small program segment corresponding to the node to generate the corresponding specification, thereby reducing the complexity of specification generation and improving the quality and efficiency of specification generation.

[0066] 3. This invention integrates static verification and fuzz testing to verify the generated formal specifications. This reduces the problem of insufficient formal specification generation caused by the limitations of a single verification method, or the problem of missing erroneous specifications, thus improving the success rate of formal specification verification. At the same time, based on the verification results of the two methods, the explanatory statements input to the large language model can be comprehensively adjusted, which is conducive to the output of more accurate specifications by the large language model. Attached Figure Description

[0067] The accompanying drawings, which are provided to further illustrate embodiments of the invention and constitute a part of this invention, are not intended to limit the scope of the invention.

[0068] Figure 1 This is a flowchart illustrating a method for generating and verifying formal program specifications based on a large language model, as described in this invention.

[0069] Figure 2 This is a schematic diagram of the composition of a program formal specification generation and verification system based on a large language model in this invention. Detailed Implementation

[0070] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, where there is no conflict, the embodiments of the present invention and the features thereof can be combined with each other.

[0071] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0072] Example 1

[0073] Please refer to Figure 1 Embodiment 1 of the present invention provides a method for generating and verifying formal specifications of programs based on a large language model. The method includes the following steps:

[0074] Step 1: Construct a program call graph based on the initial program code and expected behavior;

[0075] Step 2: Generate a prompt query statement for any node in the program call graph;

[0076] Step 3: Based on the suggested query statement, use the large language model to generate the corresponding node's specification to be verified;

[0077] Step 4: Insert the specification to be verified into the initial program code to obtain the program code to be verified. Perform static verification on the program code to be verified. If the static verification passes, output the program code to be verified as the correct program code containing the specification; if the static verification fails, proceed to step 5.

[0078] Step 5: Analyze the reasons for the static verification failure. If the reason for the static verification failure is that the specification to be verified only contains static verification error specifications that conflict with the initial program code, but does not contain timeout specifications that cause the static verification to time out, then delete the static verification error specifications in the program code to be verified, and return to step 3 to update the specification to be verified until the output of correct program code containing the specifications or the number of updates to the specification to be verified reaches the preset limit.

[0079] If the static verification fails because the specification to be verified only contains the timeout specification and not the static verification error specification, then proceed to step 6.

[0080] Step 6: Perform fuzz testing on the program code to be verified. If the fuzz test passes, output the program code to be verified as the correct program code containing the specification.

[0081] If the fuzz test fails, the fuzz test error specification that caused the fuzz test to fail is deleted from the program code to be verified, and the process returns to step 3 to update the specification to be verified until the correct program code containing the specification is output or the number of updates to the specification to be verified reaches a preset limit.

[0082] The large language model refers to a deep learning model trained with a large amount of text data, such as MetaLIama3-8B and ChatGPT. Those skilled in the art can determine the choice of large language model according to actual needs, and this embodiment does not make specific limitations.

[0083] Specifically, the specification to be verified can be inserted into the initial program code through code comments; static verification refers to the method of scanning and analyzing code through a static verification engine without executing the program code. Specific static verification engines can be selected such as Frama-c wp, Clang Static Analyzer, etc., and those skilled in the art can determine them according to actual needs.

[0084] There are two reasons for the failure of static verification. One is that the specification to be verified does not match the initial program code, and there is a deviation or conflict between the two. This type of specification is the static verification error specification, which can be directly analyzed by the static verification engine. The other is that the specification is too complex, which makes the static verification engine unable to process it and causes the verification to time out. This type of specification is the timeout specification.

[0085] The fuzzing mentioned here refers to detecting potential errors or vulnerabilities by providing a program with a large amount of random or semi-random input data and monitoring the program's response. In this embodiment, the fuzzing is performed using a fuzzing engine, such as AFL or Peach Fuzzer, depending on the actual needs.

[0086] In step 5, if the static verification fails because the specification to be verified contains both the static verification error specification and the timeout specification, then step 7 is executed.

[0087] Step 7: Delete the static verification error specification from the program code to be verified to obtain timeout program code containing the timeout specification;

[0088] Perform a fuzz test on the timeout program code. If the fuzz test passes, return to step 3 to update the specification to be verified until the correct program code containing the specification is output or the number of updates to the specification to be verified reaches a preset limit.

[0089] If the fuzz test fails, delete the fuzz test error specification that caused the fuzz test to fail, and return to step 3 to update the specification to be verified until the correct program code containing the specification is output or the number of times the specification to be verified is generated reaches the preset limit.

[0090] In step 1, constructing the program call graph based on the initial program code and expected behavior includes:

[0091] The expected behavior is inserted into the initial program code as a code comment to form program code with expected behavior comments;

[0092] The program code annotated with the expected behavior is parsed, the function containing the expected behavior is selected as the entry point, the abstract syntax tree of the source code is traversed, user-defined functions are identified, the call relationship between user-defined functions is established, and loop structures are identified.

[0093] A program call graph is constructed based on user-defined functions, the call relationships between user-defined functions, and the loop structure.

[0094] A program call graph is a graph that represents the call relationships between functions in a program. Nodes in the graph represent methods, and edges represent call relationships. For example, consider an initial program code containing a binary search function that searches for a target value in a sorted array of integers. This function works by iteratively halving the search range until the target value is found or it is determined that the target value does not exist.

[0095] The specific process is as follows:

[0096] Determine the starting and ending indexes of the search range;

[0097] Calculate the middle index and compare the middle value of the array corresponding to the middle index with the target value:

[0098] If the median value is less than the target value, it means that the target value is to the right of the current median value. Therefore, the starting index is updated to the median index plus 1.

[0099] If the median value is greater than the target value, it means that the target value is to the left of the current median value. Therefore, the index to be updated is the median index minus 1.

[0100] If the intermediate value equals the target value, it means the target value has been found, and the function returns the current intermediate index.

[0101] Repeat the above steps until the starting index is greater than the ending index. At this point, it means that the target value is not in the array, and the function returns -1 to indicate that the search failed.

[0102] The following three expected behaviors are defined for this binary search function:

[0103] Expected behavior 1: For an array {1, 2, 3, 4, 5, 6} containing 6 elements, when the function is called to find the last value 6 in the array, the expected result is index 5 (index starts from 0);

[0104] Expected behavior 2: When the function is called to look up 10, the expected result is -1;

[0105] Expected behavior 3: When a function is called to look up a randomly generated value that is definitely in the array, the expected result is a valid index, not -1.

[0106] The expected behavior is inserted into the initial code as a code comment, and the function containing the expected behavior is used as the entry point to construct a program call graph, which contains a node representing the above loop search method.

[0107] In step 2, the prompt query statement includes a query statement and several prompt phrases;

[0108] The query statement includes the code captured by the corresponding node, the existing specifications of the corresponding node, and the specifications of the direct child nodes of the corresponding node;

[0109] The prompt phrase includes sample code and the corresponding sample specification.

[0110] Taking the above initial program as an example, this embodiment constructs a prompt query statement for the node corresponding to the loop search method in the program call diagram. Specifically, since the loop node has no existing specification and no direct child node specification, the query statement is the code captured by the node.

[0111] The example code provides a function to find the maximum value in array 'a'. The main flow of this function is as follows:

[0112] After the function starts, it first initializes a variable named `max` to track and record the maximum value currently being traversed in array `a`. This `max` variable is initially set to the first element of array `a`. Simultaneously, the function also includes a loop counter `i`, which starts at 0 and is used to access each element of array `a` one by one.

[0113] Once in the loop iteration phase, the function checks each element in array `a` one by one. Specifically, it compares the current value of `max` with the element in array `a` indicated by the counter `i`. If `max` is found to be less than the current element, then the value of `max` is updated to the value of that element. This process continues until the value of the loop counter `i` reaches the length `n` of array `a`, at which point the loop iteration ends.

[0114] Finally, the function returns the value of max, which represents the maximum value in array a.

[0115] Example specifications include:

[0116] Example specification 1: i is between 0 and n (n is the number of elements in array a);

[0117] Example specification 2: At the end of each loop, max is at least as large as any one of the first i elements in array a;

[0118] Example specification 3: i is incremented in each iteration;

[0119] Example specification 4: max may be updated in each iteration.

[0120] Therefore, the suggested query for this node is: the initial program code itself, the example code mentioned above, and example specifications 1-4.

[0121] The prompt phrase is obtained in the following way:

[0122] Step 201: Collect known program code;

[0123] Step 202: Annotate the known program code to obtain a training set;

[0124] Step 203: Construct a first code structure recognition model, and train the first code structure recognition model based on the training set to obtain a second code structure recognition model;

[0125] Step 204: Input the code captured by the corresponding node into the second code structure recognition model, and output the first similar program code with the highest structural similarity to the code captured by the corresponding node, as well as the corresponding similarity value;

[0126] Step 205: Determine whether the first similar program code passes formal verification. If it passes, determine whether the similarity value between the first similar program code and the code captured by the corresponding node exceeds the first threshold.

[0127] If the first threshold is exceeded, the first similar program code is used as the example code, and the formal specification corresponding to the first similar program code is used as the example specification.

[0128] If the first similar program code fails formal verification, then proceed to step 206;

[0129] Step 206: Input the first similar program code into the second code structure recognition model, and output the second similar program code with the highest structural similarity to the first similar program code, as well as the corresponding similarity value;

[0130] Determine whether the second similar program code passes formal verification. If it passes, determine whether the similarity value between the second similar program code and the first similar code exceeds a second threshold. If it exceeds the second threshold, use the second similar program code as the example code and the formal specification corresponding to the second similar program code as the example specification.

[0131] If the second similar program code fails formal verification, then step 206 is repeated for the second similar program code until the second code structure recognition model outputs code that passes formal verification and whose structural similarity with the code captured by the corresponding node exceeds the corresponding threshold as example code.

[0132] If the structural similarity between the known code output by the second code structure recognition model and the code captured by the corresponding node cannot reach the corresponding threshold, it indicates that there is no known example code for reference, and a formal specification needs to be constructed through other means.

[0133] The known program code and its corresponding formal specifications can be obtained from code hosting platforms such as GitHub and GitLab using web crawling techniques. The code structure can be analyzed and extracted using methods described in existing technology CN115080406A, employing automated tools such as MooreData and Prodigy, supplemented by manual feature annotation. A suitable deep learning model architecture, such as a Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), or Transformer, is selected based on the actual situation to construct the first code structure recognition model. For calculating code similarity, existing technology CN117319091A can be used as a reference. The first and second thresholds can be set according to the actual situation; this invention does not limit this. Preferably, the thresholds increase accordingly with the number of iterations to ensure that the final example code still has a high similarity to the code captured by the corresponding node.

[0134] The annotation of the verified existing program code includes: annotating the structural and functional features of the verified existing program code;

[0135] The structural features include functions, classes, modules, and control flow;

[0136] The functional features include functional description, business logic, exception handling, and performance.

[0137] In step 2, generating a prompt query statement for any node in the program call graph includes:

[0138] Starting from the leaf node of the program call graph, traverse backwards to the root node, generating the prompt query statement for each node.

[0139] In step 3, generating the corresponding node's verification specification based on the prompt query statement using a large language model includes:

[0140] Input the prompt query statement and explanation statement into the large language model, and output the specification to be verified.

[0141] The specific explanatory statements can be constructed by those skilled in the art according to the actual situation, and can also be adjusted according to the feedback results of the large language model. This invention does not limit this.

[0142] Taking the initial program described above as an example, this embodiment inputs the prompt query statement into the MetaLIama3-8B large language model and adds corresponding explanatory statements. These explanatory statements include, but are not limited to, the task and requirements, the correspondence between the example code and the example specification, the similarity between the initial program code and the example code, and background information about the initial program code. Finally, MetaLIama3-8B automatically generates the following specifications:

[0143] Output specification 1: In each step of the loop, the lower bound index is always non-negative, while the upper bound index is always less than the length n of the array;

[0144] Output specification 2: If there exists an element in the array that is equal to the target value v, and this element is located at index k, then index k must be within the current search interval.

[0145] Example 2

[0146] Please refer to Figure 2 Embodiment 2 of the present invention provides a system for generating and verifying formal program specifications based on a large language model, characterized in that the system includes:

[0147] The program analysis unit is used to construct a program call graph based on the initial program code and expected behavior;

[0148] The prompt query unit is used to generate prompt query statements for any node in the program call graph;

[0149] The specification generation unit is used to generate the specification to be verified for the corresponding node based on the prompt query statement using a large language model;

[0150] A verification unit is used to verify the specification to be verified.

[0151] Specifically, the verification of the specification to be verified includes:

[0152] The specification to be verified is inserted into the initial program code to obtain the program code to be verified. Static verification is performed on the program code to be verified. If the static verification passes, the program code to be verified is output as the correct program code containing the specification.

[0153] If the static verification fails, the reason for the failure is analyzed. If the reason for the failure is that the specification to be verified only contains static verification error specifications that conflict with the initial program code, but does not contain timeout specifications that cause the static verification to time out, the static verification error specifications are deleted from the program code to be verified, and the specification generation unit is used to update the specification to be verified until the correct program code containing the specification is output or the number of updates to the specification to be verified reaches a preset limit.

[0154] If the static verification fails because the specification to be verified only contains the timeout specification and not the static verification error specification, then the program code to be verified is subjected to fuzz testing. If the fuzz test passes, the program code to be verified is output as the correct program code containing the specification.

[0155] If the fuzz test fails, the fuzz test error specification that caused the fuzz test to fail is deleted from the program code to be verified, and the specification to be verified is updated using the specification generation unit until the correct program code containing the specification is output or the number of updates to the specification to be verified reaches a preset limit.

[0156] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0157] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for generating and verifying formal program specifications based on a large language model, characterized in that, The method includes the following steps: Step 1: Construct a program call graph based on the initial program code and expected behavior; Step 2: Generate a prompt query statement for any node in the program call graph; Step 3: Based on the suggested query statement, use the large language model to generate the corresponding node's specification to be verified; Step 4: Insert the specification to be verified into the initial program code to obtain the program code to be verified. Perform static verification on the program code to be verified. If the static verification passes, output the program code to be verified as the correct program code containing the specification; if the static verification fails, proceed to step 5. Step 5: Analyze the reasons for the static verification failure. If the reason for the static verification failure is that the specification to be verified only contains static verification error specifications that conflict with the initial program code, but does not contain timeout specifications that cause the static verification to time out, then delete the static verification error specifications in the program code to be verified, and return to Step 3 to update the specification to be verified until the output of correct program code containing the specifications or the number of updates to the specification to be verified reaches the preset limit. If the static verification fails because the specification to be verified only contains the timeout specification and not the static verification error specification, then proceed to step 6. Step 6: Perform fuzz testing on the program code to be verified. If the fuzz test passes, output the program code to be verified as the correct program code containing the specification. If the fuzz test fails, the fuzz test error specification that caused the fuzz test to fail is deleted from the program code to be verified, and the process returns to step 3 to update the specification to be verified until the correct program code containing the specification is output or the number of updates to the specification to be verified reaches a preset limit.

2. The method for generating and verifying formal program specifications based on a large language model according to claim 1, characterized in that, In step 5, if the static verification fails because the specification to be verified contains both the static verification error specification and the timeout specification, then step 7 is executed. Step 7: Delete the static verification error specification from the program code to be verified to obtain timeout program code containing the timeout specification; Perform a fuzz test on the timeout program code. If the fuzz test passes, return to step 3 to update the specification to be verified until the correct program code containing the specification is output or the number of updates to the specification to be verified reaches a preset limit. If the fuzz test fails, delete the fuzz test error specification that caused the fuzz test to fail, and return to step 3 to update the specification to be verified until the correct program code containing the specification is output or the number of times the specification to be verified is generated reaches the preset limit.

3. The method for generating and verifying formal program specifications based on a large language model according to claim 1, characterized in that, In step 1, constructing the program call graph based on the initial program code and expected behavior includes: The expected behavior is inserted into the initial program code as a code comment to form program code with expected behavior comments; The program code annotated with the expected behavior is parsed, the function containing the expected behavior is selected as the entry point, the abstract syntax tree of the source code is traversed, user-defined functions are identified, the call relationship between user-defined functions is established, and loop structures are identified. A program call graph is constructed based on user-defined functions, the call relationships between user-defined functions, and the loop structure.

4. The method for generating and verifying formal program specifications based on a large language model according to claim 1, characterized in that, In step 2, the prompt query statement includes a query statement and several prompt phrases; The query statement includes the code captured by the corresponding node, the existing specifications of the corresponding node, and the specifications of the direct child nodes of the corresponding node; The prompt phrase includes sample code and the corresponding sample specification.

5. The method for generating and verifying formal program specifications based on a large language model according to claim 4, characterized in that, The prompt phrase is obtained in the following way: Step 201: Collect known program code; Step 202: Annotate the known program code to obtain a training set; Step 203: Construct a first code structure recognition model, and train the first code structure recognition model based on the training set to obtain a second code structure recognition model; Step 204: Input the code captured by the corresponding node into the second code structure recognition model, and output the first similar program code with the highest structural similarity to the code captured by the corresponding node, as well as the corresponding similarity value; Step 205: Determine whether the first similar program code passes formal verification. If it passes, determine whether the similarity value between the first similar program code and the code captured by the corresponding node exceeds the first threshold. If the first threshold is exceeded, the first similar program code is used as the example code, and the formal specification corresponding to the first similar program code is used as the example specification. If the first similar program code fails formal verification, then proceed to step 206; Step 206: Input the first similar program code into the second code structure recognition model, and output the second similar program code with the highest structural similarity to the first similar program code, as well as the corresponding similarity value; Determine whether the second similar program code passes formal verification. If it passes, determine whether the similarity value between the second similar program code and the first similar program code exceeds the second threshold. If the second threshold is exceeded, the second similar program code will be used as the example code, and the formal specification corresponding to the second similar program code will be used as the example specification. If the second similar program code fails formal verification, then step 206 is repeated for the second similar program code until the code structure recognition model outputs code that passes formal verification and whose structural similarity with the code captured by the corresponding node exceeds the corresponding threshold as the example code.

6. The method for generating and verifying formal program specifications based on a large language model according to claim 5, characterized in that, Annotating the verified existing program code includes annotating the structural and functional features of the verified existing program code; The structural features include functions, classes, modules, and control flow; The functional features include functional description, business logic, exception handling, and performance.

7. The method for generating and verifying formal program specifications based on a large language model according to claim 1, characterized in that, In step 2, generating a prompt query statement for any node in the program call graph includes: Starting from the leaf node of the program call graph, traverse backwards to the root node, generating the prompt query statement for each node.

8. The method for generating and verifying formal program specifications based on a large language model according to claim 1, characterized in that, In step 3, generating the verification specification for the corresponding node using a large language model based on the suggested query statement includes: Input the prompt query statement and explanation statement into the large language model, and output the specification to be verified.

9. A system for generating and verifying formal program specifications based on a large language model, characterized in that, The system includes: The program analysis unit is used to construct a program call graph based on the initial program code and expected behavior; The prompt query unit is used to generate prompt query statements for any node in the program call graph; The specification generation unit is used to generate the specification to be verified for the corresponding node based on the prompt query statement using a large language model; A verification unit is used to verify the specification to be verified. Specifically, the verification of the specification to be verified includes: The specification to be verified is inserted into the initial program code to obtain the program code to be verified. Static verification is performed on the program code to be verified. If the static verification passes, the program code to be verified is output as the correct program code containing the specification. If the static verification fails, the reason for the failure is analyzed. If the reason for the failure is that the specification to be verified only contains static verification error specifications that conflict with the initial program code, but does not contain timeout specifications that cause the static verification to time out, the static verification error specifications are deleted from the program code to be verified, and the specification generation unit is used to update the specification to be verified until the correct program code containing the specification is output or the number of updates to the specification to be verified reaches a preset limit. If the static verification fails because the specification to be verified only contains the timeout specification and not the static verification error specification, then the program code to be verified is subjected to fuzz testing. If the fuzz test passes, the program code to be verified is output as the correct program code containing the specification. If the fuzz test fails, the fuzz test error specification that caused the fuzz test to fail is deleted from the program code to be verified, and the specification to be verified is updated using the specification generation unit until the correct program code containing the specification is output or the number of updates to the specification to be verified reaches a preset limit.

Images