Loop invariant template generation method based on UnsatCore and reinforcement learning

Abstract

The invention discloses a loop invariant template generation method based on Unsat Core and reinforcement learning. The method comprises the following steps: S1, establishing a loop invariant template generator based on autonomous reinforcement learning; through reinforcement learning, learning how to generate an effective loop invariant template under the information guidance of Unsat Core; s2, establishing a loop invariant template solver based on an SMT solver; solving a template coefficient to generate an effective cyclic invariant candidate; when the solution fails, the Unsat Core used for providing the template cannot meet the kernel; s3, a template attribution analyzer of the Unsat Core is established; the method is used for analyzing an unsatisfactory kernel provided by an SMT, attributing the unsatisfactory kernel to a specific generation process of an invariant template, and guiding a reinforcement learning model to carry out an attribution learning and iterative template improvement process. The method has the advantages of simple principle, high stability, convenient and efficient operation and the like.

Description

Technical Field

[0001] The present invention mainly relates to the field of program correctness verification in software technology, and particularly refers to a method for generating loop invariant templates based on UnsatCore and reinforcement learning. Background Art

[0002] "Program verification" is crucial for ensuring the correctness and security of software. However, current program verification technologies are often limited by loop statements in the program. To verify a program with loop statements, a loop invariant needs to be generated to summarize the potentially infinite number of states in a loop.

[0003] In the field of program correctness verification, formal verification of some properties of a program based on axiomatic semantics is a common technical means. In Hoare logic, for a section of a program, it is regarded as a process of modifying the values of variables. For example, an assignment statement x := 1 will assign a variable x the value 1. No matter what its value was before, after executing this statement, its value must be 1. A complex program can be regarded as the accumulation of the effects of a set of basic program statements on variable modification.

[0004] The correctness verification of a program statement (denoted as S) means that: when a set of variables satisfies a formula (denoted as P, called the precondition), then after executing the program S, the values of this set of variables must be able to satisfy a formula (denoted as Q, called the postcondition). We denote the correctness verification of this program as a Hoare triple: {P}S{Q}. For example, "for any x, after executing x := 1, then x must be equal to 1" can be written as: "{true}x := 1{x == 1}".

[0005] Currently, the verification of a program is to write the precondition P and postcondition Q that the program should satisfy by humans (in a few cases, tools). And the proof of this Hoare triple is generally completed by an automated proof tool. Hoare logic provides several inference rules that allow an automated reasoning tool to perform rule reasoning. Currently, for the sequential and branching statements (IF-ELSE) of a program, existing reasoning tools are capable of proving the Hoare triple through forward and backward reasoning of the "strongest postcondition" or the "weakest precondition". However, for the loop statements (While or For) of a program, an automated reasoning tool needs to construct a "loop invariant" to prove the triple of the loop statement. The rule corresponding to the loop statement in Hoare logic is:

[0006]

[0007] The above rules require finding a loop invariant I that is always true 1) when the program first enters the loop statement, 2) during the iteration of the loop statement, and 3) implies the postcondition if the loop is exited. In this way, the correctness of the program can be proven by induction.

[0008] However, solving loop invariants in general has been proven to be an undecidable problem. Currently, the Guess-and-Check method is considered a promising approach to solving loop invariants. Such methods attempt to find the correct loop invariant by repeatedly proposing possible invariant candidates. These invariant candidates are verified by an SMT solver such as Z3.

[0009] If a candidate fails to verify the three conditions of the loop invariant, the SMT solver will provide a counterexample (a program state that violates one of the three rule premises). Then, these counterexamples are used to propose the next candidate. The Guess-and-Check method usually requires defining a hypothesis space of possible loop invariants and searching for the correct invariant in this space. To make the guessing effective, an invariant template, such as a linear template, is often given, and the invariant that can pass all the previous counterexamples is generated by solving the template coefficients. If a template can solve for an invariant that passes all the previous counterexamples, the template is called "valid".

[0010] Loop invariant generation is the most fundamental and difficult problem in program verification. Currently, the Guess-and-Check method is considered a promising approach to solving loop invariants. Such methods attempt to find the correct loop invariant by repeatedly proposing possible invariant candidates. These invariant candidates are verified by an SMT solver such as Z3. If a candidate fails to verify the three conditions of the loop invariant, the SMT solver will provide a counterexample (a program state that violates one of the three requirements). Then, these counterexamples are used to propose the next candidate. To avoid repeatedly proposing loop invariant candidates that will be falsified by the same counterexample, it is often necessary to propose a candidate invariant that can pass all the previous counterexamples when generating the next candidate invariant; the candidate that can pass all the counterexamples is called a "valid loop invariant candidate"; the loop invariant template containing the "valid loop invariant candidate" is correspondingly called a "valid loop invariant template".

[0011] The prior art "Loop Invariant Inference through SMT Solving Enhanced Reinforcement Learning" details how to generate loop invariants by combining reinforcement learning methods and SMT solvers. Specifically, this method consists of three parts: a loop invariant template generator, a template solver for generating invariant candidates from the templates, and an invariant validator. Among them, the invariant template generator is implemented by a reinforcement learning model, which is used to generate parameterized invariant templates from the grammar; the template solver is used to solve the parameters in the template to generate candidate formulas so that the generated invariant candidates can pass all the previously accumulated counterexamples; the invariant validator validates the invariant proposed in the previous step. If the validation is successful, the correct loop invariant is generated. If the validation fails, a counterexample is obtained and recorded in the counterexample set, and the template generation process starts again. Both the template solver and the invariant validator are implemented by the SMT solver, and the solution results are used to provide learning feedback for the reinforcement learning model of the template generator. However, due to the lack of means to analyze the reasons why the template is invalid in the above existing technical solutions, the reinforcement learning model can only be allowed to try how to generate valid templates through simple feedback. Therefore, the methods in the entire technical solution are inefficient, blind, and passive, and urgently need to be improved and optimized. Summary of the Invention

[0012] The technical problem to be solved by the present invention is to provide a loop invariant template generation learning method based on UnsatCore and reinforcement learning with simple principle, high stability and convenient and efficient operation in view of the deficiencies of the prior art.

[0013] To solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0014] A loop invariant template generation method based on Unsat Core and reinforcement learning, comprising:

[0015] Step S1: Establish a loop invariant template generator based on autonomous reinforcement learning; through reinforcement learning, learn how to generate "effective loop invariant templates" under the guidance of the information of Unsat Core;

[0016] Step S2: Establish a loop invariant template solver based on the SMT solver; used to solve the template coefficients to generate effective loop invariant candidates; when the solution fails, it is used to provide the unsatisfiable core of the template.

[0017] Step S3: Establish a template attribution analyzer for the Unsat Core; it is used to analyze the unsatisfiable core provided by the SMT and attribute it to the specific generation process of the invariant template, guiding the reinforcement learning model for attribution learning and iterative template refinement process.

[0018] As a further improvement of the method of the present invention: A loop invariant template is generated by a loop invariant template generator; the loop invariant template is then sent to a loop invariant template solver for solving the template coefficients; if the solution is successful, a valid invariant candidate is generated; if the coefficient solution fails, an unsatisfiable core is generated, and the attribution analyzer is called for iterative refinement.

[0019] As a further improvement of the method of the present invention: The attribution analyzer analyzes the unsatisfiable core, determines the specific predicates in the invariant template that cause the coefficient solution to be unsatisfied; the attribution analyzer is used to guide the template generator to perform relaxation operations on the specific predicates, and generate refined feedback for the reinforcement learning model according to the unsatisfiable core; the system will repeat the above process until a valid invariant template is obtained.

[0020] As a further improvement of the method of the present invention: The loop invariant template generator has a deep reinforcement learning neural network (A2C) inside; the inputs of this neural network are the precondition P, postcondition Q, loop exit condition B of the loop program, and the internal statement S of the loop body (i.e., the Hoare logic formula: {P}while B do S{Q}); the output of this neural network is a Markov decision sequence; this decision sequence corresponds to generating a loop invariant template from a domain-specific grammar DSL.

[0021] As a further improvement of the method of the present invention: The step S1 includes:

[0022] Step S11: Determine the domain-specific grammar DSL;

[0023] Step S12: Determine the invariant template, and construct an expression with a determined structure and only some values of s undetermined;

[0024] Step S13: Use the Markov process to complete the output of the template generator.

[0025] As a further improvement of the method of the present invention: the reinforcement learning model inputs an expression containing a non-terminal symbol that has not been fully expanded each time, and<P,Q,B,S> The RL model repeats this process until it generates an expression containing only the undetermined coefficients si, which is an invariant template. If the maximum depth is reached, the "fast expansion" mechanism is executed to immediately generate an expression containing only the undetermined coefficients si. Fast expansion refers to the uniform replacement of non-terminal symbols with the shortest expansion of the DSL.<P,Q,B,S> All four elements in are expressions.

[0026] As a further improvement of the method of the present invention: Step S1 includes performing feedback learning of the reinforcement learning model, and the feedback of the reinforcement learning is a triple sequence with the same length as the original Markov decision sequence That is, an expanded form e, a four-tuple<P,Q,B,S> , and a distribution And the e and quaternion of each step<P,Q,B,S> Same; the learning loss function (Loss) of RL is defined as:

[0027] ∑ i cross_entropy(h i ,r i )

[0028] Where i represents the Markov decision of step i, h i represents the probability distribution generated by RL at step i, r i Represents the probability distribution of the third element of the triplet of the i-th step in the feedback; this distribution is also called the "guidance distribution"; it has two types: penalty feedback distribution and reward feedback distribution. The penalty feedback distribution sets the probability of decision a made by the RL to 0, and the remaining decisions are evenly distributed; the reward feedback distribution sets the probability of decision a made by the RL to 1, and the probabilities of the remaining decisions are set to 0; after constructing the loss function, the reinforcement learning model updates its parameters through gradient descent.

[0029] As a further improvement of the method of the present invention: Step S2 includes:

[0030] Step S21: Counterexample set and constraint solution construction. Let V be a variable vector representing all variables in the program. The next invariant I must make the following logical expressions true:

[0031]

[0032] Step S22: When a template solver cannot find a value of a coefficient so that the template solution constraint is satisfied, an unsatisfiable kernel is provided. An unsatisfiable kernel is a conjunctive normal form.

[0033] As a further improvement of the method of the present invention: Step S3 includes:

[0034] Step S31: Construct a single-predicate conjunctive unsatisfiable core splitter. Denote the variables and coefficients of predicate p ij as v and s respectively. Then the unsatisfiable core is formalized as:

[0035] ∧∨?p ij (v, s)

[0036] where,?p ij represents one of p ij or ;

[0037] Step S32: Construct an attribution distribution director. The attribution analyzer inputs a set regarding the single-predicate unsatisfiable core Output a guiding distribution for the Markov decision involved in the predicates that appear in it, and let the reinforcement learning model directly change the generation of p t .

[0038] As a further improvement of the method of the present invention: In step S32, use the attribution analyzer to provide a specific guiding distribution for the reinforcement learning model, including the following steps. For each

[0039] Step S321: Locate the expansion e when generating p t ;

[0040] Step S322: Construct penalty feedback for all decisions in generating p t ;

[0041] Step S323: Construct a special guiding distribution {v: 0.1, s: 0.4, (t op t): 0.5} for all decisions where all non-terminals inside the generation of p t are t;

[0042] Step S324: Complete one feedback learning, and re-perform the RL generation process for p t to regenerate p' t ;

[0043] Step S325: Re-input the modified template to the template solver for template solving.

[0044] Compared with the prior art, the advantages of the present invention are:

[0045] The method for generating loop invariant templates based on Unsat Core and reinforcement learning of the present invention has a simple principle and convenient operation. By means of the analysis of the unsatisfiable core, it can iteratively and precisely modify the loop invariant templates in a targeted manner, and provide precise guidance feedback at the predicate level for RL, greatly improving the efficiency of generating effective invariants. Compared with the existing loop invariant template generation technology, when facing the situation that the template solving is unsatisfiable, the existing method can only perform a blindly randomized relaxation operation, unable to iteratively modify the loop invariant template, nor provide precise guidance feedback for RL, resulting in difficulty in efficiently generating effective invariant templates. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic flowchart of the present invention in a specific embodiment.

[0047] Figure 2 It is a schematic flowchart of the attribution analyzer providing guidance for the reinforcement learning model in a specific embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] The present invention will be further described below in conjunction with the accompanying drawings of the specification and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0049] The problem to be solved by the present invention is how to design a learning system that can automatically construct an "effective" invariant template. This system can generate, verify, and iteratively improve the loop invariant template by itself, so that it can quickly and efficiently solve the effective loop invariant, thereby completing the verification work of the loop program. The present invention uses the information of Unsat Core ("unsatisfiable core") to efficiently guide the reinforcement learning model to complete the learning and generation of effective invariant candidate templates.

[0050] As Figure 1 shown, the method for generating loop invariant templates based on Unsat Core and reinforcement learning of the present invention includes:

[0051] Step S1: Establish a loop invariant template generator based on autonomous reinforcement learning;

[0052] Through reinforcement learning, learn how to generate an "effective loop invariant template" under the guidance of the information of Unsat Core;

[0053] Step S2: Establish a loop invariant template solver based on an SMT solver;

[0054] Used to solve the template coefficients, thereby generating effective loop invariant candidates;

[0055] When the solving fails, it is used to provide the unsatisfiable core of the template, the Unsat Core;

[0056] Step S3: Establishing the template attribution analyzer of Unsat Core;

[0057] It is used to analyze the unsatisfiable cores provided by SMT and attribute them to the specific generation process of the invariant template, guiding the reinforcement learning model to perform attribution learning and iterative template improvement process.

[0058] like Figure 1 As shown, the high-level working method in this embodiment is as follows:

[0059] A loop invariant template generator generates a loop invariant template;

[0060] The loop invariant template is then sent to the loop invariant template solver to solve the template coefficients; if the solution is successful, a valid invariant candidate is generated; if the coefficient solution fails, an unsatisfiable kernel is generated, and the attribution analyzer is called for iterative improvement.

[0061] In a specific application example, the attribution analyzer analyzes the unsatisfiable kernel and identifies the specific predicates in the invariant template that cause the coefficient solution to be unsatisfied. This is used to guide the template generator in relaxing the specific predicates and to generate refined feedback to the reinforcement learning model based on the unsatisfiable kernel. The system repeats this process until a valid invariant template is obtained.

[0062] In a specific application example, by executing the template generator-template solver-unsatisfiable kernel attribution analyzer cycle in large quantities, it is expected that the generator's reinforcement learning model can learn how to generate effective invariant templates.

[0063] In a specific application example, the loop-invariant template generator incorporates a deep reinforcement learning neural network (A2C). The neural network's inputs are the precondition P, postcondition Q, loop exit condition B, and the loop body's internal statements S (i.e., the Hoare logic formula: {P}while B do S{Q}). The neural network's output is a Markov decision sequence. This decision sequence generates a loop-invariant template from a domain-specific grammar (DSL).

[0064] In this embodiment, a domain-specific syntax DSL is first introduced, and then how to generate a template with parameters from the DSL through a Markov decision sequence is introduced.

[0065] In a specific application example, step S1 may further include the following detailed process:

[0066] Step S11: Determine the domain-specific syntax DSL;

[0067] Consider the QF-NIQA theory (Quantifier-Free Nonlinear Integer Real Arithmetic Theory: "Arithmetic theory of quantifier-free non-linear integer and real number mixture"), including functions such as + (addition), - (subtraction), * (multiplication), / (division), % (modulo); including predicates such as < (less than), ≤ (less than or equal to), = (equal to); including logical connectives such as ∨ (logical disjunction), ∧ (logical conjunction). Use the character v to represent the variables of the program to be verified, and the character s to represent specific constants or undetermined template coefficients.

[0068] Among them, the DSL is defined as:

[0069]

[0070] p := t < t | t ≤ t | t = t

[0071] t := v | s | (t op t)

[0072] op := + | - | * | / | %;

[0073] The above DSL can express almost all invariants in the QF-NIQA theory. In practical applications, the maximum numbers of conjunctions and disjunctions (nc and nd) are set to 100. s can be left as the coefficient of the template, or determined as a real number or integer between -10000 and 10000, or a constant appearing in the program source code. Template is the start symbol, representing a conjunctive normal form. A conjunctive normal form is composed of multiple disjunctive forms conjoined. A disjunctive form is composed of multiple predicates p disjoined. A predicate p is composed of two arithmetic expressions t and a comparison operator <, ≤, or =. An arithmetic expression can be composed of a variable v, or a constant / coefficient s, or another two arithmetic expressions through an arithmetic operator (+ | - | * | / | %).

[0074] Step S12: Determine the invariant template and construct an expression with a determined structure and only some s values undetermined.

[0075] For example, (((x * s_1) + s_2) < (z % 3)) is a template in the above DSL, where x, y, and z are variables of the program, s_1 and s_2 are template coefficients, and 3 is a specific constant. The values of s_1 and s_2 are undetermined in the template at this time and need to be solved according to the data in the subsequent counterexample set to obtain the specific invariant.

[0076] Step S13: Use the Markov process to complete the output of the template generator;

[0077] The output of the template generator is an invariant template in the form shown above. The internal deep reinforcement learning neural network of the template generator completes the generation of the above invariant template through a Markov process. Denote this neural network as RL, the precondition of the loop to be verified as P, the postcondition as Q, the loop exit condition as B, and the statements inside the loop as S:

[0078] P -> RL(Template, <P, Q, B, S>) -> p 00

[0079] -> RL(p 00 , <P, Q, B, S>) -> (t op t)

[0080] -> RL((t op t), <P, Q, B, S>) -> ((t op t) op t)

[0081] -> RL(((t op t) op t), <P, Q, B, S>) -> (((t op t) op t) o[t)

[0082] -> RL((((t op t) op t) op t), <P, Q, B, S>) -> (((v op t) op t) op t)

[0083] -> RL((((v op t) op t) op t), <P, Q, B, S>) -> (((x op t) op t) op t)

[0084] -> RL((((x op t) op t) op t), <P, Q, B, S>) -> (((x * t) op t) op t)

[0085] -> RL((((x * t) op t) op t), <P, Q, B, S>) -> (((x * s_1) op t) op t)

[0086] -> RL((((x * s_1) op t) op t), <P, Q, B, S>) -> (((x * s_1) + t) op t)

[0087] -> RL((((x * s_1) + t) op t), <P, Q, B, S>) -> (((x * s_1) + s_2) op t)

[0088] -> RL((((x * s_1) + s_2) op t), <P, Q, B, S>) -> (((x * s_1) + s_2) < t)

[0089] ->RL((((x*s_1)+s_2)<t),<P,Q,B,S>)->(((x*s_1)+s_2)<(t op t))

[0090] ->RL((((x*s_1)+s_2)<(t op t)),<P,Q,B,S>)->(((x*s_1)+s_2)<(v op t))

[0091] ->RL((((x*s_1)+s_2)<(v op t)),<P,Q,B,S>)->(((x*s_1)+s_2)<(z op t))

[0092] ->RL((((x*s_1)+s_2)<(z op t)),<P,Q,B,S>)->(((x*s_1)+s_2)<(z%t))

[0093] ->RL((((x*s_1)+s_2)<(z%t)),<P,Q,B,S>)->(((x*s_1)+s_2)<(z%3))。

[0094] That is, each time the reinforcement learning model inputs an unexpanded expression containing non-terminals and the <P, Q, B, S> quadruple, and outputs a syntax expansion decision. For example, in the first step, Template is expanded to p 00 , and in the second step, p 00 is expanded to (t op t).

[0095] The reinforcement learning model will repeat this process until an expression containing only the undetermined coefficients s_i is generated, and the expression at this time is an invariant template. If the set deepest depth is reached, then the "fast expansion" mechanism will be executed to immediately generate an expression containing only the undetermined coefficients s_i.

[0096] Among them, "fast expansion" means uniformly replacing non-terminals with the shortest expansions of the DSL. For example, t must be expanded to s_i, so as to achieve the purpose of quickly terminating the Markov decision process.

[0097] Specifically, the four elements in <P, Q, B, S> are all expressions, which can be understood as strings. In the reinforcement learning model, feature extraction will be performed on them through a feature extraction module based on TreeLSTM.

[0098] The RL model consists of three modules, namely a feature extraction module (E) and a decision-making module (A). The feature extraction module E takes the current expanded expression of the grammar part and the quadruple <P, Q, B, S> as inputs and outputs a high-dimensional vector. This high-dimensional vector is regarded as the state semantic vector used to guide the reinforcement learning model to make decisions. The feature extraction module is implemented based on a Tree-LSTM model and an attention mechanism. The decision-making module A takes the state semantic vector and the leftmost non-terminal symbol of the current expanded expression as inputs and outputs a grammar expansion decision. Generally speaking, the grammar expansion decision is encoded into the high-dimensional semantic vector space, which has the same dimension as the state semantic vector. When making a decision, the decision-making module measures the Cosin similarity between the state semantic vector and the grammar expansion decision, and generates a probability distribution over all optional decisions through Softmax, and finally selects the decision with the highest probability.

[0099] As a preferred embodiment, the step S1 further includes performing feedback learning of the reinforcement learning model; the feedback of the reinforcement learning is a triple sequence with the same length as the original Markov decision sequence That is, an expanded expression e, a quadruple <P, Q, B, S>, and a distribution And the e and the quadruple <P, Q, B, S> are the same for each step. The learning loss function (Loss) of RL is defined as:

[0100] ∑ i cross_entropy(h i ,r i )

[0101] where i represents the i-th step of the Markov decision, h i represents the probability distribution generated by RL at the i-th step, and r i represents the probability distribution of the third element of the triple at the i-th step in the feedback. This distribution is also called the "guidance distribution". Generally, there are two types of feedback distributions: penalty feedback distribution and reward feedback distribution. The penalty feedback distribution sets the probability of the decision a made by RL to 0, and the remaining decisions are evenly distributed; the reward feedback distribution sets the probability of the decision a made by RL to 1, and the probabilities of the remaining decisions are set to 0. After constructing the loss function, the reinforcement learning model updates its parameters through gradient descent.

[0102] The input of the template solver is an invariant template and a set of counterexamples. There are two possible outputs. One is "satisfiable", which means that there is a set of values for the template coefficients such that the invariant under these values can pass all the counterexamples in the counterexample set. The other is "unsatisfiable", which means that there is no set of values for the template coefficients that can pass all the counterexamples in the counterexample set. When it is satisfiable, it indicates that the template generator generates a valid template and the system completes the predetermined task. When it is unsatisfiable, the template solver will provide an unsatisfiable core. The solving process is divided into two steps. The first step is to construct the query for solving, and the second step is SMT solving. In this embodiment, how to construct the solving constraint query according to the counterexample set and the invariant template is introduced first.

[0103] In a specific application example, step S2 further includes:

[0104] Step S21, construction of the counterexample set and the solving constraint. Let V be a vector of variables representing all variables in the program. The next invariant I must satisfy all the following logical expressions:

[0105]

[0106] Step S22, when a template solver cannot find the values of the coefficients such that the template solving constraint is satisfiable, provide an unsatisfiable core. An unsatisfiable core is a conjunctive normal form, such as:

[0107] (a ∨ b) ∧ (c ∨ d)

[0108] The meaning of this unsatisfiable core is that the four conjunctive sub-formulas (a ∧ c), (a ∧ d), (b ∧ c), and (b ∧ d) are all unsatisfiable. x > 0 ∧ x < 0 is the unsatisfiable core of (x > 0 ∧ x < 0) ∨ (x * x + y < 0). Ideally, the unsatisfiable core is minimal. In fact, the SMT solver can return a redundant unsatisfiable core (that is, there is an unsatisfiable core composed of a proper subset of the conjunctive clauses in this core).

[0109] In a specific application example, the detailed process of step S3 can further include:

[0110] Step S31: Construct a single-predicate conjunctive unsatisfiable core splitter. Denote the variables and coefficients of the predicate p ij as v and s respectively, then the unsatisfiable core can be formalized as:

[0111] ∧∨?p ij (v, s)

[0112] where? p ij represents one of p ij or ;

[0113] Also, since the assignment of variable v must come from a counterexample assignment, denoted as [v k . Note that the parameter s between any two different predicates must be independent, denoted as s ij . Then the above formula is transformed into

[0114] ∧∨?p ij ([v k , s ij )

[0115] Note that the only free variable in the above formula is s ij , and the variables between different predicates p ij have no intersection. Therefore, the above formula can be split according to the predicates to obtain the conjunctive normal form of a certain predicate p t , and it is converted into the disjunctive normal form. Any one of the conjunctive clauses is in the following form:

[0116] ?p t ([v k1 , s t ) ∧?p t ([v k2 , s t ) ∧... ∧?p t ([v kn , s t )

[0117] The meaning of the above formula is that the predicate p t (v, s t ) cannot satisfy all counterexamples when the assignments are [v k1 , [v k2 ,..., [v kn for n counterexamples. Secondly, it can be proved that there are no two identical assignments for [v ko : When they are both positive literals or both negative literals, they can be directly merged; when one is positive and the other is negative, it means that it is unsatisfiable in boolean logic, that is, the original loop invariant program is incorrect and there is no valid loop invariant.

[0118] Ideally, the SMT solver only returns the unsatisfiable core of the above formula for a certain predicate p t . If the SMT solver does not return the simplest unsatisfiable core, it can be split into a complex number of pure conjunctive unsatisfiable cores for a single predicate p t , denoted as

[0119] Step S32: Construct an attribution distribution guide. The attribution analyzer inputs the set of unsatisfiable cores for single predicates ​Output guidance distribution for the Markov decision involved in the predicate, and let the reinforcement learning model directly change p t Generation.

[0120] For an unsatisfiable core It can be seen that p t is a t <t|t≤t|t=t三者之一的比较式。其不可满足的原因在于?p t ([v k1 ],s t )∧? p t ([v k2 ],s t )∧...∧? p t ([v kn ],s t ) is empty. In order to make Satisfiable, need to p t Specifically, it is necessary to modify p t The production rule is "relaxed": more parameters s are generated, and longer expressions are generated.

[0121] For example, for the predicate p t :(x*s_1<0) and unsatisfiable core (1*s_1<0)∧((-1)*s_1<0), we can introduce an additional parameter s_2 and modify the predicate to p t :((x*s_1)+s_2<0). The unsatisfiable kernel is (1*s_1+s_2<0)∧((-1)*s_1+s_2<0), which can be satisfied by (s_1=1,s_2=-2). Satisfying the unsatisfiable kernel is a prerequisite for satisfying the template solution constraints.

[0122] In the above process, a counterexample set includes three types of counterexample sets, namely, positive and negative example sets CE p , negative counterexample set CE n , and the inductive counterexample set CE i A counterexample is a set of assignments to all variables. A positive and negative example (p) is a statement that disproves A negative counterexample (n) is one that falsifies , and the next proposed valid loop invariant must make I(n) false when assigned to n. The inductive counterexample (i_1,i_2) is a counterexample that falsifies {I∧B}S{I}. Unlike the previous two, it involves two variable assignments, i_1 and i_2. The next proposed invariant must either be false for the first assignment to i_1 or true for both assignments.

[0123] Let V be a vector of variables representing all variables in the program. The next invariant I must satisfy the following logical expressions:

[0124]

[0125] The above is the construction for solving the constraints.

[0126] For example, assume the template is s1*x + s2*y ≤ c. Here, the coefficients s1, s2, and c will be solved by the SMT so that the final invariant must be consistent with all counterexamples. To speed up the constraint-solving process, the values of the coefficients are often restricted to a narrow feasible region. For example, s k is usually restricted to [-1, 1].

[0127] Assume there is already the following set of counterexamples:

[0128] CE p : {x = 100, y = 0, n = 100}

[0129] CE n : {x = 0, y = 1, n = 2; x = 0, y = 2, n = 1}

[0130] CE i : {(x = 1, y = 0, n = 100; x = 0, y = 1, n = 100)}

[0131] Then, the SMT constraints (Query) for solving the template coefficients will be: [[ID=3�]]

[0132]

[0133] Solving this constraint gives the values of s1, s2, and c as -1, -1, and -3 respectively, making the invariant consistent with all previous counterexamples. The valid invariant candidate is (-1)*x + (-1)*y ≤ -3.

[0134] As Figure 2 shown, in the step S32, the attribution analyzer provides a specific guidance distribution for the reinforcement learning model, including the following steps:

[0135] For each

[0136] Step S321: Locate the expansion e when generating p t ; <m:math xmlns:m="http: / / www.w3.org / 1998 / Math / MathML"><m:mrow><m:mi>e< / m:mi>< / m:mrow>< / m:math><m:annotation encoding="application / x-tex">e< / m:annotation>

[0137] t Step S322: Construct penalty feedback for all decisions that generate p

[0138] Step S323: At the generation of pt Construct a special guiding distribution {v: 0.1, s: 0.4, (t op t): 0.5} for all non-terminal decision-making with t inside;

[0139] Step S324: Complete one feedback learning and re-p t Perform the RL generation process to regenerate p' t .

[0140] Step S325: Re-input the modified template to the template solver for template solving.

[0141] The above is only the preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions falling within the idea of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A method for generating loop invariant templates based on UnsatCore and reinforcement learning, characterized in that Including: Step S1: Establish a loop invariant template generator based on self-reinforcement learning; Through reinforcement learning, learn how to generate "effective loop invariant templates" under the guidance of the information of UnsatCore; Step S2: Establish a loop invariant template solver based on an SMT solver; Used to solve the template coefficients to generate effective loop invariant candidates; when the solution fails, it is used to provide the Unsat Core of the template that is unsatisfiable; Step S3: Establish a template attribution analyzer for Unsat Core; used to analyze the unsatisfiable core provided by SMT and attribute it to the specific generation process of the invariant template, guiding the reinforcement learning model for attribution learning and iterative template refinement process.

2. The method for generating loop invariant templates based on UnsatCore and reinforcement learning according to claim 1, wherein A loop invariant template is generated by the loop invariant template generator; the loop invariant template is then sent to the loop invariant template solver for solving the template coefficients; if the solution is successful, an effective invariant candidate is generated; If the coefficient solution fails, an unsatisfiable core is generated and the attribution analyzer is called for iterative refinement.

3. The method for generating loop invariant templates based on UnsatCore and reinforcement learning according to claim 2, wherein The attribution analyzer analyzes the unsatisfiable core to determine the specific predicates in the invariant template that cause the coefficient solution to be unsatisfied; the attribution analyzer is used to guide the template generator to perform relaxation operations on the specific predicates and generate refined feedback for the reinforcement learning model based on the unsatisfiable core; the system will repeat the above process until an effective invariant template is obtained.

4. The method for generating loop invariant templates based on UnsatCore and reinforcement learning according to claim 2, characterized in that There is a deep reinforcement learning neural network (A2C) inside the loop invariant template generator; the input of this neural network is the precondition P, postcondition Q, loop exit condition B, and the internal statement S of the loop body (i.e., the Hoare logic formula: {P}while B do S{Q}); the output of this neural network is a Markov decision sequence; this decision sequence corresponds to generating a loop invariant template from a domain-specific grammar DSL.

5. The method for generating loop invariant templates based on UnsatCore and reinforcement learning according to any one of claims 1-4, characterized in that, The said Step S1 includes: Step S11: Determine the domain-specific grammar DSL; Step S12: Determine the invariant template and construct an expression with a determined structure and only some values of s undetermined; Step S13: Use the Markov process to complete the output of the template generator.

6. The method for generating loop invariant templates based on UnsatCore and reinforcement learning according to claim 5, wherein Each time, the reinforcement learning model inputs an unexpanded expression containing non-terminals and the <P, Q, B, S> quadruple, and outputs a grammar expansion decision; the reinforcement learning model will repeat this process until an expression containing only undetermined coefficients s_i is generated, and at this time the expression is a loop invariant template; if the set deepest depth is reached, then the "fast expansion" mechanism will be executed to immediately generate an expression containing only undetermined coefficients s_i; the fast expansion means uniformly replacing non-terminals with the shortest expansion of DSL; the four elements in <P, Q, B, S> are all expressions.

7. The method for generating loop invariant templates based on UnsatCore and reinforcement learning according to claim 6, wherein The step S1 includes performing feedback learning of the reinforcement learning model, and the feedback of the reinforcement learning is a triple sequence with the same length as the original Markov decision sequence That is, an expansion e, a quadruple <P, Q, B, S>, and a distribution And e and the quadruple <P, Q, B, S> are the same for each step; the learning loss function (Loss) of RL is defined as: ∑ i cross-entropy(h i ,r i ) where i represents the Markov decision at the i-th step, h i represents the probability distribution generated by RL at the i-th step, r i represents the probability distribution of the third element of the triple at the i-th step in the feedback; this distribution is also called the "guidance distribution"; there are two types, the penalty feedback distribution and the reward feedback distribution. The penalty feedback distribution sets the probability of the decision a made by RL to 0, and the remaining decisions to a uniform distribution; the reward feedback distribution sets the probability of the decision a made by RL to 1, and the probabilities of the remaining decisions to 0; after constructing the loss function, the reinforcement learning model updates the parameters by gradient descent.

8. The method for generating loop invariant templates based on UnsatCore and reinforcement learning according to any one of claims 1-4, characterized in that The said Step S2 includes: Step S21, Construction of counterexample set and solution constraints. Let V be a variable vector used to represent all variables in the program. The next invariant I must make the following logical expressions all hold: Step S22: When a template solver fails to find values of coefficients such that the template solving constraints are satisfiable, provide an unsatisfiable core, where an unsatisfiable core is a conjunctive normal form.

9. The method for generating loop invariant templates based on UnsatCore and reinforcement learning according to any one of claims 1-4, characterized in that, The said step S3 includes: Step S31: Construct a single-predicate conjunctive unsatisfiable core splitter. Denote the variables and coefficients of predicate p ij as v and s respectively. Then the unsatisfiable core is formalized as: ∧V?p ij (v, s) Among them,?p ij represents p ij or one of; Step S32: Construct an attribution distribution director, and the attribution analyzer inputs a set regarding the non - satisfaction of a single predicate with respect to the kernel of Output a guiding distribution for the Markov decision involved in the predicate that appears therein, and let the reinforcement learning model directly change the generation of p t of 10. The method for generating loop invariant templates based on UnsatCore and reinforcement learning according to claim 9, wherein In the step S32, a specific guidance distribution is provided for the reinforcement learning model by using an attribution analyzer, including the following steps. For each Step S321: Locate the expansion e when generating p t is generated; Step S322: Construct penalty feedback for all decisions that generate p t ; Step S323: Generate a special guiding distribution {v: 0.1, s: 0.4, (topt): 0.5} for decisions where all non-terminals inside pt are t; Step S324: Complete one feedback learning and re-p t Perform the RL generation process to regenerate p't; Step S325: Re-input the modified template to the template solver for template solving.

Images