A method for rewriting functional convolution expressions into C code

Abstract

The present invention provides a method for expressing AI neural network operators in a functional manner in a Coq theorem prover, and implements a rewriting technology for converting functional convolution expressions into C code. It mainly includes six key parts, the formal definition of tensor primitives, the formal definition of intermediate variable binding values, the formal definition of imperative combinators of tensor primitives and intermediate variable binding values, the formal definition of rewriting rules for tensor primitives and intermediate variable binding values, the definition of functional tensor convolution expressions, and the formal definition of expressions of other AI neural network operators such as pooling and activation. It provides strong support for the subsequent verification and optimization of the correctness of neural network operators in Coq. It effectively avoids problems such as shape mismatch and unreliability common in traditional deep learning frameworks. This lays a solid foundation for building a highly reliable AI compiler.

Description

Technical Field

[0001] The present invention belongs to the field of formal methods and AI compilers, and specifically relates to a rewriting conversion method from a functional convolution expression to a C code. Background Art

[0002] One of the main challenges of existing AI compilers is to adapt AI neural network models to various complex parallel computing AI processors, including multi-core parallel computing, GPU, FPGA, TPU and other AI-specific accelerators, and optimize various operators for hardware to improve program performance and ensure the equivalence of optimized code with the original code. This requires solving the software adaptation and correctness verification problems between large-scale AI computing and AI hardware resources.

[0003] Traditionally, existing AI compilers usually use imperative languages ​​to write compiler code and optimize the code. However, with the increasing complexity of AI models and applications, imperative languages ​​have become increasingly difficult to deal with these problems. It is usually difficult to perform flexible refactoring when trying different parallelization strategies, which makes the optimization process complex and difficult to manage, increasing the difficulty of code optimization. In addition, it has become more difficult to verify the equivalence of the code before and after optimization. Existing AI compilers, such as Glow, Tensor Comprehensions, etc., use various optimization techniques to generate deep learning model code to adapt to different hardware architectures. However, these methods usually rely on manually written fixed templates to optimize the code, making it difficult to fully utilize the underlying characteristics of hardware accelerators and providing formal verification of the reliability of code optimization. Functional programming languages ​​provide a solid foundation for dealing with these challenges. They allow the design of complex optimization processes while allowing full utilization of parallelism in the underlying system. They have a strong mathematical foundation for advanced optimization and formal verification, which is almost impossible to achieve in traditional imperative programming languages. This makes them ideal for generating efficient target code for parallel systems.

[0004] Faced with the problems of difficulty in optimizing imperative code and difficulty in ensuring reliability, a new solution has been proposed internationally, namely, introducing a functional intermediate language for code transformation and optimization. In this field, Halide, TVM, and DPIA are the main representatives. They simplify the optimization process, make the optimization rules clearer and more reliable, and can explore more optimization possibilities. Among them, DPIA is a purely functional technology that generates efficient and parallel low-level imperative code through high-level functional code. However, this technology is quite complex and its reliability is difficult to guarantee. Based on the DPIA theory, Ma Yingying et al. implemented the formal representation of DPIA in the Coq theorem prover and disclosed "a formal description method of matrix code generation technology based on Coq" (Announcement No. CN114154114A). However, its functional tensor primitives based on the element-to-element mapping in DPIA are difficult to implement when expressing neural network operator operations such as convolution, pooling, and activation in AI compilers. For example, in the convolution operation, the movement process of the convolution kernel on the input tensor cannot be fully expressed. To solve this problem, the present invention defines a tensor primitive from subscript index to element value in the Coq theorem prover to simplify the construction of the entire functional tensor expression. Finally, a rewriting conversion method from the functional convolution expression represented by the tensor primitive to C code is realized, and the technology is extended to express other neural network operators. Summary of the invention

[0005] Purpose of the invention: In view of the above-mentioned prior art, a method is proposed;

[0006] Technical solution: A method for rewriting and converting a functional convolution expression into C code, comprising the following steps:

[0007] (1): Define a tensor primitive from subscript index to element value in the Coq theorem prover;

[0008] (2): Define operations for binding values ​​of intermediate variables in the Coq theorem prover;

[0009] (3): Define intermediate imperative combinators for tensor primitives and intermediate imperative combinators for intermediate variable binding values ​​in the Coq theorem prover;

[0010] (4): Define rewrite conversion rules from tensor primitives to intermediate imperative combinators and from intermediate variable binding values ​​to intermediate imperative combinators in the Coq theorem prover;

[0011] (5): Define the convolution operation of the functional tensor expression in the Coq theorem prover, and use the rewrite transformation rule in step (4) to convert the convolution operation into the corresponding C language convolution code;

[0012] (6): Define functional tensor expressions of other AI compiler core operators in the Coq theorem prover. Other AI compiler core operators include pooling, activation, and full connection.

[0013] Preferably, the implementation of step (1) is specifically as follows: a first mapping function (f:fin n->exp d) from subscript index to element value is used to construct a tensor primitive mkvSeq, where fin n is a natural number from 0 to n-1, d is an inductively defined data type, and exp is an expression type.

[0014] Preferably, the implementation of step (2) defines an intermediate variable binding value let_binding, which accepts two type parameters d1 and d2, as well as an expression e and a second mapping function, and passes the expression e of type d1 to the second mapping function. The function of the second mapping function is to map an expression from the type parameter d1 to the expression of the type parameter d2.

[0015] Preferably, the implementation of step (3) specifically includes:

[0016] Define the intermediate imperative combinator mkvSeql of the tensor primitive, which accepts two parameters: one is the third mapping function from the finite set {0,1,...,n-1} to the receiver type to the command, and the other parameter is the receiver type of the tensor with length n and data type d, and uses the seqfor command to perform the third mapping function operation on each element i in the finite set, where seqfor is an imperative loop operation;

[0017] An intermediate imperative combinator letl that defines an intermediate variable binding value is used to pass an expression e and a receiver a to a third mapping function f, where e is an expression with a type parameter of d1, a is a receiver with a type parameter of d2, and the third mapping function f is a mapping function from an expression of type d1 to an expression of type d2.

[0018] Preferably, the implementation of step (4) specifically includes:

[0019] The tensor primitive rewrites the transformation rule for the assignment function A. For a tensor constructed by the first mapping function f, use function A to assign it to the receiver a, which is equivalent to applying function A to each first mapping function f and receiver a through mkvSeql, where function A is an assignment operation used to assign an expression to the receiver;

[0020] The tensor primitive rewrites the conversion rules for the command function C. For a given tensor, the first mapping function f and the command mapping function c are constructed. When the function C is applied, a temporary variable is equivalently introduced through the command new, and mkvSeq f is assigned to the temporary variable, and then the expression value of the temporary variable is passed to the command mapping function c, where the function C is an expression-to-command mapping function;

[0021] Rewrite transformation rules for intermediate variable binding values ​​for function A: In intermediate variable binding, for a given expression e and mapping function f, assigning let_binding ef to receiver a through function A is equivalent to applying the imperative combinator letl;

[0022] Rewrite transformation rules for intermediate variable binding values ​​for function C: For a given expression e, function f, and command mapping function c, when applying function C, a temporary variable is equivalently introduced through the command new, and let_binding ef is assigned to the temporary variable, and then the expression value of the temporary variable is passed to the command mapping function c, where function C is an expression-to-command mapping operation.

[0023] Preferably, the implementation of step (5) specifically includes:

[0024] The filling function of the two-dimensional tensor constructs the output tensor through a nested two-dimensional tensor structure. The outer layer constructs rows and the inner layer constructs columns. The elements of the output tensor are determined by whether the element at the index is within the range of the input tensor. If it is within the range of the input tensor, the corresponding element of the input tensor e is used as the output element at the current index. If it is not within the range of the current input tensor, the default element zero is used as the output element;

[0025] The convolution function of a two-dimensional tensor first introduces an intermediate variable binding value to store the result after the input tensor is filled. Then a nested two-dimensional tensor structure is constructed, in which the internal function is used to represent each element of the output tensor. Finally, the rewriting transformation rule in step (4) is used to convert the functional convolution expression into the corresponding imperative combinator.

[0026] Preferably, the implementation of step (6) specifically includes:

[0027] Define the maximum pooling function of the two-dimensional tensor. The rows and columns of the output tensor are represented by constructing a nested two-dimensional tensor structure. The subscript index is used to iterate each element in the window. The maximum value in the nested tensor is used as the element of the output tensor.

[0028] Define the average pooling function of the two-dimensional tensor. First, construct a nested two-dimensional tensor. The output element at the current index is the sum of the elements in the pooling window divided by the number of elements in the pooling window.

[0029] Define the ReLU activation function of the two-dimensional tensor. First, construct a nested two-dimensional tensor structure to output the rows and columns of the tensor. Use the row index and column index to traverse the elements of the input tensor and determine whether the current element is less than or equal to zero. If so, replace it with zero. Otherwise, keep the original value.

[0030] Define the fully connected operation by constructing a one-dimensional tensor structure, where the internal element is a sum of the sum of the nested tensor elements and the corresponding bias elements, where the sum of the nested tensor elements is the product of the input element and the corresponding weight element and then the sum is accumulated.

[0031] Beneficial effects: AI neural network operators such as convolution and pooling are formally represented in the Coq theorem prover, and the corresponding C code is generated, which provides strong support for the subsequent correctness verification and optimization of neural network operators in Coq. The neural network model constructed in this way has a precise type checking mechanism, which effectively avoids the common problems of shape mismatch and unreliability in traditional deep learning frameworks. This lays a solid foundation for building a highly reliable AI compiler. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0033] The following is a detailed description with reference to the accompanying drawings and specific examples.

[0034] This paper provides a method for expressing AI neural network operators in a functional way in Coq theorem prover, and implements a rewriting technology for converting functional convolution expressions into C code. It mainly includes six key parts, such as Figure 1 As shown: the formal definition of tensor primitives, the formal definition of intermediate variable binding values, the formal definition of imperative combinators of tensor primitives and intermediate variable binding values, the formal definition of rewriting rules for tensor primitives and intermediate variable binding values, the definition of functional tensor convolution expressions, and the formal definition of expressions of other AI neural network operators such as pooling and activation. The specific steps are as follows:

[0035] Step 1: Formally define tensor primitives in the Coq theorem prover.

[0036] Construct a tensor type from subscript index to element value, where f is a mapping function from a finite set to element value. The definition of this tensor primitive is defined by the auxiliary function mk_vec, which is defined as follows:

[0037] Definition mkvSeq{d}{n:nat}(f:fin n->exp d)

[0038] :exp(ary nd):=mk_vec f.

[0039] Where mk_vec is a recursive function used to construct a vector containing m elements. It accepts a finite set fin m to a function of type A as a parameter. When m is 0, it returns an empty vector tt. When m is not 0, it constructs the first m-1 (here represented by m') elements by recursively calling mk_vec, and then constructs the last element of the vector by calling the parameter function u. The value of type fin(S m') is generated by the exist function, which is specifically defined as follows:

[0040]

[0041] Step 2: Formalize the intermediate variable binding values ​​in the Coq theorem prover.

[0042] In order to store intermediate temporary variables when generating imperative C code later, intermediate variable binding values ​​are formally defined in Coq. This definition accepts two type parameters d1 and d2, an expression e, and a function f that maps an expression of type d1 to an expression of type d2. In the let_binding function, the expression e is passed to the function f for processing, realizing the binding and conversion of intermediate variables.

[0043] In order to improve the readability of the code, a notation 'tlet'x':='e'in'f is defined for the let_binding function using Notation in Coq. This notation allows the use of tlet in Coq to bind intermediate variables. The specific definition is as follows:

[0044]

[0045] Step 3: Formalize the imperative combinators that define tensor primitives and intermediate variable binding values ​​in the Coq theorem prover.

[0046] The intermediate imperative combinator of the tensor primitive is formally defined, which accepts two parameters: one is f, which represents a mapping function from a finite set {0,1,...,n-1} to a receiver type to a command; the other parameter is the receiver type of a tensor of length n and element type d. The seqfor command is used to perform the f operation on each element i in the finite set, where seqfor is an imperative loop operation, the receiver out is the assigned object, and out|{i} is to obtain the i-th element in the receiver tensor. The specific definition is as follows:

[0047] Definition mkvSeql{n:nat}{d:data}(f:fin n->acc d->comm)

[0048] (out:acc(ary nd)):=

[0049] seqfor(fun i=>fi out|{i}).

[0050] Formally define the intermediate imperative combinator of let_binding, which accepts two implicit type parameters d1 and d2, and three parameters e, f, and out, and passes the expression e to the function f for processing. The purpose is to store intermediate temporary variables in the subsequent generated imperative C code. The specific definition is as follows:

[0051] Definition letl{d1 d2}(e:exp d1)(f:exp d1->acc d2->comm)(out:acc d2):=

[0052] fe out.

[0053] Step 4: Formalize the rewriting rules for tensor primitives and intermediate variable binding values ​​in the Coq theorem prover.

[0054] The rewriting rules for tensor primitives are mainly divided into the rewriting rules for the assignment function A and the rewriting rules for the expression to the command function C. The rewriting rules for the assignment function A are expressed as assigning the entire tensor to a receiver, which is equivalent to performing an assignment operation on each element in the tensor, where function A is an assignment operation, specifically assigning the expression to the receiver. The specific definitions are as follows:

[0055] Lemma AmkvSeq:forall(n:nat)(d:data)(f:fin n->exp d)a,

[0056] A(a,mkvSeq f)=mkvSeql(fun io=>A(o,fi))a.

[0057] For the tensor rewriting rule of function C, apply function C to convert the functional tensor into an imperative primitive operation equivalently, where function C is a mapping operation from expression to command, command comm is an inductively defined type, new is a constructor of comm, used to declare a temporary variable tmp(w,v), where w is the receiver, v is an expression, and "';" indicates the sequential execution of commands. The specific definition is as follows:

[0058] Lemma CmkvSeq:forall(n:nat)(d:data)(f:fin n->exp d)

[0059] (c:exp(ary nd)->comm),

[0060] C(mkvSeq f,c)=

[0061] new(fun tmp=>let(w,v):=tmp in A(w,mkvSeq f)';cv).

[0062] Regarding the rewriting rules of functions A and C for binding intermediate variables, they convert the operation of binding intermediate variables into an imperative assignment operation and an operation of mapping variables to commands, respectively. Functions A and C are imperative assignment and expression-to-command mapping operations, respectively. The specific definitions are as follows:

[0063]

[0064] Step 5: Formalize the functional tensor convolution expression in Coq theorem prover and convert it into the corresponding C code.

[0065] First, define the padding operation for a two-dimensional tensor, that is, add zero elements around the tensor through the tensor constructor to expand the size of the tensor, where the default element of exp_default is zero, and e|[F[ip]] is to get the (ip)th element in the tensor. The specific definition is as follows:

[0066]

[0067] To define the convolution operation of a two-dimensional tensor, first use tlet to store the padded tensor, and then use the nested tensor function to construct the convolution operation, where reduce add is a reduced accumulation operation. The specific definition is as follows:

[0068]

[0069] Finally, the functional tensor convolution expression Conv2d is converted into an imperative combinator using the intermediate imperative combinators defined in steps 3 and 4 and the rewriting conversion rules, and finally converted into the corresponding C code string using the display translation rules defined in CN114154114A. The specific two-dimensional convolution imperative C code generated is as follows:

[0070]

[0071] Step 6: Formalize functional tensor pooling, activation, and fully connected expressions in Coq theorem prover.

[0072] Define the maximum pooling functional expression of a two-dimensional tensor, where max represents the maximum value operation in the tensor. The specific definition is as follows:

[0073]

[0074] Defines the average pooling functional expression of a two-dimensional tensor, where div_n represents the division operation between a tensor element and a natural number. The specific definition is as follows:

[0075]

[0076] The functional activation operation of a two-dimensional tensor is defined as follows, where leExp is used to determine whether the value of a tensor element is less than or equal to the relationship:

[0077]

[0078] The functional fully connected operation of a two-dimensional tensor is defined as follows:

[0079]

[0080] So far, the formal definition of tensor primitives, intermediate variable binding values, intermediate imperative combinators, rewriting rules, and neural network operators such as function tensor convolution expressions, pooling, and activation has been completed in the Coq theorem prover. Thus, the rewriting conversion technology that converts functional tensor convolution expressions into corresponding C codes has been realized. This technology not only provides convenient support for the formal definition of AI neural network operators in Coq, but also lays the foundation for the subsequent verification of the correctness and optimization of AI neural network operators in Coq.

[0081] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for rewriting and converting a functional convolution expression into a C code, characterized in that: The specific steps include: (1): Define a tensor primitive from subscript index to element value in the Coq theorem prover; the implementation of step (1) is as follows: a first mapping function (f:fin n->exp d) from subscript index to element value is used to construct the tensor primitive mkvSeq, where fin n is a natural number from 0 to n-1, d is an inductively defined data type, and exp is an expression type; (2): Define operations for binding values ​​to intermediate variables in the Coq theorem prover; (3): Define intermediate imperative combinators for tensor primitives and intermediate imperative combinators for intermediate variable binding values ​​in the Coq theorem prover; (4): In the Coq theorem prover, define the rewriting transformation rule of the tensor primitive for the assignment function A, the rewriting transformation rule of the tensor primitive for the command function C, the rewriting transformation rule of the intermediate variable binding value for the function A, and the rewriting transformation rule of the intermediate variable binding value for the function C; the implementation of step (4) specifically includes: The tensor primitive rewrites the transformation rule for the assignment function A. For a tensor constructed by the first mapping function f, use function A to assign it to the receiver a, which is equivalent to applying function A to each first mapping function f and receiver a through mkvSeql, where function A is an assignment operation used to assign an expression to the receiver; The tensor primitive rewrites the conversion rules for the command function C. For a given tensor, the first mapping function f and the command mapping function c are constructed. When the function C is applied, a temporary variable is equivalently introduced through the command new, and mkvSeq f is assigned to the temporary variable, and then the expression value of the temporary variable is passed to the command mapping function c, where the function C is an expression-to-command mapping function; The rewriting transformation rule of the intermediate variable binding value for function A: In the intermediate variable binding, for the given expression e and the first mapping function f, assigning let_binding ef to the receiver a through function A is equivalent to applying the imperative combinator letl; The rewriting transformation rule of the intermediate variable binding value for function C: for a given expression e, the first mapping function f and the command mapping function c, when applying function C, equivalently introduce a temporary variable through the command new, assign let_binding ef to the temporary variable, and then pass the expression value of the temporary variable to the command mapping function c, where function C is a mapping operation from an expression to a command; (5): Define the convolution operation of the functional tensor expression in the Coq theorem prover, and use the rewrite transformation rule in step (4) to convert the convolution operation into the corresponding C language convolution code; (6): Define functional tensor expressions of other AI compiler core operators in the Coq theorem prover. Other AI compiler core operators include pooling, activation, and full connection.

2. A method for rewriting and converting a functional convolution expression into C code according to claim 1, characterized in that: The implementation of step (2) defines an intermediate variable binding value let_binding, which accepts two type parameters d1 and d2, as well as an expression e and a second mapping function, and passes the expression e of type d1 to the second mapping function. The function of the second mapping function is to map an expression from the type parameter d1 to the expression of the type parameter d2.

3. A method for rewriting and converting a functional convolution expression into C code according to claim 2, characterized in that: The implementation of step (3) specifically includes: Define the intermediate imperative combinator mkvSeql of the tensor primitive, which accepts two parameters: one is the third mapping function from the tensor receiver constructor to the command, where the third mapping function constructed by the tensor receiver is a mapping function from a finite set {0,1,...,n-1} to the receiver type, and the other parameter is the receiver type of the tensor with a length of n and a data type of d. The third mapping function operation is performed on each element i in the finite set using the seqfor command, where seqfor is an imperative loop operation; An intermediate imperative combinator letl that defines an intermediate variable binding value is used to pass an expression e and a receiver a to a third mapping function, where e is an expression with a type parameter of d1, a is a receiver with a type parameter of d2, and the third mapping function f is a mapping function from an expression of type d1 to an expression of type d2.

4. The method for rewriting and converting a functional convolution expression into a C code according to claim 1, characterized in that: The implementation of step (5) specifically includes: The filling function of the two-dimensional tensor constructs the output tensor through a nested two-dimensional tensor structure. The outer layer constructs rows and the inner layer constructs columns. The elements of the output tensor are determined by whether the element at the index is within the range of the input tensor. If it is within the range of the input tensor, the corresponding element of the input tensor e is used as the output element at the current index. If it is not within the range of the current input tensor, the default element zero is used as the output element; The convolution function of a two-dimensional tensor first introduces an intermediate variable binding value to store the result after the input tensor is filled; then a nested two-dimensional tensor structure is constructed, in which the internal function is used to represent each element of the output tensor; finally, the rewriting transformation rule in step (4) is used to convert the functional convolution expression into the corresponding imperative combinator.

5. The method for rewriting and converting a functional convolution expression into a C code according to claim 1, characterized in that: The implementation of step (6) specifically includes: Define the maximum pooling function of the two-dimensional tensor. The rows and columns of the output tensor are represented by constructing a nested two-dimensional tensor structure. The subscript index is used to iterate each element in the window. The maximum value in the nested tensor is used as the element of the output tensor. Define the average pooling function of the two-dimensional tensor. First, construct a nested two-dimensional tensor. The output element at the current index is the sum of the elements in the pooling window divided by the number of elements in the pooling window. Define the ReLU activation function of the two-dimensional tensor. First, construct a nested two-dimensional tensor structure to output the rows and columns of the tensor. Use the row index and column index to traverse the elements of the input tensor and determine whether the current element is less than or equal to zero. If so, replace it with zero. Otherwise, keep the original value. Define the fully connected operation by constructing a one-dimensional tensor structure, where the internal element is a sum of the sum of the nested tensor elements and the corresponding bias elements, where the sum of the nested tensor elements is the product of the input element and the corresponding weight element and then the sum is accumulated.

Images