Method and device for operator automatic tensor lifting and mathematical semantic verification

Abstract

The application discloses a kind of operator automatic tensor promotion and mathematical semantic verification method and device, belong to program analysis and formal verification field.This method includes: first, the bottom layer implementation of target operator is carried out symbol execution, extracts the symbol constraint between its input and output as behavior specification;Subsequently, based on the program synthesis strategy of combination of divide and conquer search and enumeration, candidate mathematical formula satisfying specification is automatically synthesized in high-level tensor operator language;Then, the correctness of synthesis formula on any scale input is formally verified in combination with Hall logic and property transformation technology;Finally, using algebraic rewriting technique based on equivalent saturation to simplify complex formula automatically, restore its natural, intuitive mathematical expression form.The present application can not only significantly improve the understandability and reliability of operator implementation, but also assist developers in auditing, debugging and reusing complex operators, improve the reliability and security of deep learning system.

Description

Technical Field

[0001] This invention relates to the field of program analysis and formal verification, and in particular to a method and apparatus for automatically upscaling the underlying implementation of deep learning operators into mathematical semantic expressions and performing formal verification and semantic simplification. Specifically, it relates to a method and apparatus for automatic tensor upscaling and mathematical semantic verification of operators. Background Technology

[0002] Deep learning operators are the core computational units in modern deep learning frameworks, essentially performing complex numerical operations on tensors. As model size and complexity increase, developers increasingly optimize performance through custom operators (such as the Triton kernel). However, these operators typically involve numerous hardware-related optimizations (such as tiling, fusion, and parallel thread scheduling) and numerical stability handling (such as the max-shift technique in softmax), resulting in a significant semantic gap between their implementation and the original mathematical definitions. Existing research mainly focuses on: 1) discovering precision or dimensionality errors based on testing or black-box methods; and 2) verifying concurrency or synchronization characteristics. However, a technique is lacking that can automatically recover the high-level mathematical semantics from the underlying operator implementation and formally verify its functional correctness. This makes it difficult for developers to understand, reuse, or verify highly optimized operator implementations, severely restricting the reliability and maintainability of deep learning systems. Summary of the Invention

[0003] The purpose of this invention is to address the shortcomings of existing technologies by providing a method and apparatus for automatic operator tensor lifting and mathematical semantic verification.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: a method for automatic tensor lifting and mathematical semantic verification of operators, comprising the following steps:

[0005] (1) By analyzing the standard operators and their implementations in the deep learning framework, the semantic patterns of high-level tensor operations and their corresponding equivalent transformation rules are extracted for precise symbolic execution.

[0006] (2) Analyze the underlying implementation of the target operator using symbolic execution techniques, extract the symbolic constraints between its input and output, and use them as behavioral specifications for tensor lifting;

[0007] (3) Based on the program synthesis method, search for candidate high-level tensor programs that satisfy the behavioral specification constraints in the predefined tensor operator language space;

[0008] (4) Formal verification techniques are used to verify the functional correctness of the candidate high-level tensor program on inputs of arbitrary scale;

[0009] (5) Using algebraic equivalent rewriting and equivalent saturation techniques, the verified high-level tensor program is automatically simplified to generate the final mathematical semantic expression of the target operator.

[0010] (6) Output the mathematical semantic expression of the target operator as its functional semantic description.

[0011] Further, step (1) specifically involves: performing static analysis and manual induction on the high-performance implementation of standard operators in the deep learning framework, identifying the corresponding high-level tensor operation modes in the underlying implementation; and extracting algebraic equivalence transformation rules corresponding to the tensor operation modes based on the optimization strategies in the implementation of deep learning operators, which are used to describe the semantic equivalence relationships between different tensor expressions.

[0012] Furthermore, in step (2), the symbolic execution includes expanding the parallel thread execution path and converting tensor access operations into index-level symbolic expressions.

[0013] Furthermore, in step (3), a hybrid synthesis strategy combining top-down divide-and-conquer search and bottom-up enumeration is used to generate candidate high-level tensor programs.

[0014] Furthermore, in step (4), the Hall logic and property transformation method is used to transform the array-level verification conditions into element-by-element verification conditions in order to reduce the verification complexity.

[0015] Furthermore, in step (5), based on the e-graph equivalence saturation mechanism, the complex tensor expression is rewritten in multiple rounds until an equivalent closure is achieved.

[0016] Furthermore, the method also includes: if a candidate high-level tensor program fails to perform functional correctness verification in step (4), it returns to step (3) to search again; if the expression size is greater than a preset value in the equivalence saturation process in step (5), it is pruned through a cost function.

[0017] To achieve the above objectives, the present invention also provides an apparatus for automatic tensor lifting and mathematical semantic verification of operators, comprising one or more processors for implementing the above method.

[0018] To achieve the above objectives, the present invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above method.

[0019] To achieve the above objectives, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0020] The beneficial effects of this invention are as follows: Existing methods struggle to recover the high-level mathematical semantics of highly optimized operator implementations, and cannot provide formal guarantees for their functional correctness. This invention, however, achieves automatic enhancement and verification of the functional semantics of deep learning operators through a combination of symbolic execution, program synthesis, formal verification, and semantic simplification. This invention not only significantly improves the understandability and credibility of operator implementations but also assists developers in auditing, debugging, and reusing complex operators, thereby enhancing the reliability and security of deep learning systems. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the overall process framework of the method of this invention.

[0022] Figure 2 This is a schematic diagram of the present invention based on the equivalent saturated e-graph structure and the semantic simplification of the Softmax operator.

[0023] Figure 3 This is a schematic diagram of the structure of the device of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0025] like Figure 1 As shown, this embodiment of the invention provides a method for automatic tensor lifting and mathematical semantic verification of operators, including the following steps:

[0026] Step 1: By analyzing the standard operators and their high-performance implementations in the deep learning framework, we extract the semantic patterns of high-level tensor operations and their corresponding equivalent transformation rules for precise symbolic execution.

[0027] Step 2: Analyze the underlying implementation of the target operator using symbolic execution techniques, and extract the symbolic constraints between its inputs and outputs as behavioral specifications for tensor lifting.

[0028] Step 3: Based on the program synthesis method, search for candidate high-level tensor programs that satisfy the behavioral specification constraints in the predefined tensor operator language space.

[0029] Step 4: Use formal verification techniques to verify the functional correctness of the candidate high-level tensor program on inputs of arbitrary size.

[0030] Step 5: Using algebraic equivalent rewriting and equivalence saturation techniques, the verified high-level tensor program is automatically simplified to generate the final mathematical semantic expression of the target operator.

[0031] Step 6: Output the mathematical semantic expression of the target operator as its functional semantic description.

[0032] This invention focuses on highly optimized operator kernels within deep learning frameworks. Experimental samples are selected from open-source code repositories, including Triton, CUDA, and PyTorch kernel implementations such as softmax, layernorm, gelu, and matmul. These operators generally incorporate complex optimizations such as block partitioning, fusion, parallel scheduling, and numerical stability handling, resulting in significant differences between their underlying implementations and high-level mathematical semantics.

[0033] As a preferred embodiment, step 1 specifically involves: performing static analysis and manual summarization of high-performance implementations of common operators in deep learning frameworks to identify the corresponding high-level tensor operation modes in the underlying implementations, including but not limited to element-wise operation mode, reduction operation mode, tensor broadcast mode, and matrix multiplication mode. Based on common optimization strategies in deep learning operator implementations, algebraic equivalence transformation rules corresponding to the aforementioned tensor operator templates are extracted. These equivalence transformation rules describe the semantic equivalence relationships between different tensor expressions, including but not limited to associative laws, distributive laws, reduction commutative laws, and numerical stability equivalence transformations.

[0034] Among them, steps 2-5 represent the four core stages of this invention: operator behavior specification generation, high-level tensor program synthesis, formal verification, and semantic simplification and equivalence saturation.

[0035] As a preferred embodiment, step 2, representing the operator behavior specification generation stage, specifically involves: firstly, performing symbolic execution on the target operator kernel, representing each element of the input tensor as a symbolic variable, and expanding the thread execution path and index calculation logic to obtain the symbolic expression of the output element with respect to the input element, thereby generating the tensor-level behavior specification.

[0036] To reduce the complexity of subsequent verification and synthesis, this invention further transforms array-level constraints into index-level expressions and eliminates control flow branches unrelated to thread scheduling. Through symbolic execution analysis, the output elements can be obtained. About input elements The symbolic expression is: The above expression constitutes the tensor-level behavioral specification of the softmax operator, describing the functional relationship between the output and input tensors. This behavioral specification does not depend on specific thread parallel structures or low-level optimizations; it only reflects the mathematical semantics of the operator. In subsequent steps, this behavioral specification will serve as a constraint in the program synthesis phase, used to search for high-level tensor programs that satisfy this specification.

[0037] Those skilled in the art will understand that the softmax operator described above is merely an example, and the method of the present invention is also applicable to other deep learning operators such as layernorm, gelu, and matrix multiplication.

[0038] As a preferred embodiment, step 3, representing the high-level tensor program synthesis stage, specifically involves: This invention defines a high-level tensor language for tensor promotion, whose syntax includes basic operators, element-wise mappings, reduction, broadcasting, index mappings, and other constructs.

[0039] In this language space, the present invention employs syntax-guided program synthesis technology, combining top-down divide-and-conquer search with bottom-up enumeration to generate subexpressions, to automatically construct candidate high-level tensor programs that meet behavioral specifications.

[0040] For example, the high-level tensor language structure is as follows:

[0041] .

[0042] As a preferred embodiment, step 4 represents the formal verification stage, which specifically involves constructing verification conditions based on Hall logic for each candidate tensor program and introducing property transformation technology to transform the tensor-level verification conditions into element-wise verification conditions, thereby significantly reducing the complexity of SMT solution.

[0043] By inductively verifying the tensor dimension parameters, it is ensured that the synthesized formula remains functionally equivalent under any input scale. After constructing the element-by-element verification conditions, the SMT solver is used to automatically determine the expressive equivalence between the candidate tensor program and the behavioral specification. Simultaneously, by inductively verifying the tensor dimension parameters, it is ensured that the synthesized tensor program remains functionally equivalent to the behavioral specification of the target operator under any input scale.

[0044] As a preferred embodiment, such as Figure 2 As shown, step 5, representing the semantic simplification and equivalence saturation stage, specifically involves: using an e-graph structure based on equivalence saturation to algebraically rewrite the validated high-level tensor program. The system repeatedly applies equivalence transformation rules to gradually merge complex expressions into equivalence classes until an equivalence closure is achieved, thereby outputting a final mathematical expression with a concise structure and natural semantics.

[0045] Figure 2 Taking the softmax operator as an example, this paper demonstrates the automatic merging and simplification process of expressions such as exponential shift, normalization, and reduction in e-graph.

[0046] In a preferred embodiment, the method further includes: if a candidate program fails in the functional correctness verification stage in step 4, returning to step 3 to search again; if the expression scale is too large in the equivalence saturation process, pruning is performed through a cost function to ensure stable system operation.

[0047] This invention can be widely applied to scenarios such as operator auditing, compiler verification, automatic document generation, and the construction of highly reliable deep learning systems.

[0048] Corresponding to the aforementioned embodiments of the method for automatic operator tensor lifting and mathematical semantic verification, the present invention also provides an embodiment of an apparatus for automatic operator tensor lifting and mathematical semantic verification.

[0049] like Figure 3 As shown in the figure, an apparatus for automatic tensor lifting and mathematical semantic verification of operators provided in this embodiment of the invention includes one or more processors for implementing the methods in the above embodiments.

[0050] Embodiments of the device of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The device embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 3 The diagram shown is a hardware structure diagram of any device with data processing capabilities, including the device of this invention. Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0051] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0052] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0053] This invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the method of the above embodiments.

[0054] This invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the method described in the above embodiments.

[0055] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0056] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions or improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0057] The above embodiments are only used to illustrate the design concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The scope of protection of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the scope of protection of the present invention.

Claims

1. A method for automatic tensor lifting and mathematical semantic verification of operators, characterized in that, Includes the following steps: (1) By analyzing the standard operators and their implementations in the deep learning framework, the semantic patterns of high-level tensor operations and their corresponding equivalent transformation rules are extracted for symbolic execution; (2) The underlying implementation of the target operator is analyzed by symbolic execution technology, its input-output relationship is extracted and symbolic constraints are generated as behavioral specifications for tensor lifting; (3) Based on the program synthesis method, search for candidate high-level tensor programs that satisfy the behavioral specification constraints in the predefined tensor operator language space; (4) Formal verification techniques are used to verify the functional correctness of the candidate high-level tensor program on inputs of arbitrary scale; (5) Using algebraic equivalent rewriting and equivalent saturation techniques, the verified high-level tensor program is simplified to generate the mathematical semantic expression of the target operator. (6) Output the mathematical semantic expression of the target operator as its functional semantic description.

2. The method according to claim 1, characterized in that, The specific steps (1) are as follows: static analysis and manual summarization of the high-performance implementation of standard operators in the deep learning framework, identifying the corresponding high-level tensor operation modes in the underlying implementation; and extracting the algebraic equivalence transformation rules corresponding to the tensor operation modes based on the optimization strategies in the implementation of deep learning operators, which are used to describe the semantic equivalence relationship between different tensor expressions.

3. The method according to claim 1, characterized in that, In step (2), symbolic execution includes expanding the parallel thread execution path and converting tensor access operations into index-level symbolic expressions.

4. The method according to claim 1, characterized in that, In step (3), a hybrid synthesis strategy combining top-down divide-and-conquer search and bottom-up enumeration is used to generate candidate high-level tensor programs.

5. The method according to claim 1, characterized in that, In step (4), the Hoare logic AND property transformation method is used to transform the array-level verification conditions into element-by-element verification conditions in order to reduce the verification complexity.

6. The method according to claim 1, characterized in that, In step (5), based on the e-graph equivalence saturation mechanism, the complex tensor expression is rewritten in multiple rounds until an equivalent closure is achieved.

7. The method according to claim 1, characterized in that, The method further includes: if a candidate high-level tensor program fails to perform functional correctness verification in step (4), it returns to step (3) to search again; if the expression size is greater than a preset value in the equivalence saturation process in step (5), it is pruned through a cost function.

8. An apparatus for automatic tensor lifting and mathematical semantic verification of operators, characterized in that, It includes one or more processors for implementing the method according to any one of claims 1-7.

9. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method described in any one of claims 1-7.

Images