Code generation method, electronic device, and storage medium

Abstract

The application relates to the field of artificial intelligence, and provides a code generation method, an electronic device and a storage medium, the method comprising the following steps: obtaining a tensor intermediate representation of a target operator; inserting a synchronization operation in the tensor intermediate representation based on a data dependency relationship of the tensor intermediate representation, so as to obtain a synchronization tensor intermediate representation; automatically splitting the synchronization tensor intermediate representation into a main function and a plurality of coroutine sub-functions, the main function being used for calling each coroutine sub-function, and different coroutine sub-functions corresponding to different cooperative thread bundles; and automatically generating running code of the target operator based on the main function and the plurality of coroutine sub-functions. The method, the electronic device and the storage medium provided by the application realize automatic generation of fine-grained asynchronous parallel code of a complex logic operator, reduce the development workload and development cost of the operator, improve the development efficiency of the operator, and optimize the performance of the operator. Moreover, the method can be used for code implementation of various operators, is not limited by the number and type of operators, and has good generalization and flexibility.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a code generation method, an electronic device and a storage medium. BACKGROUND

[0002] Existing AI (Artificial Intelligence) compilers are relatively mature in code generation and automatic optimization of simple operators. However, when facing complex operator fine-grained asynchronous parallel scenarios, the capabilities of the AI compiler are insufficient, and the operator developers resort to manually writing code.

[0003] However, when applying manual writing methods such as handwritten operators or calling operator libraries, the problems of poor generalization, low reusability, poor flexibility and high adaptation cost often occur, which greatly increases the workload of operator development and greatly reduces the development efficiency. SUMMARY

[0004] The present application provides a code generation method, an electronic device and a storage medium to solve the defects of large workload and low efficiency in code writing of complex operator fine-grained asynchronous parallel in related technologies.

[0005] The present application provides a code generation method, comprising:

[0006] obtaining a tensor intermediate representation of a target operator;

[0007] based on the data dependency relationship of the tensor intermediate representation, inserting a synchronization operation in the tensor intermediate representation to obtain a synchronization tensor intermediate representation;

[0008] automatically splitting the synchronization tensor intermediate representation into a main function and a plurality of coroutine sub-functions, the main function being used to call each coroutine sub-function, and different coroutine sub-functions corresponding to different cooperative thread bundles;

[0009] based on the main function and the plurality of coroutine sub-functions, automatically generating running code of the target operator.

[0010] According to the code generation method provided by the present application, the synchronization tensor intermediate representation is automatically split into a main function and a plurality of coroutine sub-functions, comprising:

[0011] extracting segment information of each segment of the synchronization tensor intermediate representation in serial execution;

[0012] based on the segment information of each segment, constructing a coroutine sub-function corresponding to each segment;

[0013] construct the main function based on each segment of the synchronous tensor intermediate representation in serial execution and a corresponding coroutine sub-function of each segment.

[0014] According to the code generation method provided by the application, the segment information of each segment of the synchronous tensor intermediate representation in serial execution is extracted, including:

[0015] Based on the segmentation mark in the synchronous tensor intermediate representation, each segment of the synchronous tensor intermediate representation in serial execution is segmented, and the parameter information, loop information and function body of each segment are extracted as the segment information of each segment.

[0016] According to the code generation method provided by the application, the tensor intermediate representation of the target operator is obtained, including:

[0017] The first operator and the second operator are obtained.

[0018] The first operator and the second operator are fused to obtain a target operator, and the tensor intermediate representation of the target operator is determined.

[0019] According to the code generation method provided by the application, the first operator and the second operator are fused, including:

[0020] The cache location of the output data of the first operator is determined from the serial implementation of the first operator, the data input operation of the second operator is replaced by an operation of accessing the cache location, and the data source of the write operation of the final result output by the first operator is replaced by the output data of the second operator.

[0021] According to the code generation method provided by the application, the data dependency relationship based on the tensor intermediate representation is inserted into the tensor intermediate representation to obtain a synchronous tensor intermediate representation, and the method further includes:

[0022] The tensor intermediate representation is scheduled and optimized.

[0023] According to the code generation method provided by the application, the running code of the target operator is automatically generated based on the main function and the plurality of coroutine sub-functions, including:

[0024] The main function and the plurality of coroutine sub-functions are converted into an instruction-level intermediate representation.

[0025] Based on the instruction-level intermediate representation, an executable file of the target operator is automatically generated, and the executable file includes the running code of the target operator.

[0026] The application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the code generation method according to any one of the preceding embodiments when executing the program.

[0027] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, wherein the computer program is executable on a processor to implement the code generation method according to any one of the preceding embodiments.

[0028] The application further provides a computer program product, comprising a computer program, wherein the computer program is executable on a processor to implement the code generation method according to any one of the preceding embodiments.

[0029] The code generation method, the electronic device and the storage medium provided by the application can automatically split the tensor intermediate representation into a main function and a plurality of coroutine sub-functions after inserting a synchronization operation in the tensor intermediate representation of the target operator, realize form conversion of fine-grained asynchronous parallelism of the target operator, and generate code on this basis, so as to obtain code of fine-grained asynchronous parallelism of the executable target operator, realize automatic generation of fine-grained asynchronous parallelism code of a complex logic operator, greatly reduce the workload and the cost of operator development, improve the efficiency of operator development, and optimize the performance of the operator. Moreover, the method can be used for code implementation of various operators, is not limited to the number and type of operators, has good generalization and flexibility. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the application or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or the related art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0031] Figure 1 is a structural schematic diagram of an artificial intelligence chip provided by the application.

[0032] Figure 2 is a flowchart of the code generation method provided by the application.

[0033] Figure 3 is a structural schematic diagram of the code generation device provided by the application.

[0034] Figure 4 is a structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0036] An AI compiler is a compiler system for optimizing AI models, such as deep learning neural networks. The AI compiler mainly converts operators, subgraphs, or AI models from high-level AI model descriptions (such as TensorFlow, PyTorch, etc.) to efficient low-level hardware instructions through different levels of IR (Intermediate Representation) conversion, thereby realizing automatic code generation and automatic optimization for operators, subgraphs, or AI models.

[0037] Therefore, the application of the AI compiler can greatly improve the execution efficiency and hardware utilization of the AI model. Compared with traditional compilers, the AI compiler pays more attention to tensor computation, memory access patterns, and heterogeneous hardware acceleration.

[0038] In the AI compiler, IR is an abstract formal representation between source code and target code, used for analyzing, optimizing and converting programs in the compilation process. As a core concept in the design of AI compiler, IR has the characteristics of platform independence and semantic explicitness. Currently, IR mainly includes graph-level IR, operator-level IR and instruction-level IR, etc.

[0039] Further, the graph-level IR mainly represents the data flow and data dependency between operators in the AI model through DAG (Directed acyclic graph), which facilitates graph-level optimization. The graph-level IR usually does not describe the specific implementation details of the operators. Graph-level optimization focuses on optimizing the topological relationship between operators. Common graph-level optimizations include operator fusion, operator elimination, and operator splitting, etc.

[0040] Operator-level IR can be Tensor IR (Tensor Intermediate Representation), which is an intermediate representation designed specifically for the tensor computation details of operators and plays a key role in the AI compiler. Tensor IR can convert high-level neural network operations (such as convolution, matrix multiplication, etc.) into hardware-friendly low-level tensor operations, while retaining enough abstraction to support automatic optimization and hardware adaptation. Operator-level IR focuses on optimizing the implementation details of operators, which can be optimized through scheduling optimization such as splitting, rearranging, parallelizing, unrolling, and vectorizing loops to achieve efficient memory access and parallelism on different hardware.

[0041] Instruction-Level IR is the lowest-level intermediate representation form in an AI compiler, which directly simulates the ISA (Instruction Set Architecture) of the target hardware and focuses on low-level details such as register allocation, instruction scheduling, and memory layout. Compared with high-level IR (such as graph-level IR and operator-level IR), Instruction-Level IR is closer to machine code and is the last layer of abstraction connecting compiler optimization and hardware execution. It can be understood that Instruction-Level IR finally realizes the conversion of operators, subgraphs, or AI models into machine code or assembly of the target hardware.

[0042] Generally speaking, common AI compilers such as TVM (Tensor Virtual Machine) or MLIR (Multi-Level Intermediate Representation) can automatically generate code and automatically optimize some simple operator logic through the above-mentioned techniques. However, for the case of fine-grained asynchronous execution of complex operator logic, the AI compiler is not powerful enough.

[0043] Usually, for the compilation requirements that the AI compiler cannot automatically implement, the operator developer turns to manually write the operator or call the operator library to implement the high customization of the operator. However, when customizing the operator through manual writing, the following problems often exist:

[0044] First, a manual writing can only adapt to a very limited number of operators, and different operators usually need to be re-written and developed; second, even for one type of operator, the implementation methods corresponding to different data specifications or data types are different, and each implementation method needs to be developed respectively; this makes the generalization of the manual writing method very poor, greatly increases the workload of the operator developer, and greatly reduces the development efficiency. Third, the flexibility of manual writing is poor, which is not conducive to automatic optimization, and the running performance of the code developed therefrom is usually not optimal.

[0045] To solve the above problems, the present application provides a code generation method. Figure 1 is a structural schematic diagram of an artificial intelligence chip provided by the present application, as Figure 1As shown, the execution subject of the code generation method provided by the present application can be a processor (Central Processing Unit, CPU), and the running code of the operator obtained based on the code generation method can run on an artificial intelligence chip 100, which includes a plurality of streaming processor clusters (SPC) 101 and a display memory 102. Each streaming processor cluster 101 includes a plurality of computing units 103, and each computing unit 103 includes at least an on-chip cache 104 and a register 105.

[0046] The display memory 102 can be an off-chip memory HBM (High Bandwidth Memory), or can be other types of memories. The on-chip cache 104 is a temporary memory, and its capacity is smaller than that of the display memory 102, but the data exchange speed is faster than that of the display memory 102. The on-chip cache 104 can be a general matrix main buffer (GMB). Compared with the on-chip cache 104, the capacity of the register 105 is smaller than that of the on-chip cache 104, but the data exchange speed is faster than that of the on-chip cache 104. The register 105 can be a TLR (Thread Local Register).

[0047] It should be noted that the artificial intelligence chip 100 in the present application can include other structures in addition to the above structure, and the present application does not make specific limitations thereon. The artificial intelligence chip 100 can be a graphics processor (GPU), a general-purpose computing on graphics processing units (GPGPU), a tensor processing unit (TPU), etc.

[0048] Based on Figure 1As shown in the architecture diagram of the artificial intelligence chip, the embodiment of the present application provides a code generation method, which can be applied to automatic code generation of various operators. The operators herein can be mma (Matrix Multiply-Accumulate, matrix multiplication and accumulation), conv (convolution), or other operators such as add (addition), relu (activation function), or can also be a fusion operator, for example, a fusion operator after conv and relu. Taking the fusion operator of conv and relu as an example, the fusion operator can be applied to the field of image processing, and can be applied to image processing tasks such as target detection and image segmentation. The running code of the fusion operator generated based on the code generation method provided by the present application can be applied in the inference process of image processing models such as target detection and image segmentation. By calling the running code, inference based on the fusion operator in the image processing model can be realized, and specifically, the function of image feature extraction for input images can be realized. Taking the mma operator as an example, the mma operator can be applied to the field of natural language processing, and can be applied to natural language processing tasks such as text generation and semantic understanding. The running code of the mma generated based on the code generation method provided by the present application can be applied in the inference process of natural language processing models such as text generation and semantic understanding. By calling the running code, inference based on the mma operator in the natural language processing model can be realized, for example, attention interaction for token of input text sequence can be realized to extract attention features of the text sequence.

[0049] Figure 2 is a flowchart of the code generation method provided by the present application, as Figure 2 shown, the method comprises:

[0050] Step 210, obtaining the tensor intermediate representation of the target operator.

[0051] Here, the target operator is the operator for which the corresponding code needs to be generated. Facing the fine-grained asynchronous parallel demand of complex operators, the target operator can be a complex logic operator, the target operator can be a single operator, or a fusion operator fused by multiple operators, which is not limited by the embodiment of the present application.

[0052] The tensor intermediate representation of the target operator can be directly converted by the AI compiler. In some embodiments, in the case where the target operator itself is a fused operator, operator fusion can be performed first, and then the fused operator is converted by the AI compiler to obtain the tensor intermediate representation. Alternatively, the tensor intermediate representations of multiple operators can be obtained first, and then the target operator after fusion and the tensor intermediate representation of the target operator are obtained by fusing the tensor intermediate representations of the multiple operators.

[0053] In the embodiments of the present application, the tensor intermediate representation converted by the AI compiler can be directly applied to the subsequent steps, or after obtaining the tensor intermediate representation converted by the AI compiler, scheduling optimization can be performed on the tensor intermediate representation, and the tensor intermediate representation after scheduling optimization can be applied to the subsequent steps. The present application does not make specific limitations on this.

[0054] In step 220, a synchronization operation is inserted into the tensor intermediate representation based on the data dependency relationship of the tensor intermediate representation, to obtain a synchronization tensor intermediate representation.

[0055] Specifically, after obtaining the tensor intermediate representation of the target operator, the data dependency relationship existing in the tensor intermediate representation can be analyzed. The data dependency relationship here is used to reflect the steps that exist data dependency in the execution logic of the tensor intermediate representation.

[0056] For example, in the case where the target operator is a single operator, the data dependency relationship can be the dependency relationship between the data partitioning, data loading, data calculation, result storage and other steps in the single operator. Specifically, it can be embodied that the input data of data loading is the output data of data partitioning, the input data of data calculation is the output data of data loading, the input data of result storage is the output data of data calculation, and the like.

[0057] For example, if the target operator is a fusion operator, the data dependency relationship can include not only the dependency relationship between the data split, data loading, data calculation, result storage and other steps within each operator in the fusion operator, but also the data dependency relationship between operators, such as the output of one operator being the input of another operator. Assuming that the target operator is a fusion operator of a tcore (Tensor Core) operator and a vcore (Vector Core) operator, the serial implementation of the tcore operator can already include partial synchronization logic of the tcore, such as setting different data blocks to different pipeline ids for asynchronous parallelism, and the like. In this case, the insertion of the synchronization operation in step 220 is more for the synchronization operation between the tcore and the vcore, and thus the data dependency relationship can be analyzed for the tensor intermediate representation of the fused target operator, and the synchronization operation is inserted accordingly.

[0058] After obtaining the data dependency relationship of the tensor intermediate representation, the synchronization operation can be inserted in the tensor intermediate representation based on the data dependency relationship. By inserting the synchronization operation, the synchronization logic between the steps in the target operator can be established in the tensor intermediate representation, providing the necessary premise for implementing fine-grained asynchrony, thereby avoiding errors such as data race and deadlock when implementing fine-grained asynchrony subsequently.

[0059] In an embodiment of the present application, the tensor intermediate representation after the insertion of the synchronization operation is referred to as a synchronized tensor intermediate representation.

[0060] In step 230, the synchronized tensor intermediate representation is automatically split into a main function and a plurality of coroutine sub-functions, the main function being used to call each coroutine sub-function, and different coroutine sub-functions corresponding to different cooperative thread bundles.

[0061] Specifically, after obtaining the synchronized tensor intermediate representation, the synchronized tensor intermediate representation can be automatically split, thereby converting the synchronized tensor intermediate representation into a form of a main function and a plurality of coroutine sub-functions. It can be understood that the splitting of the synchronized tensor intermediate representation does not require human operation and can be automatically performed by an AI compiler.

[0062] The main function can be regarded as a top-level scheduling function of the target operator, responsible for managing the execution flow of the target operator, specifically for calling the coroutine sub-functions in the execution flow. The coroutine sub-function is an independent computing unit split from the target operator, each coroutine sub-function representing an asynchronous operation within the target operator, and different coroutine sub-functions being able to be independently and asynchronously executed on different cooperative warp (Cwarp) bundles.

[0063] In which, threads can be organized into thread blocks (Block) or sub-thread blocks (SubBlock), and each thread block or sub-thread block is further divided into a fixed size thread group, called a thread bundle (Warp). A thread bundle usually contains 32 threads.

[0064] Cooperative thread bundle is a special thread synchronization mechanism that allows threads within the same thread bundle (Warp) to break through the limitation that traditional synchronization can only be whole thread block synchronization and cannot synchronize part of threads, and to cooperate in a more close manner. At the same time, different cooperative thread bundles in the same sub-thread block can execute different operations asynchronously and share registers and other resources. Moreover, cooperative thread bundle allows threads within the thread bundle to cooperate independently of the thread block or sub-thread block, thereby realizing fine-grained synchronization, barrier (Barrier) synchronization, and efficient data sharing. The fine-grained synchronization referred to here means synchronizing only the threads within the thread bundle or between the cooperative thread bundles, without waiting for the whole thread block to synchronize; the barrier synchronization means realizing thread bundle-level barrier through instructions; and the data sharing means efficiently sharing data within the thread bundle or between the cooperative thread bundles, reducing global memory access.

[0065] In the embodiment of the present application, the synchronous tensor intermediate representation is automatically split into a main function and a plurality of coroutine sub-functions, and different cooperative thread bundles are allocated to different coroutine sub-functions, so that in the execution flow of the target operator, different cooperative thread bundles can independently and asynchronously run different program codes, thereby independently and asynchronously realizing different coroutine sub-functions, that is, independently and asynchronously realizing different operations in the target operator, that is, realizing fine-grained asynchronous parallelism.

[0066] It can be understood that the process of automatically splitting the synchronous tensor intermediate representation into a main function and a plurality of coroutine sub-functions can be understood as the process of converting the serially executed target operator into fine-grained asynchronous parallelism.

[0067] For example, in a type of artificial intelligence chip, in the mode of cooperative thread bundle Cwarp enabled, one EU (Execution Unit) can run up to 8 Cwarps at the same time. Assuming that a group of thread bundles warp with equal cooperative thread bundle indexes cwarp_idx can be regarded as a Cwarp, and one Cwarp contains 16 warps. The 8 Cwarps (128 warps, 4096 threads) in the same EU can share 256 TLR register resources. In the case of splitting the synchronous tensor intermediate representation into multiple coroutine sub-functions, there is a corresponding Cwarp for each coroutine sub-function, and the purpose of hide-latency can be achieved by letting different Cwarps execute different coroutine sub-functions, for example, Cwarp0 executes coroutine sub-function 0 to realize data loading, Cwarp1 executes coroutine sub-function 1 to realize data calculation, and Cwarp2 executes coroutine sub-function 2 to realize data storage.

[0068] In step 240, the running code of the target operator is automatically generated based on the main function and the plurality of coroutine sub-functions.

[0069] Specifically, after obtaining the main function and the plurality of coroutine sub-functions, the running code of the target operator can be automatically generated based thereon. It can be understood that the main function and the plurality of coroutine sub-functions are function forms of the target operator at the level of the tensor intermediate representation, and the main function and the plurality of coroutine sub-functions can be converted to the level of the instruction-level IR, and then the commonly used conversion steps such as instruction selection, instruction scheduling, and register allocation at the bottom layer are performed, so as to realize the code automatic generation for the target operator. It can be understood that the generation of the running code for the target operator does not require human operation, and can be automatically implemented by the AI compiler.

[0070] In the method provided in the embodiments of the present application, after inserting the synchronization operation in the tensor intermediate representation of the target operator, the tensor intermediate representation is automatically split into the main function and the plurality of coroutine sub-functions, the form conversion of the fine-grained asynchronous parallelism of the target operator is realized, and the code generation is performed on this basis, so as to obtain the code of the fine-grained asynchronous parallelism of the executable target operator, realize the automatic generation of the fine-grained asynchronous parallelism code of the complex logic operator, greatly reduce the workload and the operator development cost of the operator development, improve the operator development efficiency, and optimize the operator performance. Moreover, the method can be used for the code implementation of various operators, is not limited to the number and type of operators, has good generalization and flexibility.

[0071] Based on the above embodiments, in step 230, the automatic splitting of the synchronous tensor intermediate representation into the main function and the plurality of coroutine sub-functions includes:

[0072] extracting segment information of each segment in serial execution of the synchronous tensor intermediate representation;

[0073] constructing a coroutine sub-function corresponding to each segment based on the segment information of each segment;

[0074] constructing the main function based on each segment in serial execution of the synchronous tensor intermediate representation and the coroutine sub-function corresponding to each segment.

[0075] Specifically, first, the serial execution process of the synchronous tensor intermediate representation can be divided, thereby obtaining a plurality of segments in the serial execution process of the synchronous tensor intermediate representation. Here, each segment is a segment of execution logic in the serial execution process of the synchronous tensor intermediate representation, thereby obtaining segment information of each segment. Here, the segment information can reflect parameters applied in the execution process, loop structures implemented, and execution logic of the segment, and the like.

[0076] For each segment, a coroutine sub-function corresponding to the segment can be constructed based on the segment information of the segment itself. Specifically, the execution logic in the segment information can be taken as the function body of the coroutine sub-function, the parameter list of the coroutine sub-function can be generated based on the parameters in the segment information, the loop logic of the coroutine sub-function can be generated based on the loop structure in the segment information, and the like, which are not limited by the embodiments of the present application.

[0077] After obtaining the coroutine sub-function corresponding to each segment, the main function for calling each coroutine sub-function can be generated based thereon. It can be understood that for construction of the main function, the calling logic of the main function for calling each coroutine sub-function can be determined based on the loop order of each segment in serial execution of the synchronous tensor intermediate representation, and the parameters of each coroutine sub-function can be converted into the arguments of the main function when calling each coroutine sub-function, and the order mapping relationship between the parameters and the arguments is preserved, thereby obtaining the main function.

[0078] Further, an Alloca Buffer can be constructed to provide memory space for tensors declared by each coroutine sub-function, and the shape, layout, and data type of the tensors and the like can be stored in the Alloca Buffer. In addition, shared data between the coroutine sub-functions can be passed through the main function in the form of function parameters.

[0079] For example, assuming that the target operator is a fusion operator of a tcore operator and a vcore operator, the target operator can be automatically split into four coroutine sub-functions, where coroutine sub-function 1 performs data reading of a type of tensor in the tcore, coroutine sub-function 2 performs data reading of another type of tensor in the tcore, coroutine sub-function 3 performs calculation of the tcore and writes the calculation result to a shared memory block, and coroutine sub-function 4 performs a post-operation of the vcore.

[0080] Based on any of the above embodiments, in step 230, the extracting of the segment information of each segment of the synchronous tensor intermediate representation in serial execution includes:

[0081] Based on the split marker in the synchronous tensor intermediate representation, each segment of the synchronous tensor intermediate representation in serial execution is split, and the parameter information, the loop information and the function body of each segment are extracted as the segment information of each segment.

[0082] Specifically, the user can set a split marker in the synchronous tensor intermediate representation in advance, so as to mark the segment split position in the synchronous tensor intermediate representation, and thus flexibly control the split case of the target operator when splitting the main function and the coroutine sub-function.

[0083] Correspondingly, the split marker can be automatically recognized in the serial execution process of the synchronous tensor intermediate representation, and the synchronous tensor intermediate representation is split based on the position marked by the split marker, so as to obtain a plurality of segments, and for each segment, a corresponding coroutine sub-function is constructed.

[0084] For a single segment, the segment information of the segment can include parameter information, loop information and a function body. The parameter information can be understood as an argument applied by the segment, where the argument refers to a variable defined when a function is defined, and the argument can be understood as a placeholder inside the function. The opposite concept is the actual argument, which is the specific value passed to the function when the argument is called. The loop information refers to the loop structure involved in the segment, and the function body refers to the implementation logic of the segment, which can usually directly use the segment itself as the function body.

[0085] Correspondingly, when constructing the coroutine sub-function, the function body can be directly applied to create the coroutine sub-function, and the parameter information can be sorted into the argument list of the coroutine sub-function, which can specifically include the input / output parameters, shared variables and semaphores of the coroutine sub-function; in addition, in the case where there is a loop structure in the loop information, the loop structure can be directly applied to the coroutine sub-function.

[0086] In the embodiment of the present application, the split marker of manual writing is introduced to split the synchronous tensor intermediate representation, so as to realize the combination of manual writing and automation, so that the code generation method can make full use of the efficiency advantage of automatic generation, and overcome the defects of automatic difficulty of complex logic operators, and further improve the flexibility of the code generation method.

[0087] Based on any of the above embodiments, in step 230, the automatic splitting of the synchronous tensor intermediate representation into a main function and a plurality of coroutine sub-functions includes:

[0088] From the serial implementation of the synchronous tensor intermediate representation, the parameter information, loop information and function body information of each segment are obtained, and the main function and each coroutine sub-function are constructed based on the information.

[0089] Considering that the main function needs to call each coroutine sub-function, the correspondence between the formal parameters and the actual parameters needs to be determined during the construction of each function. For example, for the main function, the parameters when calling the coroutine sub-function in the main function are the actual parameters; for each coroutine sub-function, the parameter list in each coroutine sub-function is actually the formal parameter list.

[0090] Suppose the target operator is a fusion operator of tcore and vcore operators, after splitting the synchronous tensor intermediate representation of the target operator, the main function fused_br_nn_matmul_br_relu and two coroutine sub-functions subroutine0 and subroutine1 are obtained, and subroutine0 and subroutine1 correspond to tcore operation and vcore operation respectively. There are parameters v0.data, v1.data, etc. in the main function, which are all actual parameters, and there are parameters v0, v1, etc. in the coroutine sub-function, which are all formal parameters. When constructing the main function and the coroutine sub-function, the formal parameters and the actual parameters must be one-to-one corresponding, and the order of the formal parameters and the order of the actual parameters should be consistent.

[0091] When constructing the coroutine sub-function, it is necessary to create the function name, the formal parameter list, create the Buffer mapping table, and create the function body. Among them, the Buffer mapping table is used to associate the received data with the type, shape and other information in the mapping table to form a complete Buffer object; the function body can be a code block containing the split segment loop, calculation logic and synchronization operation.

[0092] Moreover, for the case that the split point of the segment is inside a shared loop, that is, there is a common loop, the common loop can be identified by IR analysis, and then the corresponding loop structure is repeatedly created in the coroutine sub-function of the segment corresponding to the split point.

[0093] Based on any of the above embodiments, step 210, the tensor intermediate representation of the target operator is obtained, including:

[0094] The first operator and the second operator are obtained.

[0095] The first operator and the second operator are fused to obtain a target operator, and a tensor intermediate representation of the target operator is determined.

[0096] Specifically, the first operator and the second operator can be two operators to be fused, and the first operator and the second operator specifically represent two different mathematical operations or logical operations. For example, the first operator can be an mma operator in the tcore operator type, and the second operator can be a relu operator in the vcore operator type.

[0097] After the first operator and the second operator are obtained, the first operator and the second operator can be fused, and the fused operator is taken as the target operator. Here, the way of operator fusion can be to obtain the serial implementation of the operator running in front, and then inline the operator running behind to the output result of the operator running in front, thereby realizing the fusion of the two operators. After the target operator is obtained, the target operator can be converted into a tensor intermediate representation based on an AI compiler.

[0098] Thus, based on the code generation method provided in the embodiments of the present application, fine-grained asynchronous parallel code generation of fused operators and complex logic can be realized.

[0099] Based on any of the above embodiments, in step 210, the first operator and the second operator are fused to obtain a target operator, including:

[0100] The cache location of the output data of the first operator is determined from the serial implementation of the first operator, the data input operation of the second operator is replaced by an operation of accessing the cache location, and the data source of the write operation of the final result of the first operator is replaced by the output data of the second operator.

[0101] Specifically, for the case that the first operator runs in front and the second operator runs behind, i.e., the output data of the first operator is the input data of the second operator, in order to realize the fusion of the first operator and the second operator, the serial implementation of the first operator can be obtained through an AI compiler.

[0102] On this basis, the cache location of the output data of the first operator can be determined from the implementation of the first operator, i.e., the location of caching the output data after the first operator completes the calculation of the output data in the running process. The cache location here can be the index of Buffer.

[0103] After obtaining the cache location, the data input operation of the second operator can be adjusted to access this cache location. In this way, the output data of the first operator no longer becomes the input data of the second operator through traditional data transfer. Instead, by rewriting the input logic of the second operator, it can directly read data from the internal cache where the first operator stores its output data. Thus, the second operator is inlined to the output of the first operator, which is equivalent to injecting the computational logic of the second operator into the execution flow of the first operator without changing its structure, thereby achieving the fusion of the first and second operators.

[0104] Furthermore, based on the fusion of the first and second operators, to avoid outputting intermediate results (actually the output data of the first operator) during operator execution, the data source for the write operation of the first operator's final output can be modified. This changes the original scenario of outputting the first operator's output via a write operation to outputting the second operator's output via a write operation. Thus, after operator fusion, the final result of the operator, i.e., the output data of the second operator, can be output through the write operation originally in the first operator.

[0105] For example, operator fusion for a first operator being the tcore operator and a second operator being the vcore operator can be achieved through the following steps:

[0106] Step 1: Define the graph, nodes, and edges to reorganize the data flow graph:

[0107] Nodes represent computational operations, and edges represent the flow of data between nodes, such as tensors; a graph represents a computational process based on nodes and edges.

[0108] Step 2: Reconstruct the data flow graph based on the output tensor, and record the input nodes of the current data flow graph:

[0109] By tracing back from the final output tensor, all the nodes and input tensors necessary to compute the output tensor can be found, and the reconstructed data flow graph no longer contains any irrelevant computations.

[0110] Step 3: Using the input node of tcore in the input node as the previous dominator, traverse all paths, inline the nodes in the path, and mark the last inlined node as consumer_inline. Here, consumer_inline is the node used for tcore fusion.

[0111] Step 4: Take the input node of vcore in the input node as the dominating point, traverse the remaining path in the data flow graph, and inline the nodes in the path.

[0112] It can be understood that the calculation logic of the first operator and the second operator is sorted out through steps 1 to 4.

[0113] Step 5: Traverse the code block of tcore in the tensor intermediate representation, and query the buffer corresponding to the output tensor of tcore.

[0114] Step 6: Get the index of the buffer corresponding to the output tensor of tcore, and construct a mapping relationship between the index and the output variable of vcore.

[0115] Step 7: Traverse the code block of the tensor intermediate representation of vcore, query the output tensor of tcore, and replace it with the value output value read from the buffer index corresponding to the output tensor of tcore in step 6.

[0116] Step 8: Traverse the code block of the tensor intermediate representation of vcore, query the input tensor of vcore, and replace the input tensor according to the mapping relationship established in step 6.

[0117] It can be understood that through steps 5 to 8, the data input operation of the second number is replaced by accessing the cache location of the output data of the first operator, thereby injecting the calculation logic of the second operator into the execution flow of the first operator, and realizing the fusion of the first operator and the second operator.

[0118] Step 9: Rewrite the output value of tcore as the output value of vcore after inlining.

[0119] Step 10: Mark the scheduling handle stage of the fused tcore as kInlineAlready (processed), put the fused node into vcore, and mark the scheduling handle stage of vcore as kGroupRoot (root node).

[0120] Step 11: In the scheduling operation ScheduleOps process, bind the original tensor of vcore stage to the corresponding buffer area buffer_bind generated by the fused buffer.

[0121] It can be understood that, by steps 9 to 11, the data source of the write operation of the first operator outputting the final result is replaced by the output data of the second operator, so that the final output of the fused operator is realized.

[0122] In the method provided in the embodiment of the application, the data input operation of the second operator is replaced by an operation of accessing the cache location of the output data of the first operator, and the data source of the write operation of the first operator outputting the final result is replaced by the output data of the second operator, so that the fusion of the first operator and the second operator is realized. Since the fusion of the first operator and the second operator is embodied by injecting the calculation logic of the second operator into the execution flow of the first operator, in the target operator after fusion, the structure of the first operator remains unchanged. Since the first operator is a pre-run operator, it can be understood as a producer, and the splitting and loop order of the producer are more critical to the performance of the target operator. In the embodiment of the application, the target operator obtained by operator fusion retains the structure of the producer, providing a good foundation for subsequent fine-grained asynchronous parallel code generation.

[0123] It can be understood that, in the related art, the way adopted by operator fusion is called compute_inline, that is, the compute operation of the producer is inlined into the calculation of the consumer, avoiding the generation of an independent intermediate tensor. Here, the pre-run operator can be regarded as a producer, and the post-run operator can be regarded as a consumer. For example, the code implementation of the producer is as follows:

[0124] for i in range(10)

[0125] for j in range(20)

[0126] B[i, j]=A[i, j] 2;

[0127] The code implementation of the consumer is as follows:

[0128] for i in range(10)

[0129] for j in range(10)

[0130] C[i, j] = B[i, j]+ 5;

[0131] After the compute_inline transformation, the following is obtained:

[0132] for i in range(10)

[0133] for j in range(10)

[0134] C[i, j]=A[i, j] 2+5;

[0135] It can be seen that the compute_inline transformation is to inline the calculation of the tensor B into the tensor C, that is, to replace B[i, j] with A[i, j] 2. The loop j axis of the tensor C is 10, which is smaller than the j axis 20 of the tensor B, and the splitting mode after the compute_inline is determined by the consumer tensor C.

[0136] The fusion mode of the compute_inline transformation is easy to fail when the loop structure and calculation steps of the producer are too complex, and even if it succeeds, the finally generated loop structure follows the consumer. However, in fact, often the splitting and loop order of the producer have more critical impact on performance.

[0137] To solve this problem, a new fusion mode is proposed in the above embodiment, which is recorded as consumer_inline. Unlike the compute_inline in the related art, the consumer_inline is to inline the calculation operation of the consumer into the calculation of the producer. For example, the code of the producer is implemented as:

[0138] for i in range(10)

[0139] for k in range(15)

[0140] for j in rang(20)

[0141] A[i, j]+=A[i, k] A[k,j]

[0142] for j in range(10)

[0143] B[i, j]=A[i, j] 2;

[0144] The code of the consumer is implemented as:

[0145] for i in range(10)

[0146] for j in range(20)

[0147] C[i, j]=B[i, j]+ 5;

[0148] After the consumer_inline transformation, we get:

[0149] for i in range(10)

[0150] for k in rang(15)

[0151] for j in rang(20)

[0152] A[i, j]+=A[i, k] A[k,j]

[0153] for j in range(10)

[0154] C[i, j]=A[i, j] 2+5;

[0155] As can be seen, the consumer_inline is to inline the calculation of the tensor C into the tensor B, that is, B[i, j]+5 is replaced by C[i, j]. The code after the consumer_inline transformation retains the loop structure of the producer.

[0156] Therefore, unlike the compute_inline that inlines the calculation of the producer into the calculation of the consumer, the consumer_inline proposed in the embodiment of the present application is to inline the calculation of the consumer into the calculation of the producer.

[0157] Specifically, in the compute_inline, a post-dominance tree is constructed for the data flow graph, that is, the parent nodes of the final output are traversed from the final output, the data dependency relationship is analyzed in reverse, the anchor point is obtained, and the anchor point is inlined and fused with the final output. Specifically, the post-dominance tree can be constructed, the parent nodes of the final output output are traversed with the output output as the dominance point, the schedule of the anchor operator anchor op is applied to the output operator output op, and the compute_inline is performed on the elementwise operation, the broadcast, and the injective type operator.

[0158] In the consumer_inline, a pre-dominance tree is constructed for the data flow graph, that is, traversed backward from the input, and thus the subsequent nodes are fused in-line. Specifically, it can be expressed as follows: assuming that the anchor operator anchor op is an external operator extern op (for example, a complex operator such as tcore), the complete scheduling information in the intermediate representation ir has already been included, and thus the subgraph needs to be reconstructed. The tcore op can be taken as a pre-dominant point, all elementwise / broadcast operations in the path thereof are made compute_inline, and then consumer_inline into the tcore.

[0159] Based on any of the above embodiments, before step 220, there is further included:

[0160] The tensor intermediate representation is scheduled and optimized.

[0161] Specifically, between the two steps of obtaining the tensor intermediate representation and inserting the synchronization operation in the tensor intermediate representation, the tensor intermediate representation can be scheduled and optimized. Here, the scheduling and optimization can include at least one of the following optimization operations: splitting, vectorization, pipelining, GPU thread parallelism, loop unrolling, and the like.

[0162] The splitting refers to decomposing a large loop structure into a smaller inner and outer loop structure that is executed in blocks, where the outer loop is responsible for traversing the data blocks, and the inner loop performs calculations on the current data block. Vectorization refers to converting scalar operations that need to be executed multiple times in a loop into a single instruction that processes a data vector in parallel, thereby exploiting data-level parallelism. Pipelining refers to decomposing a computing task into multiple independent stages (such as data loading, calculation, and data storage), and allowing these stages to be executed in time overlap. GPU thread parallelism refers to mapping a computing task to parallel threads of a GPU. Loop unrolling refers to duplicating the loop body code multiple times within the loop body and correspondingly reducing the number of loop iterations.

[0163] In addition, during the scheduling and optimization, some fine-grained positions that cannot be inserted with a synchronization operation through data dependency and need to be synchronized can be manually marked with a synchronization mark. Thus, when the synchronization operation is inserted in the tensor intermediate representation in step 220, the synchronization operation can also be inserted for the positions marked with the synchronization mark during the scheduling and optimization, thereby ensuring that the synchronized tensor intermediate representation can achieve fine-grained synchronization.

[0164] Based on any of the above embodiments, in step 240, the running code of the target operator is automatically generated based on the main function and the plurality of coroutine sub-functions, including:

[0165] Converting the main function and the plurality of coroutine sub-functions into an instruction-level intermediate representation;

[0166] Based on the instruction-level intermediate representation, an executable file of the target operator is automatically generated, and the executable file includes the running code of the target operator.

[0167] Specifically, the main function and the plurality of coroutine sub-functions are still in the tensor intermediate representation for the target operator, so that after obtaining the main function and the plurality of coroutine sub-functions, the AI compiler can convert the main function and the plurality of coroutine sub-functions in the tensor intermediate representation into the instruction-level intermediate representation.

[0168] After obtaining the instruction-level intermediate representation of the target operator, the executable file of the target operator can be automatically generated through instruction selection, instruction scheduling, register allocation, and the like. Here, the executable file can be understood as a carrier of the running code of the target operator, and the running code of the target operator can be packaged into an executable file after being compiled.

[0169] In addition, in the process of automatically generating the executable file, different calling conventions can be set for the instruction-level intermediate representations corresponding to different coroutine sub-functions for register allocation and frame lowering. In addition, different cooperative thread bundle indexes can be set for the instruction-level intermediate representations corresponding to different coroutine sub-functions, so as to ensure that different coroutine sub-functions can be asynchronously parallel on different cooperative thread bundles.

[0170] Based on any of the above embodiments, the embodiment of the present application provides a code generation method, which is applied to the fusion of a first operator and a second operator and the code implementation of the fused operator, and the method includes:

[0171] Step a, based on an AI compiler, obtaining a serial implementation of a first operator.

[0172] Here, taking the AI compiler TVM as an example, the calculation logic of the first operator can be described through a tensor intermediate representation.

[0173] Step b, based on the serial implementation of the first operator, the calculation of the second operator is inlined after the output result of the first operator, the calculation fusion of the first operator and the second operator is implemented, and the fused operator is recorded as a target operator.

[0174] Step c, scheduling optimization is performed on the tensor intermediate representation of the target operator.

[0175] Step d, based on the tensor intermediate representation of the target operator, its data dependency is analyzed, and a synchronization operation is automatically inserted based on its data dependency. In addition, for some fine-grained synchronization that cannot be inserted by data dependency analysis, a synchronization operation can be inserted based on the synchronization mark in step c, thereby realizing fine-grained synchronization, and the tensor intermediate representation obtained thereby is denoted as a synchronization tensor intermediate representation.

[0176] Step e, the synchronization tensor intermediate representation is automatically split into a main function and a plurality of coroutine sub-functions, wherein the main function can call each coroutine sub-function.

[0177] In this process, the corresponding shared data and coroutine signals can be processed. The shared data and coroutine signals are passed between functions in the form of function arguments and parameter variables. In addition, common loops are processed, function bodies are divided, corresponding coroutine sub-functions coroutine caller, main function callee, memory buffer allocation alloca buffer, etc. In addition, the thread bundle index warp_idx of the coroutine sub-function corresponding to the cooperative thread bundle can be modified as the cooperative thread bundle index coroutine_warp_idx.

[0178] Step f, the main function and the plurality of coroutine sub-functions are converted into an instruction-level intermediate representation through intermediate representation conversion. Common operations such as instruction selection, instruction scheduling, and register allocation are performed on the instruction-level intermediate representation, and finally an executable file is automatically generated.

[0179] Step g, modify the scheduling optimization rules in step c, such as the split factor, etc.

[0180] In the method provided in the embodiments of the present application, after inserting the synchronization operation in the tensor intermediate representation of the target operator, the tensor intermediate representation is automatically split into a main function and a plurality of coroutine sub-functions, realizing the form conversion of fine-grained asynchronous parallelism of the target operator. On this basis, code generation is performed, and the code of fine-grained asynchronous parallelism of the executable target operator can be obtained. The method can be used for code implementation of various operators, and is not limited to the number and type of operators, and has good generalization and flexibility.

[0181] The code generation apparatus provided by the present application is described below, and the code generation apparatus described below can be correspondingly referred to the code generation method described above.

[0182] Figure 3 is a structural schematic diagram of the code generation apparatus provided by the present application, as Figure 3 indicated, the apparatus comprises:

[0183] The tensor representation unit 310 is configured to obtain a tensor intermediate representation of a target operator.

[0184] The synchronization operation unit 320 is configured to insert a synchronization operation in the tensor intermediate representation based on a data dependency relationship of the tensor intermediate representation, to obtain a synchronization tensor intermediate representation.

[0185] The function splitting unit 330 is configured to automatically split the synchronization tensor intermediate representation into a main function and a plurality of coroutine sub-functions, the main function being configured to call each coroutine sub-function, and different coroutine sub-functions corresponding to different cooperative thread bundles.

[0186] The code generation unit 340 is configured to automatically generate a running code of the target operator based on the main function and the plurality of coroutine sub-functions.

[0187] In the apparatus provided by the embodiment of the present application, after inserting a synchronization operation in the tensor intermediate representation of a target operator, the tensor intermediate representation is automatically split into a main function and a plurality of coroutine sub-functions, realizing the form conversion of fine-grained asynchronous parallelism of the target operator, and on this basis, code generation is performed, that is, the code of fine-grained asynchronous parallelism of the executable target operator can be obtained, realizing the automatic generation of the fine-grained asynchronous parallelism code of the complex logic operator, greatly reducing the workload and the operator development cost, improving the operator development efficiency, and optimizing the operator performance. Moreover, the apparatus can be used for the code implementation of various operators, and is not limited to the number and type of operators, and has good generalization and flexibility.

[0188] Based on any one of the above embodiments, the function splitting unit is specifically configured to:

[0189] extracting segment information of each segment in the serial execution of the synchronization tensor intermediate representation;

[0190] constructing a coroutine sub-function corresponding to each segment based on the segment information of each segment;

[0191] constructing the main function based on each segment in the serial execution of the synchronization tensor intermediate representation and the coroutine sub-function corresponding to each segment.

[0192] Based on any one of the above embodiments, the function splitting unit is specifically configured to:

[0193] Based on the split mark in the synchronous tensor intermediate representation, each fragment in serial execution is split based on the synchronous tensor intermediate representation, and parameter information, loop information and function body of each fragment are extracted as fragment information of each fragment.

[0194] Based on any of the above embodiments, the tensor representation unit is specifically configured to:

[0195] Obtain a first operator and a second operator;

[0196] Operator fusion is performed on the first operator and the second operator to obtain a target operator, and a tensor intermediate representation of the target operator is determined.

[0197] Based on any of the above embodiments, the tensor representation unit is specifically configured to:

[0198] The cache location of the output data of the first operator is determined from the serial implementation of the first operator, the data input operation of the second operator is replaced by an operation of accessing the cache location, and the data source of the write operation of the first operator output final result is replaced by the output data of the second operator.

[0199] Based on any of the above embodiments, the tensor representation unit is further configured to:

[0200] The tensor intermediate representation is scheduled and optimized.

[0201] Based on any of the above embodiments, the code generation unit is specifically configured to:

[0202] Convert the main function and the plurality of coroutine sub-functions into an instruction-level intermediate representation;

[0203] Based on the instruction-level intermediate representation, an executable file of the target operator is automatically generated, and the executable file includes running code of the target operator.

[0204] Figure 4 An example of an entity structure diagram of an electronic device is shown in Figure 4 As shown, the electronic device can include a processor (processor) 410, a communication interface (communications interface) 420, a memory (memory) 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call the logical instructions in the memory 430 to execute the code generation method, which includes:

[0205] Obtain a tensor intermediate representation of a target operator;

[0206] inserting a synchronization operation in the tensor intermediate representation based on a data dependency relationship of the tensor intermediate representation, to obtain a synchronized tensor intermediate representation;

[0207] automatically splitting the synchronized tensor intermediate representation into a main function and a plurality of coroutine sub-functions, the main function being used to call each coroutine sub-function, and different coroutine sub-functions corresponding to different cooperative thread bundles;

[0208] generating running code of the target operator automatically based on the main function and the plurality of coroutine sub-functions.

[0209] In addition, the logic instructions in the memory 430 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part that contributes to the related art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0210] On the other hand, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program is executed by a processor, and the computer can execute the code generation method provided by the above-mentioned method, the method comprises:

[0211] obtaining a tensor intermediate representation of a target operator;

[0212] inserting a synchronization operation in the tensor intermediate representation based on a data dependency relationship of the tensor intermediate representation, to obtain a synchronized tensor intermediate representation;

[0213] automatically splitting the synchronized tensor intermediate representation into a main function and a plurality of coroutine sub-functions, the main function being used to call each coroutine sub-function, and different coroutine sub-functions corresponding to different cooperative thread bundles;

[0214] generating running code of the target operator automatically based on the main function and the plurality of coroutine sub-functions.

[0215] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the code generation method provided by each of the above methods, and the method comprises:

[0216] obtaining a tensor intermediate representation of a target operator;

[0217] inserting a synchronization operation in the tensor intermediate representation based on a data dependency relationship of the tensor intermediate representation, to obtain a synchronization tensor intermediate representation;

[0218] automatically splitting the synchronization tensor intermediate representation into a main function and a plurality of coroutine sub-functions, the main function being configured to call each of the coroutine sub-functions, and different coroutine sub-functions corresponding to different cooperative thread bundles;

[0219] generating running code of the target operator automatically based on the main function and the plurality of coroutine sub-functions.

[0220] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0221] From the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software plus a necessary general hardware platform, and of course, it can also be realized by hardware. Based on such understanding, the above technical solutions, or the essential part or contribution to the related art, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0222] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A code generation method, characterized in that, include: Obtain the tensor intermediate representation of the target operator; Based on the data dependencies of the tensor intermediate representation, a synchronization operation is inserted into the tensor intermediate representation to obtain a synchronized tensor intermediate representation; Based on the segmentation marker in the intermediate representation of the synchronization tensor, each segment of the intermediate representation of the synchronization tensor in the serial execution is segmented, and the parameter information, loop information and function body of each segment are extracted as the segment information of each segment; Based on the fragment information of each fragment, construct the corresponding coroutine subfunction for each fragment; Based on the intermediate representation of the synchronization tensor in each segment during serial execution, and the corresponding coroutine subfunctions of each segment, a main function is constructed; each coroutine subfunction represents an asynchronous operation inside the target operator, and the main function is used to call each coroutine subfunction. Different coroutine subfunctions correspond to different cooperative thread bundles, and the different cooperative thread bundles run independently and asynchronously to achieve fine-grained asynchronous parallelism of different operations in the target operator. Based on the main function and the coroutine sub-functions corresponding to each segment, the running code of the target operator is automatically generated.

2. The code generation method according to claim 1, characterized in that, The process of obtaining the tensor intermediate representation of the target operator includes: Obtain the first and second operators; Operator fusion is performed on the first operator and the second operator to obtain the target operator, and the tensor intermediate representation of the target operator is determined.

3. The code generation method according to claim 2, characterized in that, The operator fusion of the first operator and the second operator includes: The cache location of the output data of the first operator is determined from the serial implementation of the first operator. The data input operation of the second operator is replaced with the operation of accessing the cache location. The data source of the write operation of the final output result of the first operator is replaced with the output data of the second operator.

4. The code generation method according to any one of claims 1 to 3, characterized in that, The process of inserting synchronization operations into the tensor intermediate representation based on the data dependencies of the tensor intermediate representation to obtain a synchronized tensor intermediate representation includes, prior to: The scheduling of the tensor intermediate representation is optimized.

5. The code generation method according to any one of claims 1 to 3, characterized in that, The automatic generation of the target operator's execution code based on the main function and the corresponding coroutine sub-functions of each segment includes: Convert the main function and the coroutine sub-functions corresponding to each segment into instruction-level intermediate representations; Based on the instruction-level intermediate representation, an executable file for the target operator is automatically generated, and the executable file includes the running code of the target operator.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the code generation method as described in any one of claims 1 to 5.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the code generation method as described in any one of claims 1 to 5.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the code generation method as described in any one of claims 1 to 5.

Images