Codegen-based operator framework, construction method and operator generation method

Through the Codegen operator framework, the development of assembly language operators is simplified, the problems of cumbersome development process and limited performance optimization in the existing technology are solved, and efficient and general operator generation is achieved.

CN118963738APending Publication Date: 2024-11-15北京壁仞科技开发有限公司 +1

Patent Information

Application Number
CN202411433151.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-15
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

In the process of building artificial intelligence models, developers need to customize operators and data processing logic. The existing technology has cumbersome processes, prone to errors, poor universality and limited performance optimization in the development of assembly language operators.

Method used

Provide an operator framework based on code generator Codegen, which obtains thread bundle group index and global configuration through preset code units, combines the functional code units corresponding to different thread bundle groups to realize flexible instructions and hardware operations, improve performance optimization and Universality.

Benefits of technology

The development process of assembly language operators is simplified, development efficiency and versatility is improved, the maintainability of operator generation programs is enhanced, and high-performance operators can be built based on the operator's functions and performance optimization strategies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118963738A_ABST
    Figure CN118963738A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an operator framework based on a code generator Codegen, a construction method and an operator generation method. The operator framework comprises a preset code unit which is configured to obtain a thread bundle group index and perform global configuration; the at least one first code unit is configured to carry function codes, and different first code units correspond to different thread bundle groups; and a first selection unit configured to select a first code unit to be executed according to the thread bundle group index. According to the operator framework and the construction method thereof provided by the embodiment of the invention, a series of designed and verified operator frameworks can be generated in advance, and developers can utilize the preset operator frameworks and perform development on the basis of the preset operator frameworks when developing operators, so that the development process of assembly language operators can be simplified, the development efficiency can be improved, and the development cost can be reduced. The error probability is reduced, and the universality and maintainability of an operator generation program can be enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present disclosure relate to an operator framework, a construction method and an operator generation method based on a code generator Codegen. Background Art

[0002] With the rapid development of artificial intelligence technology, more and more industry scenarios need to build artificial intelligence models. In the process of building artificial intelligence models, users need to use custom algorithms and data processing logic to connect a complete modeling process. Among them, these custom algorithms and data processing logic are usually called "operators". In the process of batch development of assembly language operators, developers usually need to first develop an assembly language operator generation program, and use the program to generate the assembly language code corresponding to the operator. Summary of the invention

[0003] At least one embodiment of the present disclosure provides an operator framework based on a code generator Codegen, including: a preset code unit, configured to obtain a thread bundle group index and perform global configuration; at least one first code unit, configured to carry functional codes, wherein different first code units correspond to different thread bundle groups; and a first selection unit, configured to select a first code unit to be executed according to the thread bundle group index.

[0004] In the operator framework provided by at least one embodiment of the present disclosure, the preset code unit is also configured to obtain a thread bundle index, and some of the first code units in the at least one first code unit include: at least one second code unit, configured to carry the function code, wherein different second code units correspond to different thread bundles; and a second selection unit, configured to select a second code unit to be executed according to the thread bundle index.

[0005] In the operator framework provided in at least one embodiment of the present disclosure, the function code includes at least one of a memory access instruction, a calculation instruction, a synchronization instruction, or a local resource configuration instruction.

[0006] In the operator framework provided by at least one embodiment of the present disclosure, the global configuration includes global resource configuration.

[0007] In the operator framework provided in at least one embodiment of the present disclosure, the operator framework is written in assembly language.

[0008] In the operator framework provided in at least one embodiment of the present disclosure, different operator frameworks correspond to different types of operators or different implementation strategies for the same type of operators.

[0009] In the operator framework provided in at least one embodiment of the present disclosure, a corresponding target operator can be generated based on the operator framework. In the process of generating the target operator, the function code carried by the at least one first code unit can be edited, and / or the function code carried by the at least one second code unit can be edited.

[0010] In the operator framework provided in at least one embodiment of the present disclosure, the operator framework is embedded in the code generator Codegen.

[0011] At least one embodiment of the present disclosure provides a method for constructing an operator framework as described in at least one embodiment above, comprising: constructing a corresponding operator framework according to the function and performance optimization strategy of a target operator to be generated.

[0012] In the construction method provided in at least one embodiment of the present disclosure, the construction of a corresponding operator framework according to the function and performance optimization strategy of the target operator to be generated includes: The number of the at least one first code unit and the function and corresponding code of each first code unit are determined according to the function of the target operator and the performance optimization strategy.

[0013] In the construction method provided in at least one embodiment of the present disclosure, the construction of the corresponding operator framework according to the function and performance optimization strategy of the target operator to be generated also includes: The number of second code units in each first code unit and the function and corresponding code of each second code unit are determined according to the function of the target operator and the performance optimization strategy.

[0014] At least one embodiment of the present disclosure provides an operator generation method, which is executed by a code generator Codegen and is based on the operator framework described in at least one embodiment above, including: selecting a corresponding operator framework according to the function and performance optimization strategy of a target operator to be generated; inserting function code into the operator framework to generate the target operator.

[0015] In the operator generation method provided by at least one embodiment of the present disclosure, the inserting of the function code in the operator framework includes: inserting the function code in the at least one first code unit to update the at least one first code unit; and / or inserting the function code in a second code unit included in a part of the first code units in the at least one first code unit to update the second code unit.

[0016] At least one embodiment of the present disclosure provides a device for constructing an operator framework as described in at least one embodiment above, including: a construction module configured to construct a corresponding operator framework according to the function and performance optimization strategy of a target operator to be generated.

[0017] At least one embodiment of the present disclosure provides an operator generation device, which is implemented by a code generator Codegen and is based on the operator framework described in at least one of the above embodiments, including: a selection module, configured to select a corresponding operator framework according to the function and performance optimization strategy of a target operator to be generated; an insertion module, configured to insert a function code into the operator framework to generate the target operator.

[0018] At least one embodiment of the present disclosure provides an electronic device, comprising: at least one processor; at least one memory storing one or more computer program modules; wherein the one or more computer program modules are configured to be executed by the at least one processor to execute instructions of the construction method or operator generation method described in at least one embodiment above.

[0019] At least one embodiment of the present disclosure provides a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by at least one processor, implement the construction method or operator generation method described in at least one embodiment above. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0021] Figure 1 It is a schematic diagram of the structure of an assembly language operator; Figure 2A A schematic diagram of an operator framework based on a code generator Codegen provided for at least one embodiment of the present disclosure; Figure 2B A schematic diagram of an operator framework based on a code generator Codegen provided for at least one embodiment of the present disclosure; Figure 3A A flowchart of a method for constructing an operator framework based on a code generator Codegen provided in at least one embodiment of the present disclosure; Figure 3B A flowchart of a method for constructing an operator framework based on a code generator Codegen provided in at least one embodiment of the present disclosure; Figure 3C A schematic diagram of an operator framework obtained according to the method for constructing an operator framework provided in at least one embodiment of the present disclosure; Figure 4A A flowchart of an operator generation method provided for at least one embodiment of the present disclosure; Figure 4B A schematic diagram of an operator obtained according to an operator generation method provided in at least one embodiment of the present disclosure; Figure 5 A schematic block diagram of a device for constructing an operator framework provided in at least one embodiment of the present disclosure; Figure 6 A schematic block diagram of an operator generating device provided in at least one embodiment of the present disclosure; Figure 7 A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure; Figure 8 A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure; and Fig. 9 A schematic block diagram of a non-transitory computer-readable storage medium is provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0023] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0024] The present disclosure is described below through several specific embodiments. In order to keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and known components (elements) may be omitted. When any component (component) of the embodiments of the present disclosure appears in more than one drawing, the component (component) is represented by the same or similar reference numeral in each drawing.

[0025] Figure 1 A schematic diagram of the structure of an assembly language operator.

[0026] In the process of batch developing assembly language operators, developers usually need to first develop an assembly language operator generation program, and use the program to generate the assembly language code corresponding to the operator. Figure 1 As shown in the figure, an assembly language operator is usually composed of multiple parallel or nested code blocks (the assembly language operator itself is also the outermost and largest code block). When developing an operator generation program, developers need to build code blocks one by one according to the functional characteristics of the operator, and fill in nested or parallel code blocks or codes inside them.

[0027] In the development process of existing assembly language operators and their generation programs, developers usually need to write the various components of the operator generation program from scratch, that is, build a framework and fill in specific instruction details based on the framework, and finally complete the overall development of the operator. This process requires developers to take into account the entire process from framework construction to instruction details, and completely insert and configure assembly instructions. The code development volume of a single operator is large, the development process is relatively cumbersome and error-prone, and the versatility is also poor. In addition, there are usually common logic and abstractions between different operators, and developing operators from scratch will lead to a waste of resources and efficiency. Moreover, in terms of performance optimization, the effect of operator development depends largely on the experience or ability of the operator developer, which leads to significant performance differences even for the same operator developed by different personnel.

[0028] Currently, some methods generate operators based on the AI ​​compiler. The AI ​​compiler is a compiler optimized for artificial intelligence and machine learning applications. It can meet the different needs of reasoning scenarios and training scenarios, and convert programs written in high-level languages ​​or trained model files into programs that can be executed on specific hardware.

[0029] However, the inventors of the present disclosure noticed that, for developers, although the AI ​​compiler has a high degree of automation, its flexibility at the instruction and hardware operation levels is relatively low, which leads to certain limitations in the performance of operators generated based on the AI ​​compiler.

[0030] In the process of high-performance parallel computing using a graphics processing unit (GPU) or a general-purpose computing on graphics processing unit (GPGPU), a large number of threads are used to achieve high concurrency. Multiple threads can be combined into warps, each of which includes a fixed number of threads (or less than the fixed number), for example, 32 threads. These threads execute the same instructions in a single instruction multiple thread (SIMT) manner, but process different data. In a warp, each thread can share some register resources with other threads, and each thread also has independent register resources for loading or computing different data. The warps are scheduled and allocated in each computing unit so that multiple computing cores of the computing unit can run the warps. Depending on the number of computing cores in the computing unit, multiple warps in a thread block can be executed simultaneously or in time-sharing. Multiple warps can be combined into warp groups, each of which includes a fixed number of warps (or less than the fixed number), for example, 4 warps. Multiple warp groups can be executed in the same computing unit or in different computing units.

[0031] At least one embodiment of the present disclosure provides an operator framework based on a code generator Codegen, wherein the operator framework includes: a preset code unit, configured to obtain a thread warp group index and perform global configuration; at least one first code unit, configured to carry functional codes, wherein different first code units correspond to different thread warp groups; and a first selection unit, configured to select a first code unit to be executed according to the thread warp group index.

[0032] The operator framework provided in at least one embodiment of the present disclosure is implemented based on the code generator Codegen. The code generator CodeGen is a technology different from the AI ​​compiler, which can generate code according to predefined rules and templates, which can be customized according to project requirements. Compared with compilers that focus more on versatility, the code generator CodeGen focuses more on performance optimization. Assembly language operators can be written directly for hardware and instruction sets. This method provides greater flexibility, and the performance of the generated operators is usually significantly better than that of operators generated based on the AI ​​compiler. Through the operator framework and its construction method provided in at least one embodiment of the present disclosure, a series of designed and verified operator frameworks can be generated in advance. When developing operators, developers can use these preset operator frameworks and develop on their basis, which can not only simplify the development process of assembly language operators, improve development efficiency, and reduce the probability of errors, but also enhance the versatility and maintainability of the operator generation program. In addition, the operator framework is constructed according to the functions and performance optimization strategies of the operators, so that operator developers can focus more on algorithm optimization and workload, and it is easier to achieve high-performance operators.

[0033] Figure 2A A schematic diagram of an operator framework based on a code generator Codegen is provided for at least one embodiment of the present disclosure.

[0034] For example, Figure 2A As shown, the operator framework 100 includes a preset code unit 101 , a first selection unit 102 , and at least one first code unit 103 .

[0035] For example, the operator framework 100 may be generated using relevant modules in a code generator Codegen, and may be used to generate a target operator.

[0036] In the operator framework 100 provided in at least one embodiment of the present disclosure, the preset code unit 101 is configured to obtain the thread warp group index and perform global configuration. For example, the preset code unit can be used to obtain the index value of the thread warp group to which the current thread belongs, and the current thread refers to the thread currently executing the code when running the target operator generated by the above operator framework.

[0037] For example, global configuration includes global resource configuration. Global resource configuration refers to the allocation and initialization of various resources required for the execution of operators, such as determining the storage location of input data or output data. Global resources can be, for example, shared memory, global memory, registers, etc. For example, an example of global resource configuration is global register configuration, which refers to the allocation and initialization of register space for operators. For example, global resource configuration can also include configuration of the number of computing cores and the organizational form of computing cores required by operators.

[0038] In the operator framework 100 provided in at least one embodiment of the present disclosure, the first selection unit 102 is configured to select a first code unit to be executed according to a warp group index. At least one first code unit 103 is configured to carry a function code, and different first code units correspond to different warp groups.

[0039] For example, when running the target operator, the first selection unit determines the first code unit corresponding to the warp group according to the warp group index obtained by the preset code unit, so that the function code in the first code unit can be executed.

[0040] For example, the operator framework includes n first code units A1~An (n is a positive integer), which correspond to thread bundle groups WG1~thread bundle groups WGn respectively. When the thread bundle group index obtained by the preset code unit is "1", the first selection unit determines the first code unit A1 corresponding to the thread bundle group WG1 according to the thread bundle group index, so that the function code in the first code unit A1 can be executed. When the thread bundle group index obtained by the preset code unit is "n", the first selection unit determines the first code unit An corresponding to the thread bundle group WGn according to the thread bundle group index, so that the function code in the first code unit An can be executed. It should be noted that the number of first code units can be set according to actual needs, and the embodiments of the present disclosure are not limited to this. The above-mentioned situation where the first code unit corresponds one-to-one with the thread bundle group is only an example. One first code unit can also correspond to multiple thread bundle groups, but one thread bundle group can only correspond to one first code unit.

[0041] For example, the function code is responsible for implementing a specific function of the operator. In some examples, the function code may include at least one of a memory access instruction, a synchronization instruction, a calculation instruction, or a local resource configuration instruction.

[0042] In parallel computing, the target operator is usually designed for a single thread, which means that each thread will execute the same target operator function. In order to improve the flexibility and efficiency of parallel computing, the embodiment of the present disclosure provides an operator framework that supports the selection and execution of corresponding function codes according to different thread bundle groups. Under this mechanism, even if all threads are executing the same target operator function, different thread bundle groups can execute function codes in different first code units according to their thread bundle group indexes, so that threads in different thread bundle groups exhibit different behaviors, thereby realizing task allocation in parallel computing and improving the flexibility and efficiency of parallel computing.

[0043] Figure 2B A schematic diagram of an operator framework based on a code generator Codegen is provided for at least one embodiment of the present disclosure.

[0044] In the operator framework 100 provided in at least one embodiment of the present disclosure, the preset code unit 101 is further configured to obtain a warp index.

[0045] For example, the index value of the thread bundle where the current thread is located may be obtained through a preset code unit.

[0046] like Figure 2B As shown, in the operator framework 100 provided in at least one embodiment of the present disclosure, part of the first code units in at least one first code unit 103 includes a second selection unit 201 and at least one second code unit 202 .

[0047] For example, the first code unit may include the second selection unit and the second code unit, or may not include the second selection unit and the second code unit, which depends on the performance optimization strategy at the underlying hardware level. For example, if all the thread bundles in a thread bundle group are expected to be responsible for different functions, the first code unit corresponding to the thread bundle group may include a second selection unit and multiple second code units; if all the thread bundles in a thread bundle group are expected to be responsible for the same function, the first code unit corresponding to the thread bundle group does not need to include the second selection unit and the second code unit. It should be noted that the number of second code units included in the first code unit can be set according to actual needs, and the embodiments of the present disclosure are not limited to this.

[0048] In the operator framework 100 provided in at least one embodiment of the present disclosure, the second selection unit 201 is configured to select a second code unit to be executed according to a warp index. At least one second code unit 202 is configured to carry a function code, wherein different second code units correspond to different warps.

[0049] For example, when the target operator is executed, the second selection unit determines the second code unit corresponding to the thread warp according to the thread warp index obtained by the preset code unit, so that the function code in the second code unit can be executed.

[0050] For example, the first code unit A1 includes m second code units B1~Bm (m is a positive integer), which respectively correspond to the thread bundles W1~thread bundles Wm in the thread bundle group WG1. When the thread bundle index obtained by the preset code unit is "1", the second selection unit determines the second code unit B1 corresponding to the thread bundle W1 according to the thread bundle index, so that the function code in the second code unit B1 can be executed. When the thread bundle index obtained by the preset code unit is "m", the second selection unit determines the second code unit Bm corresponding to the thread bundle Wm according to the thread bundle index, so that the function code in the second code unit Bm can be executed. It should be noted that the number of second code units in each first code unit can be set according to actual needs, and can be the same as the number of thread bundles or different, and the embodiments of the present disclosure are not limited to this. The above-mentioned situation where the second code unit corresponds to the thread bundle one by one is only an example. One second code unit can also correspond to multiple thread bundles, but one thread bundle can only correspond to one second code unit.

[0051] In the operator framework provided in at least one embodiment of the present disclosure, the operator framework is written in assembly language. For example, the operator generated by using the operator framework is an assembly language operator.

[0052] In the operator framework provided in at least one embodiment of the present disclosure, different operator frameworks correspond to different types of operators or different implementation strategies of the same type of operators. For example, according to different operator types, operators can be divided into element-by-element operation operators, matrix multiplication operators, convolution operators, normalization operators, pooling operators, fusion operators, etc. According to different operator types, different operator frameworks can be constructed. For example, according to different performance optimization strategies at the underlying hardware level, multiple different operator frameworks can also be constructed for the same type of operator.

[0053] In the operator framework provided in at least one embodiment of the present disclosure, a corresponding target operator can be generated based on the operator framework. In the process of generating the target operator, the function code carried by at least one first code unit can be edited, and / or the function code carried by at least one second code unit can be edited.

[0054] For example, in some examples, in the process of generating a target operator based on an operator framework, the function code in the first code unit may be edited (e.g., inserted, deleted, or modified, etc.) to implement the function of the target operator. For example, in some examples, if the first code unit includes a second code unit, the function code in the second code unit may be edited to implement the function of the target operator.

[0055] In the operator framework provided in at least one embodiment of the present disclosure, the operator framework is embedded in the code generator Codegen. The operator framework is a part of the code generator Codegen, and the operator framework can be used to generate operators on the code generator Codegen.

[0056] Figure 3A A flowchart of a method for constructing an operator framework based on a code generator Codegen is provided for at least one embodiment of the present disclosure. Figure 3A As shown, at least one embodiment of the present disclosure provides a method for constructing an operator framework, including the following step S301. The construction method is executed by a code generator Codegen.

[0057] Step S301: construct a corresponding operator framework according to the function and performance optimization strategy of the target operator to be generated.

[0058] For example, in step S301, it is first necessary to clarify the specific functions that the target operator to be generated needs to implement, as well as the performance optimization strategy at the underlying hardware level. For example, depending on the performance optimization strategy at the underlying hardware level, multiple corresponding operator frameworks can be constructed for the same operator. In the process of constructing the operator framework, the developer can control the underlying instruction logic and hardware logic.

[0059] Figure 3B A flowchart of a method for constructing an operator framework based on a code generator Codegen is provided for at least one embodiment of the present disclosure.

[0060] For example, Figure 3B As shown, Figure 3A An example of step S301 may include the following step S3011.

[0061] Step S3011: Determine the number of at least one first code unit and the function and corresponding code of each first code unit according to the function of the target operator and the performance optimization strategy.

[0062] For example, in step S3011, it is first determined how to decompose the operator into at least one thread warp group to be implemented, thereby determining at least one first code unit corresponding to at least one thread warp group. For example, the task amount of each thread warp group can be balanced to avoid performance bottlenecks.

[0063] For example, Figure 3B As shown, Figure 3A An example of step S301 may further include the following step S3012.

[0064] Step S3012: Determine the number of second code units in each first code unit and the function and corresponding code of each second code unit according to the function of the target operator and the performance optimization strategy.

[0065] For example, in step S3012, it is possible to further determine how the tasks for which a single warp group is responsible are decomposed into tasks to be implemented by multiple warps, thereby determining at least one second code unit corresponding to at least one warp. For example, the amount of tasks for each warp can be balanced to avoid performance bottlenecks.

[0066] Figure 3C This is a schematic diagram of an operator framework obtained according to the operator framework construction method provided in at least one embodiment of the present disclosure, which is an example of the above steps S3011 to S3012.

[0067] For example, Figure 3C As shown, step S3011 is executed. For the target operator, it can be decomposed into thread bundle group 1 to thread bundle group 3 for execution. Thread bundle group 1 to thread bundle group 3 are responsible for different tasks, such as memory access tasks or computing tasks, etc., so that it can be determined that the operator framework to be constructed includes three first code units A1 to A3, which are respectively used to carry the corresponding functional codes that can realize the tasks that thread bundle group 1 to thread bundle group 3 are responsible for. After determining the number of first code units, the specific code implementation of the first selection unit can be determined. It should be noted that the above is only an example. According to different performance optimization strategies, operators with the same function can be decomposed into more or fewer thread bundle groups for implementation, and the corresponding number and function of the first code units will also change accordingly. The embodiments of the present disclosure are not limited to this.

[0068] For example, Figure 3C As shown, continue to execute step S3012, the tasks that thread bundle group 1 is responsible for can be further decomposed into threads bundle 1 to thread bundle 3, and thread bundle 1 to thread bundle 3 are responsible for different tasks respectively, so that it can be determined that the first code unit A1 includes three second code units B1 to B3, which are respectively used to carry the corresponding functional codes that can implement the tasks that thread bundle 1 to thread bundle 3 are responsible for. After determining the number of second code units, the specific code implementation of the second selection unit can be determined. It should be noted that the above is only an example. According to different performance optimization strategies, the tasks that a single thread bundle group is responsible for can be decomposed into more or fewer thread bundles for implementation, and the corresponding number and function of the second code units will also change accordingly. The embodiments of the present disclosure are not limited to this.

[0069] For example, Figure 3C As shown, the code implementation of the preset code unit can be determined according to the actual needs of the specific hardware architecture and global configuration.

[0070] Figure 4AA flowchart of an operator generation method provided in at least one embodiment of the present disclosure. Figure 4A As shown, the operator generation method provided by at least one embodiment of the present disclosure is executed by a code generator Codegen and is based on the operator framework of at least one embodiment described above, and includes the following steps S401-S402.

[0071] Step S401: Select a corresponding operator framework according to the function and performance optimization strategy of the target operator to be generated.

[0072] For example, according to the operator framework construction method provided in the above embodiment of the present disclosure, multiple operator frameworks can be pre-built and embedded in the code generator Codegen according to different operator functions and performance optimization strategies at the underlying hardware level. In the process of operator development, a suitable operator framework can be directly selected from the pre-built operator frameworks according to specific needs.

[0073] Step S402: inserting function code into the operator framework to generate a target operator.

[0074] An example of step S402 may include the following step S4021.

[0075] Step S4021: inserting a function code into at least one first code unit to update at least one first code unit; and / or inserting a function code into a second code unit included in part of the first code units in at least one first code unit to update the second code unit.

[0076] For example, the operator framework specifies the functions that each of the first code unit and the second code unit contains needs to implement, and the corresponding function code can be inserted into it according to the specific computing requirements. If the first code unit does not include the second code unit, the corresponding function code is directly inserted into the first code unit. If the first code unit includes the second code unit, the corresponding function code is inserted into the second code unit. In the process of operator development, the code in the preset code unit can also be adaptively modified.

[0077] Figure 4B A schematic diagram of an operator obtained according to an operator generation method provided in at least one embodiment of the present disclosure.

[0078] For example, executing step S401, selecting a corresponding operator framework according to the function and performance optimization strategy of the target operator to be generated. Figure 4BAs shown, in this example, the target operator to be generated is a matrix multiplication operator (or convolution operator), and there are three available thread bundle groups. The operator framework thus selected includes three first code units A1~A3, corresponding to three thread bundle groups WG1~WG3 respectively. Thread bundle group WG1 is responsible for memory access tasks and local resource configuration tasks, and thread bundle groups WG2 and thread bundle groups WG3 are responsible for calculation and local resource configuration tasks. The first code unit A1 also includes three second code units B1~B3, corresponding to three thread bundles 1~3 respectively. Thread bundle W1 is responsible for loading the first input data and local resource configuration tasks, thread bundle W2 is responsible for loading the second input data and local resource configuration tasks, and thread bundle W3 is responsible for storing the output data and local resource configuration tasks.

[0079] For example, continue to execute step S402 to insert the function code into the operator framework to generate the target operator. Figure 4B As shown, the function code for loading the first input data and configuring local resources is inserted into the second code unit B1 included in the first code unit A1, the function code for loading the second input data and configuring local resources is inserted into the second code unit B2, the function code for storing the output data and configuring local resources is inserted into the second code unit B3, and the function code for vector and tensor calculation and configuring local resources is inserted into the first code units A2~A3. After all the function codes are inserted, a complete matrix multiplication operator (or convolution operator) can be obtained.

[0080] In the operator generation method provided in at least one embodiment of the present disclosure, the operator framework is constructed by the instruction generator Codegen, but the specific functional codes of each part of the operator framework are inserted by the developer, the logical details and strategies are more controllable, and there is more freedom when performing detailed performance tuning.

[0081] Figure 5 A schematic block diagram of a device for constructing an operator framework provided in at least one embodiment of the present disclosure.

[0082] For example, Figure 5 As shown, the operator framework construction device 500 provided in the embodiment of the present disclosure includes a construction module 501.

[0083] For example, the construction module 501 is configured to construct a corresponding operator framework according to the function and performance optimization strategy of the target operator to be generated. For the relevant content of the construction module 501, reference can be made to the relevant description of step S301 in the embodiment of the operator framework construction method, which will not be repeated here.

[0084] For example, in at least one embodiment of the present disclosure, the construction module 501 includes a first construction unit, which is configured to determine the number of at least one first code unit and the function of each first code unit and the corresponding code according to the function of the target operator and the performance optimization strategy.

[0085] For example, in at least one embodiment of the present disclosure, the construction module 501 also includes a second construction unit, which is configured to determine the number of second code units in each first code unit and the function of each second code unit and the corresponding code according to the function of the target operator and the performance optimization strategy.

[0086] Figure 6 A schematic block diagram of an operator generation device provided for at least one embodiment of the present disclosure.

[0087] For example, Figure 6 As shown, the operator generation device 600 provided in the embodiment of the present disclosure is implemented by a code generator Codegen and is based on the operator framework of at least one of the above embodiments, including a selection module 601 and an insertion module 602.

[0088] For example, the selection module 601 is configured to select a corresponding operator framework according to the function and performance optimization strategy of the target operator to be generated. For the relevant content of the selection module 601, reference can be made to the relevant description of step S401 in the embodiment of the operator generation method, which will not be repeated here.

[0089] For example, the insertion module 602 is configured to insert the function code into the operator framework to generate the target operator. For the relevant content of the insertion module 602, reference may be made to the relevant description of step S402 in the embodiment of the operator generation method, which will not be repeated here.

[0090] For example, in at least one embodiment of the present disclosure, the insertion module 602 is further configured to insert a function code into at least one first code unit to update at least one first code unit; and / or insert a function code into a second code unit included in a portion of the first code units in at least one first code unit to update the second code unit.

[0091] It should be noted that the above Figure 5 and Figure 6 The various modules shown may be implemented by software, hardware, firmware or any combination thereof. For example, they may be implemented as construction circuits, selection circuits and insertion circuits, respectively. The embodiments of the present disclosure do not limit their specific implementations.

[0092] It should be understood that the operator framework construction device 500 provided in at least one embodiment of the present disclosure can be used to implement the aforementioned operator framework construction method, and can also achieve technical effects similar to the aforementioned operator framework construction method; the operator generation device 600 provided in at least one embodiment of the present disclosure can be used to implement the aforementioned operator generation method, and can also achieve technical effects similar to the aforementioned operator generation method, which will not be elaborated here.

[0093] It should be noted that in the embodiments of the present disclosure, the operator framework construction device 500 and the operator generation device 600 may include more or fewer modules or units, and the connection relationship between the modules or units is not limited and can be determined according to actual needs. The specific configuration of each module is not limited and can be composed of analog devices according to circuit principles, or can be composed of digital chips, or in other applicable ways.

[0094] Figure 7 A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.

[0095] For example, Figure 7 As shown, the electronic device 700 includes at least one processor 701 and at least one memory 702. The at least one memory 702 includes one or more computer program modules. The one or more computer program modules are stored in the memory 702 and are configured to be executed by the at least one processor 701. The one or more computer program modules include instructions for executing the above-mentioned operator framework construction method and operator generation method. When executed by the at least one processor 701, one or more steps in the operator framework construction method or operator generation method provided in at least one embodiment of the present disclosure can be executed. The memory 702 and the processor 701 can be interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0096] For example, the processor 701 may be a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), a general purpose graphics processing unit (GPGPU), an artificial intelligence (AI) accelerator, or other forms of processing units with data processing capabilities and / or program execution capabilities, such as a field programmable gate array (FPGA), etc.; for example, the central processing unit (CPU) may be an X86, ARM, RISC-V architecture, etc. The processor 701 may be a general-purpose processor or a dedicated processor, and may control other components in the electronic device 700 to perform desired functions.

[0097] For example, the memory 702 may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (Cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc.

[0098] Figure 8 A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0099] The electronic devices in at least one embodiment of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0100] The electronic device includes at least one processor and a memory. The processor here may be referred to as the processing device 801 described below, and the memory may include at least one of the read-only memory (ROM) 802, the random access memory (RAM) 803, and the storage device 808 described below. The memory is used to store programs for executing the methods described in the above-mentioned various method embodiments; the processor is configured to execute the programs stored in the memory. The processor may include a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0101] like Figure 8 As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0102] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a display (e.g., a liquid crystal display (LCD) or an organic light emitting diode display (OLED)), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 800 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0103] In particular, according to at least one embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, at least one embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 809, or installed from the storage device 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of at least one embodiment of the present disclosure are executed.

[0104] It should be noted that the computer-readable medium of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In at least one embodiment of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In at least one embodiment of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0105] The computer-readable medium may be included in the electronic device 800 ; or may exist independently without being assembled into the electronic device 800 .

[0106] Fig. 9 A schematic block diagram of a non-transitory computer-readable storage medium is provided for at least one embodiment of the present disclosure.

[0107] For example, Fig. 9 As shown, a non-transitory computer-readable storage medium 900 stores computer-readable instructions 901, which, when executed by at least one processor, perform one or more steps of the above-mentioned operator framework construction method or operator generation method.

[0108] For example, the storage medium may include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a flash memory, or any combination of the above storage media, or other applicable storage media. For example, the readable storage medium may also be Figure 7 For the memory 702 in, the related description can be referred to the aforementioned content and will not be repeated here.

[0109] Although the disclosure has been described in detail above with general descriptions and specific implementation methods, it is obvious to those skilled in the art that some modifications or improvements may be made to the embodiments of the disclosure. Therefore, these modifications or improvements made without departing from the spirit of the disclosure are within the scope of protection claimed by the disclosure.

[0110] There are a few points to note about this disclosure: (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure. Other structures may refer to the general design.

[0111] (2) For the sake of clarity, in the drawings used to describe the embodiments of the present disclosure, the thickness of layers or regions is enlarged or reduced, that is, these drawings are not drawn according to the actual scale.

[0112] (3) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to obtain new embodiments.

[0113] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. An operator framework based on a code generator Codegen, comprising: A preset code unit is configured to obtain a thread warp group index and perform global configuration; at least one first code unit configured to carry a function code, wherein different first code units correspond to different warp groups; as well as A first selection unit is configured to select a first code unit to be executed according to the warp group index.

2. The operator framework according to claim 1, wherein: The preset code unit is further configured to obtain a thread warp index, Some of the first code units in the at least one first code unit include: at least one second code unit configured to carry the function code, wherein different second code units correspond to different warps; and A second selection unit is configured to select a second code unit to be executed according to the thread warp index.

3. The operator framework according to claim 1, wherein: The function code includes at least one of a memory access instruction, a calculation instruction, a synchronization instruction or a local resource configuration instruction.

4. The operator framework according to claim 1, wherein: The global configuration includes global resource configuration.

5. The operator framework according to claim 1, wherein: The operator framework is written in assembly language.

6. The operator framework according to claim 1, wherein: Different operator frameworks correspond to different types of operators or different implementation strategies for the same type of operators.

7. The operator framework according to claim 2, wherein: Based on the operator framework, the corresponding target operator can be generated. In the process of generating the target operator, the function code carried by the at least one first code unit may be edited, and / or The function code carried by the at least one second code unit can be edited.

8. The operator framework according to claim 1, wherein: The operator framework is embedded in the code generator Codegen.

9. A method for constructing an operator framework according to any one of claims 1 to 8, comprising: Construct the corresponding operator framework according to the functions and performance optimization strategies of the target operator to be generated.

10. The construction method according to claim 9, wherein: The corresponding operator framework is constructed according to the function and performance optimization strategy of the target operator to be generated, including: The number of the at least one first code unit and the function and corresponding code of each first code unit are determined according to the function of the target operator and the performance optimization strategy.

11. The construction method according to claim 10, wherein: The method of constructing a corresponding operator framework according to the function and performance optimization strategy of the target operator to be generated also includes: The number of second code units in each first code unit and the function and corresponding code of each second code unit are determined according to the function of the target operator and the performance optimization strategy.

12. An operator generation method, executed by a code generator Codegen and based on the operator framework according to any one of claims 1 to 8, comprising: Select the corresponding operator framework according to the function and performance optimization strategy of the target operator to be generated; Inserting function code into the operator framework to generate the target operator.

13. The operator generation method according to claim 12, wherein: The inserting the function code into the operator framework includes: inserting the function code into the at least one first code unit to update the at least one first code unit; and / or The function code is inserted into a second code unit included in a portion of the at least one first code unit to update the second code unit.

14. An electronic device comprising: at least one processor; at least one memory storing one or more computer program modules; The one or more computer program modules are configured to be executed by the at least one processor to execute instructions of the construction method described in any one of claims 9 to 11 or the operator generation method described in any one of claims 12 to 13.

15. A non-transitory computer-readable storage medium having computer-readable instructions stored thereon, wherein: When the computer-readable instructions are executed by at least one processor, the construction method described in any one of claims 9 to 11 or the operator generation method described in any one of claims 12 to 13 is implemented.

Citation Information

Patent Citations

  • An Elasticsearch code generation method and tool based on a java language

    CN109725894A

  • Special back-end code generation method and device supporting machine learning training

    CN113885871A

  • Model reasoning optimization method and system, electronic equipment and storage medium

    CN114020465A

  • Method for generating program codes, equipment and computer program product

    CN114063997A

  • Inference optimization method and system for model, and electronic device and storage medium

    WO2023078116A1

Cited By

  • Code generation method and device, electronic device, storage medium and electronic equipment

    CN121300806A