Operator fusion method and device, electronic equipment and storage medium
By adding synchronization instructions to the context program of the artificial intelligence chip and using preset synchronization instruction groups to perform operator operations in parallel, the problem of low operator fusion efficiency on the artificial intelligence chip is solved, and efficient training and reasoning efficiency and versatility are achieved.
Patent Information
- Application Number
- CN202411244705.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-05
- Publication Date
- 2024-11-26
AI Technical Summary
In the prior art, operator fusion efficiency on artificial intelligence chips is low, developers have a lot of time to perform manual fusion, and compiler automatic fusion is difficult to handle multiple operator combinations, resulting in low training and inference efficiency.
By adding synchronization instructions to the context program of the artificial intelligence chip, operator fusion is realized, without the need for developers to develop fusion operators specifically or use compilers, and the operations of the to-fused operators are executed in parallel using preset synchronization instruction groups.
This greatly improves the training and inference efficiency of artificial intelligence models on artificial intelligence chips, simplifies the operator fusion process, and improves the universality and efficiency of operator fusion.
Smart Images

Figure CN119026084A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an operator fusion method, device, electronic equipment and storage medium. Background Art
[0002] At present, the operator fusion method is a common performance optimization method for artificial intelligence operators. This operator fusion method is often used in deep learning and other intensive tasks. Its basic operation is to merge multiple consecutive operators into a single operator; this can reduce computing and memory overhead, and can also improve the training efficiency and reasoning efficiency of artificial intelligence models on artificial intelligence chips. Therefore, how to efficiently fuse different operators on artificial intelligence chips is particularly important.
[0003] In the related technology, developers can manually fuse different operators in the artificial intelligence chip and then develop the fused operators; or, they can use a compiler to obtain a loop program that fuses different operators, and run the loop program in a context program in the artificial intelligence chip to achieve the automatic fusion process of multiple operators.
[0004] However, when there are too many types of operators to be fused, the manual fusion method by developers requires a lot of development work. When the computational complexity of an operator to be fused is high, it is difficult to achieve automatic fusion through the compiler. Therefore, it is difficult for the compiler to automatically fuse all operator combinations, resulting in low training efficiency and inference efficiency of the artificial intelligence model on the artificial intelligence chip. Summary of the invention
[0005] The present invention provides an operator fusion method, device, electronic device and storage medium to solve the defects of low training efficiency and reasoning efficiency of artificial intelligence models on artificial intelligence chips caused by manual fusion of operators by developers and automatic fusion of operators by compilers. Only a small amount of modification is required to add synchronization instructions to the context program developed in the artificial intelligence chip to achieve the purpose of operator fusion between different operator combinations. There is no need to rely on developers to specially develop fused operators, and there is no need to use a compiler in the operator fusion process. The entire operator fusion process is simple and easy to implement, and is also suitable for operator fusion scenarios of different operator combinations in the field of artificial intelligence, thereby greatly improving the training efficiency and reasoning efficiency of artificial intelligence models on artificial intelligence chips, and also improving the versatility of operator fusion.
[0006] The present invention provides an operator fusion method, which is applied to an artificial intelligence chip that runs multiple context programs simultaneously, and the method comprises the following steps.
[0007] In response to the operator fusion request, a plurality of target context programs are determined from the plurality of context programs as operators to be fused respectively.
[0008] Based on the preset synchronization instruction group corresponding to the multiple operators to be fused, the fusion process of the multiple operators to be fused is implemented.
[0009] According to an operator fusion method provided by the present invention, it also includes: based on the execution order of each of the multiple operators to be fused, determining the preset synchronization instruction corresponding to each of the operators to be fused, and determining each of the preset synchronization instructions as the preset synchronization instruction group.
[0010] According to an operator fusion method provided by the present invention, the preset synchronization instruction corresponding to each of the operators to be fused is determined based on the execution order of each of the multiple operators to be fused, including: based on the synchronization instructions pre-configured for multiple operators arranged in sequence in the artificial intelligence model, and the execution order of each of the multiple operators to be fused, determining the preset synchronization instruction corresponding to each of the operators to be fused.
[0011] According to an operator fusion method provided by the present invention, the fusion process of multiple operators to be fused is implemented based on the preset synchronization instruction group corresponding to the multiple operators to be fused, including: generating first indication information based on the preset synchronization instruction group corresponding to the multiple operators to be fused, the first indication information is used to instruct the user to insert the multiple preset synchronization instructions contained in the preset synchronization instruction group into the corresponding operators to be fused in the form of API respectively; based on the insertion operation completion mark, executing the fusion operation of the multiple operators to be fused in parallel.
[0012] According to an operator fusion method provided by the present invention, the fusion operation of multiple operators to be fused is performed in parallel, including: when the multiple operators to be fused include a preceding operator to be fused and a subsequent operator to be fused, the target data block and the second indication information are sent to the preceding operator to be fused into which the corresponding preset synchronization instruction has been inserted, and the third indication information is sent to the subsequent operator to be fused into which the corresponding preset synchronization instruction has been inserted; the second indication information is used to instruct the preceding operator to be fused to calculate the target data block, and send a synchronization signal carrying the calculation result to the subsequent operator to be fused into which the corresponding preset synchronization instruction has been inserted; the third indication information is used to indicate that the subsequent operator to be fused is in a waiting state for the synchronization signal during the calculation process of the preceding operator to be fused, and performs calculation based on the calculation result when the synchronization signal is received; a new target data block is re-determined, and the above steps are repeated; until all target data blocks to be fused have completed the calculation of the preceding operator to be fused and the calculation of the subsequent operator to be fused.
[0013] The present invention also provides an artificial intelligence chip, comprising a processor for simultaneously running multiple context programs and a control unit, wherein the processor contains multiple processing units and the multiple processing units are respectively communicated with the control unit; the control unit is used to respond to an operator fusion request and determine multiple target context programs from the multiple context programs as operators to be fused; the multiple target processing units corresponding to the multiple operators to be fused are used to implement the fusion process of the multiple operators to be fused based on the preset synchronization instruction groups corresponding to the multiple operators to be fused.
[0014] According to an artificial intelligence chip provided by the present invention, when the multiple operators to be fused include a preceding operator to be fused and a subsequent operator to be fused, the target processing unit corresponding to the preceding operator to be fused, into which the corresponding preset synchronization instruction has been inserted, is used to calculate the target data block, and send a synchronization signal carrying the calculation result to the subsequent operator to be fused, into which the corresponding preset synchronization instruction has been inserted; the calculation result represents that the preceding operator to be fused completes the calculation of the target data block; the target processing unit corresponding to the subsequent operator to be fused, into which the corresponding preset synchronization instruction has been inserted, is used to be in a waiting state during the calculation process of the preceding operator to be fused, and perform calculation based on the calculation result when the synchronization signal is received.
[0015] The present invention also provides an operator fusion device, which is applied to an artificial intelligence chip that runs multiple context programs simultaneously. The device includes: an operator determination unit and an operator fusion unit.
[0016] The operator determination unit is used to determine, in response to the operator fusion request, a plurality of target context programs from the plurality of context programs as operators to be fused.
[0017] The operator fusion unit is used to implement the fusion process of multiple operators to be fused based on the preset synchronization instruction group corresponding to the multiple operators to be fused.
[0018] The present invention also provides an electronic device, comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, wherein the artificial intelligence chip implements any of the above-described operator fusion methods when executing the computer program.
[0019] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the operator fusion method described in any one of the above is implemented.
[0020] The present invention provides an operator fusion method, device, electronic device and storage medium, wherein the operator fusion method includes: when the artificial intelligence chip responds to the operator fusion request, it first determines that multiple target context programs are operators to be fused from multiple context programs, and then further implements the fusion process of multiple operators to be fused based on the preset synchronization instruction group corresponding to the multiple operators to be fused. In this way, the purpose of operator fusion between different operator combinations can be achieved by adding synchronization instructions to the context programs developed in the artificial intelligence chip with a small amount of modification, without relying on developers to specially develop the fused operators, and there is no need to use a compiler during the operator fusion process. The entire operator fusion process is simple and easy to implement, and is also applicable to operator fusion scenarios of different operator combinations in the field of artificial intelligence, thereby greatly improving the training efficiency and reasoning efficiency of the artificial intelligence model on the artificial intelligence chip, and also improving the versatility of operator fusion. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0022] Figure 1 It is a flowchart of the operator fusion method provided by the present invention.
[0023] Figure 2 This is a schematic diagram of the artificial intelligence operator fusion provided by the present invention.
[0024] Figure 3 It is a schematic diagram for comparing the number of operator fusions provided by the present invention.
[0025] Figure 4 It is a schematic diagram of the operator fusion process provided by the present invention.
[0026] Figure 5 It is a structural schematic diagram of the operator fusion device provided by the present invention.
[0027] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0029] In the embodiments of the present invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B may be singular or plural. In the textual description of the present invention, the character " / " generally indicates that the previous and next associated objects are in an "or" relationship. In addition, it should be noted that the serial numbers themselves, such as "first", "second", etc., used to distinguish the objects described in the present invention are only used to distinguish the objects described, and do not have any order or technical meaning.
[0030] At present, the operator fusion method is a common performance optimization method for artificial intelligence operators. This operator fusion method is often used in deep learning and other intensive tasks. Its basic operation is to merge multiple consecutive operators into a single operator; this can reduce computing and memory overhead, and can also improve the training efficiency and reasoning efficiency of artificial intelligence models on artificial intelligence chips. Therefore, how to efficiently fuse different operators on artificial intelligence chips is particularly important.
[0031] In the related technology, developers can manually fuse different operators in the artificial intelligence chip and then develop a fusion program that fuses multiple operator operations; or, they can use a compiler to obtain a for loop program that fuses different operators, and run the for loop program in a context program in the artificial intelligence chip to achieve an automatic fusion process of multiple operators.
[0032] For example, when fusing operator A and operator B, the developer can manually fuse operator A and operator B to obtain fused operator C; the for loop program for fusing operator A and operator B can also be implemented in a context program, and the fusion process of the two operators can be completed by initiating a calculation request for operator A and operator B.
[0033] However, since there are various different operators in the field of artificial intelligence technology and there are many combinations between operators, when there are too many types of operators to be fused, developers have to manually fuse them one by one, which greatly increases the workload of developers. When the computational complexity of an operator to be fused is high, it is difficult to achieve automatic fusion through the compiler, so the compiler's automatic fusion method is difficult to automatically fuse all operator combinations, resulting in low training efficiency and inference efficiency of artificial intelligence models on artificial intelligence chips.
[0034] In order to solve the above technical problems, the present invention provides an operator fusion method, device, electronic device and storage medium. It only needs to make a small modification to the context program developed in the artificial intelligence chip by adding synchronization instructions, so as to achieve the purpose of operator fusion between different operator combinations. There is no need to rely on developers to specially develop the fused operators, and there is no need to rely on the automatic fusion function of the compiler during the operator fusion process. The entire operator fusion process is simple and easy to implement, and is also suitable for operator fusion scenarios of different operator combinations in the field of artificial intelligence, thereby greatly improving the training efficiency and reasoning efficiency of the artificial intelligence model on the artificial intelligence chip, and also improving the versatility of operator fusion.
[0035] Combine the following Figure 1-Figure 6 The operator fusion method, device, electronic device and storage medium of the present invention are described, wherein the execution subject of the operator fusion method is an artificial intelligence chip, which has at least the hardware characteristics of running multiple context programs at the same time and the data block calculation function, and the artificial intelligence chip can be called an artificial intelligence accelerator or computing card, such as GPU (Graphics Processing Unit), TPU (Tensor Processing Unit), NPU (Neural network Processing Unit), DPU (Deeplearning Processing Unit), APU (Accelerated Processing Unit), and GPGPU (General-Purpose computing on Graphics Processing Unit). Furthermore, the execution subject of the operator fusion method can also be applied to an operator fusion device set in an artificial intelligence chip, and the operator fusion device can be implemented by software, hardware or a combination of the two. The following describes the operator fusion method by taking the execution subject of the operator fusion method as an artificial intelligence chip that has at least the hardware characteristics of running multiple context programs at the same time and the data block calculation function as an example.
[0036] In order to facilitate understanding of the operator fusion method provided by the embodiment of the present invention, the operator fusion method provided by the present invention will be described in detail below through the following exemplary embodiments. It can be understood that the following exemplary embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.
[0037] Reference Figure 1 , which is a flow chart of the operator fusion method provided by the present invention, such as Figure 1 As shown, the operator fusion method includes the following steps 110 and 120.
[0038] Step 110 : In response to the operator fusion request, a plurality of target context programs are determined from a plurality of context programs as operators to be fused.
[0039] Step 120: Based on the preset synchronization instruction group corresponding to the multiple operators to be fused, the fusion process of the multiple operators to be fused is implemented.
[0040] It can be understood that the operator fusion request can be a request automatically generated when multiple operators need to be merged into a single operator during model training or model inference; for example, when the calculation result output by one operator is the input data of another operator, the two operators can be merged into a single operator, and an operator fusion request can be automatically generated for the two operators.
[0041] Artificial intelligence operator fusion technology reduces the number of memory accesses and improves computing density by merging multiple independent computing operations into a larger computing operation, thereby making full use of the hardware resources of artificial intelligence chips and accelerating the training and reasoning process of deep learning models.
[0042] Considering that some AI chip architectures in the field of AI technology support multi-context programming, that is, allowing multiple threads or processes to access the GPU in the AI chip at the same time, each thread or process has an execution context, which can improve the utilization of GPU resources. Therefore, the above-mentioned AI chip has the hardware feature of running multiple context programs at the same time because it supports multi-context programming. The present invention can combine this hardware feature to perform operator fusion.
[0043] Reference Figure 2 The schematic diagram of artificial intelligence operator fusion is shown in Figure 2As shown in the figure, the left side of the dotted line is the unfused operation implementation process. At this time, operator A, operator B and operator C are each an independent context program (such as context program A, context program B and context program C). If operator A, operator B and operator C are not fused, context program A, context program B and context program C need to be executed sequentially, that is, three context programs need to be executed; on the contrary, if operator A, operator B and operator C have been fused and a fused operator is obtained, it can be as follows Figure 2 The fused operation implementation process shown on the right side of the dotted line only requires one context program to execute the operations of operator A, operator B, and operator C on the data block (Tile) in sequence within the loop.
[0044] When performing operator fusion in combination with the above-mentioned hardware characteristics of the artificial intelligence chip, based on the at least two operators carried in the operator fusion request, a target context program that implements the respective computing functions of the at least two operators can be selected from a plurality of independent context programs of the artificial intelligence chip, and each selected target context program can be used as an operator to be fused.
[0045] It can be understood that the number of target context programs is the same as the number of operators to be fused and corresponds one to one.
[0046] The number of operators to be fused is at least two, and the at least two operators to be fused may be operators of the same type or operators of different types.
[0047] It should be noted that, considering that the types of processing units in artificial intelligence chips are usually not unique, and the types of artificial intelligence operators are also not unique, the processing unit that runs the corresponding operator to be fused can be determined based on the pre-established mapping relationship between the type of artificial intelligence operator and the type of processing unit; for example, the processing unit that runs the corresponding operator to be fused can be a large-grained control unit (CU) or a small-grained execution unit (EU).
[0048] Exemplarily, common operator fusions include but are not limited to the fusion of convolutional layer operators and pooling layer operators, the fusion of multiple fully connected layer operators, the fusion of convolutional layer operators and batch normalization layer operators, the fusion of convolutional operators and activation operators, the fusion of multiple convolutional layer operators, the fusion of convolutional layer operators and fully connected layer operators, the fusion of multiple element-by-element operators and fully connected layer operators, the fusion of multiple convolutional layer operators and pooling layer operators, and the like.
[0049] Although operator fusion can be achieved through the hardware feature of running multiple independent context programs simultaneously in the artificial intelligence chip without the need to specially develop the fused operator, the data dependency problem of multiple independent operators to be fused (that is, the calculation of the next operator to be fused requires the calculation result of the previous operator to be fused) can be solved by adding corresponding preset synchronization instructions to each operator to be fused. This ensures that multiple operators to be fused are executed synchronously and sequentially, thereby ensuring that the final output calculation result after the operator fusion is more accurate and reliable.
[0050] It can be understood that the preset synchronization instruction group corresponding to the multiple operators to be fused contains multiple preset synchronization instructions, and each operator to be fused may correspond to at least one preset synchronization instruction.
[0051] For example, refer to Figure 3 The diagram shows the comparison of the number of operator fusions. When operators A1, A2, A3, ..., A m These m operators, as well as operator B1, operator B2, operator B3, operator B4, ..., operator B n When these n operators are fused, Figure 3 The existing operator fusion method on the left side of the dotted line has a total of m×n operator combinations, which means that m×n fusion operators need to be developed. Figure 3 The operator fusion method of the present invention on the right side of the dotted line can realize operator fusion by only modifying m+n operators.
[0052] The operator fusion method provided by the present invention, when responding to the operator fusion request, the artificial intelligence chip first determines from multiple context programs that multiple target context programs are operators to be fused, and then further implements the fusion process of multiple operators to be fused based on the preset synchronization instruction group corresponding to the multiple operators to be fused. In this way, the purpose of operator fusion between different operator combinations can be achieved by adding synchronization instructions to the context programs developed in the artificial intelligence chip with a small amount of modification, without relying on developers to specially develop the fused operators, and during the operator fusion process, there is no need to rely on the automatic fusion function of the compiler. The entire operator fusion process is simple and easy to implement, and is also applicable to operator fusion scenarios of different operator combinations in the field of artificial intelligence, thereby greatly improving the training efficiency and reasoning efficiency of the artificial intelligence model on the artificial intelligence chip, and also improving the versatility of operator fusion.
[0053] Based on the above Figure 1 In the operator fusion method shown, in an exemplary embodiment, an artificial intelligence chip determines a preset synchronization instruction group corresponding to multiple operators to be fused, and its specific implementation process can be achieved through the following steps.
[0054] Based on the execution order of each of the multiple operators to be fused, a preset synchronization instruction corresponding to each operator to be fused is determined, and the preset synchronization instructions are determined as a preset synchronization instruction group.
[0055] Specifically, since the artificial intelligence model is developed based on model design, algorithm implementation and specific application scenarios of the model, and in the whole process of model development, the selection and sequence arrangement of operators is one of the key links, involving multiple stages such as algorithm selection, architecture design and model training and tuning, when the artificial intelligence model has been developed and set in the artificial intelligence chip, all operators involved are arranged in execution order and can be read by the artificial intelligence chip. In this way, the artificial intelligence chip can read the execution order of each operator to be fused, and determine at least one preset synchronization instruction to be added to the corresponding operator to be fused according to different execution orders, and use all the determined preset synchronization instructions as the preset synchronization instruction group corresponding to all operators to be fused.
[0056] In an example embodiment, the artificial intelligence chip determines the preset synchronization instruction corresponding to each operator to be fused based on the execution order of each of the multiple operators to be fused, and the determination process can be implemented through the following steps.
[0057] Based on the synchronization instructions pre-configured for multiple operators arranged in sequence in the artificial intelligence model, and the execution order of multiple operators to be fused, the preset synchronization instructions corresponding to each operator to be fused are determined.
[0058] Specifically, the artificial intelligence chip can pre-configure at least one synchronization instruction based on all operators involved in the artificial intelligence model that are arranged in the order of execution. For example, operator 1 with an execution order of 1 is configured with a pass synchronization instruction, which can be recorded as a pass instruction, which is used to indicate that after the calculation of operator 1 is completed, the corresponding calculation result 1 is synchronously sent to operator 2 with an execution order of 2; operator 2 with an execution order of 2 is configured with a pass synchronization instruction and a wait synchronization instruction, where the wait synchronization instruction can be recorded as a wait instruction. The added pass instruction and wait instruction are used to indicate that operator 2 waits for operator 1 to complete part of the calculation and receive the corresponding calculation result 1 before starting the calculation, and after the calculation is completed The calculation result 2 is synchronized to the operator 3 with the execution order of 3; the operator 3 with the execution order of 3 is also configured with the pass instruction and the wait instruction, which is used to indicate that the operator 3 waits for the operator 2 to complete part of the calculation and receive the corresponding calculation result 2 before starting the calculation, and after the calculation is completed, the calculation result 3 is synchronized to the operator 4 with the execution order of 4; and so on, until the operator M with the execution order of M is configured with the wait instruction, which is used to indicate that the operator M waits for the operator M-1 to complete part of the technology and receive the corresponding calculation result M-1 before starting the calculation; here, M represents the maximum execution order of the last operator among all the operators arranged in the execution order involved in the artificial intelligence model. In this way, based on the execution order of multiple operators to be fused, at least one preset synchronization instruction corresponding to each operator to be fused can be quickly determined.
[0059] Exemplarily, when multiple operators to be merged are operator A ranked 1 and operator B ranked 2, the preset synchronization instruction added or inserted on operator A is a pass instruction, and the preset synchronization instruction added or inserted on operator B is a wait instruction.
[0060] Based on the above Figure 1 In the operator fusion method shown, in an exemplary embodiment, step 120 can be specifically implemented through the following steps.
[0061] First, based on the preset synchronization instruction group corresponding to multiple operators to be fused, the first indication information is generated, and the first indication information is used to instruct the user to insert multiple preset synchronization instructions contained in the preset synchronization instruction group into the corresponding operators to be fused in the form of APIs respectively; then, based on the insertion operation completion mark, the fusion operation of multiple operators to be fused is executed in parallel.
[0062] Specifically, when the artificial intelligence chip determines that at least one preset synchronization instruction needs to be added to each operator to be fused, it can automatically generate first indication information to instruct the user to insert each preset synchronization instruction into the corresponding operator to be fused in the form of an application programming interface (API); the first indication information here can be transmitted to the user's corresponding terminal device in text communication, and the first indication information can carry at least one preset synchronization instruction that needs to be added to each operator to be fused. In addition, the user here can be a programmer or developer, etc.
[0063] In this way, all preset synchronization instructions are inserted into the corresponding operators to be fused by the user manually inserting the API until the insertion operation is completed. At this time, the artificial intelligence chip can start the fusion operation of multiple operators to be fused in parallel.
[0064] In an exemplary embodiment, the artificial intelligence chip executes the fusion operation of multiple operators to be fused in parallel, and the specific process is implemented through the following steps.
[0065] In the case where multiple operators to be fused include preceding operators to be fused and subsequent operators to be fused, first, the target data block and the second indication information are sent to the preceding operator to be fused in which the corresponding preset synchronization instruction has been inserted, and the third indication information is sent to the subsequent operator to be fused in which the corresponding preset synchronization instruction has been inserted; the second indication information is used to instruct the preceding operator to be fused to calculate the target data block, and send a synchronization signal carrying the calculation result to the subsequent operator to be fused in which the corresponding preset synchronization instruction has been inserted; the third indication information is used to indicate that the subsequent operator to be fused is in a waiting state for the synchronization signal during the calculation process of the preceding operator to be fused, and calculates based on the calculation result when the synchronization signal is received; a new target data block is re-determined, and the above steps are repeated; until all target data blocks to be fused have completed the calculation of the preceding operator to be fused and the calculation of the subsequent operator to be fused.
[0066] For example, when the current operator to be fused is operator A and the subsequent operator to be fused is operator B, you can refer to Figure 4 The schematic diagram of the operator fusion process is shown in Figure 4As shown, the left side of the dotted line is the operator fusion process provided by the present invention, that is, operator A and operator B are two independent target context programs running simultaneously on the artificial intelligence chip. At this time, if operator A and operator B are both tensor core (tcore) type operators, it is determined that operator A and operator B run on different CUs, or, if operator A and operator B are not tcore type operators, it is determined that operator A and operator B run on different EUs; and, in order to ensure the accuracy of the calculation results (strictly ensure the execution order of operator A and operator B), operator A and operator B need to be synchronized, that is, a pass instruction in the form of API has been inserted into operator A, and a wait instruction in the form of API has been inserted into operator B. In this way, when operator A completes the calculation of the target data block on its CU or EU, it synchronizes the calculation result to the CU or EU running operator B, and operator B starts the calculation; and during the calculation process of operator A, operator B is always in a waiting state for the synchronization signal until it receives the synchronization signal sent by operator A, and operator B starts the calculation based on the received calculation result.
[0067] At the same time, when the CU or EU running operator A sends the calculation result of the target data block to the CU or EU running operator B, it can receive the new target data block and calculate it again. When the calculation result is sent to the CU or EU running operator B, operator B can start the calculation again. Repeat the above process until all target data blocks to be fused are calculated by operator A and operator B in turn. In this way, by making a small modification to the single operator program that has been developed in the artificial intelligence chip (that is, adding preset synchronization instructions or inserting APIs), the fusion purpose of any two or more operators can be achieved, and the development workload has been reduced by an order of magnitude.
[0068] It should be noted that Figure 4 The traditional fusion operator process on the right side of the dotted line is to fuse operator A and operator B through a context program. The developer can either manually fuse operator A and operator B to obtain a fused operator, and then use the fused operator to calculate each target data block; or, the compiler can compile a loop program that sequentially executes operator A and operator B, and then run the loop program to calculate each target data block. Figure 4 The operator fusion process on the right side of the dotted line, the traditional fusion operator process can neither achieve the fusion purpose of any two or more operators, nor reduce the development workload. However, using the operator fusion method provided by the present invention, only m+n operators need to be modified, and the development workload is greatly reduced; and for the fusion of a larger number of operators, the advantages of operator fusion are more obvious.
[0069] The operator fusion method provided by the present invention can complete the development of the fusion operator by keeping the original program unchanged on the basis of the original context program and only adding synchronization instructions (such as adding a pass instruction to the preceding operator and a wait instruction to the subsequent operator). The effect of operator fusion can be achieved by running multiple operators simultaneously on the artificial intelligence chip.
[0070] The artificial intelligence chip provided by the present invention is described below. The artificial intelligence chip described below and the operator fusion method described above can be referenced to each other.
[0071] An artificial intelligence chip includes a processor and a control unit that simultaneously runs multiple context programs. The processor contains multiple processing units, and the multiple processing units are respectively communicatively connected to the control unit.
[0072] The control unit is used to determine, in response to an operator fusion request, a plurality of target context programs from a plurality of context programs as operators to be fused.
[0073] The multiple target processing units corresponding to the multiple operators to be fused are used to implement the fusion process of the multiple operators to be fused based on the preset synchronization instruction groups corresponding to the multiple operators to be fused.
[0074] In an exemplary embodiment, when the multiple operators to be fused include a preceding operator to be fused and a subsequent operator to be fused,
[0075] The target processing unit corresponding to the preceding operator to be fused into which the corresponding preset synchronization instruction has been inserted is used to calculate the target data block and send a synchronization signal carrying the calculation result to the subsequent operator to be fused into which the corresponding preset synchronization instruction has been inserted; the calculation result represents that the preceding operator to be fused completes the calculation of the target data block; The target processing unit corresponding to the subsequent operator to be fused, into which the corresponding preset synchronization instruction has been inserted, is used to be in a waiting state during the calculation process of the preceding operator to be fused, and to perform calculation based on the calculation result when a synchronization signal is received.
[0076] In an exemplary embodiment, the control unit is used to determine the preset synchronization instruction corresponding to each operator to be fused based on the execution order of each of the multiple operators to be fused, and determine the preset synchronization instructions as a preset synchronization instruction group.
[0077] In an exemplary embodiment, a control unit is used to determine a preset synchronization instruction corresponding to each operator to be fused based on synchronization instructions pre-configured for multiple sequentially arranged operators in an artificial intelligence model and the execution order of each of the multiple operators to be fused.
[0078] In an exemplary embodiment, multiple target processing units corresponding to multiple operators to be fused are used to generate first indication information based on a preset synchronization instruction group corresponding to the multiple operators to be fused, and the first indication information is used to instruct the user to insert the multiple preset synchronization instructions contained in the preset synchronization instruction group into the corresponding operators to be fused in the form of APIs respectively; based on the insertion operation completion mark, the fusion operation of the multiple operators to be fused is executed in parallel.
[0079] The operator fusion device provided by the present invention is described below. The operator fusion device described below and the operator fusion method described above can be referenced to each other.
[0080] Reference Figure 5 , which is a schematic diagram of the structure of the operator fusion device provided by the present invention, such as Figure 5 As shown, the operator fusion device 500 is applied to an artificial intelligence chip that runs multiple context programs simultaneously, and includes an operator determination unit 510 and an operator fusion unit 520.
[0081] The operator determination unit 510 is used to determine, in response to an operator fusion request, a plurality of target context programs from a plurality of context programs as operators to be fused.
[0082] The operator fusion unit 520 is used to implement the fusion process of multiple operators to be fused based on the preset synchronization instruction group corresponding to the multiple operators to be fused.
[0083] Optionally, the operator determination unit 510 is specifically configured to determine a preset synchronization instruction corresponding to each operator to be fused based on the respective execution orders of the multiple operators to be fused, and determine the preset synchronization instructions as a preset synchronization instruction group.
[0084] Optionally, the operator determination unit 510 is specifically used to determine the preset synchronization instructions corresponding to each operator to be fused based on the synchronization instructions pre-configured for multiple sequentially arranged operators in the artificial intelligence model, and the execution order of each of the multiple operators to be fused.
[0085] Optionally, the operator fusion unit 520 is specifically used to generate a first indication message based on a preset synchronization instruction group corresponding to multiple operators to be fused, and the first indication information is used to instruct the user to insert multiple preset synchronization instructions contained in the preset synchronization instruction group into the corresponding operators to be fused in the form of APIs respectively; based on the insertion operation completion mark, the fusion operation of multiple operators to be fused is executed in parallel.
[0086] Optionally, the operator fusion unit 520 is specifically used to send the target data block and the second indication information to the preceding operator to be fused into which the corresponding preset synchronization instruction has been inserted, and to send the third indication information to the subsequent operator to be fused into which the corresponding preset synchronization instruction has been inserted, when multiple operators to be fused include a preceding operator to be fused and a subsequent operator to be fused; the second indication information is used to instruct the preceding operator to be fused to calculate the target data block, and send a synchronization signal carrying the calculation result to the subsequent operator to be fused into which the corresponding preset synchronization instruction has been inserted; the third indication information is used to indicate that the subsequent operator to be fused is in a waiting state for a synchronization signal during the calculation process of the preceding operator to be fused, and performs calculation based on the calculation result when the synchronization signal is received; re-determine a new target data block, and repeat the above steps; until all target data blocks to be fused have completed the calculation of the preceding operator to be fused and the calculation of the subsequent operator to be fused.
[0087] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: an artificial intelligence (AI) chip 610, a communication interface (Communications Interface) 620, a memory 630 and a communication bus 640, wherein the AI chip 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The AI chip 610 may call the logic instructions in the memory 630 to execute the operator fusion method, which includes: in response to an operator fusion request, determining from a plurality of context programs that a plurality of target context programs are operators to be fused respectively; based on a preset synchronization instruction group corresponding to the plurality of operators to be fused, implementing the fusion process of the plurality of operators to be fused.
[0088] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0089] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the operator fusion method provided by the above methods, and the method includes: in response to an operator fusion request, determining from multiple context programs that multiple target context programs are operators to be fused; based on a preset synchronization instruction group corresponding to the multiple operators to be fused, implementing the fusion process of the multiple operators to be fused.
[0090] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the operator fusion method provided by the above-mentioned methods. The method includes: in response to an operator fusion request, determining from a plurality of context programs that a plurality of target context programs are operators to be fused respectively; based on a preset synchronization instruction group corresponding to the plurality of operators to be fused, implementing the fusion process of the plurality of operators to be fused.
[0091] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. An operator fusion method, characterized in that: An artificial intelligence chip applied to simultaneously running multiple context programs, the method comprising: In response to the operator fusion request, determining a plurality of target context programs from the plurality of context programs as operators to be fused respectively; Based on the preset synchronization instruction group corresponding to the multiple operators to be fused, the fusion process of the multiple operators to be fused is implemented.
2. The operator fusion method according to claim 1, characterized in that: The method further comprises: Based on the respective execution orders of the multiple operators to be fused, a preset synchronization instruction corresponding to each of the operators to be fused is determined, and the preset synchronization instructions are determined as the preset synchronization instruction group.
3. The operator fusion method according to claim 2, characterized in that: The step of determining a preset synchronization instruction corresponding to each of the operators to be fused based on the respective execution orders of the multiple operators to be fused includes: Based on the synchronization instructions pre-configured for multiple operators arranged in sequence in the artificial intelligence model, and the execution order of each of the multiple operators to be fused, the preset synchronization instructions corresponding to each of the operators to be fused are determined.
4. The operator fusion method according to any one of claims 1 to 3, characterized in that: The step of implementing the fusion process of the multiple operators to be fused based on the preset synchronization instruction group corresponding to the multiple operators to be fused includes: Generate first indication information based on the preset synchronization instruction groups corresponding to the multiple operators to be merged, wherein the first indication information is used to instruct the user to insert the multiple preset synchronization instructions contained in the preset synchronization instruction group into the corresponding operators to be merged in the form of APIs respectively; Based on the insertion operation completion flag, the fusion operations of the multiple operators to be fused are executed in parallel.
5. The operator fusion method according to claim 4, characterized in that: The parallel execution of the fusion operation of the multiple operators to be fused includes: In the case where the multiple operators to be fused include preceding operators to be fused and subsequent operators to be fused, Send the target data block and the second indication information to the preceding operator to be fused into which the corresponding preset synchronization instruction has been inserted, and send the third indication information to the succeeding operator to be fused into which the corresponding preset synchronization instruction has been inserted; the second indication information is used to instruct the preceding operator to be fused to calculate the target data block, and send a synchronization signal carrying the calculation result to the succeeding operator to be fused into which the corresponding preset synchronization instruction has been inserted; the third indication information is used to indicate that the succeeding operator to be fused is in a waiting state for the synchronization signal during the calculation process of the preceding operator to be fused, and performs calculation based on the calculation result when the synchronization signal is received; A new target data block is re-determined, and the above steps are repeated until all target data blocks to be fused have completed the calculation of the preceding operator to be fused and the calculation of the subsequent operator to be fused.
6. An artificial intelligence chip, characterized in that: It includes a processor and a control unit for simultaneously running multiple context programs, wherein the processor includes multiple processing units and the multiple processing units are respectively connected to the control unit for communication; The control unit is used to determine, in response to the operator fusion request, that a plurality of target context programs are respectively operators to be fused from the plurality of context programs; The multiple target processing units corresponding to the multiple operators to be fused are used to implement the fusion process of the multiple operators to be fused based on the preset synchronization instruction groups corresponding to the multiple operators to be fused.
7. The artificial intelligence chip according to claim 6, characterized in that: In the case where the multiple operators to be fused include preceding operators to be fused and subsequent operators to be fused, The target processing unit corresponding to the preceding operator to be fused into which the corresponding preset synchronization instruction has been inserted is used to calculate the target data block, and send a synchronization signal carrying the calculation result to the subsequent operator to be fused into which the corresponding preset synchronization instruction has been inserted; the calculation result represents that the preceding operator to be fused completes the calculation of the target data block; The target processing unit corresponding to the subsequent operator to be fused, into which the corresponding preset synchronization instruction has been inserted, is used to be in a waiting state during the calculation process of the preceding operator to be fused, and to perform calculation based on the calculation result when the synchronization signal is received.
8. An operator fusion device, characterized in that: An artificial intelligence chip for simultaneously running multiple context programs, the device comprising: An operator determination unit, configured to determine, in response to an operator fusion request, a plurality of target context programs from the plurality of context programs as operators to be fused; The operator fusion unit is used to implement the fusion process of multiple operators to be fused based on the preset synchronization instruction group corresponding to the multiple operators to be fused.
9. An electronic device comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, characterized in that: When the artificial intelligence chip executes the computer program, the operator fusion method as described in any one of claims 1 to 5 is implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the operator fusion method according to any one of claims 1 to 5 is implemented.
