Instruction generation methods and devices for artificial intelligence accelerators, electronic devices

Abstract

A method, apparatus, and electronic device for generating instructions for an artificial intelligence accelerator are disclosed, relating to the field of integrated circuits. The method includes: determining a first intermediate representation instruction sequence corresponding to the neural network model to be processed by the AI ​​accelerator; based on the hardware resources of the AI ​​accelerator, converting the first intermediate representation instruction sequence into at least one second intermediate representation instruction sequence using at least one code generation method; and generating executable instructions for the AI ​​accelerator based on the hardware resources of the AI ​​accelerator and the at least one second intermediate representation instruction sequence. Because this technical solution compiles the neural network model to generate executable instructions based on a full consideration of the AI ​​accelerator's hardware resources, the final generated executable instructions can better utilize the AI ​​accelerator's performance and improve the user experience.

Description

Technical Field

[0001] This disclosure relates to the field of integrated circuit technology, and in particular to an instruction generation method and apparatus for an artificial intelligence accelerator, and an electronic device. Background Technology

[0002] The optimization process of a compiler in a high-performance Artificial Intelligence (AI) accelerator (or simply "AI accelerator") involves several steps, including front-end source code parsing, code generation, instruction scheduling, resource allocation, and assembly. Most of these steps operate independently, with some lacking coordination or interdependence. This results in the compiler's final output failing to meet global optimality requirements, leading to non-optimal machine instructions and underperforming the AI ​​accelerator's performance. Summary of the Invention

[0003] In the optimization process of the compiler for AI accelerators in related technologies, each key step is run and solved independently. This results in the compiler ultimately producing machine instructions that are not globally optimal, thus failing to fully utilize the performance of the AI ​​accelerator.

[0004] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide a method, apparatus, and electronic device for generating instructions for an artificial intelligence accelerator. In the instruction generation scheme for the AI ​​accelerator provided by this disclosure, hardware resources are fully considered during the compilation of the neural network model, enabling the final compiled machine instructions to achieve a globally optimal solution, thereby allowing the machine instructions to better utilize the performance of the AI ​​accelerator.

[0005] According to one aspect of this disclosure, an instruction generation method for an artificial intelligence accelerator is provided. The method includes: firstly determining a first intermediate representation instruction sequence corresponding to a neural network model that the AI ​​accelerator needs to process; then, based on the hardware resources of the AI ​​accelerator, converting the first intermediate representation instruction sequence into at least one second intermediate representation instruction sequence using at least one code generation method; and finally, generating instructions executable by the AI ​​accelerator based on the hardware resources of the AI ​​accelerator and at least one second intermediate representation instruction sequence.

[0006] According to one aspect of this disclosure, an instruction generation apparatus for an artificial intelligence accelerator is provided, comprising: a determining module, a code generation module, and an instruction generation module. The determining module is configured to determine a first intermediate representation instruction sequence corresponding to a neural network model that the AI ​​accelerator needs to process; the code generation module is configured to, based on the hardware resources of the AI ​​accelerator, convert the first intermediate representation instruction sequence determined by the determining module into at least one second intermediate representation instruction sequence using at least one code generation method; and the instruction generation module is configured to, based on the hardware resources of the AI ​​accelerator and the at least one second intermediate representation instruction sequence converted by the code generation module, generate instructions executable by the AI ​​accelerator.

[0007] According to one aspect of this disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the methods provided in any of the above aspects.

[0008] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the method provided in any of the preceding aspects.

[0009] Based on the above scheme, when compiling neural network models that require AI accelerator computation, the hardware resources of the AI ​​accelerator are taken into account as a whole. Multiple code generation methods are used to obtain various low-level intermediate representation instruction sequences (i.e., second intermediate representation instruction sequences), which in turn generate the final instructions that the AI ​​accelerator can execute. It can be seen that because the hardware resources of the AI ​​accelerator are fully considered throughout the entire process of compiling the neural network model, the final executable instructions can better utilize the performance of the AI ​​accelerator when running on it, thus improving the user experience. Attached Figure Description

[0010] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0011] Figure 1 It is a schematic diagram of a compilation process provided by related technologies.

[0012] Figure 2 This is a schematic diagram of a compilation process provided in this publication.

[0013] Figure 3 This is a schematic diagram of the structure of a compiler-side device provided in this disclosure.

[0014] Figure 4 This is a schematic diagram of the structure of an operating side device provided in this disclosure.

[0015] Figure 5 This is a flowchart illustrating an instruction generation method for an artificial intelligence accelerator provided in this disclosure. Figure 1 .

[0016] Figure 6 This is another compilation process diagram provided in this publication.

[0017] Figure 7 This is a flowchart illustrating an instruction generation method for an artificial intelligence accelerator provided in this disclosure. Figure 2 .

[0018] Figure 8 This is a flowchart of the operation of a neural network model provided in this publication.

[0019] Figure 9 This is a flowchart illustrating an instruction generation method for an artificial intelligence accelerator provided in this disclosure. Figure 3 .

[0020] Figure 10 This is a flowchart illustrating an instruction generation method for an artificial intelligence accelerator provided in this disclosure. Figure 4 .

[0021] Figure 11 This is a schematic diagram of the partitioning result of the objects to be processed provided in this disclosure.

[0022] Figure 12 This is a flowchart illustrating an instruction generation method for an artificial intelligence accelerator provided in this disclosure. Figure 5 .

[0023] Figure 13 This is an iterative diagram illustrating an iterative code generation method provided in this publication.

[0024] Figure 14 This is a flowchart illustrating an instruction generation method for an artificial intelligence accelerator provided in this disclosure. Figure 6 .

[0025] Figure 15 This is a schematic diagram of a computing window provided in this publication.

[0026] Figure 16 This is a schematic diagram of the generation of sub-intermediate representation instruction sequences in a sliding window code generation method provided in this disclosure. Figure 1 .

[0027] Figure 17 This is a schematic diagram of the generation of sub-intermediate representation instruction sequences in a sliding window code generation method provided in this disclosure. Figure 2 .

[0028] Figure 18 This is a schematic diagram of the generation of sub-intermediate representation instruction sequences in a sliding window code generation method provided in this disclosure. Figure 3 .

[0029] Figure 19 This is a schematic diagram of the generation of sub-intermediate representation instruction sequences in a sliding window code generation method provided in this disclosure. Figure 4 .

[0030] Figure 20 This is a schematic diagram of the generation of sub-intermediate representation instruction sequences in a sliding window code generation method provided in this disclosure. Figure 5 .

[0031] Figure 21 This is a schematic diagram of the generation of sub-intermediate representation instruction sequences in a sliding window code generation method provided in this disclosure. Figure 6 .

[0032] Figure 22 This is a schematic diagram of the generation of sub-intermediate representation instruction sequences in a sliding window code generation method provided in this disclosure. Figure 7 .

[0033] Figure 23 This is a schematic diagram of the generation of sub-intermediate representation instruction sequences in a sliding window code generation method provided in this disclosure. Figure 8 .

[0034] Figure 24 This is a schematic diagram of the generation of sub-intermediate representation instruction sequences in a sliding window code generation method provided in this disclosure. Figure 9 .

[0035] Figure 25 This is a schematic diagram of the generation of sub-intermediate representation instruction sequences in a sliding window code generation method provided in this disclosure. Figure 10 .

[0036] Figure 26 This is a flowchart illustrating an instruction generation method for an artificial intelligence accelerator provided in this disclosure. Figure 7 .

[0037] Figure 27 This is a flowchart illustrating an instruction generation method for an artificial intelligence accelerator provided in this disclosure. Figure 8 .

[0038] Figure 28 This is a flowchart illustrating an instruction generation method for an artificial intelligence accelerator provided in this disclosure. Figure 9 .

[0039] Figure 29 This is a flowchart illustrating an instruction generation method for an artificial intelligence accelerator provided in this disclosure. Figure 10 .

[0040] Figure 30 This is a schematic diagram of the execution result of an instruction generated by a sliding window code generation method provided in this disclosure.

[0041] Figure 31 This is a schematic diagram of the structure of an instruction generation device for an artificial intelligence accelerator provided in this disclosure. Detailed Implementation

[0042] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0043] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0044] Application Overview

[0045] The process of compilers compiling neural network models in related technologies is as follows: Figure 1 As shown, the entire process can include the following steps in sequence: front-end parsing 101, instruction legalization 102, code generation 103, pre-scheduling 104, resource allocation 105 (e.g., SRAM allocation), post-scheduling 106, and assembly 107.

[0046] Specifically, front-end parsing 101 is mainly used to parse the neural network model input by the user to the compiler and obtain a high-level intermediate representation instruction sequence. Instruction legalization 102 is mainly used to convert the high-level intermediate representation instruction sequence into a high-level intermediate representation instruction sequence that can be legally executed on the artificial intelligence (AI) accelerator using specific legalization methods. Code generation 103 is mainly used to reduce the legalized high-level intermediate representation instruction sequence to a low-level intermediate representation instruction sequence that can be executed on the AI ​​accelerator (allocated resources and virtualized). Pre-scheduling 104 is mainly used to determine the scheduling result of each low-level intermediate representation instruction, i.e., the execution order of each low-level intermediate representation instruction, based on the dependencies between the low-level intermediate representation instructions in the low-level intermediate representation instruction sequence. Resource allocation 105 is mainly used to allocate corresponding resources (e.g., SRAM resources, including SRAM addresses and SRAM space) in the AI ​​accelerator based on the tensors to be processed and computed by each low-level intermediate representation instruction. Post-scheduling 106 is mainly used to combine the scheduling results of pre-scheduling and the resource allocation results, as well as the AI ​​accelerator modules (i.e., modules that can perform specific operations, such as convolution modules, pooling modules, etc.) corresponding to each low-level intermediate representation instruction, to virtually run the low-level intermediate representation instruction sequence and adjust the execution order of each low-level intermediate representation instruction in the sequence based on the running results, so as to obtain the final scheduling result. Assembly 107 is mainly used to convert each low-level intermediate representation instruction in each low-level intermediate representation instruction sequence into machine instructions that can be directly executed in the AI ​​accelerator, according to the final scheduling result obtained from post-scheduling.

[0047] It can be seen that in the compilation process of existing compilers, each step is mostly running and solving independently, lacking inter-step coordination and failing to comprehensively consider the impact of AI accelerator hardware resources on instruction generation. As a result, the compilation scheme of existing compilers cannot fully utilize the performance of AI accelerators.

[0048] To address the aforementioned technical issues, refer to Figure 2 As shown, this disclosure provides an instruction generation scheme for an artificial intelligence accelerator. In this scheme, while considering the overall hardware resources of the AI ​​accelerator, multiple code generation methods are used to obtain multiple low-level intermediate representation instruction sequences (i.e., second intermediate representation instruction sequences), thereby generating the final instructions that the AI ​​accelerator can execute.

[0049] Thus, because the hardware resources of the AI ​​accelerator are fully considered throughout the entire process of compiling the neural network model to generate executable instructions for the AI ​​accelerator, the final executable instructions can better utilize the hardware resources of the AI ​​accelerator, maximize the performance of the AI ​​accelerator, and improve the user experience.

[0050] Exemplary System

[0051] This disclosure relates to a compilation-side device (i.e., a compiler in this disclosure) and a runtime-side device. The compilation-side device is used to generate instructions executable by the runtime-side device, and the runtime-side device is used to perform neural network operations by running the instructions provided by the compilation-side device. These operations include, but are not limited to, convolution operations, pooling operations, activation function operations, etc.

[0052] In some embodiments, the compiler-side device may include, but is not limited to, a personal computer, a server computer, a multiprocessor system, a microprocessor-based system, etc.

[0053] In some embodiments, the operating side device may include, but is not limited to, AI accelerators or neural network chips.

[0054] Figure 3 This is a schematic diagram of the structure of an electronic device 30 provided in an embodiment of this disclosure. The electronic device 30 can be used to implement the functions of a compiler-side device. For example... Figure 3 As shown, the electronic device 30 includes at least one processor 31 and a memory 32. In some embodiments, such as Figure 3 As shown, the electronic device 30 may also include an input device 33 and an output device 34.

[0055] Figure 4 This is a schematic diagram of the structure of an electronic device 40 provided in an embodiment of this disclosure. The electronic device 40 can be used to implement the functions of a running-side device. For example... Figure 4 As shown, the electronic device 40 may include a control unit 41 and a computing unit 42. The control unit 41 is used to execute instructions provided by the compilation-side device to control the computing unit 42 to perform calculations on the neural network model. In some embodiments, the electronic device 40 may also include a buffer memory 43 and an off-chip memory 44.

[0056] Exemplary methods

[0057] The methods described in the following embodiments of this disclosure can be applied to many fields that require neural network model calculations, such as image processing, intelligent driving, speech recognition, text conversion, or multimodal processing. For example, image data can be processed in the field of image processing by performing the methods provided in the embodiments of this disclosure.

[0058] Figure 5This disclosure provides an instruction generation method for an artificial intelligence accelerator, which can be applied to the compiler-side devices disclosed in the foregoing embodiments. Figure 5 As shown, the method may include S501-S503:

[0059] S501. Determine the first intermediate representation instruction sequence corresponding to the neural network model that the AI ​​accelerator needs to process.

[0060] Existing processors (e.g., central processing units, CPUs) possess certain data processing capabilities, but for specific processing tasks, such as image processing and speech processing, CPUs exhibit low execution efficiency and poor performance. Therefore, artificial intelligence accelerators are needed to perform the data operations corresponding to these specific processing tasks. Exemplarily, an AI accelerator can be any accelerator capable of executing neural network models, such as a brain processing unit (BPU) or a neural network processing unit (NPU). In this disclosure, the type and structure of the AI ​​accelerator are not specifically limited.

[0061] In some embodiments, the neural network model can be a model applied in multiple different fields such as natural language processing, computer vision, speech recognition, and image processing. Different neural network models can be designed in various fields based on application scenarios and requirements; this disclosure does not limit the application scenarios or model structures of the neural network model.

[0062] In the field of artificial intelligence, because the source language corresponding to a neural network model is a highly abstract language relative to the executable instructions of an AI accelerator, it is necessary to use intermediate representations (IR) to gradually reduce the level of abstraction of the source language. For example, intermediate representations include the first intermediate representation instruction sequence and the second intermediate representation instruction sequence mentioned in this application. The first intermediate representation instruction sequence (i.e., the high-level intermediate representation instruction sequence in this application) has a higher level of abstraction than the second intermediate representation instruction sequence (i.e., the low-level intermediate representation instruction sequence in this application).

[0063] Reference Figure 2 As shown, the steps performed by S501 are front-end parsing 201 and instruction validation 202.

[0064] In the front-end parsing of 201 errors, the compilation-side device can use specific parsing methods or derivation rules to analyze the text and syntax of the neural network model, thereby obtaining a high-level intermediate representation instruction sequence in a unified format. In this embodiment, the specific form of the high-level intermediate representation instruction sequence can be any of the following: graph IR, linear IR, and hybrid IR. Graph IR primarily stores the information obtained after parsing the neural network model in a graph, where each instruction corresponding to the neural network model is represented by nodes, edges, lists, trees, etc., in the graph. Linear IR is assembly code similar to an abstract machine, specifically converting the information obtained after parsing the neural network model into an ordered sequence of operations (i.e., operation instructions), while specifying a clear order for the operation instructions in this sequence. Hybrid IR is a combination of linear IR and graph IR.

[0065] The high-level intermediate representation instruction sequence obtained from the front-end parsing is directly parsed from the syntax and text of the neural network model. Some of the instructions or data types it indicates may not be directly supported by the AI ​​accelerator. Therefore, before further compiling and processing this high-level intermediate representation instruction sequence, it is necessary to use specific legalization methods in the instruction legalization process 202 to convert the high-level intermediate representation instruction sequence into a legal first intermediate representation instruction sequence.

[0066] In some possible scenarios, the high-level intermediate representation instruction sequence parsed by the front end may contain some instructions that cannot be legally converted. Therefore, after the instruction legalization 202 error, it is necessary to check for successful legalization. If it is determined that some instructions cannot be legalized (i.e., legalization failed), it likely indicates that the AI ​​accelerator cannot support the execution of that instruction. In this case, error messages can be directly provided to the user to inform them which instructions cannot be legalized. The user can then modify the neural network model based on the error messages to avoid the occurrence of illegal high-level intermediate representation instructions. After modifying the neural network model, the user can recompile the neural network model using the compilation-side device until all high-level intermediate representation instructions are legal before proceeding with subsequent compilation steps.

[0067] If it is determined that all instructions are valid, proceed with the subsequent compilation steps.

[0068] A neural network model consists of multiple neural network layers. The operations corresponding to one neural network layer, after being parsed and validated by the front end of the compilation device, can yield one or more high-level intermediate representation instructions. The operations corresponding to multiple neural network layers in the neural network model, when transformed into all the high-level intermediate representation instructions, can form a high-level intermediate representation instruction sequence, i.e., the first intermediate representation instruction sequence.

[0069] In this embodiment of the disclosure, the first intermediate representation instruction sequence can be presented in any feasible form. For details, please refer to the relevant descriptions of the high-level intermediate representation instruction sequence in the foregoing embodiments, which will not be repeated here.

[0070] S502. Based on the hardware resources of the AI ​​accelerator, the first intermediate representation instruction sequence is converted into at least one second intermediate representation instruction sequence using at least one code generation method.

[0071] In this embodiment of the disclosure, the second intermediate representation instruction sequence is a low-level intermediate representation instruction sequence. For example, refer to... Figure 2 As shown, the steps performed in step S502 can specifically be code generation 2031 by the compilation-side device during the compilation process.

[0072] The purpose of Code Generation 2031 is to reduce the highly abstract, high-level intermediate representation instruction sequence obtained after the compiler-side device performs front-end parsing and instruction legalization of the neural network model to a low-level intermediate representation instruction sequence (i.e., the second intermediate representation instruction sequence) that can be executed on an AI accelerator. It should be noted that "execution" here refers to allocating corresponding resources (including computational and storage resources) based on the content of the low-level intermediate representation instructions in the low-level intermediate representation instruction sequence, and virtually running it after scheduling. The instructions in this low-level intermediate representation instruction sequence can correspond to the operators supported by the AI ​​accelerator; therefore, it is considered that the instructions in this low-level intermediate representation instruction sequence can be executed on the AI ​​accelerator.

[0073] Specifically, converting the legalized high-level first intermediate representation instruction sequence into a low-level second intermediate representation instruction sequence can include instruction selection and instruction generation. Instruction selection involves using a pre-defined selection strategy (or operator selection algorithm) to match each first intermediate representation instruction in the first intermediate representation instruction sequence with at least one low-level intermediate representation instruction. Instruction generation involves converting all first intermediate representation instructions into their matching low-level intermediate representation instructions based on the matching results of instruction selection, thereby obtaining the low-level second intermediate representation instruction sequence.

[0074] Unlike related technologies, in this embodiment, to ensure that the executable instructions compiled by the compilation-side device can better utilize the performance of the AI ​​accelerator, the hardware resources of the AI ​​accelerator are taken into account during the code generation step. In this embodiment, the hardware resources of the AI ​​accelerator may include computing resources and storage resources.

[0075] For example, computing resources may include: the number of conv modules that can perform convolution operations, the number of pool modules that can perform pooling operations, the number of load modules that can perform data loading, the number of store modules that can perform storage, etc.

[0076] For example, storage resources may include: the number of registers, the number of static random-access memory (SRAM), the size of the SRAM, etc. The SRAM may include L1 SRAM and L2 SRAM.

[0077] The hardware resources of the AI ​​accelerator can be pre-acquired by the compilation-side device before executing the instruction generation method of the AI ​​accelerator provided in the embodiments of this disclosure.

[0078] Furthermore, in order to exhaust all compilation possibilities of the neural network model as much as possible and obtain better compilation results, in this embodiment of the disclosure, considering the hardware resources of the AI ​​accelerator, at least one code generation method can be used to convert the first intermediate representation instruction sequence into at least one second intermediate representation instruction sequence. Each code generation method corresponds to at least one second intermediate representation instruction sequence converted from the first intermediate representation instruction sequence.

[0079] For example, at least one code generation method may include: simple code generation method, iterative split code generation method, and sliding window code generation method.

[0080] In this case, combination Figure 2 , refer to Figure 6 As shown, the code generation in the joint optimization 203 of the compiler-side device can specifically include: generating instruction sequence 1 in at least one second intermediate representation instruction sequence using a simple code generation method, generating instruction sequence 2 in at least one second intermediate representation instruction sequence using an iterative split code generation method, and generating instruction sequence 3 in at least one second intermediate representation instruction sequence using a sliding window code generation method. The specific implementations of the code generation methods mentioned here will be described in detail in subsequent embodiments, and will not be elaborated here.

[0081] S503, based on the hardware resources of the AI ​​accelerator and at least one second intermediate representation instruction sequence, generates instructions executable by the AI ​​accelerator.

[0082] After obtaining at least one low-level second intermediate representation instruction sequence, in order to determine the execution order of each second intermediate representation instruction in the second intermediate representation instruction sequence, the compilation-side device needs to determine the execution order of each second intermediate representation instruction based on the dependencies between the low-level intermediate representation instructions. This process can be called pre-scheduling.

[0083] Meanwhile, in order for each second intermediate representation instruction to run smoothly in the AI ​​accelerator, resources (such as SRAM resources, including SRAM addresses and SRAM space) need to be allocated to each second intermediate representation instruction according to its resource requirements. This process can be called resource allocation.

[0084] After pre-scheduling and successful resource allocation, the scheduling results of pre-scheduling, the resource allocation results, and the AI ​​accelerator modules (i.e., modules capable of performing specific operations, such as convolution modules and pooling modules) corresponding to each second intermediate representation instruction need to be combined to virtually run the second intermediate representation instruction sequence. Based on the running results, the execution order of each low-level intermediate representation instruction in the low-level intermediate representation instruction sequence is adjusted (i.e., the low-level intermediate representation instructions in the low-level intermediate representation instruction sequence are re-scheduled) to obtain a scheduling result that most efficiently utilizes the AI ​​accelerator performance. This process can be called post-scheduling.

[0085] After the scheduling is completed, the low-level intermediate representation instructions can be converted into instructions that the AI ​​accelerator can directly execute, i.e. instructions constructed from machine language, based on the scheduling results obtained from the post-scheduling.

[0086] Based on this, by way of example, refer to Figure 2 As shown, the steps performed by S502 can specifically be pre-scheduling 2032, resource allocation 2033, post-scheduling 204, and assembly 205 of the compilation process by the compilation-side device.

[0087] During the execution of S502, in order to ensure that the final executable instructions of the AI ​​accelerator can fully utilize the performance of the AI ​​accelerator, it is necessary to consider the hardware resources of the AI ​​accelerator. This will allow the final executable instructions to fully utilize the performance of the AI ​​accelerator and improve the user experience.

[0088] Based on the technical solution provided in this disclosure, when compiling a neural network model that requires AI accelerator computation, the hardware resources of the AI ​​accelerator are considered as a whole, and multiple code generation methods are used to obtain multiple low-level intermediate representation instruction sequences (i.e., second intermediate representation instruction sequences), thereby generating the final instructions that the AI ​​accelerator can execute. It can be seen that because the hardware resources of the AI ​​accelerator are incorporated into the entire process of compiling the neural network model, the final executable instructions can better utilize the performance of the AI ​​accelerator, improving the user experience.

[0089] In some embodiments, combined with Figure 5 , refer to Figure 7 As shown, the above S502 may specifically include S701 and S702:

[0090] S701. Determine the operation flow of the neural network model based on the first intermediate representation instruction sequence.

[0091] The first intermediate representation instruction sequence is a highly abstract representation of the neural network model, so the operation flowchart of the neural network model can be obtained based on the first intermediate representation instruction sequence.

[0092] For example, the flowchart of the operation of the neural network model indicated by the first intermediate representation instruction sequence can be as follows: Figure 8 As shown. Based on Figure 8 As can be seen, this neural network model consists of four layers, from input to output: a first convolutional layer (conv1), a first pooling layer (pool2), a second convolutional layer (conv3), and a second pooling layer (pool4). During the process of converting the input to the output of the neural network model, features will appear sequentially. Figure 1 ,feature Figure 2 and characteristics Figure 3 Among them, input, output, and features Figure 1 ,feature Figure 2 and characteristics Figure 3 It can be a tensor.

[0093] Tensors can be multidimensional arrays. For example, in this disclosure, the tensor has three dimensions: H, W, and C, where H is the feature map height, W is the feature map width, and C is the feature map channel (e.g., C is 3 for RGB images).

[0094] In this disclosure, to enable the AI ​​accelerator to successfully complete the computation of the neural network model, the input of the neural network model is pre-stored in a specific memory of the AI ​​accelerator. For example, this specific memory can be a dual-channel synchronous dynamic random access memory (DDR) or any other feasible off-chip memory. Subsequent embodiments will use DDR as an example of the specific memory, and will not be described further.

[0095] S702, based on the hardware resources of the AI ​​accelerator and the operation flow of the neural network model, generates at least one fourth intermediate representation instruction sequence in the second intermediate representation instruction sequence.

[0096] One of the simplest code generation strategies is to directly generate a sequence of instructions that conforms to the computational flow. This approach is the code generation strategy of the simple code generation method mentioned in the previous embodiments.

[0097] In this disclosure, considering the large computational load of neural network models, to improve the computational efficiency of the AI ​​accelerator, the tensors needed for each computation can be stored in the SRAM of the AI ​​accelerator when processing the neural network model. Of course, in practice, other high-speed cache resources similar to SRAM in the AI ​​accelerator can also be used; this disclosure does not impose specific limitations. Subsequent embodiments will use the example of storing tensors in SRAM as required by the AI ​​accelerator's computation, and will not be elaborated further.

[0098] Based on this, the computational flow of the neural network model is as follows: Figure 8 The computational resources in the AI ​​accelerator's hardware resources, as shown in the diagram, include the conv module, pool module, load module, and store module. The storage resources in the AI ​​accelerator's hardware resources include SRAM. The low-level intermediate representation instructions supported by the AI ​​accelerator include: load instructions that support the load module in loading tensors from the AI ​​accelerator's DDR to the AI ​​accelerator's SRAM; store instructions that support the store module in storing tensors from the AI ​​accelerator's SRAM to the AI ​​accelerator's DDR; conv instructions that support the conv module in performing convolution operations; and pool instructions that support the pool module in performing pooling operations. (Input, features...) Figure 1 -F1, Features Figure 2 -F2, Features Figure 3 --Taking the data volume of F3 and output as requiring 8 SRAMs as an example, the fourth intermediate representation instruction sequence generated by the simplest code generation strategy can specifically include the following six intermediate representation instructions:

[0099] 1. Load input; This instruction specifically instructs the AI ​​accelerator to load input from DDR into SRAM.

[0100] 2. `conv F1,input`: This instruction specifically tells the AI ​​accelerator to use the `conv` module to perform a convolution operation on the `input` to obtain features. Figure 1 -F1.

[0101] 3. pool F2, F1; This instruction specifically instructs the AI ​​accelerator to use the pool module to perform a pooling operation on F1 to obtain features. Figure 2 -F2.

[0102] 4. `conv F3,F2`; This instruction specifically instructs the AI ​​accelerator to use the `conv` module to perform a convolution operation on `F2` to obtain the features. Figure 3 -F3.

[0103] 5. pool output, F3; This instruction specifically instructs the AI ​​accelerator to use the pool module to perform a pooling operation on F3 to obtain the output.

[0104] 6. Store output. This instruction specifically instructs the AI ​​accelerator to store the output from SRAM into DDR.

[0105] The above six instructions have a necessary sequential execution relationship, with the execution order being 1-6. It can be seen that the SRAM requirements for input, F1-F3, and output are 8, 8, 8, 8, and 4 respectively. In completing these instructions and obtaining the output, the SRAM must retain at least one instruction's input and output tensors. Therefore, while this implementation can quickly generate the fourth intermediate representation instruction sequence and consumes less computational resources from the AI ​​accelerator, the SRAM resource requirements of the AI ​​accelerator depend on the specific structure of the neural network model. For example, if the maximum data size corresponding to the input and output of the second layer in the neural network model is M units, then the minimum SRAM resource requirement of this fourth intermediate representation instruction sequence for the AI ​​accelerator is M units.

[0106] Based on the technical solutions corresponding to S701 and S702 described above, the fourth intermediate representation instruction sequence can be directly generated based on the computational flow of the neural network model and the hardware resources of the AI ​​accelerator. Because the execution order of each instruction in the fourth intermediate representation instruction sequence is consistent with the computational flow of the neural network model, there is no need to consider instruction parallelism and the number of instructions is small. Therefore, instruction scheduling during the subsequent compilation process can be completed quickly, thereby enabling faster generation of AI accelerator-executable instructions corresponding to the fourth intermediate representation instruction sequence, thus improving the compilation speed of the compilation-side device.

[0107] In some embodiments, combined with Figure 5 , refer to Figure 9 As shown, the above S502 may specifically include S901 and S902:

[0108] S901. Based on the first intermediate representation instruction sequence, determine the objects to be processed corresponding to each neural network layer in the neural network model, and the first dependency relationship between the objects to be processed corresponding to each neural network layer.

[0109] In this embodiment of the disclosure, the input object and output object of the neural network layer can be referred to as the object to be processed.

[0110] For example, the flowchart of the operation of the neural network model indicated by the first intermediate representation instruction sequence can be as follows: Figure 8 As shown. The input object of conv1 is the input of the entire neural network model, and the output object of conv1 is the features. Figure 1 The input object of pool2 is the output object of conv1 - features. Figure 1 The output of pool2 is features. Figure 2 The input to conv3 is the output of pool2 – the features. Figure 2 The output of conv3 is features. Figure 3 The input to pool4 is the output of conv3 – the features. Figure 3 The output of pool4 is the output of the entire neural network model.

[0111] Therefore, the first dependency relationship between the various objects to be processed can be derived as follows: Features Figure 1 Depends on input, features Figure 2 Dependency features Figure 1 ,feature Figure 3 Dependency features Figure 2 Output-dependent features Figure 3 In addition, this first dependency relationship should also include input and features. Figure 1 ,feature Figure 2 ,feature Figure 3 The operations between output and feature Figure 1 It is obtained by performing a convolution operation on the input, and the size of the convolution kernel is 2×2. The first dependency relationship mentioned in subsequent embodiments is similar and will not be described in detail hereafter.

[0112] S902. Based on the hardware resources of the AI ​​accelerator and the first dependency relationship, generate at least one second intermediate representation instruction sequence including multiple first sub-intermediate representation instruction sequences in the third intermediate representation instruction sequence.

[0113] To allow for more parallel opportunities and greater optimization space during pre-scheduling, the compiler, after determining the hardware resources and primary dependencies of the AI ​​accelerator, can employ appropriate strategies to split the objects to be processed in the neural network layers. This results in multiple sets of sub-operational flows, which in turn generate multiple sets of sub-intermediate representation instruction sequences. Because each sub-intermediate representation instruction sequence can include multiple intermediate representation instructions, and many instructions in different sets of intermediate representation instructions do not necessarily have dependencies, the compiler can find more parallel opportunities during pre-scheduling.

[0114] The generation idea of ​​the third intermediate representation instruction sequence in S901 and S902 is the same as the code generation principle of the iterative split code generation method and the sliding window code generation method in the previous embodiments. The specific details will be explained in the subsequent embodiments.

[0115] Based on the technical solutions corresponding to S901 and S902 described above, a first dependency relationship can be obtained through the first intermediate representation instruction sequence. Based on this first dependency relationship and hardware resources, multiple first sub-intermediate representation instruction sequences that can form at least one third intermediate representation instruction sequence within a second intermediate representation instruction sequence can be derived. In this way, because there are multiple sets of first sub-intermediate representation instruction sequences that do not necessarily have a dependency relationship, more parallel opportunities can be provided for subsequent instruction scheduling. Consequently, the instructions ultimately generated that can be executed by the AI ​​accelerator can better utilize the performance of the AI ​​accelerator.

[0116] In some embodiments, taking iterative code generation as an example, combined with Figure 9 ,reference Figure 10 As shown, the above S902 may specifically include S9021A-S9023A:

[0117] S9021A: Based on the hardware resources of the AI ​​accelerator and the first dependency relationship, the object to be processed is split into multiple sub-objects to be processed.

[0118] The purpose of splitting the object to be processed is to obtain multiple sub-operational processes of the neural network model, and each sub-operational process must conform to the original operation process of the neural network model. Therefore, the first dependency relationship needs to be considered when splitting the object to be processed. In addition, whether the multiple sub-operational processes obtained after splitting can run smoothly depends on whether the hardware resources of the AI ​​accelerator can support the requirements of all sub-operational processes.

[0119] For example, S9021A will be specifically explained using the following conditions: The first dependency is Figure 8 The flowchart of the neural network model shown illustrates the dependencies. Figure 8 The four objects to be processed, excluding the output object, have the same data size of 8 units and an H-dimensional size of 8. The output object can have a data size of 4 units and an H-dimensional size of 4. The computational resources in the AI ​​accelerator's hardware resources include a conv module, a pool module, a load module, and a store module. The storage resources in the AI ​​accelerator's hardware resources include 16 SRAMs (or SRAM cells, as in subsequent embodiments) capable of storing one unit of data. The low-level intermediate representation instructions supported by the AI ​​accelerator include: load instructions, store instructions, conv instructions, pool instructions, and pool2 instructions.

[0120] It is important to clarify that the purpose of neural network model operations is to obtain the final output object. Therefore, it is necessary to segment the various objects to be processed that can be obtained before the output object based on the output object.

[0121] Based on the above conditions, we can first try dividing the output object into two equal sub-objects, output0 and output1, with each sub-object containing two units of data. Based on the size and stride of the kernel (convolution kernel or pooling kernel) corresponding to each instruction in the low-level intermediate representation instructions, we know that output0 requires features. Figure 3 The features are derived from the four units of data. Figure 3 The four units of data in the middle require features Figure 2 The five units of data, features Figure 2 The 5 units of data in the middle require features Figure 1 The 6 units of data, features Figure 1 The 6 units of data require 7 units of data from the input. In this case, the input and features need to be combined. Figure 1 ,feature Figure 2 ,feature Figure 3The system is divided into four equal parts. Therefore, it's clear that to obtain output 0, the AI ​​accelerator needs at least 2+4+5+6+7 = 24 SRAMs. Obviously, based on these conditions, the AI ​​accelerator cannot meet this requirement, making this partitioning strategy unfeasible.

[0122] Following the same logical deduction, it is concluded that when the output object is divided into four identical output sub-objects (output0-output3), each sub-object represents one unit of data. Since only the first sub-object, output1, requires calculation of all intermediate data except for input, the calculation of output0 requires the most SRAM, specifically 1+2+3+4+5=15 SRAM. This partitioning method meets the hardware resource requirements of the AI ​​accelerator and is therefore feasible. Now, referring to... Figure 11 As shown, input and features Figure 1 ,feature Figure 2 ,feature Figure 3 It is equally divided into eight parts. The specific sub-objects to be processed within the input partition include input0, input1, input2, input3, input4, input5, input6, and input7; features Figure 1 The sub-objects to be processed include F1_0, F1_1, F1_2, F1_3, F1_4, F1_5, F1_6, and F1_7; features Figure 2 The sub-objects to be processed are divided into F2_0, F2_1, F2_2, F2_3, F2_4, F2_5, F2_6, and F2_7; features Figure 3 The sub-objects to be processed include F3_0, F3_1, F3_2, F3_3, F3_4, F3_5, F3_6 and F3_7.

[0123] Of course, dividing the output object into four or more parts can also meet the hardware resource requirements of the AI ​​accelerator.

[0124] It should be noted that the above example illustrates how to determine the partitioning method for the sub-objects to be processed based on the storage resources in the AI ​​accelerator's hardware resources. In practice, the partitioning method also needs to be considered in conjunction with the computing resources in the AI ​​accelerator's hardware resources. For example, if there are only two conv modules in the AI ​​accelerator's computing resources, in order to better utilize the AI ​​accelerator's performance, then in order to better utilize the parallelizable operations of the multiple groups of sub-objects to be processed, the number of parallelizable convolution operations should ideally not exceed two.

[0125] S9022A. Based on the first dependency relationship, determine the second dependency relationship between multiple sub-objects to be processed corresponding to each neural network layer.

[0126] This embodiment aims to divide the object to be processed into multiple groups of sub-objects to be processed based on the division results, and then obtain multiple first sub-intermediate representation instruction sequences. There should be a dependency relationship between the sub-objects to be processed in each group, and this dependency relationship must be derived from the first dependency relationship between the objects to be processed.

[0127] For example, taking the first dependency as... Figure 8 The flowchart of the neural network model shown illustrates the dependencies, and the partitioning results of the sub-objects to be processed are as follows: Figure 11 As shown, the second dependency can include:

[0128] output0 is obtained by pooling operations F3_0 and F3_1 (with the same parameters as the pool2 instruction). F3_0 and F3_1 are obtained by convolution operations F2_0-F2_2 (i.e., F2_0, F2_1, and F2_2) (with the same parameters as the conv instruction). F2_0-F2_2 is obtained by pooling operations F1_0-F1_3 (i.e., F1_0, F1_1, F1_2, and F1_3) (with the same parameters as the pool instruction). F1_0-F1_3 is obtained by convolution operations input0-input4 (i.e., input0, input1, input2, input3, and input4) (with the same parameters as the conv instruction).

[0129] `output1` is obtained by pooling operations F3_2 and F3_3 (with the same parameters as the `pool2` instruction). `output1` is obtained by convolution operations F2_2-F2_4 (F2_2, F2_3, and F2_4) (with the same parameters as the `conv` instruction). `output1` is obtained by pooling operations F1_2-F1_5 (F1_2, F1_3, F1_4, and F1_5) (with the same parameters as the `pool` instruction). `output1` is obtained by convolution operations input2-input6 (input2, input3, input4, input5, and input6) (with the same parameters as the `conv` instruction).

[0130] output2 is obtained by pooling operations F3_4 and F3_5 (with the same parameters as the pool2 instruction). F3_4 and F3_5 are obtained by convolution operations F2_4-F2_6 (i.e., F2_4, F2_5, and F2_6) (with the same parameters as the conv instruction). F2_4-F2_6 is obtained by pooling operations F1_4-F1_7 (i.e., F1_4, F1_5, F1_6, and F1_7) (with the same parameters as the pool instruction). F2_4-F1_7 is obtained by convolution operations input4-input7 (i.e., input4, input5, input6, and input7) (with the same parameters as the conv instruction).

[0131] output3 is obtained by pooling operations F3_6 and F3_7 (with the same parameters as the pool2 instruction). F3_6 and F3_7 are obtained by convolution operations F2_6 and F2_7 (with the same parameters as the conv instruction). F2_6 and F2_7 are obtained by pooling operations F1_6 and F1_7 (with the same parameters as the pool instruction). F1_6 and F1_7 are obtained by convolution operations input6 and input7 (with the same parameters as the conv instruction).

[0132] S9023A: Generate multiple first sub-intermediate representation instruction sequences based on the second dependency relationship.

[0133] After determining the second dependency, multiple sets of interdependent sub-objects to be processed can be determined based on the second dependency. Based on the dependency between each sub-object in each set of sub-objects to be processed, multiple first sub-intermediate representation instruction sequences can be generated.

[0134] Based on the technical solutions corresponding to S9021A-S9023A mentioned above, the strategy for splitting the object to be processed and the second dependency relationship of different sub-objects to be processed after splitting can be determined based on hardware resources and the first dependency relationship. Then, since the second dependency relationship can represent the generation process of different sub-objects to be processed, multiple first sub-intermediate representation instruction sequences can be successfully generated. This also provides more parallel scheduling opportunities for the instruction scheduling of the subsequent compilation-side device, so that the final generated AI accelerator executable instructions can better utilize the performance of the AI ​​accelerator.

[0135] In one possible implementation, combining Figure 10 , refer to Figure 12 As shown, S9023A may include S90231A and S90232A:

[0136] S90231A. Based on the second dependency relationship, determine the second sub-objects that have a dependency relationship with multiple first sub-objects to be processed in the output object of the neural network model.

[0137] The second sub-object to be processed that has a dependency relationship with the first sub-object to be processed includes sub-objects to be processed that have a direct dependency relationship with the first sub-object to be processed, and sub-objects to be processed that have an indirect dependency relationship with the first sub-object to be processed. For example, if the first sub-object to be processed depends on sub-object A to be processed through convolution, and sub-object A to be processed depends on sub-object B to be processed through pooling, then sub-object B to be processed has an indirect dependency relationship with the first sub-object to be processed.

[0138] For example, taking the second dependency relationship shown after S102 as an example, the first child object to be processed includes output0, output1, output2 and output4.

[0139] The second sub-objects that depend on output0 include: F3_0, F3_1, F2_0, F2_1, F2_2, F1_0, F1_1, F1_2, F1_3, input0, input1, input2, input3, and input4.

[0140] The second sub-objects that depend on output1 include: F3_2, F3_3, F2_2, F2_3, F2_4, F1_2, F1_3, F1_4, F1_5, input2, input3, input4, input5, and input6.

[0141] The second sub-objects that depend on output2 include: F3_4, F3_5, F2_4, F2_5, F2_6, F1_4, F1_5, F1_6, F1_7, input4, input5, input6, and input7.

[0142] The second sub-objects that depend on output3 include: F3_6, F3_7, F2_6, F2_7, F1_6, F1_7, input6, and input7.

[0143] S90232A: Based on multiple first sub-objects to be processed, second sub-objects to be processed that have dependencies on each of the first sub-objects to be processed, and the dependencies between the first sub-objects to be processed and the second sub-objects to be processed, a sequence of multiple first sub-intermediate representation instructions corresponding to the multiple first sub-objects to be processed is generated.

[0144] After S90232A is executed, the compiler-side device has determined which sub-objects were obtained from each first sub-object of the partitioned output object through which operations, thus determining the dependencies between each first and second sub-object. Based on this, and combined with the low-level intermediate representation instructions supported by the AI ​​accelerator, a first-word intermediate representation instruction sequence corresponding to each first sub-object can be generated.

[0145] For example, taking the second dependency relationship shown in the example after S90231A and the example after SS9022A, a total of four iterations will be performed for output0-output3, generating four first sub-intermediate representation instruction sequences. The relationship between the sub-objects to be processed and the iteration order in each sub-intermediate representation instruction sequence can be as follows: Figure 13 As shown, input0-input4, F1_0-F1_3, F2_0-F2_2, F3_0-F3_1, and output0 correspond to the first iteration; input5-input6, F1_4-F1_5, F2_3-F2_4, F3_2-F3_3, and output1 correspond to the second iteration; input7, F1_6-F1_7, F2_5-F2_6, F3_4-F3_5, and output2 correspond to the third iteration; and F2_7, F3_6-F3_7, and output3 correspond to the fourth iteration. The four first sub-intermediates corresponding to output0-output3 represent the instruction sequence as iteration0, iteration1, iteration2, and iteration3, respectively, as detailed below:

[0146] #iteration0

[0147] 1. Load input_0_to_4; (5 SRAMs are required at this time).

[0148] 2. Convex F1_0_to_3, input_0_to_4; (This requires 9 SRAMs)

[0149] 3. Pool F2_0_to_2, F1_0_to_3; (This requires 12 SRAMs)

[0150] 4. Convex F3_0_to_1, F2_0_to_2; (This requires 14 SRAM blocks)

[0151] 5. pool2 output_0, F3_0_to_1; (15 SRAMs are required at this point)

[0152] 6. Store output_0. (At this point, release the 12 SRAM bytes that are not needed for subsequent iterations.)

[0153] #iteration1

[0154] 1. Load input_5_to_6; (At this point, input4, input5, and input6 need to be stored, requiring 3 SRAMs).

[0155] 2. Convex F1_4_to_5, input_4_to_6; (At this point, F1_3, F1_4, and F1_5 need to be stored, requiring a total of 6 SRAMs)

[0156] 3. Pool F2_3_to_4, F1_3_to_5; (At this point, F2_2, F2_3, and F2_4 need to be stored, requiring a total of 9 SRAMs)

[0157] 4. Convex F3_2_to_3, F2_2_to_4; (At this point, a total of 11 SRAM blocks are required)

[0158] 5. pool 2output_1, F3_2_to_3; (At this point, a total of 12 SRAMs are required)

[0159] 6. Store output_1. (At this point, release the 9 SRAM bytes that will not be needed for the next iteration.)

[0160] #iteration2

[0161] 1. Load input_7; (At this point, input6 and input7 need to be stored, requiring 2 SRAMs).

[0162] 2. Convex F1_6_to_7, input_6_to_7; (At this point, F1_5, F1_6, and F1_7 need to be stored, requiring a total of 5 SRAMs)

[0163] 3. Pool F2_5_to_6, F1_5_to_7; (At this point, F2_4, F2_5, and F2_6 need to be stored, requiring a total of 8 SRAMs)

[0164] 4. Convex F3_4_to_5, F2_4_to_6; (At this point, a total of 10 SRAMs are required)

[0165] 5. pool2 output_2, F3_4_to_5; (At this point, a total of 11 SRAMs are required)

[0166] 6. Store output_2. (At this point, release the 10 SRAM units that will not be needed for the next iteration.)

[0167] #iteration3

[0168] 1. Pool F2_7, F1_7; (Further storage of F2_6 and F2_7 is required at this point, totaling 2 SRAMs)

[0169] 2. Convex F3_6_to_7, F2_6_to_7; (This requires a total of 4 SRAMs)

[0170] 3. pool2 output_3, F3_6_to_7; (At this point, a total of 5 SRAMs are required)

[0171] 6. Store output_3. (At this point, release the 5 SRAMs that are not needed.)

[0172] Among them, load A means loading A from DDR into SRAM, conv Y,X means performing a conv convolution operation on X to obtain Y, pool Y,X means performing a pooling operation on X to obtain Y, pool2Y,X means performing a pool2 pooling operation on X to obtain Y, and store Z means storing Z into DDR.

[0173] Based on the technical solutions corresponding to S90231A and S90232A, interruptions in the code generation process during compilation are avoided. Furthermore, since the instruction sequences generated by different iterations have no data dependencies, more parallel opportunities are provided for subsequent instruction scheduling. This allows the final executable instructions of the AI ​​accelerator to leverage the advantages of parallel instruction execution, better utilizing the AI ​​accelerator's performance.

[0174] In some embodiments, taking the sliding window code generation method as an example, combined with Figure 9 ,reference Figure 14 As shown, the above S902 may specifically include S9021B-S9023B:

[0175] S9021B: Based on the hardware resources and primary dependencies of the AI ​​accelerator, determine the computation window for the object to be processed corresponding to each neural network layer.

[0176] The purpose of splitting the object to be processed is to obtain multiple sub-operational processes of the neural network model, and each sub-operational process must conform to the original operation process of the neural network model. Therefore, when determining the operation window of the object to be processed, it is necessary to consider the primary dependency relationship with the operation process of the neural network model. In addition, after the determined operation window is used, whether the multiple sub-operational processes can run smoothly depends on whether the hardware resources of the AI ​​accelerator can support the requirements of all sub-operational processes.

[0177] Furthermore, the basic idea behind the sliding window code generation method is to set a fixed-size computation window for each object to be processed. The size of this computation window must at least ensure that the low-level intermediate representation instructions of the object to be processed can complete one effective calculation. As mentioned above, the kernel size of the `conv` instruction is 2×2, so it requires at least two data units to perform one calculation. Based on this, the `input` object to be processed needs to occupy at least two SRAMs to store two data units. In other words, for `input`, its computation window size is at least 2 data units (referred to as 2).

[0178] After determining the sliding window, each sub-object to be processed can be split into multiple sub-objects based on this sliding window. Then, using... Figure 8 Taking the flowchart of the neural network model shown as an example, at least one calculation window is moved sequentially to gradually complete the calculation of the output. Each move must ensure that at least one sub-object of the object to be processed corresponding to the sliding window can be processed once, and that new sub-objects to be processed that depend on the object to be processed are calculated. The calculation process that needs to be performed after each sliding window move is a sub-calculation process.

[0179] Furthermore, the computation window also needs to ensure that, when all computation windows move to the point where each object to be processed participates in the computation, the amount of storage resources required for the computation of all objects to be processed meets the hardware resource requirements of the AI ​​accelerator, that is, the available storage resources in the hardware resources of the AI ​​accelerator meet this amount.

[0180] For example, S9021B will be specifically explained using the following conditions: The first dependency is... Figure 8 The flowchart of the neural network model shown illustrates the dependencies. Figure 8 The four objects to be processed, excluding the output object, have the same data size of 8 units and an H dimension of 8. The output data size can be 4 units and the H dimension is 4.

[0181] The computing resources in the hardware resources of the AI ​​accelerator include the conv module, pool module, load module, and store module. The storage resources in the hardware resources of the AI ​​accelerator include 10 SRAMs (or SRAM cells) that can store one unit of data. The low-level intermediate representation instructions supported by the AI ​​accelerator include the load instruction, store instruction, conv instruction, pool instruction, and pool2 instruction.

[0182] It is important to clarify that the purpose of neural network model operations is to obtain the final output object, so the sliding window corresponding to each object to be processed needs to be determined based on the output object.

[0183] First, you can try setting the size of the output object's sliding window to two data units. Then, the `store` directive can store the contents of one sliding window in the output object, thus defining the feature. Figure 3 The window size must be at least 4 units to obtain 2 units of data from the output object through a single execution of the pool2 instruction. Therefore, using the sliding window of the output object as a reference, the input and features can be derived. Figure 1 ,feature Figure 2 The calculation windows are all 4 units of data size.

[0184] In this way, when the various calculation windows are moved so that the content of a sliding window in the output object is calculated, it can be deduced that: input, features Figure 1 ,feature Figure 2 and characteristics Figure 3 Each part involved in the computation requires 4 SRAMs. This results in a total of 18 SRAMs being needed. This contradicts the amount of SRAM available in the AI ​​accelerator's hardware resources mentioned above, meaning the AI ​​accelerator's hardware resources cannot support this type of computation.

[0185] Then, using the same logic, it was deduced that when input and features... Figure 1 ,feature Figure 2 ,feature Figure 3 When the sizes of the computation windows corresponding to the output are 2, 2, 2, 2, and 1 respectively, if the content of a sliding window of the output is calculated, the input and features... Figure 1 ,feature Figure 2 ,feature Figure 3 The size of each data point involved in the computation is 2 units. This means a total of 9 SRAM units are required, which the AI ​​accelerator's hardware resources can handle.

[0186] S9022B: Based on the computation window of the object to be processed and the first dependency relationship, the object to be processed is split into multiple sub-objects to be processed, and the second dependency relationship between the multiple sub-objects to be processed corresponding to each neural network layer is determined.

[0187] Specifically, after determining the operation window for each object to be processed, the objects can be split based on the size of the operation window of the output object. For example, taking the instance shown in S9022B, the specific implementation of the split sub-objects to be processed can be as follows: Figure 11 As shown, the specific details can be referred to the relevant descriptions in the foregoing embodiments, and will not be repeated here. The correspondence between the calculation window and the split sub-objects to be processed can be as follows: Figure 15 As shown.

[0188] Then, based on the partitioning results and the first dependency relationship, the second dependency relationship between different sub-objects to be processed can be obtained.

[0189] For example, taking the first dependency as... Figure 8 The flowchart of the neural network model shown illustrates the dependencies, and the partitioning results of each sub-object to be processed are as follows: Figure 11 As shown, the second dependency relationship can be referred to the relevant description after S9022A, which will not be repeated here.

[0190] S9023B: Generate multiple first sub-intermediate representation instruction sequences based on the second dependency relationship and the operation window of the object to be processed.

[0191] The specific implementation of S9023B can be to move the operation window to the corresponding object to be processed according to certain rules. Each move requires that the sub-objects to be processed covered by each operation window can complete at least one operation (convolution operation, pooling operation, etc.).

[0192] For example, the second dependency is shown after S9022B, and the operation window of the object to be processed is the operation window shown after S9021B (e.g. Figure 15 Taking the example shown, each operation window can move ten times, generating ten corresponding first sub-intermediate representation instruction sequences, as detailed below:

[0193] The result after the first move can be as follows Figure 16 As shown, the first sub-intermediate representation instruction sequence generated by this move is as follows:

[0194] #input / F1 window slide

[0195] Load input_0; (At this point, a total of 1 SRAM is required)

[0196] Load input_1; (At this point, a total of 2 SRAMs are required)

[0197] conv F1_0, input_0_to_1. (At this point, a total of 3 SRAMs are required.)

[0198] The result after the second move can be as follows Figure 17 As shown, the first sub-intermediate representation instruction sequence generated by this move is as follows:

[0199] #input / F1 / F2 window slide

[0200] Load input_2; (At this point, release the SRAM occupied by input0, so a total of 3 SRAMs are needed)

[0201] conv F1_1, input_1_to_2; (At this point, a total of 4 SRAMs are needed)

[0202] Pools F2_0 and F1_0_to_1 are needed. (5 SRAMs are required at this point.)

[0203] The result after the third move can be as follows Figure 18 As shown, the first sub-intermediate representation instruction sequence that this move can generate is as follows:

[0204] #input / F1 / F2 / F3window slide

[0205] Load input_3; (At this point, releasing the cumulative SRAM requires 5 units).

[0206] conv F1_2, input_2_to_3; (At this point, a total of 5 SRAMs are needed)

[0207] pool F2_1, F1_1_to_2; (At this point, a total of 6 SRAMs are needed)

[0208] conv F3_0, iF2_0_to_1. (At this point, a total of 7 SRAMs are required.)

[0209] The result after the fourth move can be as follows Figure 19 As shown, the first sub-intermediate representation instruction sequence that this move can generate is as follows:

[0210] #input / F1 / F2 / F3 / output window slide

[0211] Load input_4; (At this point, a total of 7 SRAM units are required)

[0212] conv F1_3, input_3_to_4; (At this point, a total of 7 SRAMs are needed)

[0213] pool F2_2, F1_2_to_3; (At this point, a total of 7 SRAMs are needed)

[0214] conv F3_1, F2_1_to_2; (At this point, a total of 8 SRAMs are needed)

[0215] pool2 output_0, F3_0_to_1; (At this point, a total of 9 SRAMs are required)

[0216] Store output_0. (At this point, a total of 8 SRAM units are required.)

[0217] The result after the fifth move can be as follows Figure 20 As shown, the first sub-intermediate representation instruction sequence generated by this move is as follows:

[0218] #input / F1 / F2 / F3 window slide

[0219] Load input_5; (At this point, a total of 8 SRAMs are required)

[0220] conv F1_4, input_4_to_5; (At this point, a total of 8 SRAMs are needed)

[0221] pool F2_3, F1_3_to_4; (At this point, a total of 8 SRAMs are needed)

[0222] conv F3_2, F2_2_to_3. (At this point, a total of 8 SRAMs are needed.)

[0223] The result after the sixth move can be as follows Figure 21 As shown, the first sub-intermediate representation instruction sequence generated by this move is as follows:

[0224] #input / F1 / F2 / F3 / output window slide

[0225] Load input_6; (At this point, a total of 8 SRAM units are required)

[0226] conv F1_5, input_5_to_6; (At this point, a total of 8 SRAMs are needed)

[0227] pool F2_4, F1_4_to_5; (At this point, a total of 8 SRAMs are needed)

[0228] conv F3_3, F2_3_to_4. (At this point, a total of 8 SRAMs are needed.)

[0229] pool2 output_1, F3_2_to_3; (At this point, a total of 9 SRAMs are required)

[0230] Store output_1. (At this point, a total of 8 SRAMs are required.)

[0231] The result after the seventh move can be as follows Figure 22 As shown, the first sub-intermediate representation instruction sequence generated by this move is as follows:

[0232] #input / F1 / F2 / F3 window slide

[0233] Load input_7; (At this point, a total of 8 SRAMs are required)

[0234] conv F1_6, input_6_to_7; (At this point, a total of 8 SRAMs are needed)

[0235] pool F2_5, F1_5_to_6; (At this point, a total of 8 SRAMs are needed)

[0236] conv F3_4, F2_4_to_5. (At this point, a total of 8 SRAMs are needed.)

[0237] The result after the eighth move can be as follows Figure 23 As shown, the first sub-intermediate representation instruction sequence generated by this move is as follows:

[0238] #F1 / F2 / F3 / output window slide

[0239] conv F1_7, input_7; (At this point, a total of 7 SRAMs are needed)

[0240] pool F2_6, F1_6_to_7; (At this point, a total of 7 SRAMs are needed)

[0241] conv F3_5, F2_5_to_6; (At this point, a total of 7 SRAMs are needed)

[0242] pool2 output_2, F3_4_to_5; (At this point, a total of 8 SRAMs are required)

[0243] Store output_2. (At this point, a total of 7 SRAMs are required.)

[0244] The result after the ninth move can be as follows Figure 24As shown, the first sub-intermediate representation instruction sequence generated by this move is as follows:

[0245] #F2 / F3 window slide

[0246] pools F2_7 and F1_7; (A total of 5 SRAMs are needed at this point)

[0247] conv F3_6, F2_6_to_7; (At this point, a total of 5 SRAMs are needed)

[0248] The result after the tenth move can be as follows Figure 25 As shown, the first sub-intermediate representation instruction sequence generated by this move is as follows:

[0249] #F3 / output window slide

[0250] conv F3_7, F2_7; (At this point, a total of 3 SRAMs are needed)

[0251] pool2 output_3, F3_6_to_7; (At this point, a total of 4 SRAMs are required)

[0252] Store output_3. (At this point, release all 4 occupied SRAMs.)

[0253] As can be seen, the aforementioned ten sliding operations of the computation window can generate ten first sub-intermediate representation instruction sequences. Because the instructions in different first sub-intermediate representation instruction sequences are independent, there are more opportunities for parallel execution, leveraging the parallel execution advantage of the AI ​​accelerator. Furthermore, compared to the technical solutions corresponding to S9021A-S9023A, this technical solution requires less storage resources (e.g., SRAM) from the AI ​​accelerator (a maximum of nine SRAMs are needed simultaneously), thus saving more resources for the AI ​​accelerator (or, in other words, achieving higher resource utilization).

[0254] Based on the technical solutions corresponding to S9021B-S9023B, the computation window for each object to be processed can be determined based on hardware resources and the first dependency relationship, meeting the requirements of the available hardware resources. Then, based on this computation window and the first dependency relationship, the object to be processed can be split, and a second dependency relationship can be determined. Since this second dependency relationship can represent the generation process of different sub-objects to be processed, multiple first sub-intermediate representation instruction sequences can be successfully generated. This avoids interruptions to the code generation process during compilation and provides more parallel scheduling opportunities for subsequent instruction scheduling on the compilation side, enabling the final executable instructions of the AI ​​accelerator to better utilize the performance of the AI ​​accelerator.

[0255] In some embodiments, combined with Figure 5 , refer to Figure 26 As shown, S503 may specifically include S5031 and S5032:

[0256] S5031. Based on the hardware resources of the AI ​​accelerator, determine the target intermediate representation instruction sequence in at least one second intermediate representation instruction sequence.

[0257] To ensure that the final instructions used to generate executable instructions for the AI ​​accelerator maximize its performance, the best-performing sequence from at least one second intermediate representation instruction sequence must be selected as the target intermediate representation instruction sequence. This target intermediate representation instruction sequence is then used as the basis for generating executable instructions for the AI ​​accelerator.

[0258] In one possible implementation, combining Figure 26 , refer to Figure 27 As shown, S5031 may include S271 and S272:

[0259] S271. Based on the hardware resources of the AI ​​accelerator, schedule and allocate resources for each instruction in each second intermediate representation instruction sequence to obtain the running results of each second intermediate representation instruction sequence.

[0260] Combination Figure 2 , refer to Figure 6 As shown, after the compilation-side device generates at least one second intermediate representation instruction sequence (e.g., instruction sequence 1, instruction sequence 2, and instruction sequence 3) using multiple code generation methods, the instructions in each second intermediate representation instruction sequence can be scheduled and resource-allocated based on the hardware resources of the AI ​​accelerator, i.e., joint optimization. (Refer to...) Figure 6 As shown, joint optimization can specifically include code generation 2031, pre-scheduling 2032, resource allocation 2033, and result collection and selection 2034.

[0261] In this embodiment of the disclosure, in order to improve the effectiveness of pre-scheduling and resource allocation and increase the performance utilization of the AI ​​accelerator, pre-scheduling and resource allocation can be performed in conjunction with the hardware resources of the AI ​​accelerator. That is, resource allocation is performed simultaneously with pre-scheduling, so that the two are interdependent during operation or solution, resulting in better pre-scheduling and resource allocation results.

[0262] Furthermore, since at least one second intermediate representation instruction sequence is generated using at least one code production method, the results of different second intermediate representation instruction sequences after undergoing the same pre-scheduling and resource allocation will inevitably differ. Some may fail to complete resource allocation, while those that do will have varying degrees of impact on the AI ​​accelerator's performance. Second intermediate representation instruction sequences that fail to complete resource allocation can be discarded. For multiple second intermediate representation instruction sequences that complete resource allocation, virtual execution can be performed simultaneously with pre-scheduling and resource allocation to obtain the execution results.

[0263] Finally, in the result collection and selection process, the sequence with the best execution result from multiple second intermediate representation instruction sequences that have completed resource allocation is selected as the target intermediate representation instruction sequence based on the execution result. That is, S272 is executed.

[0264] In some embodiments, combined with Figure 27 , refer to Figure 28 As shown, S271 may specifically include S2711-S2713:

[0265] S2711. Based on the second intermediate representation instruction sequence, determine the third dependency relationship between each instruction in the second intermediate representation instruction sequence.

[0266] Taking SRAM as an example of the hardware resources of an AI accelerator, when scheduling instructions in the second intermediate representation instruction sequence, the goal is to ensure that the entire second intermediate representation instruction sequence is fully parallelized on the AI ​​accelerator, so as to better utilize the performance of the AI ​​accelerator. At the same time, it is necessary to allocate different SRAMs (or SRAM resources) to the tensors corresponding to several parallel instructions, so as to avoid resource conflicts caused by tensors using the same SRAM, which would prevent parallelism and affect the performance of the AI ​​calculator.

[0267] Therefore, to ensure the full parallelism of the entire second intermediate representation instruction sequence on the AI ​​accelerator, it is necessary to first determine the dependencies between the instructions in the second intermediate representation instruction sequence, i.e., the third dependency relationships. For example, a certain conv instruction needs to convolve the feature maps input0 and input1 to obtain F1_0; another load instruction needs to load input0 and input1 from DDR to SRAM. Thus, there is a dependency relationship between the load instruction and the conv instruction: the conv instruction depends on the load instruction, the conv instruction is the successor of the load instruction, and the load instruction is the predecessor of the conv instruction. The same logic applies to the remaining instructions.

[0268] In some embodiments, this third dependency can be represented in the form of a graph, which may be called an instruction dependency graph.

[0269] S2712. Determine the resource requirements of each instruction in the second intermediate representation instruction sequence.

[0270] Specifically, the resource requirements of a particular instruction refer to the specific modifications that instruction makes to the computational and storage resources of the AI ​​accelerator. For example, a certain conv instruction requires the computational resource of the conv module in the AI ​​accelerator, as well as three SRAMs in the AI ​​accelerator.

[0271] Of course, in order to achieve the joint scheduling and resource allocation, in addition to determining the resource requirements of each instruction in the second intermediate instruction sequence, it is also necessary to determine the hardware resources of the AI ​​accelerator, that is, to determine the available hardware resources of the AI ​​accelerator (which may include computing resources and storage resources).

[0272] In this way, based on the third dependency relationship, the resource requirements of each instruction, and the hardware resources available to the AI ​​accelerator, the instructions in the second intermediate representation instruction sequence can be scheduled and resources allocated to obtain the running result of the second intermediate representation instruction sequence, that is, to execute S2713.

[0273] S2713. Based on the third dependency relationship and the resource requirements of each instruction in the second intermediate representation instruction sequence, schedule and allocate resources for each instruction in the second intermediate representation instruction sequence to obtain the running result of the second intermediate representation instruction sequence.

[0274] Based on the technical solutions corresponding to S2711-S2713 above, the third dependency relationship and the resource requirements of each instruction can be determined first. Since this third dependency relationship reflects the order of execution among the instructions, a more suitable method can be used for pre-scheduling and resource allocation based on this third dependency relationship and the resource requirements of each instruction to obtain the execution result of the second intermediate representation instruction sequence. Furthermore, based on the execution results of different second intermediate representation instruction sequences, a target intermediate representation instruction sequence can be selected to obtain executable instructions that better utilize the performance of the AI ​​accelerator.

[0275] In some embodiments, combined with Figure 28 , refer to Figure 29 As shown, S2713 may specifically include S27131 and S27132:

[0276] S27131. Based on the third dependency relationship, determine the priority of each instruction in the second intermediate representation instruction sequence.

[0277] Specifically, in this third dependency relationship, the instruction that is executed earlier has higher priority. If two instructions are executed in the same order, they can be considered to have the same priority. Alternatively, their priorities can be determined by combining the resource requirements of the two instructions, or by any feasible method.

[0278] S27132. Based on the priority and resource requirements of each instruction in the second intermediate representation instruction sequence, schedule and allocate resources for each instruction in the second intermediate representation instruction sequence in order of priority to obtain the running result of the second intermediate representation instruction sequence.

[0279] In one possible implementation, S27132 may specifically include the following steps:

[0280] 1. Instructions that do not have a predecessor instruction in the second intermediate instruction sequence are identified as instructions to be scheduled.

[0281] Specifically, instructions that do not have a predecessor instruction can be organized into a set of instructions to be scheduled, ready_nodes, according to their priority.

[0282] 2. Based on the priority order of all scheduled instructions, the resource requirements of each scheduled instruction and the hardware resources of the AI ​​accelerator, each instruction to be run is determined and hardware resources are allocated before execution to obtain the sub-running result of each scheduled instruction.

[0283] Specifically, you can first create a set of running instructions called `running_nodes`. Initially, this set of running instructions is empty.

[0284] Then, according to the priority order of the instructions to be scheduled, the hardware resources required by each instruction to be scheduled are allocated one by one;

[0285] If the hardware resources required by a scheduled instruction can be allocated, then the scheduled instruction is identified as a running instruction, removed from the set of scheduled instructions, and added to the set of running instructions. At the same time, the running instructions in the set of running instructions are run virtually.

[0286] If the hardware resources required by a scheduled instruction cannot be allocated, the instruction is skipped, and an attempt is made to allocate the hardware resources required by a lower priority scheduled instruction.

[0287] During this process, a Compute Resources Table (CRT) and a Storage Resources Table (SRT) can be established. The CRT and SRT can be updated in real time as hardware resources are allocated to instructions awaiting scheduling. The CRT includes the allocation status of all available computing resources in the AI ​​accelerator, specifically including when any computing resource was allocated to which instruction. The SRT includes the allocation status of all available storage resources in the AI ​​accelerator, specifically including when any storage resource was allocated to which instruction. The compiler-side device can determine whether a given instruction can be allocated the necessary hardware resources based on the SRT and CRT.

[0288] 3. After each instruction execution is completed, determine the new instruction to be scheduled based on the third dependency relationship.

[0289] Specifically, when a running instruction in the set of running instructions finishes running, its successor node can be identified as a scheduled instruction and added to the set of scheduled instructions for processing; at the same time, the CRT and SRT also need to be updated synchronously.

[0290] It should be noted that the execution of the instructions here specifically refers to the virtual execution within the AI ​​accelerator.

[0291] 4. The set of sub-run results corresponding to each instruction in the second intermediate representation instruction sequence is determined as the run result of the second intermediate representation instruction sequence.

[0292] For example, taking the second intermediate representation instruction sequence composed of multiple first sub-intermediate representation instruction sequences shown after S9023B in the aforementioned embodiment as an example, the running result obtained after executing the technical solution corresponding to S27132 can be referred to Figure 30 As shown.

[0293] Assuming each instruction takes one unit of time to execute, refer to... Figure 30 As can be seen, based on the technical solution provided in this disclosure, pre-scheduling and resource allocation are performed in conjunction. Therefore, in the final execution result, multiple parallel instructions can exist in each time unit, and all instructions can be executed in a total of 25 time units. This is something that cannot be achieved when pre-scheduling and resource allocation run separately, resulting in a reduction in the chance of parallelism.

[0294] Based on the technical solutions corresponding to S27131 and S27132 above, the priority of each instruction in the second intermediate representation instruction sequence can be obtained based on the third dependency relationship. A higher priority indicates that the corresponding instruction needs to be scheduled and allocated resources earlier. Instructions with similar or equal priorities may run in parallel. Therefore, this solution can successfully complete instruction scheduling and resource allocation based on this priority and resource requirements, obtaining the execution result of the second intermediate representation instruction sequence. This avoids compilation interruptions during the joint optimization of pre-scheduling and resource allocation by the compilation-side device, thus improving compilation efficiency.

[0295] S272. Determine the target intermediate representation instruction sequence based on the execution results of each second intermediate representation instruction sequence.

[0296] S272 can specifically be as follows: Figure 6 The results shown are collected and selected in 2034. For details, please refer to [link / reference]. Figure 6 As shown, when collecting and selecting results 2034, the results of each second intermediate instruction sequence are first collected, and then it is determined whether each result is successful. If it is unsuccessful, the process ends, and if it is successful, the result is selected.

[0297] Specifically, the result selection can be as follows: after obtaining the execution results of each second intermediate representation instruction sequence, the effect parameters corresponding to the execution results of each second intermediate representation instruction sequence can be obtained, such as execution time, memory access size, etc.

[0298] Based on these performance parameters, the performance utilization coefficient of the running results can be derived. For example, the performance utilization coefficient can be a weighted average obtained by normalizing the running time and memory access size. The larger the performance utilization coefficient, the better the running results, and the better the corresponding second intermediate indicates that the instruction sequence performs for the AI ​​accelerator.

[0299] Finally, the second intermediate representation instruction sequence with the optimal performance utilization coefficient of the running result (e.g., the maximum or minimum) can be determined as the target intermediate representation instruction sequence.

[0300] Based on the technical solutions corresponding to S271 and S272 above, each instruction in the second intermediate representation instruction sequence can be scheduled and resources allocated simultaneously to obtain the execution results of all second intermediate representation instruction sequences, thereby determining the target intermediate representation instruction sequence. In this way, the target intermediate representation instruction sequence with the best effect can be used as the basis for generating executable instructions for the AI ​​accelerator, thus better leveraging the performance of the AI ​​accelerator.

[0301] S5032. Based on the target intermediate representation instruction sequence, generate instructions that can be executed by the AI ​​accelerator.

[0302] Specifically, refer to Figure 6 As shown, after obtaining the target intermediate instruction sequence, the target intermediate instruction sequence can be post-scheduled first. The specific implementation of post-scheduling can be referred to the relevant description in the previous embodiment, and will not be repeated here.

[0303] Once scheduling is complete, assembly techniques can be used to convert the intermediate representation instruction sequence of the target into machine language instructions, which are then executed by the AI ​​accelerator.

[0304] Based on the technical solutions corresponding to S5031 and S5032, a better target intermediate representation instruction sequence can be selected from multiple second intermediate representation instruction sequences to generate instructions executable by the AI ​​accelerator, so that the final executable instructions of the AI ​​accelerator can better exert the performance of the AI ​​accelerator.

[0305] It is understood that, in order to achieve the aforementioned functions, the electronic device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, the embodiments of this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure.

[0306] Exemplary device

[0307] In the case of dividing each functional module according to its corresponding function, this disclosure also provides an instruction generation device for an artificial intelligence accelerator. For example... Figure 31 The diagram shown is a structural schematic of an instruction generation device for an artificial intelligence accelerator provided in an embodiment of this disclosure. The device may include: a determination module 311, a code generation module 312, and an instruction generation module 313.

[0308] Among them, the determining module 311 is used to determine the first intermediate representation instruction sequence corresponding to the neural network model that the artificial intelligence AI accelerator needs to process;

[0309] The code generation module 312 is used to convert the first intermediate representation instruction sequence determined by the determining module 311 into at least one second intermediate representation instruction sequence based on the hardware resources of the AI ​​accelerator and using at least one code generation method.

[0310] The instruction generation module 313 is used to generate instructions executable by the AI ​​accelerator based on the hardware resources of the AI ​​accelerator and at least one second intermediate representation instruction sequence converted by the code generation module 312.

[0311] In some embodiments, the code generation module 312 may specifically include a first processing unit 3121 and a second processing unit 3122; wherein, the first processing unit 3121 is used to determine the objects to be processed corresponding to each neural network layer in the neural network model, and the first dependency relationship between the objects to be processed corresponding to each neural network layer, based on the first intermediate representation instruction sequence determined by the determining module 311; the second processing unit 3122 is used to generate at least one first sub-intermediate representation instruction sequence included in the third intermediate representation instruction sequence of at least one second intermediate representation instruction sequence, based on the hardware resources of the AI ​​accelerator and the first dependency relationship determined by the first processing unit 3121.

[0312] In some embodiments, the second processing unit 3122 includes a splitting subunit 31221, a determining subunit 31222, and a generating subunit 31223; wherein, the splitting subunit 31221 is used to split the object to be processed into multiple sub-objects to be processed according to the hardware resources of the AI ​​accelerator and the first dependency relationship determined by the first processing unit 3121; the determining subunit 31222 is used to determine the second dependency relationship between the multiple sub-objects to be processed corresponding to each neural network layer according to the first dependency relationship; and the generating subunit 31223 is used to generate multiple first sub-intermediate representation instruction sequences according to the second dependency relationship determined by the determining subunit 31222.

[0313] In some embodiments, the generation subunit 31223 is specifically used to: determine a second sub-object that has a dependency relationship with a plurality of first sub-objects to be processed in the output object of the neural network model according to a second dependency relationship; and generate a plurality of first sub-intermediate representation instruction sequences corresponding to the plurality of first sub-objects to be processed based on the plurality of first sub-objects to be processed, the second sub-objects to be processed that have a dependency relationship with each of the first sub-objects to be processed, and the dependency relationship between the first sub-objects to be processed and the second sub-objects to be processed.

[0314] In some embodiments, a determining subunit 31222 is used to determine the computation window of the object to be processed corresponding to each neural network layer based on the hardware resources of the AI ​​accelerator and the first dependency relationship; a splitting subunit 31221 is used to split the object to be processed into multiple sub-objects to be processed based on the computation window of the object to be processed determined by the determining subunit 31222 and the first dependency relationship, and to determine the second dependency relationship between the multiple sub-objects to be processed corresponding to each neural network layer; and a generating subunit 31223 is used to generate multiple first sub-intermediate representation instruction sequences based on the second dependency relationship determined by the splitting subunit 31221 and the computation window of the object to be processed determined by the determining subunit 31222.

[0315] In some embodiments, the first processing unit 3121 is further configured to determine the operation flow of the neural network model according to the first intermediate representation instruction sequence; the second processing unit 3122 is further configured to generate at least one fourth intermediate representation instruction sequence in the second intermediate representation instruction sequence based on the hardware resources of the AI ​​accelerator and the operation flow of the neural network model determined by the first processing unit 3121.

[0316] In some embodiments, the instruction generation module 313 includes a selection unit 3131 and a generation unit 3132; wherein, the selection unit 3131 is used to determine a target intermediate representation instruction sequence from at least one second intermediate representation instruction sequence generated by the code generation module 312 based on the hardware resources of the AI ​​accelerator; the generation unit 3132 is used to generate instructions executable by the AI ​​accelerator based on the target intermediate representation instruction sequence selected by the selection unit 3131.

[0317] In some embodiments, the selection unit 3131 includes a first subunit 31311 and a second subunit 31312; wherein, the first subunit 31311 is used to schedule and allocate resources for each instruction in each second intermediate representation instruction sequence generated by the code generation module 312 based on the hardware resources of the AI ​​accelerator, and obtain the running result of each second intermediate representation instruction sequence; the second subunit 31312 is used to determine the target intermediate representation instruction sequence based on the running result of each second intermediate representation instruction sequence obtained by the first subunit 31311.

[0318] In some embodiments, the first subunit 31311 is specifically used to: determine a third dependency relationship between each instruction in the second intermediate representation instruction sequence based on the second intermediate representation instruction sequence; determine the resource requirements of each instruction in the second intermediate representation instruction sequence; and schedule and allocate resources for each instruction in the second intermediate representation instruction sequence based on the third dependency relationship and the resource requirements of each instruction in the second intermediate representation instruction sequence to obtain the running result of the second intermediate representation instruction sequence.

[0319] In some embodiments, when the first subunit 31311 schedules and allocates resources for each instruction in the second intermediate representation instruction sequence based on the third dependency relationship and the resource requirements of each instruction in the second intermediate representation instruction sequence to obtain the running result of the second intermediate representation instruction sequence, it is specifically used to: determine the priority of each instruction in the second intermediate representation instruction sequence based on the third dependency relationship; and schedule and allocate resources for each instruction in the second intermediate representation instruction sequence according to the priority and resource requirements of each instruction in the second intermediate representation instruction sequence in order of priority to obtain the running result of the second intermediate representation instruction sequence.

[0320] Regarding the instruction generation device for the artificial intelligence accelerator in the above embodiments, the specific methods of operation of each module and the corresponding beneficial effects have been described in detail in the embodiments of the instruction generation method for the artificial intelligence accelerator mentioned above, and will not be repeated here.

[0321] Exemplary electronic devices

[0322] This disclosure also provides an electronic device, which can be the instruction generation device of the aforementioned artificial intelligence accelerator, or a terminal or server including the instruction generation device of the artificial intelligence accelerator. The electronic device includes a processor and a memory. The memory stores instructions executable by the processor, which can implement the functions of each module in the instruction generation device of the artificial intelligence accelerator in the above embodiments. The memory stores at least one instruction, which is loaded and executed by the processor to implement the instruction generation method of the artificial intelligence accelerator provided in the above method embodiments.

[0323] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute one or more computer program instructions to implement the instruction generation methods of the artificial intelligence accelerators of the various embodiments of this disclosure described above.

[0324] In some embodiments, the electronic device may further include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanisms. The input devices may include, for example, a keyboard, a mouse, or other means for users to input information. The output devices may output various information to the outside, and may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc. In addition, depending on the specific application, the electronic device 10 may include any other suitable components.

[0325] Exemplary computer program products and computer-readable storage media

[0326] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the instruction generation methods for artificial intelligence accelerators according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0327] In addition to the methods and apparatus described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, cause the processor to perform the steps in the instruction generation method for an artificial intelligence accelerator described in the various embodiments of this disclosure in the "Exemplary Methods" section above.

[0328] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0329] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the instruction generation methods for the artificial intelligence accelerators of various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0330] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0331] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0332] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. An instruction generation method of an artificial intelligence accelerator, the method comprising: determining a first intermediate representation instruction sequence corresponding to a neural network model to be processed by an artificial intelligence (AI) accelerator; converting the first intermediate representation instruction sequence into at least one second intermediate representation instruction sequence based on hardware resources of the AI accelerator by using at least one code generation manner; generating instructions executable by the AI accelerator based on the hardware resources of the AI accelerator and the at least one second intermediate representation instruction sequence; the generating instructions executable by the AI accelerator based on the hardware resources of the AI accelerator and the at least one second intermediate representation instruction sequence comprises: scheduling and allocating resources for each instruction in each of the second intermediate representation instruction sequences based on the hardware resources of the AI accelerator to obtain a running result of each of the second intermediate representation instruction sequences; determining a target intermediate representation instruction sequence according to the running result of each of the second intermediate representation instruction sequences; generating instructions executable by the AI accelerator based on the target intermediate representation instruction sequence. the converting the first intermediate representation instruction sequence into at least one second intermediate representation instruction sequence based on the hardware resources of the AI accelerator by using at least one code generation manner comprises:

2. The method of claim 1, wherein, determining to-be-processed objects corresponding to each neural network layer in the neural network model and a first dependency relationship between the to-be-processed objects corresponding to the each neural network layer according to the first intermediate representation instruction sequence; generating a plurality of first sub-intermediate representation instruction sequences included in a third intermediate representation instruction sequence in the at least one second intermediate representation instruction sequence according to the hardware resources of the AI accelerator and the first dependency relationship. the generating the plurality of first sub-intermediate representation instruction sequences in the third intermediate representation instruction sequence according to the hardware resources of the AI accelerator and the first dependency relationship comprises:

3. The method of claim 2, wherein, splitting the to-be-processed objects to obtain a plurality of to-be-processed sub-objects according to the hardware resources of the AI accelerator and the first dependency relationship; determining a second dependency relationship between the plurality of to-be-processed sub-objects corresponding to the each neural network layer according to the first dependency relationship; generating the plurality of first sub-intermediate representation instruction sequences according to the second dependency relationship. the generating the plurality of first sub-intermediate representation instruction sequences according to the second dependency relationship comprises:

4. The method of claim 3, wherein, determining second to-be-processed sub-objects having a dependency relationship with a plurality of first to-be-processed sub-objects in an output object of the neural network model according to the second dependency relationship; generating the plurality of first sub-intermediate representation instruction sequences corresponding to the plurality of first to-be-processed sub-objects based on the plurality of first to-be-processed sub-objects, the second to-be-processed sub-objects having a dependency relationship with each of the first to-be-processed sub-objects, and the dependency relationship between the first to-be-processed sub-objects and the second to-be-processed sub-objects. the generating the plurality of first sub-intermediate representation instruction sequences in the third intermediate representation instruction sequence according to the hardware resources of the AI accelerator and the first dependency relationship comprises:

5. The method of claim 2, wherein, ​ determine, according to the hardware resources of the AI accelerator and the first dependency relationship, an operation window of a to-be-processed object corresponding to each neural network layer; split the to-be-processed object according to the operation window of the to-be-processed object and the first dependency relationship to obtain a plurality of to-be-processed sub-objects, and determine a second dependency relationship between a plurality of to-be-processed sub-objects corresponding to each neural network layer; generate the plurality of first sub-intermediate representation instruction sequences according to the second dependency relationship and the operation window of the to-be-processed object.

6. The method of claim 1, wherein, The at least one second intermediate representation instruction sequence is generated by at least one code generation method based on the hardware resources of the AI accelerator, including: determining an operation flow of the neural network model according to the first intermediate representation instruction sequence; generating a fourth intermediate representation instruction sequence in the at least one second intermediate representation instruction sequence based on the hardware resources of the AI accelerator and the operation flow of the neural network model.

7. The method of claim 1, wherein, The running result of each second intermediate representation instruction sequence is obtained by scheduling and resource allocation of each instruction in each second intermediate representation instruction sequence based on the hardware resources of the AI accelerator, including: determining a third dependency relationship between each instruction in the second intermediate representation instruction sequence based on the second intermediate representation instruction sequence; determining required resources of each instruction in the second intermediate representation instruction sequence; scheduling and resource allocation of each instruction in the second intermediate representation instruction sequence based on the third dependency relationship and the required resources of each instruction in the second intermediate representation instruction sequence to obtain the running result of the second intermediate representation instruction sequence.

8. The method of claim 7, wherein, The running result of the second intermediate representation instruction sequence is obtained by scheduling and resource allocation of each instruction in the second intermediate representation instruction sequence based on the third dependency relationship and the required resources of each instruction in the second intermediate representation instruction sequence, including: determining a priority of each instruction in the second intermediate representation instruction sequence based on the third dependency relationship; scheduling and resource allocation of each instruction in the second intermediate representation instruction sequence according to the priority and the required resources of each instruction in the second intermediate representation instruction sequence in a priority order to obtain the running result of the second intermediate representation instruction sequence.

9. An instruction generation apparatus of an artificial intelligence accelerator, comprising: a determination module configured to determine a first intermediate representation instruction sequence corresponding to a neural network model to be processed by an artificial intelligence (AI) accelerator; a code generation module configured to convert the first intermediate representation instruction sequence determined by the determination module into at least one second intermediate representation instruction sequence by at least one code generation method based on hardware resources of the AI accelerator; an instruction generation module configured to generate an instruction executable by the AI accelerator based on the hardware resources of the AI accelerator and the at least one second intermediate representation instruction sequence converted by the code generation module; the instruction generation module is specifically configured to: scheduling and resource allocation are performed on each instruction in each second intermediate representation instruction sequence converted by the code generation module based on hardware resources of the AI accelerator, to obtain a running result of each second intermediate representation instruction sequence; determining a target intermediate representation instruction sequence according to the running result of each second intermediate representation instruction sequence; generating an instruction executable by the AI accelerator based on the target intermediate representation instruction sequence.

10. A computer readable storage medium, the storage medium storing a computer program, the computer program being used to execute the instruction generation method of the artificial intelligence accelerator in any one of the above claims 1-8.

11. An electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the instruction generation method of the artificial intelligence accelerator in any one of the above claims 1-8.

Images