A compilation method, system, terminal, and storage medium for generating hardware accelerator executable code for multimodal large language models.

Abstract

This invention relates to the field of data processing technology, and discloses a compilation method, system, terminal, and storage medium for generating executable code for hardware accelerators based on multimodal large language models. The method includes: constructing a computation graph based on the structure of the large language model to be deployed and the operators supported by the hardware platform; traversing the computation graph to infer the output tensor type information of each operator and verifying type consistency between adjacent operators; based on the verified type information, performing static memory allocation to determine the static storage address of weight data, and simultaneously performing dynamic memory allocation to determine the dynamic storage address of state data; subsequently, combining the verified type information, static and dynamic storage addresses, generating an intermediate representation instruction sequence for configuring hardware acceleration platform registers; and finally converting the intermediate representation instruction sequence into a target code file that can be directly loaded and executed by the hardware acceleration platform. This invention improves compilation efficiency, deployment reliability, and resource utilization.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a compilation method, system, terminal, and computer-readable storage medium for generating executable code for hardware accelerators based on multimodal large language models. Background Technology

[0002] As the scale of Large Language Models (LLMs) continues to expand, their enormous computational and storage requirements pose a significant challenge to hardware inference speed. Field-Programmable Gate Array (FPGA) platforms, with their customizability, low latency, and high energy efficiency, have become one of the ideal hardware platforms for deploying LLMs. However, mapping complex LLMs to FPGA hardware presents significant challenges, primarily in the following aspects: Deep learning models, after being deployed to FPGA-based large language model accelerators, require extensive tensor computations during inference, involving the arrangement of operators, address allocation and concatenation between different operators, and how to efficiently perform these computations.

[0003] Existing deep learning compilation and optimization techniques, such as polyhedral compilation and automated tuning tools (e.g., AutoTVM, Automated Tensor Virtual Machine), while improving model execution efficiency to some extent, have significant limitations when dealing with specific FPGA hardware accelerators. Polyhedral compilation is highly complex, requires a strong mathematical background from developers, and performs poorly in heterogeneous computing environments. Tools like AutoTVM, with their large search space and high time costs, offer unsatisfactory tuning results, and the tuning may not reach the level of manual optimization. Furthermore, their effectiveness is uncertain when migrated to different hardware platforms, often requiring additional manual intervention.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a compilation method, system, terminal, and computer-readable storage medium for generating executable code for hardware accelerators based on multimodal large language models, aiming to solve the problem that existing technologies cannot efficiently and accurately compile and deploy large language models on specific hardware accelerators.

[0006] To achieve the above objectives, the present invention provides a compilation method for generating hardware accelerator executable code for multimodal large language models. The compilation method for generating hardware accelerator executable code for multimodal large language models includes the following steps:

[0007] Based on the model structure of the large language model to be deployed and the hardware operators supported by the hardware acceleration platform, a computation graph is constructed, wherein the nodes of the computation graph are operators, and the directed edges of the computation graph represent tensor data flows.

[0008] Traverse the computation graph, infer the output tensor type information of each operator, and verify the consistency of tensor type information between adjacent operators;

[0009] If they match, based on the verified tensor type information, static memory allocation is performed to determine the static storage address of the weight data, and dynamic memory allocation is performed to determine the dynamic storage address of the state data. Based on the verified tensor type information, the static storage address, and the dynamic storage address, an intermediate representation instruction sequence for configuring the hardware acceleration platform registers is generated.

[0010] The intermediate instruction sequence is converted into a target code file that can be loaded and executed by the hardware acceleration platform.

[0011] Furthermore, the construction of the computation graph based on the model structure of the large language model to be deployed and the hardware operators supported by the hardware acceleration platform includes:

[0012] An initial computation graph is derived or defined from the large language model to be deployed. The initial computation graph is parsed into a directed acyclic graph. The computational operations in the directed acyclic graph are abstracted into initial operators. The execution sequence of each initial operator is determined based on the computational logic of the large language model to be deployed, and an operator execution sequence table is obtained.

[0013] If the operator execution sequence list contains a new operator that does not match the hardware operator, then the new operator is registered to obtain the target operator, and the target operator is included in the support scope of the hardware acceleration platform.

[0014] Define attributes for each operator in the operator execution sequence list, and obtain the final computation graph based on the attributes and the operator execution sequence list.

[0015] Furthermore, the step of traversing the computation graph, inferring the output tensor type information of each operator, and verifying the consistency of tensor type information between adjacent operators includes:

[0016] The computation graph is traversed in a recursive descent manner to obtain the input tensor type information for each operator;

[0017] Invoke the inference function corresponding to each operator, generated by the hardware acceleration platform register configuration task, to calculate the output tensor type information of each operator;

[0018] The output tensor type information of the predecessor operator is compared with the input tensor type information of the successor operator.

[0019] Furthermore, the step of performing static memory allocation to determine the static storage address of the weight data includes:

[0020] Traverse the computation graph to identify weight data;

[0021] Calculate the required storage space for the weight data based on the tensor type information of the weight data;

[0022] Based on the storage space, static storage addresses are allocated for the weight data in the hardware acceleration platform storage space according to the graph traversal order.

[0023] Furthermore, the step of performing dynamic memory allocation to determine the dynamic storage address of the state data includes:

[0024] After the static storage address allocation is completed, the computation graph is traversed again to identify all state data, wherein the state data includes input data, output data and intermediate result data;

[0025] During the lifecycle of the state data, a dynamic storage address is allocated for the state data in the hardware acceleration platform storage space, and the dynamic storage address is reclaimed after the lifecycle ends.

[0026] Furthermore, the intermediate instruction sequence is composed of predefined statement expression objects, which are used to abstractly represent the operation of configuring hardware acceleration platform registers; the statement expression objects include assignment statement objects, loop control statement objects, and conditional judgment statement objects.

[0027] Further, the step of converting the intermediate representation instruction sequence into an object code file that can be loaded and executed by the hardware acceleration platform includes:

[0028] Iterate through each statement expression object in the intermediate instruction sequence;

[0029] Based on the type of each statement expression object, the corresponding C or C++ code statements are dynamically generated using the visitor pattern, and all the C or C++ code statements are combined into a target header file that can be loaded and executed by the hardware acceleration platform.

[0030] Furthermore, to achieve the above objectives, the present invention also provides a compilation system for generating hardware accelerator executable code for multimodal large language models, wherein the compilation system for generating hardware accelerator executable code for multimodal large language models includes:

[0031] The computation graph construction module is used to construct a computation graph based on the model structure of the large language model to be deployed and the hardware operators supported by the hardware acceleration platform. The nodes of the computation graph are operators, and the directed edges of the computation graph represent tensor data flows.

[0032] The type information verification module is used to traverse the computation graph, infer the output tensor type information of each operator, and verify the consistency of tensor type information between adjacent operators. If they are consistent, the process continues; otherwise, the process terminates.

[0033] The instruction sequence generation module is used to perform static memory allocation to determine the static storage address of weight data and dynamic memory allocation to determine the dynamic storage address of state data based on the verified tensor type information, the static storage address and the dynamic storage address, and generate an intermediate representation instruction sequence for configuring hardware acceleration platform registers based on the verified tensor type information, the static storage address and the dynamic storage address.

[0034] The code file generation module is used to convert the intermediate representation instruction sequence into a target code file that can be loaded and executed by the hardware acceleration platform.

[0035] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a compiler for generating hardware accelerator executable code for a multimodal large language model, stored in the memory and executable on the processor, wherein when the compiler for generating hardware accelerator executable code for a multimodal large language model is executed by the processor, it implements the steps of the compilation method for generating hardware accelerator executable code for a multimodal large language model as described above.

[0036] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a compiler for generating hardware accelerator executable code for a multimodal large language model, wherein when the compiler for generating hardware accelerator executable code for a multimodal large language model is executed by a processor, it implements the steps of the compilation method for generating hardware accelerator executable code for a multimodal large language model as described above.

[0037] The beneficial effects of this invention are as follows: By mapping the model structure of a large language model to specific operators on a hardware acceleration platform and constructing a computation graph with operators as nodes and tensor data streams as edges, efficient adaptation between the model and hardware is achieved. By automatically inferring the output tensor type information of operators and verifying the consistency between adjacent operators, errors due to data stream mismatches are detected in advance, avoiding runtime failures. This significantly improves deployment reliability and robustness, reduces debugging costs, and ensures the correctness of the generated code. Combining static memory allocation (for fixed-weight data) and dynamic memory allocation (for variable-state data) optimizes storage space utilization and reduces memory consumption. The generation of intermediate representation instruction sequences abstracts hardware configuration details, making the compilation process platform-independent and improving code portability and maintainability. Directly converting the intermediate representation instruction sequences into executable object code files for the hardware accelerator simplifies the deployment process and improves code generation efficiency. Attached Figure Description

[0038] Figure 1 This is a flowchart of a preferred embodiment of the compilation method for generating hardware accelerator executable code for multimodal large language models according to the present invention;

[0039] Figure 2 This is a flowchart illustrating the overall operation of the compilation method for generating executable code for hardware accelerators, which is oriented towards multimodal large language models according to the present invention.

[0040] Figure 3 This is a flowchart of constructing a computation graph for an FPGA hardware platform in the compilation method for generating executable code for hardware accelerators for multimodal large language models in this invention;

[0041] Figure 4 This is a structural diagram of a preferred embodiment of the compilation system for generating hardware accelerator executable code for multimodal large language models according to the present invention;

[0042] Figure 5 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0043] This application provides a compilation method, system, terminal, and storage medium for generating executable code for hardware accelerators based on multimodal large language models. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description, with reference to the accompanying drawings and embodiments, further illustrates this application. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0044] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0045] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0046] The preferred embodiment of the present invention describes a compilation method for generating hardware accelerator executable code for multimodal large language models, such as... Figure 1 and Figure 2 As shown, the compilation method for generating hardware accelerator executable code for multimodal large language models includes the following steps:

[0047] S10. Based on the model structure of the large language model to be deployed and the hardware operators supported by the hardware acceleration platform, construct a computation graph, wherein the nodes of the computation graph are operators, and the directed edges of the computation graph represent tensor data flows.

[0048] The purpose of this step is to map the computational flow of the large language model onto the operators of the FPGA hardware acceleration platform, achieve end-to-end model deployment, and ensure that the model structure matches the hardware characteristics.

[0049] In this embodiment, the operations are as follows: An initial computation graph is exported from the large language model to be deployed or manually defined. The computation graph is represented as a directed acyclic graph, where nodes represent operators (such as matrix multiplication, layer normalization, etc.) and directed edges represent tensor data flows. The computation graph includes node attributes, such as function calls, variables, and constants (operators, inputs, weights). When constructing the computation graph, constraints need to be placed on the input data, including the shape, data type, and storage format of the input data, to ensure compatibility with the hardware acceleration platform. If operators not registered in the compilation system are used in the computation graph, operator registration is required: the operator's scheduling or inference function is defined in the source file of the operator inference function, and then the registration function is called to associate the operator name with the function pointer, thereby bringing the new operator into the scope of the compilation system's support. Finally, a complete computation graph is obtained, where all operators are adapted to the FPGA hardware accelerator.

[0050] It should be noted that the process of constructing a computation graph may involve graph analysis and operator abstraction, but since the computation graph is constructed manually, the graph optimization step can be omitted, thereby simplifying the process and improving efficiency.

[0051] Further, step S10 includes:

[0052] S11. Derive or define an initial computation graph from the large language model to be deployed, parse the initial computation graph into a directed acyclic graph, abstract the computational operations in the directed acyclic graph into initial operators, and determine the execution sequence of each initial operator based on the computational logic of the large language model to be deployed, thereby obtaining an operator execution sequence table.

[0053] The purpose of this step is to convert the model into a graph structure, which facilitates subsequent processing and analysis.

[0054] In this embodiment, the operations are as follows: After the model is exported, graph analysis is performed to abstract the computational operations into operator nodes. Based on the model's computational logic (such as the forward propagation order), the execution sequence of the operators is determined, and an operator execution sequence table is generated. For example, for the attention mechanism layer, the operator sequence may include linear transformations of queries, keys, and values, attention calculations, and output transformations.

[0055] It should be noted that this step ensures the logical correctness of the computation graph, laying the foundation for subsequent shape inference and address allocation.

[0056] S12. If the operator execution sequence list contains a new operator that does not match the hardware operator, then the new operator is registered to obtain a target operator, and the target operator is included in the support scope of the hardware acceleration platform.

[0057] The purpose of this step is to expand the support scope of the hardware acceleration platform and ensure that the new operators can be processed by the compilation system.

[0058] In this embodiment, the operations are as follows: Identify operators in the operator execution sequence list that do not match existing hardware operators. For each new operator, operator registration is performed: define the operator's inference function in the compiler system's operator library. This function describes the operator's behavior (e.g., input / output shape calculation), and call the registration interface to register it. After registration, the new operator becomes a target operator, which can be recognized and used by the compiler system.

[0059] It should be noted that the operator registration mechanism improves the flexibility and scalability of the compilation system, allowing adaptation to continuously evolving hardware operators.

[0060] S13. Define attributes for each operator in the operator execution sequence list, and obtain the final computation graph based on the attributes and the operator execution sequence list.

[0061] The purpose of this step is to improve the metadata of the computation graph, which will facilitate subsequent shape inference and address allocation.

[0062] In this embodiment, the operations are as follows: Attributes are defined for each operator, including the shape, data type, weight information, and operator type (such as matrix multiplication or layer normalization) of the input tensor. Based on the operator execution sequence and attributes, the final computation graph is constructed, ensuring the integrity of nodes and edges.

[0063] It should be noted that attribute definition is crucial to the compilation process, as it directly affects subsequent shape verification and memory allocation.

[0064] In one embodiment, such as Figure 3 As shown, the steps for generating the computation graph are as follows:

[0065] Step 1: Perform operator mapping based on the model structure.

[0066] Action: Analyze the network structure of the large language model to be deployed and identify all computational operations (i.e., operators) that need to be performed in the model.

[0067] Output: A list of operators to be mapped.

[0068] Step 2: Traverse the compiler system operator library.

[0069] Action: For the operator that needs to be mapped, search in the existing hardware operator library of the FPGA compilation system.

[0070] Objective: To confirm whether the operator is supported by the platform.

[0071] Step 3: Determine whether the query contains the corresponding operator.

[0072] Action: Check if there is a hardware implementation in the system operator library that matches the current model operator.

[0073] If not, proceed to step 4.

[0074] If so, proceed to step 5.

[0075] Step 4: Register the operator.

[0076] Action: Register the new operator into the compiler system's operator library so that it can be used later.

[0077] Step 5: Call the corresponding operator function to construct the computation graph.

[0078] Action: Regardless of whether the operator is existing or newly registered, its specific hardware implementation function is called to add it as a node to the computation graph being built.

[0079] Step 6: Determine whether all operator mappings have been completed.

[0080] Action: Check if the initial list of operators for the model has been fully processed.

[0081] If not (not completed), the process returns to step 1 to continue processing the next operator to be mapped, forming a processing loop.

[0082] If so (completed), proceed to the final step.

[0083] Step 7: Generate the computational graph corresponding to the FPGA hardware platform.

[0084] Action: Once all operators in the model have been successfully mapped and constructed as nodes, output the final computation graph.

[0085] Output: A complete computation graph consisting of FPGA hardware operators, used for subsequent compilation and deployment.

[0086] Process Summary: The process begins with model parsing, iterates through each operator, checks its support status in the system library, registers any missing operators, calls operator functions to construct graph nodes, and continues until all operators have been processed, ultimately generating a computation graph adapted for the FPGA.

[0087] S20. Traverse the computation graph, infer the output tensor type information of each operator, and verify the consistency of tensor type information between adjacent operators.

[0088] The purpose of this step is to ensure data flow compatibility between operators in the computation graph and to avoid deployment errors caused by shape and data type mismatches.

[0089] In this embodiment, the operation is as follows: The computation graph is traversed recursively, visiting each operator node. For each operator, based on the shape and data type of its input tensor, and the operator's type, the corresponding shape inference function is called to calculate the shape and data type of the output tensor. The shape inference function is generated based on the FPGA hardware accelerator's register configuration task, ensuring that the output matches the hardware expectation. For example, for a matrix multiplication operator with an input shape of (1, 1014, 2048) and a data type of float16, and a weight shape of (2048, 2048) and a data type of int4, the output shape will be (1, 1014, 2048) and the data type of float16. Then, the consistency of adjacent operators is verified: the output tensor type information of the predecessor operator is compared with the input tensor type information of the successor operator. If they match, the process continues; otherwise, the process terminates and an error is reported.

[0090] It should be noted that the tensor type information refers to shape and data type. Shape and data type verification is a critical checkpoint in the compilation process to prevent runtime errors and improve deployment reliability.

[0091] Further, step S20 includes:

[0092] S21. Traverse the computation graph in a recursive descent manner to obtain the input tensor type information of each operator.

[0093] The purpose of this step is to collect the input specifications for each operator, providing a basis for shape inference.

[0094] In this embodiment, the operations are as follows: starting from the initial node of the computation graph, each node is recursively visited to obtain the shape and data type of its input tensor. The traversal order follows the dependencies of the computation graph to ensure that all nodes are processed.

[0095] It should be noted that recursive descent traversal ensures the correct transmission of data stream.

[0096] S22. Call the inference function corresponding to each operator, generated by the hardware acceleration platform register configuration task, to calculate the output tensor type information of each operator.

[0097] The purpose of this step is to accurately infer the output type and ensure hardware compatibility.

[0098] In this embodiment, the operation is as follows: Each operator has a corresponding inference function, which is generated based on the hardware accelerator's register configuration task. These inference functions are called, taking into account the input tensor type information, and the output tensor type information is calculated. The inference functions are hard-coded with hardware-specific rules, such as considering the impact of quantization type (e.g., int4) on the computation.

[0099] It should be noted that the inference function based on hardware configuration ensures that the output is perfectly adapted to the hardware accelerator.

[0100] S23. Compare the output tensor type information of the predecessor operator with the input tensor type information of the successor operator.

[0101] The purpose of this step is to verify the continuity of the data stream.

[0102] In this embodiment, the operation is as follows: For each directed edge, check whether the output of the predecessor node matches the input of the successor node in terms of shape and data type. If they do not match, the compilation process terminates and an error message is output, indicating the location and details of the mismatch.

[0103] It should be noted that this step enhances the robustness of the compilation process and avoids invalid deployments.

[0104] S30. If they match, perform static memory allocation to determine the static storage address of the weight data based on the verified tensor type information, and perform dynamic memory allocation to determine the dynamic storage address of the state data. Based on the verified tensor type information, the static storage address, and the dynamic storage address, generate an intermediate representation instruction sequence for configuring the hardware acceleration platform registers.

[0105] The purpose of this step is to efficiently manage memory resources and generate configuration instructions for each operator, thereby optimizing storage space utilization.

[0106] In this embodiment, the operations are as follows: First, static memory allocation is performed: the computation graph is traversed to identify all weight data nodes. Based on the tensor type information of the weight data (such as shape and data type), the required storage space (in bytes) is calculated. For example, for a weight with a shape of (2048, 2048) and a data type of int4, the required space is 2048 * 2048 * (4 or 8) bytes. Then, in a depth-first traversal order, static storage addresses are tightly allocated in the storage space of the hardware acceleration platform (such as HBM). The weight data is fixed, so the static address is determined at compile time and remains unchanged at runtime. Second, dynamic memory allocation is performed: after static allocation, the computation graph is traversed again to identify state data (including input data, output data, and intermediate result data). The space requirement is calculated based on the data type (such as float16). Dynamic allocation considers the data lifecycle: dynamic storage addresses are allocated within the data's lifecycle and reclaimed after the lifecycle ends for reuse. For example, if input data is used only by one operator, its address is released after that operator is executed. Finally, based on the verified tensor type information, static memory address, and dynamic memory address, an intermediate representation (IR) instruction sequence is generated. The IR consists of predefined statement expression objects, such as assignment statement objects, loop control statement objects, and conditional statement objects. These objects abstractly represent operations that configure hardware registers. IR generation is achieved by calling the configuration task for each operator. The configuration task is imported from a testbench written in Verilog or System Verilog and converted into an array of IR objects.

[0107] It should be noted that dynamic address allocation significantly improves memory utilization, especially for resource-constrained FPGA platforms. IR, as a platform-independent abstraction layer, facilitates subsequent code generation.

[0108] Further, the step of performing static memory allocation to determine the static storage address of the weight data includes:

[0109] Traverse the computation graph to identify weight data;

[0110] Calculate the required storage space for the weight data based on the tensor type information of the weight data;

[0111] Based on the storage space, static storage addresses are allocated for the weight data in the hardware acceleration platform storage space according to the graph traversal order.

[0112] In the above embodiment, the operations are as follows: A depth-first search algorithm is used to traverse the computation graph and collect all weight nodes. For each weight node, the space size is calculated based on its shape and data type. Then, starting from the base address, static addresses are allocated tightly in traversal order to avoid fragmentation.

[0113] It should be noted that static allocation ensures efficient access to weight data at runtime.

[0114] Furthermore, the step of performing dynamic memory allocation to determine the dynamic storage address of the state data includes:

[0115] After the static storage address allocation is completed, the computation graph is traversed again to identify all state data, wherein the state data includes input data, output data and intermediate result data;

[0116] During the lifecycle of the state data, a dynamic storage address is allocated for the state data in the hardware acceleration platform storage space, and the dynamic storage address is reclaimed after the lifecycle ends.

[0117] In the above embodiment, the operations are as follows: traverse the computation graph, analyze the producers and users of each state data, and determine the lifecycle. When allocating addresses, use a memory management algorithm (such as the first-fit algorithm) to allocate addresses in a dynamic memory pool. When data is no longer in use, immediately reclaim its address for subsequent allocation.

[0118] It should be noted that dynamic allocation is based on lifecycle management, which reduces memory usage and supports the deployment of larger models.

[0119] Furthermore, the intermediate instruction sequence is composed of predefined statement expression objects, which are used to abstractly represent the operation of configuring hardware acceleration platform registers; the statement expression objects include assignment statement objects, loop control statement objects, and conditional judgment statement objects.

[0120] In the above embodiments, the intermediate instruction sequence is a custom instruction sequence, with each instruction corresponding to a statement expression object. For example, an assignment statement object represents a register write operation (such as setting a register address and value), and a loop statement object represents repeated configuration (such as for loop control). These objects are generated by parsing configuration tasks in the Verilog testbench: each task is converted into an array of IR objects, which contains sub-expression objects (such as loop variable initialization and conditional judgment). The IR objects are organized in a tree structure for easy traversal and code generation.

[0121] It should be noted that IR design improves the maintainability and scalability of the compilation system, allowing support for multiple hardware platforms.

[0122] S40. The intermediate representation instruction sequence is converted into a target code file that can be loaded and executed by the hardware acceleration platform.

[0123] The purpose of this step is to generate executable code files for configuring the hardware accelerator and completing model deployment.

[0124] In this embodiment, the operations are as follows: Each statement expression object in the intermediate IR instruction sequence is traversed. Using the visitor pattern, corresponding C or C++ code statements are dynamically generated based on the object's type. For example, assignment statement objects generate code like "*addr = value;", and loop statement objects generate for loop code. All generated code statements are combined into a C or C++ header file (.h file), which contains all configuration instructions and can be loaded and executed by the hardware acceleration platform. During the generation process, the consistency between the IR and the hardware design can be verified: a testbench is run using simulation tools (such as ModelSim), and the generated configuration statements are compared to ensure correctness.

[0125] It should be noted that the code generation process ensures a complete match with the hardware register configuration, enabling seamless deployment.

[0126] Further, step S40 includes:

[0127] S41. Traverse each statement expression object in the intermediate representation instruction sequence;

[0128] S42. Based on the type of each statement expression object, dynamically generate corresponding C or C++ code statements using the visitor pattern, and combine all the C or C++ code statements into a target header file that can be loaded and executed by the hardware acceleration platform.

[0129] In the above embodiments, the operations are as follows: For each IR object, the visitor pattern calls the corresponding code generation method. For example, when accessing an assignment statement object, register write code is generated; when accessing a loop statement object, the loop structure and inner statements are generated. The final header file contains a complete initialization sequence for setting model parameters on the hardware.

[0130] It should be noted that the visitor pattern separates the IR structure and code generation logic, improving the modularity and testability of the code.

[0131] Furthermore, step S40 also includes the following sub-steps:

[0132] S43. Verify the consistency between the generated target code file and the expected behavior of the hardware accelerator using simulation software.

[0133] The purpose of this step is to verify functional correctness. The compiler can automatically generate simulation scripts, call simulation tools such as Modelsim, and run the hardware testbench. The simulation will generate a register configuration log. By writing a comparison function, the configuration sequence in the log is compared with the configuration statements in the generated .h file. If they match completely, it proves that the compilation process is accurate and the generated code can correctly configure the hardware.

[0134] The beneficial effects of this invention are as follows:

[0135] 1. By mapping the model structure of a large language model to specific operators on a hardware acceleration platform and constructing a computation graph with operators as nodes and tensor data streams as edges, efficient adaptation between the model and hardware is achieved. This simplifies the complexity of model deployment, ensures an end-to-end deployment process, improves the systematicity and manageability of the compilation process, and enhances cross-hardware platform compatibility.

[0136] 2. By automatically inferring the output tensor type information (such as shape and data type) of operators and verifying the consistency between adjacent operators, errors caused by data flow mismatches are detected in advance, avoiding runtime failures. This significantly improves the reliability and robustness of deployment, reduces debugging costs, and ensures the correctness of generated code.

[0137] 3. By combining static memory allocation (for data with fixed weights) and dynamic memory allocation (for data with variable states), storage space utilization is optimized and memory usage is reduced. The generation of intermediate representation instruction sequences abstracts hardware configuration details, making the compilation process platform-independent, improving code portability and maintainability, and providing a unified interface for subsequent code generation.

[0138] 4. Directly converting intermediate instruction sequences into executable object code files (such as C or C++ header files) for hardware accelerators simplifies the deployment process and improves code generation efficiency. This ensures precise matching between code and hardware registers, enabling efficient inference and reducing the need for manual intervention.

[0139] In summary, this invention addresses the issues of poor compatibility and suboptimal performance of deep learning compilers on specific hardware platforms. Through computation graph-driven processing, type verification, intelligent memory allocation, and automated code generation, this invention improves compilation efficiency, deployment reliability, and resource utilization, making it particularly suitable for resource-constrained hardware acceleration platforms such as FPGAs.

[0140] Furthermore, such as Figure 4As shown, based on the above-described compilation method for generating hardware accelerator executable code for multimodal large language models, this invention also provides a compilation system for generating hardware accelerator executable code for multimodal large language models. The compilation system for generating hardware accelerator executable code for multimodal large language models includes:

[0141] The computation graph construction module 51 is used to construct a computation graph based on the model structure of the large language model to be deployed and the hardware operators supported by the hardware acceleration platform, wherein the nodes of the computation graph are operators and the directed edges of the computation graph represent tensor data flows.

[0142] The type information verification module 52 is used to traverse the computation graph, infer the output tensor type information of each operator, and verify the consistency of tensor type information between adjacent operators. If they are consistent, the process continues; otherwise, the process terminates.

[0143] The instruction sequence generation module 53 is used to perform static memory allocation to determine the static storage address of weight data and dynamic memory allocation to determine the dynamic storage address of state data based on the verified tensor type information, the static storage address and the dynamic storage address, and generate an intermediate representation instruction sequence for configuring hardware acceleration platform registers based on the verified tensor type information, the static storage address and the dynamic storage address.

[0144] The code file generation module 54 is used to convert the intermediate representation instruction sequence into a target code file that can be loaded and executed by the hardware acceleration platform.

[0145] Furthermore, such as Figure 5 As shown, based on the above-mentioned compilation method and system for generating hardware accelerator executable code for multimodal large language models, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 5 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0146] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a compiler 40 for generating hardware accelerator executable code for a multimodal large language model. This compiler 40 can be executed by the processor 10 to implement the compilation method for generating hardware accelerator executable code for a multimodal large language model as described in this application.

[0147] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the compilation method for generating hardware accelerator executable code for multimodal large language models.

[0148] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0149] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a compiler for generating hardware accelerator executable code for a multimodal large language model, wherein when the compiler for generating hardware accelerator executable code for a multimodal large language model is executed by a processor, it implements the steps of the compilation method for generating hardware accelerator executable code for a multimodal large language model as described above.

[0150] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0151] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0152] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A compilation method for generating hardware accelerator executable code for multimodal large language models, characterized in that, The compilation method for generating hardware accelerator executable code for multimodal large language models includes: Based on the model structure of the large language model to be deployed and the hardware operators supported by the hardware acceleration platform, a computation graph is constructed, wherein the nodes of the computation graph are operators, and the directed edges of the computation graph represent tensor data flows. Traverse the computation graph, infer the output tensor type information of each operator, and verify the consistency of tensor type information between adjacent operators; If they match, based on the verified tensor type information, static memory allocation is performed to determine the static storage address of the weight data, and dynamic memory allocation is performed to determine the dynamic storage address of the state data. Based on the verified tensor type information, the static storage address, and the dynamic storage address, an intermediate representation instruction sequence for configuring the hardware acceleration platform registers is generated. The intermediate instruction sequence is converted into a target code file that can be loaded and executed by the hardware acceleration platform. The step of constructing a computation graph based on the model structure of the large language model to be deployed and the hardware operators supported by the hardware acceleration platform includes: An initial computation graph is derived or defined from the large language model to be deployed. The initial computation graph is parsed into a directed acyclic graph. The computational operations in the directed acyclic graph are abstracted into initial operators. The execution sequence of each initial operator is determined based on the computational logic of the large language model to be deployed, and an operator execution sequence table is obtained. If the operator execution sequence list contains a new operator that does not match the hardware operator, then the new operator is registered to obtain the target operator, and the target operator is included in the support scope of the hardware acceleration platform. Define attributes for each operator in the operator execution sequence list, and obtain the final computation graph based on the attributes and the operator execution sequence list; The step of traversing the computation graph, inferring the output tensor type information of each operator, and verifying the consistency of tensor type information between adjacent operators includes: The computation graph is traversed in a recursive descent manner to obtain the input tensor type information for each operator; Invoke the inference function corresponding to each operator, generated by the hardware acceleration platform register configuration task, to calculate the output tensor type information of each operator; Compare the output tensor type information of the predecessor operator with the input tensor type information of the successor operator; The step of performing static memory allocation to determine the static storage address of the weight data includes: Traverse the computation graph to identify weight data; Calculate the required storage space for the weight data based on the tensor type information of the weight data; Based on the storage space, static storage addresses are allocated for the weight data in the hardware acceleration platform storage space according to the graph traversal order; The step of performing dynamic memory allocation to determine the dynamic storage address of the state data includes: After the static storage address allocation is completed, the computation graph is traversed again to identify all state data, wherein the state data includes input data, output data and intermediate result data; During the lifecycle of the state data, a dynamic storage address is allocated for the state data in the hardware acceleration platform storage space, and the dynamic storage address is reclaimed after the lifecycle ends.

2. The compilation method for generating hardware accelerator executable code for multimodal large language models according to claim 1, characterized in that, The intermediate instruction sequence is composed of predefined statement expression objects, which are used to abstractly represent the operation of configuring hardware acceleration platform registers; the statement expression objects include assignment statement objects, loop control statement objects, and conditional judgment statement objects.

3. The compilation method for generating hardware accelerator executable code for multimodal large language models according to claim 1, characterized in that, The step of converting the intermediate representation instruction sequence into a target code file that can be loaded and executed by the hardware acceleration platform includes: Iterate through each statement expression object in the intermediate instruction sequence; Based on the type of each statement expression object, the corresponding C or C++ code statements are dynamically generated using the visitor pattern, and all the C or C++ code statements are combined into a target header file that can be loaded and executed by the hardware acceleration platform.

4. A compilation system for generating hardware accelerator executable code for multimodal large language models, characterized in that, The compilation system for generating hardware accelerator executable code for multimodal large language models is used to implement the compilation method for generating hardware accelerator executable code for multimodal large language models as described in any one of claims 1-3, wherein the compilation system for generating hardware accelerator executable code for multimodal large language models includes: The computation graph construction module is used to construct a computation graph based on the model structure of the large language model to be deployed and the hardware operators supported by the hardware acceleration platform. The nodes of the computation graph are operators, and the directed edges of the computation graph represent tensor data flows. The type information verification module is used to traverse the computation graph, infer the output tensor type information of each operator, and verify the consistency of tensor type information between adjacent operators. If they are consistent, the process continues; otherwise, the process terminates. The instruction sequence generation module is used to perform static memory allocation to determine the static storage address of weight data and dynamic memory allocation to determine the dynamic storage address of state data based on the verified tensor type information, the static storage address and the dynamic storage address, and generate an intermediate representation instruction sequence for configuring hardware acceleration platform registers based on the verified tensor type information, the static storage address and the dynamic storage address. The code file generation module is used to convert the intermediate representation instruction sequence into a target code file that can be loaded and executed by the hardware acceleration platform.

5. A terminal, characterized in that, The terminal includes: a memory, a processor, and a compiler for generating hardware accelerator executable code for a multimodal large language model, stored in the memory and executable on the processor. When the compiler for generating hardware accelerator executable code for a multimodal large language model is executed by the processor, it implements the steps of the compilation method for generating hardware accelerator executable code for a multimodal large language model as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a compiler for generating hardware accelerator executable code for a multimodal large language model, wherein when the compiler for generating hardware accelerator executable code for a multimodal large language model is executed by a processor, it implements the steps of the compilation method for generating hardware accelerator executable code for a multimodal large language model as described in any one of claims 1-3.

Images