Large language model operator conversion system and method and AI accelerator

Abstract

The invention provides a large language model operator conversion system and method and an AI accelerator, the system comprises an operator identification module, a conversion module and a model reconstruction module, the operator identification module analyzes a computational graph structure of an input model to identify convertible operators meeting preset conversion conditions in the computational graph structure; the conversion module converts the convertible operator into an equivalent operator sequence based on a preset conversion strategy; and the model reconstruction module replaces nodes corresponding to convertible operators in the computational graph of the input model with an equivalent operator sequence, updates the connection relation and the weight initializer, and generates an updated model. According to the system, pipeline processing of model optimization is realized through an identification rule and a conversion strategy, and adaptation for different hardware platforms can be realized by updating a preset conversion condition and a conversion strategy, so that the problems of low deployment efficiency and low expandability on a CNN (Convolutional Neural Network) accelerator are solved.

Description

Technical Field

[0001] This application relates to the field of model technology, and in particular to a large language model operator conversion system, method and AI accelerator. Background Technology

[0002] Large language models are used in edge and edge AI applications. When large language models are deployed to edge and edge devices, if they rely on traditional hardware accelerators optimized for convolutional neural networks, such as neural processing units, digital signal processors and application-specific integrated circuits, they will face the problem of mismatch between the model's execution characteristics and the hardware capabilities.

[0003] In the relevant technical solutions, the model structure is analyzed manually to identify performance bottleneck operators. Adaptation strategies such as model rewriting, operator fusion, or replacement with hardware-friendly expressions are then adopted. The compilation and verification steps are completed with the help of an inference framework. If the performance does not meet expectations, the optimization strategy needs to be adjusted, the model structure needs to be modified, and the test needs to be repeated, forming a multi-round trial and error cycle.

[0004] This deployment method relies on manual knowledge of model semantics and hardware characteristics, resulting in inefficient and costly optimization processes. The optimization results are difficult to reuse in model updates or multi-platform heterogeneous deployment scenarios, leading to low deployment efficiency and scalability on CNN network accelerators. Summary of the Invention

[0005] This application provides a large language model operator transformation system, method, and AI accelerator to address the issues of low deployment efficiency and scalability on CNN network accelerators.

[0006] In a first aspect, this application provides a large language model operator conversion system, comprising: The operator identification module is used to parse the computation graph structure of the input model to identify convertible operators in the computation graph structure that meet preset conversion conditions. A conversion module is used to convert the convertible operator into an equivalent operator sequence based on a preset conversion strategy; The model reconstruction module is used to replace the nodes corresponding to the convertible operators in the computation graph structure of the input model with the equivalent operator sequence, update the connection relationships and weight initializers, and generate an updated model.

[0007] By automatically identifying convertible operators in the computation graph structure and converting them into equivalent operator sequences, the system can automatically generate hardware-adapted models from the original model, thereby improving deployment efficiency.

[0008] In some feasible embodiments, a verification module is also included, which is used to perform forward inference on the updated model to obtain a transformation output; Perform forward inference on the original model to obtain the original output; Compare the numerical error between the converted output and the original output; When the numerical error is less than or equal to a preset threshold, the target model is obtained.

[0009] By comparing the numerical error between the transformed output and the original output through forward inference, the target model is confirmed when the error does not exceed the threshold, and the numerical consistency of the transformation process is automatically verified.

[0010] In some feasible embodiments, the operator identification module identifies convertible operators in the computation graph structure that satisfy preset transformation conditions, including: Traverse the nodes in the computation graph structure; If the operation type of the node matches the preset operator type, extract the shape of the associated weight tensor; The shape of the weight tensor is matched with a preset shape pattern to generate a shape matching result; If the shape matching result is a successful match, mark the current node as a convertible operator.

[0011] Transformable operators are identified by traversing nodes and matching operation types with weight tensor shapes, enabling automated operator filtering based on preset shape patterns.

[0012] In some feasible embodiments, the conversion module performs a conversion based on a preset conversion strategy to convert the convertible operator into an equivalent operator sequence, including: Obtain the input tensor and the weight tensor, wherein the input tensor is the dynamic data flowing to the convertible operator, and the weight tensor is the model parameters associated with the convertible operator; Perform a dimensionality transformation on the input tensor to reshape the input stream as an output, the dimensionality transformation including changing the dimensional arrangement; Element rearrangement is performed on the weight tensor to output rearranged convolution kernel parameters, wherein the element rearrangement includes adjusting the distribution position of the elements; The convolution operator optimized for the target hardware platform is invoked to perform convolution operations on the reshaped input stream and rearranged convolution kernel parameters to generate an equivalent operator sequence, which includes the convolution operator and connection relationships.

[0013] By performing dimensionality transformation on the input tensor, rearranging the elements of the weight tensor, and invoking the convolution operator, a mathematical equivalence transformation is achieved to map the convertible operator to a hardware-optimized equivalent sequence.

[0014] In some feasible embodiments, the preset conversion conditions are defined by a rule base, which includes at least the identification and conversion rules corresponding to matrix multiplication operators, biased matrix multiplication operators, and parameterized activation function operators; The operator identification module parses the computation graph structure of the input model to identify convertible operators in the computation graph structure that satisfy preset transformation conditions, including: If the node operation type is matrix multiplication and the associated weight tensor has a first preset shape, identify the matrix multiplication operator; If the node operation type is biased matrix multiplication and there exists a non-empty biased tensor, identify the biased matrix multiplication operator. If the node operation type is a parameterized activation function and has a learnable parameter tensor, identify the parameterized activation function operator.

[0015] The system defines recognition and conversion rules for various operator types through a rule base, supporting unified processing of matrix multiplication operators, biased matrix multiplication operators, and parameterized activation function operators, thereby enhancing system scalability.

[0016] In some feasible embodiments, the model reconstruction module performs the replacement of the nodes corresponding to the convertible operators in the computational graph structure of the input model with the equivalent operator sequence, including: From the computation graph structure, delete the node corresponding to the convertible operator, and remove the weight initializer associated with the convertible operator; Insert new nodes into the equivalent operator sequence, and add weight initializers associated with the nodes corresponding to the transformable operators.

[0017] By deleting the original operator nodes and weight initializers and inserting new nodes and new weights, the safe execution of operator replacement in the computation graph structure is achieved, ensuring the integrity of the model topology.

[0018] In some feasible embodiments, the conversion module is configured to process multiple convertible operators in an iterative manner; For each convertible operator, the transformation module is used to perform: Create the first reshaping node to expand the dimensions of the input tensor to a target format, which is a format adapted to convolution operations; Create convolution nodes and configure convolution weight parameters to perform computations mathematically equivalent to the original operators; Create a second reshaping node to restore the output tensor dimension of the convolution node to the output shape of the original operator; The model reconstruction module iteratively updates the computation graph structure until all identified convertible operators have been converted.

[0019] By iteratively creating remodeling nodes and convolutional nodes and updating the computation graph structure, batch processing of all convertible operators is achieved, thus completing the automated transformation of the entire model.

[0020] Secondly, this application provides a method for large language model operator transformation, including: The computation graph structure of the input model is analyzed by the operator recognition module to identify the convertible operators in the computation graph structure that meet the preset transformation conditions. The conversion module converts the convertible operator into an equivalent operator sequence based on a preset conversion strategy. The model reconstruction module replaces the nodes corresponding to the convertible operators in the computation graph structure of the input model with the equivalent operator sequence, updates the connection relationships and weight initializers, and generates an updated model.

[0021] Thirdly, this application provides an AI accelerator, comprising: Hardware computing units are used to perform operator calculations in neural network models; The driving layer includes the large language model operator transformation system described in any of the first aspects; The large language model operator conversion system performs operator conversion on the input model during the model deployment phase, converting operators that are not suitable for execution by the hardware computing unit into equivalent operator sequences optimized by the hardware computing unit. The driver layer updates the model configuration and executes it on the hardware computing unit.

[0022] As can be seen from the above technical solutions, this application provides a large language model operator conversion system, method, and AI accelerator. The system includes: an operator recognition module, a conversion module, and a model reconstruction module. The operator recognition module is used to parse the computation graph structure of the input model to identify convertible operators in the computation graph structure that satisfy preset conversion conditions. The conversion module is used to convert the convertible operators into equivalent operator sequences based on a preset conversion strategy. The model reconstruction module is used to replace the nodes corresponding to the convertible operators in the computation graph structure of the input model with the equivalent operator sequences, and update the connection relationships and weight initializers to generate an updated model. The system realizes pipelined processing of model optimization through recognition rules and conversion strategies. Adaptation to different hardware platforms can be achieved by updating preset conversion conditions and conversion strategies, thereby solving the problems of low deployment efficiency and scalability on CNN network accelerators. Attached Figure Description

[0023] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A schematic diagram comparing the present application's embodiments with the prior art; Figure 2 This is a schematic diagram of the structure of the large language model operator conversion system provided in the embodiments of this application; Figure 3 This is a flowchart illustrating the large language model operator conversion method provided in an embodiment of this application. Detailed Implementation

[0025] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following examples do not represent all embodiments consistent with this application.

[0026] This application applies to the deployment of deep learning model inference, and is suitable for the efficient deployment of large language models based on the Transformer architecture on traditional CNN network accelerators. In modern deep learning models such as Vision Transformer (ViT), BERT, and GPT, the matrix multiplication (MatMul) operator is a core computational unit, and its execution time accounts for a significant proportion of the total model inference time, urgently requiring efficient execution on various AI inference hardware platforms. However, traditional operator deployment methods, when faced with operators not optimized for specific hardware, typically revert to execution on general-purpose processors such as CPUs, rather than utilizing the efficient computational units in dedicated AI accelerators (such as convolution acceleration engines). This strategy not only leads to idle dedicated hardware resources and low utilization of computational resources, but also easily creates performance bottlenecks, making it difficult to meet the requirements of cross-platform, high-performance inference deployment.

[0027] This application provides an operator transformation framework designed to address these challenges by reconstructing operators that are difficult to execute efficiently into equivalent operator forms that have been deeply optimized in hardware, thereby fully utilizing the high-performance computing units in dedicated AI accelerators to achieve efficient inference deployment of models on heterogeneous hardware platforms.

[0028] like Figure 1As shown, traditional operator transformation methods typically require extensive manual operator optimization when facing new hardware platforms, resulting in deployment cycles that can last for days or even weeks. Furthermore, this process heavily relies on engineers' experience, requiring repeated testing and manual tuning to approximate performance targets. The lack of automation and standardization mechanisms severely restricts deployment efficiency and cross-platform compatibility. Early optimization solutions attempted to use general-purpose operator libraries (such as BLAS) or simple replacement strategies, but these often fail to activate the core acceleration units (such as convolutional computation engines) on dedicated AI accelerators. Instead, they rely on general-purpose computing resources to execute key operators, leading to idle hardware resources, low computational efficiency, and limited performance improvements.

[0029] This application provides a large language model operator conversion system in some embodiments, such as... Figure 2 As shown, it includes an operator recognition module, a conversion module, and a model reconstruction module. Through the collaborative work of these three modules, the automatic conversion from the original model to the hardware-optimized model is achieved.

[0030] The operator recognition module is used to parse the computational graph structure of the input model, which is the data representation of the deep learning model to be deployed. The input model includes the model's computational logic and parameter information, and its format can be the Open Neural Network Exchange format or a model serialization format defined by other frameworks.

[0031] The computation graph structure is a graphical representation of the computational logic within the input model. The computation graph structure consists of nodes and edges. Nodes represent computational operators, and edges represent the flow paths of data tensors. This structured representation allows the system to access and process each computational unit in the model through graph traversal and analysis techniques.

[0032] In this embodiment, the computation graph structure is based on the ONNX model graph structure, which is an open neural network exchange format computation graph data structure. This is a standardized, cross-platform intermediate representation format for deep learning models.

[0033] The system then identifies convertible operators in the computation graph structure that meet preset transformation conditions. Convertible operators are computation nodes in the computation graph structure that have optimization potential. The identification of convertible operators depends on preset transformation conditions, which can be based on operator type, weight shape features, or data flow pattern.

[0034] In large language models, the matrix multiplication operator corresponding to the fully connected layer is often identified as a convertible operator because its computational characteristics are suitable for conversion into a hardware-optimized form.

[0035] In this embodiment, the preset transformation conditions are a set of rules for determining whether an operator is transformable. These preset transformation conditions are stored in the system's rule base and may include operator type matching rules, weight shape verification rules, and data structure constraint rules.

[0036] The operator identification module loads the model file and constructs an internal graph representation. It traverses each node in the computation graph structure and evaluates the conversion feasibility of each node according to preset conversion conditions. The evaluation process comprehensively considers factors such as node type, weight characteristics and computation context. Only nodes that meet all conditions will be marked as convertible operators.

[0037] It is understandable that there can be one or more convertible operators. When there are multiple convertible operators, the convertible operators at this moment are a set of convertible operators.

[0038] For example, by traversing and analyzing the nodes in the ONNX model graph, a mechanism can be constructed to identify the type of associated operator, the shape of the weights, and the feasibility of the transformation. For instance, it can identify MatMul operators that meet the transformation conditions, such as 768×3072 or 3072×768 linear layer configurations in the Transformer, and verify the existence and accessibility of the weight initializer, providing input guarantees for subsequent operator transformations.

[0039] After the operator recognition module completes its processing, the conversion module, based on a preset conversion strategy, transforms the convertible operators into equivalent operator sequences. These equivalent operator sequences are computational subgraphs composed of one or more hardware-native optimized operators. While mathematically and semantically equivalent to the original convertible operators, the equivalent operator sequences offer significantly improved execution efficiency.

[0040] The conversion module receives the list of convertible operators output by the operator recognition module and generates a corresponding equivalent operator sequence for each convertible operator. The generation process is based on a preset conversion strategy, which defines the mapping rules from the source operator to the target operator sequence. The conversion strategy maintains mathematical equivalence, ensuring that the converted calculation process remains consistent with the original calculation within the same numerical precision range. The conversion module executes the specific operator rewriting logic, including parameter conversion, shape derivation, and attribute mapping operations.

[0041] After generating the equivalent operator sequence, the equivalent operator sequence is sent to the model reconstruction module. The model reconstruction module replaces the nodes corresponding to the convertible operators in the computation graph of the input model with the equivalent operator sequence. The model reconstruction module first locates the position of the convertible operators in the original graph, and then performs graph editing operations to remove the original nodes and insert the newly generated equivalent operator sequence. During the editing process, the model reconstruction module needs to handle the update of node connection relationships to ensure that the data flow correctly flows from the upstream nodes to the newly inserted operator sequence, and from the sequence output to subsequent nodes.

[0042] Update the connectivity relationships and weight initializers to generate the updated model. The connectivity relationships describe the data flow and dependencies between nodes in the computation graph. The connectivity relationships are reflected through the reference relationships of tensor names. During the model reconstruction process, it is necessary to maintain the correctness and consistency of these reference relationships. When replacing operators, the references of upstream and downstream nodes to these connectivity relationships must be updated.

[0043] A weight initializer is a data container in the model that stores persistent parameters. Weight initializers are associated with specific computation nodes and exist in the model graph as initializers. During operator transformations, these parameters may need to be extracted, transformed, and re-stored as new weight initializers.

[0044] The model reconstruction module manages the lifecycle of the weight initializer, removes unwanted original weights, adds newly generated computational parameters, and finally generates an updated model. The updated model is a new model generated after operator transformation and graph structure reconstruction. The updated model maintains the interface and computational semantics of the original model, but some internal computational paths have been optimized. The updated model can be directly deployed to the target hardware platform and achieve performance improvement.

[0045] For example, when processing a large language model based on the Transformer architecture, the operator recognition module identifies the matrix multiplication operator corresponding to the linear layer in the model computation graph. The weight shape of this operator conforms to the fully connected layer pattern in the preset transformation conditions. The transformation module generates an equivalent operator sequence consisting of input reshaping nodes, one-to-one convolution nodes, and output reshaping nodes, according to the transformation rules from matrix multiplication to convolution. The model reconstruction module then replaces the original matrix multiplication nodes with this sequence and updates the references of adjacent layers to the new output tensor.

[0046] Traditional operator optimization relies on expert experience, requiring manual analysis of model structure, design of replacement strategies, and verification of results—a time-consuming and error-prone process. This application constructs a pipeline encompassing identification, transformation, and reconstruction to achieve end-to-end automatic conversion from the original model to a hardware-adapted model, reducing deployment barriers and labor costs. Furthermore, existing simple operator replacement methods often neglect mathematical equivalence, easily introducing accuracy loss or behavioral bias. This application is based on the principle of mathematical equivalence transformation and introduces a forward inference comparison mechanism after conversion to ensure consistent output results within the floating-point precision range, meeting the reliability requirements of industrial-grade deployments.

[0047] In some embodiments, the system further includes a verification module, which is used to confirm the correctness of the transformation. The verification module receives the updated model output by the model reconstruction module and verifies whether the transformation process maintains numerical consistency by performing calculation and comparison operations.

[0048] The validation module is used to perform forward inference on the updated model to obtain the transformed output; Perform forward inference on the original model to obtain the original output; Compare the numerical error between the converted output and the original output; The target model is obtained when the numerical error is less than or equal to a preset threshold.

[0049] The test input is fed into both the updated model and the original model, triggering forward inference in both models. Forward inference is the process by which the model performs computations sequentially according to the computation graph. It starts from the input layer, proceeds through the intermediate layers, and finally yields the output. During validation, forward inference is used to activate the complete computational path of the model and obtain the actual output data.

[0050] During forward inference, the verification module monitors the correct execution of the computation process, ensuring that both models complete the computation in the same environment. After forward inference is complete, the verification module collects the transformed output generated by the updated model and the original output generated by the original model. The transformed output and / or the original output can be a single tensor or a collection of multiple output tensors.

[0051] The verification module pairs the transformed output with the original output according to the output name or index to ensure that the output results at the corresponding positions are compared. For each output pair, the numerical error is calculated, which reflects the degree of change in the model behavior before and after the transformation.

[0052] Numerical error is calculated using a standard difference metric, which includes calculating the absolute or relative differences between corresponding elements of the output tensors, statistically aggregating these differences, and using the aggregation result as a scalar value representing the overall error level of the output pair. The verification module compares the calculated numerical error with a preset threshold to determine whether the error is within an acceptable range.

[0053] When the numerical error of all output pairs does not exceed a preset threshold, the verification module confirms the transformation is successful. At this point, the updated model is marked as the target model and prepared for subsequent deployment. If the numerical error of any output pair exceeds the preset threshold, the verification module determines that there is a problem with the transformation and needs to check the transformation process.

[0054] The target model obtained is a validated updated model that has both the performance optimization brought about by the transformation and maintains the numerical behavior consistent with the original model. The target model is the deployable model that the system finally outputs.

[0055] For example, the system transforms a visual Transformer model that involves multiple layers of linear computation. The validation module uses a batch of typical images as test inputs, inputting them into the original model and the updated model respectively. During forward inference, both models produce classification probability outputs. The validation module calculates the cosine similarity between the two probability vectors as a numerical error, which is far below a preset threshold. The validation module thus confirms that the transformation is correct and outputs the validated target model.

[0056] The verification module confirms whether the transformation process maintains the behavioral characteristics of the model through forward inference and numerical comparison. This automatic verification replaces the traditional manual verification method, improves verification efficiency, and eliminates inconsistencies caused by subjective judgment.

[0057] Before the conversion process, convertible operators that meet preset conversion conditions are identified. Specifically, in some embodiments, the operator identification module identifies convertible operators in the computation graph structure that meet the preset conversion conditions, including: Traverse the nodes in the computation graph structure; If the operation type of a node matches the preset operator type, extract the shape of the associated weight tensor; Match the shape of the weight tensor with a preset shape pattern to generate a shape matching result; If the shape matching result is successful, mark the current node as a convertible operator.

[0058] Nodes are the basic processing units in a computation graph structure. Nodes include operation type attributes and input / output connection information. In implementation, nodes can represent operation types such as convolution, matrix multiplication, or activation functions. As the processing object of the operator recognition module, the attributes of a node determine whether it meets the conversion conditions. The operation type is the category identifier of the computation operation represented by the node. The operation type can be stored in the node attributes as a string or an enumeration value, for example, including matrix multiplication operators, convolution operators, or pooling operators.

[0059] The system traverses each node in the computation graph structure in topological order. During the traversal, each node undergoes the same analysis process, including operation type checking and weight shape verification.

[0060] For each accessed node, the operator identification module first checks its operation type attribute. The module matches the node operation type with a preset set of operator types. Only nodes whose operation types appear in the preset set will enter the subsequent processing stage. It can quickly filter out unsupported operator types, such as activation functions or normalization layers, and other nodes that do not need to be converted. The operation type matching can be implemented using precise string comparison or pattern matching algorithms.

[0061] After a node is filtered by operation type, the operator identification module extracts the shape of the associated weight tensor. The shape of the weight tensor is the dimensional information of the parameter data associated with the node. The shape of the weight tensor is obtained by accessing the model initializer and is represented as an integer tuple.

[0062] The system queries the weight inputs in the node input list, searches for the corresponding tensor data in the model initializer, obtains the weight data, analyzes the dimensional information and records the shape features. For dynamic shapes, the actual tensor dimensions can be derived using a type inference tool.

[0063] The actual weight shape is compared with the preset shape pattern, taking into account changes and transpose of batch dimensions. The matching process supports wildcard mode, allowing specific dimensions to match any value. The shape matching engine generates shape matching results, indicating whether the current node meets the shape conditions.

[0064] Nodes with successful shape matching are marked as convertible operators. Marking operations may include setting node attributes and adding the node to the processing queue. The operator identification module continues to traverse the remaining nodes until the entire computational graph structure has been processed. Finally, the set of all marked convertible operators is the output of this module. By automating node traversal and condition matching, the efficiency problem of manually identifying target operators is solved.

[0065] For operators that meet preset transformation conditions, in some embodiments, these preset transformation conditions are defined through a rule base. The rule base is a centralized knowledge base storing transformation rules, including complete definitions of identification and transformation rules. The rule base adopts a modular design, supporting different processing strategies for different types of operators, providing the system with scalable rule management capabilities. For example, the rule base supports multiple transformation modes such as MatMul to Conv, Gemm to Conv, and PRelu to Relu. In this embodiment, the rule base includes at least the identification and transformation rules corresponding to matrix multiplication operators, biased matrix multiplication operators, and parameterized activation function operators.

[0066] The operator identification module parses the computational graph structure of the input model to identify convertible operators in the computational graph structure that satisfy preset transformation conditions, including: If the node operation type is matrix multiplication and the associated weight tensor has a first preset shape, identify the matrix multiplication operator; If the node operation type is biased matrix multiplication and there exists a non-empty biased tensor, identify the biased matrix multiplication operator. If the node operation type is a parameterized activation function and has a learnable parameter tensor, identify the parameterized activation function operator.

[0067] Matrix multiplication operators are fundamental linear operational units in deep learning, performing matrix multiplication on two two-dimensional tensors. In typical implementations, matrix multiplication operators possess specific operation type identifiers and weight tensor shape characteristics, making them a key operator type supported by rule bases.

[0068] For the identification of matrix multiplication operators, the corresponding identification rules in the rule base are applied. The rule engine checks whether the node operation type is matrix multiplication and verifies whether the associated weight tensor has a preset two-dimensional shape feature. When a node satisfies both the operation type condition and the shape condition, the rule engine identifies it as a matrix multiplication operator. During the identification process, the rule engine considers the batch dimensionality changes in actual deployment to ensure the accuracy of the identification results.

[0069] The biased matrix multiplication operator is an extended linear layer that includes a bias term. It adds a biased addition operation to the basic matrix multiplication. This operator is identified by the node operation type and the existence of the bias tensor, and corresponding processing rules exist in a rule base.

[0070] Identifying biased matrix multiplication operators requires the application of more complex rule logic. The rule engine first checks whether the node operation type matches the characteristics of biased matrix multiplication, and then verifies the existence of a non-empty bias tensor. The rule engine analyzes the data flow relationships in the computation graph to confirm the correct connection between biased addition operations and matrix multiplication. Only nodes that simultaneously satisfy the operation type condition and the bias existence condition are identified as target operators.

[0071] Parametric activation function operators are activation units with learnable parameters. These include nonlinear functions with parameters such as PReLU. These operators are identified by their operation type and the tensor of learnable parameters, and specific transformation rules are configured in a rule base.

[0072] The identification of parameterized activation function operators relies on specialized rule configuration. The rule engine checks whether the node operation type belongs to a preset set of parameterized activation function types, and also verifies whether the node is associated with a learnable parameter tensor. For operators such as PReLU, the rule engine further analyzes the shape features of the parameter tensor to ensure that the parameter structure matches the operator semantics. The identification process comprehensively considers both operation type and parameter characteristics to achieve accurate operator classification.

[0073] The rule base enables the system to uniformly handle recognition tasks for multiple operator types. Recognition rules for different operator types exist in parallel within the rule base and are coordinated and scheduled by the rule engine. The rule engine applies each recognition rule according to its priority, avoiding rule conflicts and duplicate recognition. This unified rule processing mechanism ensures the consistency and integrity of the recognition process.

[0074] For example, in a complex model involving multiple operator types, a rule base coordinates the identification of matrix multiplication operators, biased matrix multiplication operators, and parameterized activation function operators. The rule engine sequentially applies various identification rules, scanning each node in the computation graph. Nodes conforming to the matrix multiplication rule are marked as first-class transformable operators; nodes conforming to the biased matrix multiplication rule are marked as second-class transformable operators; and nodes conforming to the parameterized activation function rule are marked as third-class transformable operators. The entire process is completed under the unified scheduling of the rule base, ensuring the accuracy and consistency of the identification results.

[0075] In some embodiments, after obtaining the set of convertible operators, specifically, the conversion module performs a conversion based on a preset conversion strategy to convert the convertible operators into an equivalent operator sequence, including: Obtain the input tensor and the weight tensor; Perform a dimensionality transformation on the input tensor to reshape the input stream with the output. The dimensionality transformation includes changing the arrangement of dimensions. Perform element rearrangement on the weight tensor to output rearranged convolution kernel parameters. Element rearrangement includes adjusting the distribution of elements. The target hardware platform-optimized convolution operator is invoked to reshape the input stream and rearrange the convolution kernel parameters, and then perform convolution operations to generate an equivalent operator sequence, which includes convolution operators and connection relationships.

[0076] The input tensor is the dynamic data flowing to the convertible operator, and its dimension is related to the batch size and feature dimension. The weight tensor is the model parameters associated with the convertible operator. The weight tensor is stored in the model initializer, including the knowledge representation obtained from network training.

[0077] In this embodiment, the preset conversion strategy is the mathematical equivalence transformation principle. This principle ensures that the mathematical operations remain completely equivalent while changing the form of computational representation. The mathematical equivalence transformation principle is established through rigorous mathematical proof, which guarantees that the computational process before and after the transformation produces identical output results within the floating-point precision range. This principle enables the system to optimize the computational structure while maintaining the accuracy of the model.

[0078] By introducing a dual reshape transformation strategy as a mechanism to equivalently map the MatMul operator to a convolution (Conv) operator, this strategy achieves a mathematically equivalent transformation from matrix multiplication to convolution by employing dimensionality expansion at the input end (2D to 4D tensor transformation) and dimensionality restoration at the output end (4D to 2D tensor restoration), combined with weight transpose. This includes maintaining consistency between computational semantics and floating-point precision. The mathematically equivalent transformation principles upon which the dual reshape strategy relies can be applied analogously to other operators with fixed weight structures and linear semantics, providing a technical foundation for subsequent extensions.

[0079] Dimensional transformation is a concrete manifestation of the mathematical principle of equivalent transformation in data organization. It changes the dimensional arrangement of data through tensor reshaping operations without altering its mathematical essence. In the transformation from matrix multiplication to convolution, the input tensor is reorganized from a two-dimensional structure into a four-dimensional structure. This reshaping only changes the data view without altering the mathematical relationships between elements. The reshaped data retains the same element values ​​and relative positions, ensuring that subsequent calculations are mathematically identical to the original calculations.

[0080] Element rearrangement adapts to different computational paradigms by adjusting the distribution of elements in the weight tensor. In the process of converting matrix multiplication to convolution, the transpose of the weight matrix and the reshaping of the convolution kernel together ensure the equivalence of computational semantics. This rearrangement is based on the matrix operation rules in linear algebra, ensuring that the convolution operation and matrix multiplication correspond completely in mathematical expression.

[0081] The verification of the principle of mathematical equivalence transformation is achieved through numerical consistency checks. The system compares the output results of the models before and after the transformation through forward inference to confirm that the transformation process does not introduce mathematical bias. This verification mechanism provides an empirical basis for the practical application of the principle of mathematical equivalence transformation.

[0082] For example, the dimension transformation operation unfolds the input tensor. The transformation module sets a dimension recombination scheme according to the input requirements of the target convolution operator. For example, it may recombine the three-dimensional tensor into a four-dimensional form, add height and width dimensions and set them to the size of a single element. The dimension transformation is achieved by creating a reshaping node, which receives the original input tensor and outputs the data stream after the structure transformation.

[0083] Element rearrangement is performed on the weight tensor. The transformation module analyzes the dimensionality of the original weight tensor and rearranges the elements according to the convolution kernel parameter format. For weights in matrix multiplication operators, rearrangement typically involves matrix transpose and dimension splitting operations. Element rearrangement generates new parameter data conforming to the convolution kernel format. These parameters retain the same values ​​as the original weights but have a different organizational structure.

[0084] After preprocessing the input and weights, the transformation module calls the convolution operator optimized for the target hardware platform, creates a convolution computation node, and configures the correct convolution parameters, including stride, number of padding groups, and kernel size. The convolution node receives the reshaped input stream and rearranged kernel parameters, and performs the actual mathematical calculations. This computation stage makes full use of hardware optimization features to achieve an efficient computation process.

[0085] The target hardware platform can be an AI accelerator. By leveraging the deep optimization features of dedicated AI accelerators for convolution operators, the transformed operators can directly match the native optimization path of the target AI accelerator for convolution operations. This fully utilizes its dedicated computing units, achieving efficient inference acceleration while maintaining model semantic consistency, and supports cross-hardware platform deployment, significantly reducing reliance on manual optimization.

[0086] Finally, the transformation module generates a complete sequence of equivalent operators, which includes combinations of input reshaping nodes, convolution computation nodes, and output reshaping nodes, forming a computational subgraph that can replace the original operators. The sequence of equivalent operators maintains the same interface specifications as the original operators, ensuring seamless integration into the original computational graph.

[0087] For example, the input tensor has a three-dimensional structure consisting of batch size multiplied by sequence length multiplied by feature dimension. The transformation module first creates a reshaping node to convert the input tensor into a four-dimensional form, adding a single element spatial dimension. The weight tensor undergoes transpose and reshaping operations, transforming it from a matrix of input dimension multiplied by output dimension into a convolution kernel format of output channel number multiplied by input channel number multiplied by 1 multiplied by 1. Then, a 1 multiplied by 1 convolution node is created, and the calculation is performed using the rearranged parameters. The final output equivalent operator sequence achieves the same computational function, but adopts a hardware-optimized execution path.

[0088] In some embodiments, the conversion module is configured to process multiple convertible operators in an iterative manner, that is, the system processes multiple targets sequentially, and the iterative manner processes the convertible operators one by one through a loop control structure. This processing mode can ensure that each convertible operator goes through the same conversion process and maintain the consistency of the processing results.

[0089] For each transformable operator, the transformation module is used to perform: Create the first reshaping node to expand the dimensions of the input tensor to the target format, which is a format suitable for convolution operations; Create convolution nodes and configure convolution weight parameters to perform computations mathematically equivalent to the original operators; Create a second reshaping node to restore the output tensor dimension of the convolution node to the output shape of the original operator; The model reconstruction module iteratively updates the computation graph structure until all identified convertible operators have been converted.

[0090] The system first performs a topological sort on the set of convertible operators. The sorting process is based on the data dependencies in the computation graph to ensure that operators without cyclic dependencies can be processed independently. The system establishes an iterative processing state machine to track the transformation progress and processing results of each operator.

[0091] For each convertible operator, the transformation module creates a first reshaping node, which is the first computational unit in the equivalent operator sequence. The first reshaping node is used to expand the dimensions of the input tensor to the target format. This node adapts the data format to the requirements of subsequent convolution operations by inserting new dimensions or adjusting the order of existing dimensions.

[0092] The transformation module configures the parameters of the convolution nodes based on the computational semantics of the original operators. For the transformation from matrix multiplication to convolution, a 1×1 convolution kernel size, unit stride, and zero padding are typically used. The convolution weight parameters are obtained through precise element rearrangement, ensuring that the convolution operation is mathematically equivalent to the original matrix multiplication. The convolution node receives the output of the first reshaping node and performs hardware-optimized computational operations.

[0093] The module analyzes the output tensor dimensions of the original operator and designs a corresponding dimension restoration scheme. The second reshaping node removes the extra dimensions introduced by the convolution operation, restoring the output data to the format expected by the original operator. In other words, the second reshaping node is used to restore the output tensor dimensions to their original shape. This node ensures that the output data maintains consistency with the output format of the original operator by removing or merging dimensions. During the dimension restoration process, the system verifies whether the output shape matches the input requirements of downstream nodes, ensuring that the connectivity of the computation graph is not affected.

[0094] After each transformation of a transformable operator is completed, the module performs graph editing operations to integrate the newly generated sequence of equivalent operators into the computation graph. Update operations include removing original nodes, inserting new sequences, and updating connectivity relationships. The iterative update mechanism ensures that the computation graph is always in a complete and usable state, supporting subsequent transformation operations.

[0095] An iterative control mechanism monitors the progress of the entire processing. The system maintains a complete record of processed and pending operators to ensure that each convertible operator is processed correctly. The iterative process terminates normally once all identified convertible operators have been transformed. The system outputs a complete transformation report, recording the transformation status of each operator and the final result.

[0096] After obtaining the equivalent operator sequence, in some embodiments, the nodes corresponding to the convertible operators in the computation graph of the input model are replaced with the equivalent operator sequence, including: From the computation graph structure, delete the node corresponding to the convertible operator, and remove the weight initializer associated with the convertible operator; Insert new nodes into the equivalent operator sequence, and add weight initializers associated with the nodes corresponding to the transformable operators.

[0097] The module traverses all nodes in the computation graph structure, identifies computational units that have been marked as convertible operators, and records its complete connection relationships for each target node, including all input sources and output targets.

[0098] The model reconstruction module removes the target node from the node list of the computation graph structure and simultaneously disconnects the node from all upstream and downstream connections. A safe graph editing algorithm can be used to maintain the topological consistency of the computation graph structure during the deletion process.

[0099] The removal of weight initializers is carried out simultaneously with node deletion. The model reconstruction module queries all weight initializers associated with the convertible operator and removes these parameter data from the model initializer list. The removal of weight initializers needs to check the reference count to ensure that only parameters that are no longer referenced by any node are cleared.

[0100] The model reconstruction module adds each node in the equivalent operator sequence to the computation graph structure sequentially, while establishing internal connections between nodes. Each new node needs to be configured with the correct operation attributes, including operation type, computation parameters, and dimension information. During the insertion process, the module generates a unique identifier name for the new node to avoid naming conflicts with existing nodes.

[0101] The addition of new weight initializers is coordinated with node insertion. The model reconstruction module encapsulates the transformed parameter data into new weight initializers and adds them to the model initializer set. Each new weight initializer needs to establish a correct reference relationship with the corresponding new node to ensure that the computing node can access the required parameter data.

[0102] The model reconstruction module connects the newly inserted equivalent operator sequence to the original computational graph structure, linking the sequence's input nodes to the upstream source of the original convertible operator and the sequence's output nodes to the downstream target of the original convertible operator. This connection update process requires precise matching of tensor shapes and data types to ensure that data can flow correctly between the old and new computational units.

[0103] To address the shortcomings of previous technologies, this application aims to provide an automated operator transformation framework for heterogeneous AI accelerators. This framework can reconstruct computational operators inefficiently executed within a model into hardware-optimized equivalent operator forms. By introducing a transformation mechanism based on tensor reshaping and mathematical equivalence transformations (e.g., double reshape and 1×1 convolution mapping), semantically preserved transformation from source to target operators is achieved, replacing the traditional manual optimization process and supporting rapid, automated model adaptation. This framework can significantly shorten deployment cycles, improve hardware resource utilization, and enhance portability across various AI acceleration platforms while ensuring floating-point precision and computational correctness. The framework employs a pipelined design, automating the entire process from inefficient operator identification and equivalent structure replacement to post-transformation verification, ensuring reliable transformation execution and overcoming the fundamental deficiencies of simple operator replacement in terms of equivalence guarantees and performance gains.

[0104] Based on the above-mentioned large language model operator transformation system, such as Figure 3 As shown, some embodiments of this application also provide a method for large language model operator transformation, including: The operator recognition module parses the computation graph structure of the input model to identify convertible operators in the computation graph structure that meet the preset transformation conditions. The conversion module converts convertible operators into equivalent operator sequences based on a preset conversion strategy. The model reconstruction module replaces the nodes corresponding to the convertible operators in the computation graph structure of the input model with equivalent operator sequences, updates the connection relationships and weight initializers, and generates an updated model.

[0105] For details on the specific steps of this system, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0106] Based on the aforementioned large language model operator conversion system, some embodiments of this application also provide an AI accelerator, including: Hardware computing units are used to perform operator calculations in neural network models; The driver layer includes a large language model operator conversion system; Among them, the large language model operator conversion system performs operator conversion on the input model during the model deployment phase, converting operators that are not suitable for execution by the hardware computing unit into equivalent operator sequences optimized by the hardware computing unit; The driver layer updates the model configuration and executes it on the hardware computing unit.

[0107] AI accelerators are hardware devices designed for neural network computing. They consist of highly parallel computing units and optimized memory subsystems. This hardware architecture is optimized for specific computing patterns and can significantly improve the inference speed of neural network models.

[0108] In this embodiment, the AI ​​accelerator is a traditional hardware accelerator specifically optimized for convolutional neural networks (CNNs). The hardware computing units of this type of accelerator (such as computing cores, data paths, and memory hierarchy) are designed and optimized for the regular data flow and high parallelism of convolution operations. They can take the form of NPUs, DSPs, or ASICs, but they all share the common feature of CNNs. Therefore, when executing operators such as matrix multiplication commonly found in large language models, they will encounter architectural mismatch problems.

[0109] Hardware computing units are physical modules in AI accelerators that perform specific computational operations. Each hardware computing unit consists of multiple processing cores and dedicated circuitry. These computing units are optimized for common neural network operations such as convolution and matrix operations, achieving high-performance computing through parallel processing and data reuse mechanisms.

[0110] The driver layer is a software interface layer that sits above the hardware computing units. It manages hardware resources and schedules computing tasks. The driver layer provides standardized application programming interfaces (APIs), enabling upper-layer software to use accelerator functions without needing to understand the hardware details.

[0111] The large language model operator conversion system is a software component integrated into the driver layer. It includes an operator recognition module, a conversion module, and a model reconstruction module. During the model deployment phase, this system analyzes and converts the input model, enabling it to fully utilize the characteristics of the hardware computing units.

[0112] The AI ​​accelerator's driver layer loads the large language model operator conversion system during system initialization. The driver layer detects the available resources and characteristics of the hardware computing units and passes this information to the conversion system for optimization decisions. The conversion system configures corresponding conversion rules based on the hardware characteristics to ensure that the generated equivalent operator sequence can fully utilize the parallel processing capabilities of the hardware computing units.

[0113] When a new input model needs to be deployed, the driver layer activates the large language model operator transformation system. The system first analyzes the computational graph structure of the input model, identifying operation nodes that are unsuitable for direct execution by hardware computing units. This identification process is based on a list of supported capabilities of the hardware computing units, focusing on operator types that, while semantically clear, are inefficient to execute.

[0114] For identified unsuitable operators, the conversion system initiates the conversion process. Based on the architectural characteristics of the hardware computing unit, the system selects the most suitable equivalent operator sequence generation scheme. The conversion process considers the memory layout preferences, data parallelism granularity, and computational pipeline characteristics of the hardware computing unit to ensure that the generated sequence can be seamlessly integrated into the hardware execution environment.

[0115] After the transformation is complete, the driver layer receives the updated model and configures hardware resources. The driver layer analyzes the computational requirements of the updated model and allocates processing cores and memory space within the hardware computing units. Resource configuration optimization considers computational dependencies and data locality, minimizing data transfer overhead and maximizing computing unit utilization.

[0116] The driver layer loads the configured update model into the hardware computing unit for execution. It manages the scheduling and execution of computing tasks and monitors the working status of the hardware computing unit. During execution, the driver layer handles the transmission of input data, the distribution of computing tasks, and the collection of output results, ensuring the entire inference process runs efficiently and stably.

[0117] For example, the input model is a large language model based on the Transformer architecture to be deployed. The input model includes a large number of matrix multiplication operators and other non-linear operations. The computation mode of these operators is significantly different from the regular convolution computation optimized by the hardware computing unit of CNN accelerator, and cannot be directly and efficiently executed.

[0118] An equivalent operator sequence is a subgraph generated by the transformation system and composed of operators natively optimized by the hardware computing unit. The equivalent operator sequence is mathematically equivalent to the original operator, but the computational form has been transformed into a convolution operation that the hardware computing unit is good at. For example, a matrix multiplication operator is transformed into a sequence of a reshaping operator, a one-to-one convolution operator, and another reshaping operator.

[0119] The updated model is a model composed of CNN accelerator-friendly operators after being processed by the transformation system. All computational nodes in the updated model can be executed by the hardware computing unit. The updated model is the output of the transformation system and is also the form that the driver layer can directly load and configure for execution by the hardware computing unit.

[0120] The driver layer finally obtains an updated model output by the transformation system. At this point, the driver layer's work becomes efficient; it only needs to map the convolutional operators in the updated model to a large number of parallel multiply-accumulators in the hardware computing unit for execution, just like processing an optimized CNN model. The hardware computing unit then completes the complex inference tasks of the large language model.

[0121] This embodiment solves the architectural mismatch between large language models and such hardware by integrating an automated operator transformation system into the driver layer of a CNN accelerator. This enables a large number of traditional CNN accelerators that have been deployed and are designed for vision tasks to efficiently run cutting-edge large language models, thus expanding their application scenarios and lifecycle.

[0122] By leveraging built-in mathematical equivalence-based transformation rules within the driver layer, performance is improved while maintaining the model's computational accuracy. The transformed updated model, given the original model, produces numerically consistent outputs, meeting the reliability requirements of industrial applications and enabling the automated transformation results to be directly used in production environments.

[0123] This embodiment creatively translates the computational features in a large language model into a language that a CNN accelerator can understand, thereby fully releasing the computing power in the hardware computing unit that was originally idle due to insufficient operator support, and avoiding the cost of replacing hardware to support new models.

[0124] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. A large language model operator conversion system, characterized in that, include: The operator identification module is used to parse the computation graph structure of the input model to identify convertible operators in the computation graph structure that meet preset conversion conditions. A conversion module is used to convert the convertible operator into an equivalent operator sequence based on a preset conversion strategy; The model reconstruction module is used to replace the nodes corresponding to the convertible operators in the computation graph structure of the input model with the equivalent operator sequence, update the connection relationships and weight initializers, and generate an updated model.

2. The large language model operator conversion system according to claim 1, characterized in that, It also includes a verification module, which performs forward inference on the updated model to obtain a transformation output; Perform forward inference on the original model to obtain the original output; Compare the numerical error between the converted output and the original output; When the numerical error is less than or equal to a preset threshold, the target model is obtained.

3. The large language model operator conversion system according to claim 1, characterized in that, The operator identification module identifies convertible operators in the computation graph structure that meet preset transformation conditions, including: Traverse the nodes in the computation graph structure; If the operation type of the node matches the preset operator type, extract the shape of the associated weight tensor; The shape of the weight tensor is matched with a preset shape pattern to generate a shape matching result; If the shape matching result is a successful match, mark the current node as a convertible operator.

4. The large language model operator conversion system according to claim 1, characterized in that, The conversion module performs a conversion based on a preset conversion strategy, converting the convertible operator into an equivalent operator sequence, including: Obtain the input tensor and the weight tensor, wherein the input tensor is the dynamic data flowing to the convertible operator, and the weight tensor is the model parameters associated with the convertible operator; Perform a dimensionality transformation on the input tensor to reshape the input stream as an output, the dimensionality transformation including changing the dimensional arrangement; Element rearrangement is performed on the weight tensor to output rearranged convolution kernel parameters, wherein the element rearrangement includes adjusting the distribution position of the elements; The convolution operator optimized for the target hardware platform is invoked to perform convolution operations on the reshaped input stream and rearranged convolution kernel parameters to generate an equivalent operator sequence, which includes the convolution operator and connection relationships.

5. The large language model operator conversion system according to claim 1, characterized in that, The preset conversion conditions are defined by a rule base, which includes at least the recognition and conversion rules corresponding to matrix multiplication operators, biased matrix multiplication operators, and parameterized activation function operators. The operator identification module parses the computation graph structure of the input model to identify convertible operators in the computation graph structure that satisfy preset transformation conditions, including: If the node operation type is matrix multiplication and the associated weight tensor has a first preset shape, identify the matrix multiplication operator; If the node operation type is biased matrix multiplication and there exists a non-empty biased tensor, identify the biased matrix multiplication operator. If the node operation type is a parameterized activation function and has a learnable parameter tensor, identify the parameterized activation function operator.

6. The large language model operator conversion system according to claim 1, characterized in that, The model reconstruction module performs the following steps: replacing the nodes corresponding to the convertible operators in the computational graph structure of the input model with the equivalent operator sequence, including: From the computation graph structure, delete the node corresponding to the convertible operator, and remove the weight initializer associated with the convertible operator; Insert new nodes into the equivalent operator sequence, and add weight initializers associated with the nodes corresponding to the transformable operators.

7. The large language model operator conversion system according to claim 1, characterized in that, The conversion module is configured to process multiple convertible operators in an iterative manner; For each convertible operator, the transformation module is used to perform: Create the first reshaping node to expand the dimensions of the input tensor to a target format, which is a format adapted to convolution operations; Create convolution nodes and configure convolution weight parameters to perform computations mathematically equivalent to the original operators; Create a second reshaping node to restore the output tensor dimension of the convolution node to the output shape of the original operator; The model reconstruction module iteratively updates the computation graph structure until all identified convertible operators have been converted.

8. A method for operator transformation of a large language model, characterized in that, include: The computation graph structure of the input model is analyzed by the operator recognition module to identify the convertible operators in the computation graph structure that meet the preset transformation conditions. The conversion module converts the convertible operator into an equivalent operator sequence based on a preset conversion strategy. The model reconstruction module replaces the nodes corresponding to the convertible operators in the computation graph structure of the input model with the equivalent operator sequence, updates the connection relationships and weight initializers, and generates an updated model.

9. An AI accelerator, characterized in that, include: Hardware computing units are used to perform operator calculations in neural network models; The driving layer includes the large language model operator conversion system as described in any one of claims 1-7; The large language model operator conversion system performs operator conversion on the input model during the model deployment phase, converting operators that are not suitable for execution by the hardware computing unit into equivalent operator sequences optimized by the hardware computing unit. The driver layer updates the model configuration and executes it on the hardware computing unit.

Images