Operator fusion method, system, device, and medium

Abstract

The application provides an operator fusion method, system, device and medium, comprising obtaining acceleration chip configuration information of a current hardware platform and operator type information contained in a calculation graph; inputting the obtained acceleration chip configuration information, operator type information and calculation graph to a trained operator fusion model to output a target operator; wherein the operator fusion model adjusts low-rank weight adjustment parameters in the operator fusion model by loading a target first configuration file matched with the acceleration chip configuration information and a target second configuration file matched with the operator type information to adapt to the hardware platform and the operator type. By dynamically loading the configuration file related to the current hardware and the operator type, the operator implementation is automatically adjusted when facing changes in algorithm and application requirements and different acceleration chip architectures, ensuring performance consistency and reliability in different task scenarios.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an operator fusion method, system, device and medium. Background Technology

[0002] In various complex applications of modern high-performance computing, computation graphs are relied upon to manage and optimize the computation process. As a directed, non-commutative graph of computational tasks, the computation graph not only defines the relationships between operators but also directly impacts the execution efficiency of the computational tasks. Meanwhile, modern high-performance computing systems are gradually evolving towards heterogeneous computing architectures, composed of a central processing unit and various dedicated acceleration chips. These acceleration chips differ significantly in computing power and storage architecture, enabling heterogeneous computing systems to achieve excellent performance across different types of tasks. Therefore, the appropriate fusion of operators within the computation graph is particularly important in heterogeneous computing systems.

[0003] Since different accelerator chips have different characteristics in terms of hardware architecture, storage hierarchy, data transfer speed and parallel computing capabilities, the same computation graph may require different operator fusion strategies on different accelerator chips. Therefore, it is necessary to tailor a suitable operator fusion strategy for different accelerator chips in order to give full play to the best computing power of different accelerator chips.

[0004] Therefore, there is an urgent need for an operator fusion method to solve the above-mentioned technical problems. Summary of the Invention

[0005] Therefore, it is necessary to provide an operator fusion method to address the aforementioned technical problems and improve adaptability.

[0006] Firstly, this application provides an operator fusion method, the method comprising:

[0007] Obtain the acceleration chip configuration information and operator type information contained in the computation graph of the current hardware platform;

[0008] The obtained acceleration chip configuration information, operator type information, and computation graph are input into the trained operator fusion model to output the target operator;

[0009] The operator fusion model adjusts the low-rank weight adjustment parameters within the operator fusion model to adapt to the hardware platform and operator type by loading a target first configuration file that matches the acceleration chip configuration information and a target second configuration file that matches the operator type information.

[0010] In some embodiments, the operator fusion model includes:

[0011] The input preprocessing module is used to encode the input acceleration chip configuration information and operator type information to generate chip encoding vectors and operator encoding vectors;

[0012] A chip architecture adaptation module is used to generate a first configuration file during training and load a target first configuration file that matches the configuration information of the acceleration chip during runtime.

[0013] The operator type adaptation module is used to generate a second configuration file during training and load a target second configuration file that matches the operator type information during runtime.

[0014] The post-processing adaptation module is used to generate a fused encoding vector based on the chip encoding vector and the operator encoding vector;

[0015] The operator generation module is used to generate target operators based on the fused encoding vector.

[0016] In some embodiments, encoding the input acceleration chip configuration information and operator type information to generate chip encoding vectors and operator encoding vectors includes:

[0017] The encoder encodes the input accelerator chip configuration information according to a first preset formula to generate a chip encoding vector. The first preset formula is:

[0018] O arch =TE((I config ⊕E(I config ))⊕(I operator ⊕E(I operator ))),

[0019] Among them, I config This indicates the configuration information of the acceleration chip, E(I) config ) represents the encoding vector corresponding to the accelerator chip, I operator Represents operator type information, E(I) operator ) represents the encoding vector corresponding to the operator type, TE represents the transformation operation, ⊕ represents the concatenation operation, and O arch Represents the chip encoding vector;

[0020] The encoder encodes the input operator type information according to a second preset formula to generate an operator encoding vector. The second preset formula is:

[0021] O operator =TE(I operator ⊕E(I operator )),

[0022] Among them, O operator This represents the operator encoding vector.

[0023] In some embodiments, generating a fused encoding vector based on the chip encoding vector and the operator encoding vector includes:

[0024] The chip encoding vector and the operator encoding vector are concatenated to generate the first concatenated vector;

[0025] The first concatenation vector is input into the multilayer perceptron to generate a transform-coded vector;

[0026] The transform coding vector is input into a preset selection network to generate a first weight corresponding to the chip coding vector and a second weight corresponding to the operator coding vector;

[0027] The chip encoding vector is weighted according to the first weight, and the operator encoding vector is weighted according to the second weight.

[0028] The weighted chip encoding vector and the operator encoding vector are concatenated to generate a second concatenated vector;

[0029] A second concatenation vector is input into the multilayer perceptron to generate a fusion coding vector.

[0030] In some embodiments, inputting the transform coding vector into a preset selection network to generate a first weight corresponding to the chip coding vector and a second weight corresponding to the operator coding vector includes:

[0031] The selection network generates a first weight corresponding to the chip encoding vector based on a third preset formula, which is:

[0032]

[0033] Where A represents the network parameters, x represents the first concatenated input vector, and s1 represents the first weight;

[0034] The selection network generates second weights corresponding to the operator encoding vector based on a fourth preset formula, which is:

[0035]

[0036] Where B represents the network parameters and s2 represents the second weight.

[0037] In some embodiments, the method further includes a method for training the operator fusion model:

[0038] The operator fusion model is pre-trained based on a large-scale dataset of original operators to ensure that the operator fusion model can output fusion operators based on the input computation graph;

[0039] A first operator dataset that meets the preset conditions is selected according to preset parameters, and the pre-trained operator fusion model is trained a second time based on the first operator dataset to optimize the operator fusion model.

[0040] After the optimized operator fusion model is verified, a second operator dataset that meets the requirements of the adaptation scenario is selected according to different adaptation scenarios.

[0041] The optimized operator fusion model is trained a third time based on the second operator dataset to ensure that the operator fusion model is adaptively adjusted according to the applicable scenario.

[0042] In some embodiments, before performing a third training on the optimized operator fusion model based on the second operator dataset, the following steps are included:

[0043] Freeze all parameters of the operator fusion model and add low-rank weight adjustment parameters to the chip architecture adaptation module and operator adaptation module in the operator fusion model;

[0044] The third training of the optimized operator fusion model based on the second operator dataset further includes:

[0045] Save the first low-rank weight adjustment parameter in the chip architecture adaptation module that matches the adaptation scenario as the first configuration file;

[0046] The second low-rank weight adjustment parameters that match the adaptation scenario in the operator type adaptation module are saved as the second configuration file.

[0047] Secondly, this application provides an operator fusion system, the system comprising:

[0048] The data preparation module is used to obtain the acceleration chip configuration information of the current hardware platform and the operator type information contained in the computation graph;

[0049] The operator fusion module is used to input the acquired acceleration chip configuration information, operator type information, and computation graph into the trained operator fusion model to output the target operator;

[0050] The operator fusion model adjusts the low-rank weight adjustment parameters within the operator fusion model to adapt to the hardware platform and operator type by using a target first configuration file that matches the acceleration chip configuration information and a target second configuration file that matches the operator type information.

[0051] Thirdly, this application provides a computer program product that, when executed by a processor, implements the steps of the following method:

[0052] Obtain the acceleration chip configuration information and operator type information contained in the computation graph of the current hardware platform;

[0053] The obtained acceleration chip configuration information, operator type information, and computation graph are input into the trained operator fusion model to output the target operator;

[0054] The operator fusion model adjusts the low-rank weight adjustment parameters within the operator fusion model to adapt to the hardware platform and operator type by loading a target first configuration file that matches the acceleration chip configuration information and a target second configuration file that matches the operator type information.

[0055] Fourthly, this application provides an electronic device, which includes:

[0056] One or more processors;

[0057] and memory associated with one or more processors, the memory being used to store program instructions, which, when read and executed by one or more processors, perform the following operations:

[0058] Obtain the acceleration chip configuration information and operator type information contained in the computation graph of the current hardware platform;

[0059] The obtained acceleration chip configuration information, operator type information, and computation graph are input into the trained operator fusion model to output the target operator;

[0060] The operator fusion model adjusts the low-rank weight adjustment parameters within the operator fusion model to adapt to the hardware platform and operator type by loading a target first configuration file that matches the acceleration chip configuration information and a target second configuration file that matches the operator type information.

[0061] Fifthly, this application also provides a computer-readable storage medium storing a computer program that causes a computer to perform the following operations:

[0062] Obtain the acceleration chip configuration information and operator type information contained in the computation graph of the current hardware platform;

[0063] The obtained acceleration chip configuration information, operator type information, and computation graph are input into the trained operator fusion model to output the target operator;

[0064] The operator fusion model adjusts the low-rank weight adjustment parameters within the operator fusion model to adapt to the hardware platform and operator type by loading a target first configuration file that matches the acceleration chip configuration information and a target second configuration file that matches the operator type information.

[0065] The beneficial effects achieved by this application are as follows:

[0066] This application provides an operator fusion method, including obtaining the acceleration chip configuration information of the current hardware platform and the operator type information contained in the computation graph; inputting the obtained acceleration chip configuration information, the operator type information, and the computation graph into a trained operator fusion model to output a target operator; wherein, the operator fusion model adjusts the low-rank weight adjustment parameters within the operator fusion model by loading a target first configuration file matching the acceleration chip configuration information and a target second configuration file matching the operator type information to adapt to the hardware platform and operator type. By dynamically loading configuration files related to the current hardware and operator type, the operator implementation can be automatically adjusted when facing changes in algorithm and application requirements and different acceleration chip architectures, ensuring performance consistency and reliability in different task scenarios. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein:

[0068] Figure 1 This is a schematic diagram of an operator fusion method provided in an embodiment of this application;

[0069] Figure 2 This is a diagram of an operator fusion model architecture provided in an embodiment of this application;

[0070] Figure 3 This is a schematic diagram of an encoding processing method provided in an embodiment of this application;

[0071] Figure 4 This is a schematic diagram of a fusion processing method provided in an embodiment of this application;

[0072] Figure 5 This is a diagram of an operator fusion system architecture provided in an embodiment of this application;

[0073] Figure 6 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0075] It should be understood that, in the description of this application, unless the context explicitly requires it, the words "comprising," "including," and similar terms throughout the specification and claims should be interpreted as encompassing rather than being exclusive or exhaustive; that is, meaning "including but not limited to."

[0076] It should also be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0077] It should be noted that the terms "S1," "S2," etc., are used only for descriptive purposes and do not specifically refer to the order or sequence, nor are they intended to limit this application. They are merely for the convenience of describing the method of this application and should not be construed as indicating the sequential order of the steps. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0078] As modern high-performance computing systems gradually evolve towards heterogeneous architectures, it is necessary to rationally merge operators in the computation graph to improve the computing power of these systems. Operator fusion is an optimization technique that combines multiple adjacent operators into a larger one, thereby reducing data migration and exchange between different operators, lowering memory bandwidth requirements, increasing computational density, and significantly improving computational speed. Therefore, a reasonable operator fusion strategy plays an indispensable role in improving the efficiency of high-performance computing.

[0079] However, due to the different architectures and performance characteristics of various accelerator chips, the same computation graph may require different operator fusion strategies on different accelerator chips. For example, some chips may be suitable for fusion of more operators to reduce memory accesses, while others may need to keep operators separate to fully utilize their parallel processing capabilities. Therefore, operator fusion strategies tailored to each accelerator chip can better leverage their computational potential and ultimately achieve a more efficient computation process. Operator fusion strategies in heterogeneous computing systems not only directly affect the execution efficiency of the computation graph but also effectively improve computational speed, maximizing the hardware advantages of dedicated accelerator chips.

[0080] Existing operator fusion methods mainly include the following: graph pattern fusion, dynamic fusion, static fusion, and hierarchical fusion. Graph pattern fusion utilizes predefined computation graph patterns for optimization, reducing computation and data transfer overhead by identifying and fusing common patterns. It is simple to implement and highly effective, suitable for handling fixed and common graph structures. However, it is limited by the fixed pattern library and cannot handle all computation graph structures, exhibiting poor adaptability. It may not be effective in optimizing complex or uncommon patterns. Dynamic fusion performs operator fusion at runtime based on the actual computation graph, offering high flexibility and adaptability, capable of handling complex and dynamically changing computational tasks. It improves computational performance through real-time optimization, but also suffers from high runtime overhead and implementation complexity, potentially impacting overall performance. Static fusion optimizes the computation graph globally during the compilation phase, reducing runtime computation and data transfer overhead and significantly improving performance. However, static fusion is less adaptable to dynamically changing computation graphs, and the complex compilation process can lead to extended compilation time, limiting its effectiveness in handling dynamic tasks. Hierarchical fusion systematically improves computational efficiency by progressively optimizing the computation graph according to a hierarchical structure, merging from low to high levels. It allows optimization at different levels, enhancing overall performance. However, hierarchical fusion is complex to implement, requiring effective hierarchical management. The choice of different levels directly affects the final optimization effect, and inappropriate choices may reduce the performance improvement.

[0081] Therefore, this application discloses an adaptive operator fusion method that automatically adjusts the operator implementation to adapt to the characteristics of various acceleration chips, including GPUs (Graphics Processing Units), FPGAs (Field Programmable Gate Arrays), and TPUs (Tensor Processing Units), to ensure optimal performance on different platforms.

[0082] Example 1

[0083] This application provides an operator fusion method, specifically, as shown in the embodiments below. Figure 1 As shown, the method disclosed in the embodiments of this application is used to perform operator fusion on operators contained in a computation graph, including the following:

[0084] S1. Obtain the acceleration chip configuration information and operator type information contained in the computation graph of the current hardware platform.

[0085] To improve the adaptability of the same computation graph across different platforms, this application requires the collection of corresponding acceleration chip configuration information on the platform. This acceleration chip configuration information refers to configuration information related to the acceleration chip and affecting its computational performance, including but not limited to hardware architecture information, storage hierarchy, and data transfer characteristics. This acceleration chip configuration information is provided by the hardware platform and can be obtained through query requests, etc. This application does not limit the specific method for obtaining the acceleration chip configuration information.

[0086] In this context, "acceleration chip" refers to different types of computing accelerators, such as GPUs, TPUs, and FPGAs. A computation graph is a directed acyclic graph representing the computation process, where each operation or operator (such as addition, multiplication, convolution, etc.) represents an operation, and each edge represents the flow of data between operations / operators. Computation graphs are widely used in deep learning, automatic differentiation, and data computation. The operator types mentioned above have defined type attributes during computation graph construction, which can be used to obtain the operator type. Specifically, obtaining the operator type can be implemented programmatically, which is a conventional technique in this field and will not be elaborated upon here.

[0087] S2. Input the obtained acceleration chip configuration information, operator type information, and computation graph into the trained operator fusion model to output the target operator.

[0088] Based on the output target operator, the computation graph is updated, and the updated computation graph is executed on the hardware platform. Specifically, the operator fusion model adjusts the low-rank weight adjustment parameters within the model by loading a first target configuration file matching the acceleration chip configuration information and a second target configuration file matching the operator type information to adapt to the hardware platform and operator type.

[0089] This application achieves automatic adjustment of operator implementation based on acceleration chip configuration and operator type through a pre-trained operator fusion model, ensuring optimal performance of the computation graph across different platforms. By automatically adapting to hardware platform characteristics, optimizing performance, reducing development and maintenance costs, and responding to hardware evolution and changing requirements, it ensures consistent performance and stability of the computation graph in diverse computing environments.

[0090] Furthermore, by optimizing specific hardware architectures, the execution efficiency of operators is effectively improved, overcoming the complexity and limitations of traditional manual optimization methods, enabling operators to execute more efficiently. In addition, by reducing manual optimization and adjustments for each platform, the complexity and cost of development and maintenance are reduced. Simultaneously, it can quickly adapt to updates in accelerator chip technology, avoiding the challenge of traditional methods failing to keep up with hardware evolution. Furthermore, the method disclosed in this application can automatically adjust operator implementation according to changes in algorithm and application requirements, ensuring the system maintains high efficiency and adaptability in different scenarios, guaranteeing application flexibility and scalability. It ensures performance consistency across diverse hardware platforms, improves the reliability and stability of cross-platform applications, and enables the system to maintain optimal performance in various computing environments.

[0091] In a specific embodiment, such as Figure 2 As shown in the model architecture diagram, the operator fusion model includes an input preprocessing module for encoding the input accelerator chip configuration information and operator type information to generate chip encoding vectors and operator encoding vectors; a chip architecture adaptation module for generating a first configuration file during training and loading a target first configuration file matching the accelerator chip configuration information during runtime; an operator type adaptation module for generating a second configuration file during training and loading a target second configuration file matching the operator type information during runtime; a post-processing adaptation module for generating a fused encoding vector based on the chip encoding vector and operator encoding vector; and an operator generation module for decoding the fused encoding vector to generate the target operator. Through the collaboration of these five modules, the operator fusion model ensures that the final fused target operator output by the model achieves optimal performance on different chip architectures and operator types.

[0092] The above-mentioned input preprocessing module, such as Figure 3 The diagram illustrates the encoding process, which encodes the acceleration chip configuration information and operator type information in the input model.

[0093] Specifically, the encoder encodes the input accelerator chip configuration information according to a first preset formula to generate a chip encoding vector. The first preset formula is:

[0094] O arch =TE((I config ⊕E(I config ))⊕(I operator ⊕E(I ooperator ))), where I config This indicates the configuration information of the acceleration chip, E(I) config ) represents the encoding vector corresponding to the accelerator chip, I operator Represents operator type information, E(I)operator ) represents the encoding vector corresponding to the operator type, TE represents the transformation operation, ⊕ represents the concatenation operation, and O arch This represents the chip encoding vector.

[0095] The encoder encodes the input operator type information according to a second preset formula to generate an operator encoding vector. The second preset formula is:

[0096] O operator =TE(I operator ⊕E(I operator ), where O operator This represents the operator encoding vector.

[0097] It is understood that in the encoding process disclosed in the embodiments of this application, marker information is added, that is, an encoding vector E(I) is uniformly added when encoding the chip configuration information. config When encoding operator type information, E(I) is uniformly added. operator Ensure the independence between the chip encoding vector and the operator encoding vector to avoid deep coupling.

[0098] The aforementioned chip architecture adaptation module is specifically designed to adapt to specific chip architectures, ensuring the system can run efficiently on different hardware platforms. This module employs a Transformer-based architecture combined with LoRA (Low-Rank Adaptation) to improve the adaptability and scalability of the operator fusion model. More specifically, during the training phase, the chip architecture adaptation module uses LoRA for refined training based on different selected adaptation scenarios (i.e., including different chip architectures), and saves a first configuration file for each architecture. This first configuration file contains the low-rank weight adjustment parameters required for adapting to the specific chip architecture, thus eliminating the need to retrain the entire model during training; only a small portion of the parameters need to be adjusted to complete the adaptation. The Transformer is a revolutionary architectural design in Natural Language Processing (NLP). It abandons traditional methods based on Recurrent Neural Networks (RNNs) and instead uses self-attention mechanisms, enabling the model to process sequential data more effectively. Low-rank adaptation is a technique for fine-tuning a pre-trained model, aiming to adapt to new tasks or datasets by adjusting a subset of the model's parameters while preserving as much of the pre-trained model's knowledge as possible. In the Transformer model, low-rank adaptation techniques can be used to reduce the computational resources and time costs required during fine-tuning while maintaining good performance.

[0099] In actual operation, the chip architecture adaptation module can dynamically load the corresponding target primary configuration file based on the currently running chip architecture, achieving automatic adaptation to the chip architecture. Through this dynamic loading mechanism, the system can seamlessly switch between different hardware environments without needing to redesign or deploy the entire system every time the architecture changes. This significantly reduces the cost and time required to adapt to different hardware platforms and ensures optimal performance across diverse chip architectures. It further improves the flexibility of heterogeneous computing systems and enhances their ability to cope with constantly changing hardware environments, guaranteeing the stability and efficiency of heterogeneous computing systems under various architectures.

[0100] The main function of the operator type adaptation module is similar to that of the chip architecture adaptation module, but it focuses on handling adaptation tasks related to different operator types. The operator type adaptation module also adopts a Transformer-based architecture and incorporates low-rank adaptation techniques to learn and optimize strategies for different operators. During training, the operator type adaptation module performs refined learning for various operator types and saves the optimization strategies as a second configuration file. The second configuration file records the low-rank weight adjustment parameters corresponding to a specific operator type, enabling rapid loading and application of these optimization strategies in practical applications. At runtime, the operator type adaptation module can dynamically load the target second configuration file that matches the current operator type, thereby achieving automatic optimization for specific operator types. This greatly enhances the scalability of the operator type adaptation module, allowing the target operators generated by the model to flexibly handle various computational tasks and ensure optimal operator performance in each task. By combining the powerful modeling capabilities of the Transformer architecture with the efficient adaptability of low-rank adaptation techniques, the operator type adaptation module can achieve the most efficient performance with minimal computational overhead when facing different operator requirements. This dynamic adaptability ensures the system's flexibility and superiority in diverse computing environments, enabling it to maintain its optimal state when handling various complex tasks.

[0101] By utilizing low-rank adaptation for fine-tuning and saving the configuration file locally, redundant model parameters are effectively avoided during inference, thus significantly alleviating the pressure on on-chip storage. This optimization not only improves storage utilization but also ensures the system's high efficiency and stability during runtime.

[0102] like Figure 4The diagram illustrates the fusion processing method. The post-processing adaptation module effectively fuses the chip encoding vector and operator encoding vector already encoded in the input pre-processing module. Specifically, the chip encoding vector and operator encoding vector are concatenated to generate a first concatenated vector; the first concatenated vector is input into the multilayer perceptron to generate a transform encoding vector; the transform encoding vector is input into a preset selection network to generate a first weight corresponding to the chip encoding vector and a second weight corresponding to the operator encoding vector; the chip encoding vector is weighted according to the first weight, and the operator encoding vector is weighted according to the second weight; the weighted chip encoding vector and operator encoding vector are concatenated to generate a second concatenated vector; the second concatenated vector is input into the multilayer perceptron to generate a fused encoding vector. The multilayer perceptron (MLP) is part of a feed-forward network, and its function is to perform nonlinear transformations on the input data to capture more complex feature representations.

[0103] The above-mentioned input of a transform encoding vector to a preset selection network to generate a first weight corresponding to the chip encoding vector and a second weight corresponding to the operator encoding vector includes: the selection network generating the first weight corresponding to the chip encoding vector based on a third preset formula; and generating the second weight corresponding to the operator encoding vector based on a fourth preset formula.

[0104] The third preset formula mentioned above is:

[0105]

[0106] The fourth preset formula mentioned above is:

[0107]

[0108] Here, A and B represent network parameters, x represents the first concatenated input vector, s1 represents the first weight, and s2 represents the second weight. Specifically, A and B can be understood as matrices and are learnable parameters. Through learning and optimization during backpropagation, this network structure can better determine the weights of the chip encoding vector and the operator encoding vector during fusion, i.e., whether the current fusion strategy should favor chip adaptation or operator adaptation. Finally, the chip encoding vector and the operator encoding vector are weighted and summed to achieve information fusion. The post-processing adaptation module provides a unified and optimized input basis for the subsequent operator generation module, namely the fused encoding vector O. fusion This ensures that the target operator output by the operator fusion model can run efficiently in environments with various architectures and operator types.

[0109] Similarly, the operator generation module described above uses the Transformer-Decoder as its backbone. Based on the fused encoding vector generated by the post-processing adaptation module, it comprehensively considers various adaptation information and fusion rules to ultimately generate the optimal operator implementation (i.e., the target operator) suitable for different chip architectures and operator types. Understandably, the operator fusion model maintains a vocabulary, where the word segmentation method is BPE, and the final output vector is a string of token IDs. Based on the token ID, we need to query the corresponding token from the vocabulary, concatenate the decoded tokens to obtain the final operator, which is typically a piece of CUDA code. Since the features output by the post-processing module may not be suitable for direct decoding, the operator generation module here leverages the characteristics of the Transformer module to process the fused encoding vector and ultimately decode it into an operator.

[0110] In a specific embodiment, this application also discloses a training method for the above-mentioned operator fusion model:

[0111] The operator fusion model is pre-trained using a large-scale dataset of raw operators to ensure that it can output fused operators based on the input computation graph. Specifically, the pre-training process endows the model with the general ability to generate operators from the computation graph. During the pre-training phase, the model learns basic operator generation rules and patterns through a large amount of diverse data. These rules and patterns provide a solid foundation for subsequent optimization, enabling the model to adapt widely to different computational tasks. The operator dataset includes at least the accelerator chip type, operator type, and computation graph.

[0112] A first operator dataset that meets preset conditions is selected based on preset parameters. The pre-trained operator fusion model is then trained a second time using this dataset to optimize the model. The preset parameters include, but are not limited to, memory access ratio, throughput, memory bandwidth utilization, branch efficiency, and kernel function execution time. The preset conditions are thresholds set by those skilled in the art for different parameters based on operational requirements. Each preset parameter exceeding its corresponding threshold indicates that it meets the operational requirements, i.e., it conforms to the preset conditions.

[0113] After the optimized operator fusion model passes validation, a second operator dataset matching the requirements of different adaptation scenarios is selected. The optimized operator fusion model is then trained a third time using this second operator dataset to ensure adaptive adjustments based on the adaptation scenarios. Here, an adaptation scenario refers to a scenario adapted to different acceleration chips and / or different operator types. The second operator dataset is the dataset of operators already running in that adaptation scenario, such as a dataset for TPU chips, a dataset suitable for GPUs, a reduce-type operator dataset, or an element-wise operation operator dataset. The third training of the optimized operator fusion model using the second operator dataset also includes: saving the first low-rank weight adjustment parameters matching the adaptation scenario in the chip architecture adaptation module as a first configuration file; and saving the second low-rank weight adjustment parameters matching the adaptation scenario in the operator type adaptation module as a second configuration file. This training phase requires freezing the parameters of all modules in the operator fusion model so that they are not updated during training. Low-rank adaptation techniques are used to fine-tune specific adaptation modules (i.e., chip architecture adaptation modules and operator type adaptation modules), and LoRA parameters are saved for loading during subsequent inference. For example, to train LoRA parameters suitable for TPU chips, LoRA parameters are added to the QKV matrix of the transformer structure in the chip architecture adaptation module, while all model parameters are frozen. Only these LoRA parameters are updated during training. The same principle applies to training the LoRA of the operator type adaptation module. Afterwards, a series of different LoRA parameter configuration files are saved, and different LoRA configuration files (i.e., target first configuration file and target second configuration file) are dynamically loaded during inference. For example, to generate a reduce type operator suitable for TPU chips, the chip architecture adaptation module loads the target first configuration file corresponding to the TPU during inference, and the operator type adaptation module loads the target second configuration file matching the reduce operator type.

[0114] To further reduce storage pressure, the model size can be compressed, specifically including the following: In response to the detection that the operator fusion model has completed training and passed validation, determine the tensors to be quantized for each layer of the operator fusion model; calibrate the operator fusion model based on a pre-set validation dataset to determine quantization parameters, such as scaling factors and zeros; quantize the model's weights and / or parameters based on the quantization parameters obtained during the calibration phase; in response to the detection that quantization is complete, test the above operator fusion model based on the validation dataset; if the model performance does not significantly decrease, derive the quantized operator fusion model. This allows the model's parameters to be quantized from high precision (e.g., 32-bit floating-point) to low precision (e.g., 8-bit or 4-bit integer), thereby reducing storage requirements and computational complexity, and also reducing the model size, thus reducing the storage pressure on the model. Furthermore, methods such as weight pruning, knowledge distillation, and network architecture search can also be used to compress the model.

[0115] The above steps train the model, enabling the final operator fusion model to flexibly adapt to diverse computing environments and ensure that the generated operators perform optimally under any circumstances. This layer-by-layer optimization and dynamic adaptation approach provides robust support for modern computing tasks, demonstrating significant advantages, especially when faced with constantly updated accelerator chips and diverse operator requirements.

[0116] 6) Alleviating on-chip storage pressure: By utilizing LoRA for fine-tuning and saving the configuration file locally, this method effectively avoids repeatedly loading redundant model parameters during inference, thus significantly alleviating the pressure on on-chip storage. This optimization not only improves storage utilization but also ensures the system's high efficiency and stability during operation.

[0117] Example 2

[0118] Corresponding to the above embodiments, such as Figure 5 As shown in the embodiments of this application, an operator fusion system is also provided, including:

[0119] The data preparation module 510 is used to obtain the acceleration chip configuration information of the current hardware platform and the operator type information contained in the computation graph;

[0120] The operator fusion module 520 is used to input the acquired acceleration chip configuration information, operator type information and computation graph into the trained operator fusion model to output the target operator;

[0121] The operator fusion model adjusts the low-rank weight adjustment parameters within the operator fusion model to adapt to the hardware platform and operator type by using a target first configuration file that matches the acceleration chip configuration information and a target second configuration file that matches the operator type information.

[0122] Example 3

[0123] This application also provides a computer program product that, when executed by a processor, implements the steps of the following method:

[0124] Obtain the acceleration chip configuration information and operator type information contained in the computation graph of the current hardware platform;

[0125] The obtained acceleration chip configuration information, operator type information, and computation graph are input into the trained operator fusion model to output the target operator;

[0126] The operator fusion model adjusts the low-rank weight adjustment parameters within the operator fusion model to adapt to the hardware platform and operator type by loading a target first configuration file that matches the acceleration chip configuration information and a target second configuration file that matches the operator type information.

[0127] The operator fusion model includes:

[0128] The input preprocessing module is used to encode the input acceleration chip configuration information and operator type information to generate chip encoding vectors and operator encoding vectors;

[0129] A chip architecture adaptation module is used to generate a first configuration file during training and load a target first configuration file that matches the configuration information of the acceleration chip during runtime.

[0130] The operator type adaptation module is used to generate a second configuration file during training and load a target second configuration file that matches the operator type information during runtime.

[0131] The post-processing adaptation module is used to generate a fused encoding vector based on the chip encoding vector and the operator encoding vector;

[0132] The operator generation module is used to generate target operators based on the fused encoding vector.

[0133] In some implementation scenarios, when the computer program product is executed by a processor, it further implements the steps of the following method:

[0134] The encoder encodes the input accelerator chip configuration information according to a first preset formula to generate a chip encoding vector. The first preset formula is:

[0135] O arch =TE((I config ⊕E(I config ))⊕(I operator ⊕E(I operator ))),

[0136] Among them, I config This indicates the configuration information of the acceleration chip, E(I) config ) represents the encoding vector corresponding to the accelerator chip, I operator Represents operator type information, E(I) operator ) represents the encoding vector corresponding to the operator type, TE represents the transformation operation, ⊕ represents the concatenation operation, and O arch Represents the chip encoding vector;

[0137] The encoder encodes the input operator type information according to a second preset formula to generate an operator encoding vector. The second preset formula is:

[0138] O operator =TE(I operator ⊕E(I operator )),

[0139] Among them, O operator This represents the operator encoding vector.

[0140] In some implementation scenarios, when the computer program product is executed by a processor, it further implements the steps of the following method:

[0141] The chip encoding vector and the operator encoding vector are concatenated to generate the first concatenated vector;

[0142] The first concatenation vector is input into the multilayer perceptron to generate a transform-coded vector;

[0143] The transform coding vector is input into a preset selection network to generate a first weight corresponding to the chip coding vector and a second weight corresponding to the operator coding vector;

[0144] The chip encoding vector is weighted according to the first weight, and the operator encoding vector is weighted according to the second weight.

[0145] The weighted chip encoding vector and the operator encoding vector are concatenated to generate a second concatenated vector;

[0146] A second concatenation vector is input into the multilayer perceptron to generate a fusion coding vector.

[0147] In some implementation scenarios, when the computer program product is executed by a processor, it further implements the steps of the following method:

[0148] The selection network generates a first weight corresponding to the chip encoding vector based on a third preset formula, which is:

[0149]

[0150] Where A represents the network parameters, x represents the first concatenated input vector, and s1 represents the first weight;

[0151] The selection network generates second weights corresponding to the operator encoding vector based on a fourth preset formula, which is:

[0152]

[0153] Where B represents the network parameters and s2 represents the second weight.

[0154] In some implementation scenarios, when the computer program product is executed by a processor, it further implements the steps of the following method:

[0155] The operator fusion model is pre-trained based on a large-scale dataset of original operators to ensure that the operator fusion model can output fusion operators based on the input computation graph;

[0156] A first operator dataset that meets the preset conditions is selected according to preset parameters, and the pre-trained operator fusion model is trained a second time based on the first operator dataset to optimize the operator fusion model.

[0157] After the optimized operator fusion model is verified, a second operator dataset that meets the requirements of the adaptation scenario is selected according to different adaptation scenarios.

[0158] The optimized operator fusion model is trained a third time based on the second operator dataset to ensure that the operator fusion model is adaptively adjusted according to the applicable scenario.

[0159] In some implementation scenarios, when the computer program product is executed by a processor, it further implements the steps of the following method:

[0160] Freeze all parameters of the operator fusion model and add low-rank weight adjustment parameters to the chip architecture adaptation module and operator adaptation module in the operator fusion model;

[0161] The third training of the optimized operator fusion model based on the second operator dataset further includes:

[0162] Save the first low-rank weight adjustment parameter in the chip architecture adaptation module that matches the adaptation scenario as the first configuration file;

[0163] The second low-rank weight adjustment parameters that match the adaptation scenario in the operator type adaptation module are saved as the second configuration file.

[0164] Example 4

[0165] Corresponding to all the above embodiments, this application provides an electronic device, including: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, which, when read and executed by the one or more processors, perform the following operations:

[0166] Obtain the acceleration chip configuration information and operator type information contained in the computation graph of the current hardware platform;

[0167] The obtained acceleration chip configuration information, operator type information, and computation graph are input into the trained operator fusion model to output the target operator;

[0168] The operator fusion model adjusts the low-rank weight adjustment parameters within the operator fusion model to adapt to the hardware platform and operator type by loading a target first configuration file that matches the acceleration chip configuration information and a target second configuration file that matches the operator type information.

[0169] in, Figure 6 An exemplary architecture of an electronic device is shown, which may include a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, and a memory 620. The processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620 can communicate with each other via a bus 630.

[0170] The processor 610 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to achieve the technical solution provided in this application.

[0171] The memory 620 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 620 can store the operating system 621 for controlling the execution of the electronic device 600, and the basic input / output system (BIOS) 622 for controlling the low-level operations of the electronic device 600. Additionally, it can store a web browser 623, a data storage management system 624, and an icon font processing system 625, etc. The aforementioned icon font processing system 625 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 620 and called and executed by the processor 610.

[0172] Input / output interface 613 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0173] Network interface 614 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0174] Bus 630 includes a pathway for transmitting information between various components of the device, such as processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620.

[0175] In addition, the electronic device 600 can also obtain information on specific claim conditions from the virtual resource object claim condition information database for use in condition judgment, etc.

[0176] It should be noted that although the above-described device only shows the processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, memory 620, bus 630, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0177] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, a cloud server, or a network device, etc.) to execute the methods of various embodiments or some parts of the embodiments of this application.

[0178] Example 5

[0179] Corresponding to all the above embodiments, this application also provides a computer-readable storage medium storing a computer program that causes a computer to perform the following operations:

[0180] Obtain the acceleration chip configuration information and operator type information contained in the computation graph of the current hardware platform;

[0181] The obtained acceleration chip configuration information, operator type information, and computation graph are input into the trained operator fusion model to output the target operator;

[0182] The operator fusion model adjusts the low-rank weight adjustment parameters within the operator fusion model to adapt to the hardware platform and operator type by loading a target first configuration file that matches the acceleration chip configuration information and a target second configuration file that matches the operator type information.

[0183] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0184] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An operator fusion method, characterized in that, The method includes: Obtain the acceleration chip configuration information and operator type information contained in the computation graph of the current hardware platform; The obtained acceleration chip configuration information, operator type information, and computation graph are input into the trained operator fusion model to output the target operator; The operator fusion model adjusts the low-rank weight adjustment parameters within the operator fusion model to adapt to the hardware platform and operator type by loading a target first configuration file that matches the acceleration chip configuration information and a target second configuration file that matches the operator type information. The operator fusion model includes: The input preprocessing module is used to encode the input acceleration chip configuration information and operator type information to generate chip encoding vectors and operator encoding vectors; A chip architecture adaptation module is used to generate a first configuration file during training and load a target first configuration file that matches the configuration information of the acceleration chip during runtime. The operator type adaptation module is used to generate a second configuration file during training and load a target second configuration file that matches the operator type information during runtime. The post-processing adaptation module is used to generate a fused encoding vector based on the chip encoding vector and the operator encoding vector; The operator generation module is used to generate target operators based on the fused coding vector; The step of generating a fused encoding vector based on the chip encoding vector and the operator encoding vector includes: The chip encoding vector and the operator encoding vector are concatenated to generate the first concatenated vector; The first concatenation vector is input into the multilayer perceptron to generate a transform-coded vector; The transform coding vector is input into a preset selection network to generate a first weight corresponding to the chip coding vector and a second weight corresponding to the operator coding vector; The chip encoding vector is weighted according to the first weight, and the operator encoding vector is weighted according to the second weight. The weighted chip encoding vector and the operator encoding vector are concatenated to generate a second concatenated vector; A second concatenation vector is input into the multilayer perceptron to generate a fusion coding vector.

2. The method according to claim 1, characterized in that, The process of encoding the input acceleration chip configuration information and operator type information to generate chip encoding vectors and operator encoding vectors includes: The encoder encodes the input accelerator chip configuration information according to a first preset formula to generate a chip encoding vector. The first preset formula is: ， in, This indicates the configuration information of the acceleration chip. This represents the encoding vector corresponding to the acceleration chip. Indicates operator type information, This represents the encoding vector corresponding to the operator type. Indicates a conversion operation. This indicates a splicing operation. Represents the chip encoding vector; The encoder encodes the input operator type information according to a second preset formula to generate an operator encoding vector. The second preset formula is: ， in, This represents the operator encoding vector.

3. The method according to claim 1, characterized in that, The step of inputting the transform coding vector into a preset selection network to generate a first weight corresponding to the chip coding vector and a second weight corresponding to the operator coding vector includes: The selection network generates a first weight corresponding to the chip encoding vector based on a third preset formula, which is: ， in, Represents network parameters, This represents the first concatenated input vector. Indicates the first weight; The selection network generates second weights corresponding to the operator encoding vector based on a fourth preset formula, which is: ， in, Represents network parameters, This indicates the second weight.

4. The method according to any one of claims 2-3, characterized in that, The method also includes a training method for the operator fusion model: The operator fusion model is pre-trained based on a large-scale dataset of original operators to ensure that the operator fusion model can output fusion operators based on the input computation graph; A first operator dataset that meets the preset conditions is selected according to preset parameters, and the pre-trained operator fusion model is trained a second time based on the first operator dataset to optimize the operator fusion model. After the optimized operator fusion model is verified, a second operator dataset that meets the requirements of the adaptation scenario is selected according to different adaptation scenarios. The optimized operator fusion model is trained a third time based on the second operator dataset to ensure that the operator fusion model is adaptively adjusted according to the applicable scenario.

5. The method according to claim 4, characterized in that, Before performing a third training on the optimized operator fusion model based on the second operator dataset, the following steps are included: Freeze all parameters of the operator fusion model and add low-rank weight adjustment parameters to the chip architecture adaptation module and operator adaptation module in the operator fusion model; The third training of the optimized operator fusion model based on the second operator dataset further includes: Save the first low-rank weight adjustment parameter in the chip architecture adaptation module that matches the adaptation scenario as the first configuration file; The second low-rank weight adjustment parameters that match the adaptation scenario in the operator type adaptation module are saved as the second configuration file.

6. An operator fusion system for implementing the operator fusion method as described in any one of claims 1-5, characterized in that, The system includes: The data preparation module is used to obtain the acceleration chip configuration information of the current hardware platform and the operator type information contained in the computation graph; The operator fusion module is used to input the acquired acceleration chip configuration information, operator type information, and computation graph into the trained operator fusion model to output the target operator; The operator fusion model adjusts the low-rank weight adjustment parameters within the operator fusion model to adapt to the hardware platform and operator type by using a target first configuration file that matches the acceleration chip configuration information and a target second configuration file that matches the operator type information.

7. An electronic device, characterized in that, The electronic device includes: One or more processors; And a memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, It stores a computer program that causes a computer to perform the method described in any one of claims 1-5.

Images