Operator fusion method and system, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides an operator fusion method and system, electronic equipment and a storage medium, and the method comprises the steps: receiving the operator number of to-be-fused operators and an equipment function identifier sent by a host, and the equipment function identifier is determined by the host based on the operator identifier of the to-be-fused operators; based on the device function identifier, loading a device function corresponding to the to-be-fused operator, the device function being developed in advance based on the to-be-fused operator, and being compiled and loaded into a memory of the device; and fusing the operators to be fused based on the number of the operators and the equipment function. According to the invention, automatic fusion during operator operation can be realized, the development amount of operator fusion is greatly reduced, and the efficiency of operator fusion is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an operator fusion method, system, electronic device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence (AI), model performance optimization has become a key link in improving algorithm efficiency and accuracy. Operator fusion, as an important technology in this field, aims to reduce computational redundancy, reduce memory access times, and accelerate the computational process by combining multiple computational steps or operations (i.e., operators) into a single operation. Operator fusion technology is widely used in deep learning frameworks, machine learning models, and high-performance computing environments, and plays a significant role in improving model reasoning and training speed.

[0003] Currently, operator fusion mainly relies on manual implementation by developers. Developers need to have a deep understanding of the working principle of each operator and the dependencies between them. Based on this understanding, they manually write fusion code for each possible combination. However, due to the wide variety of operators in the AI ​​field, the combination of different operators is complex and varied. The manual one-to-one fusion method not only takes a lot of time and manpower, but is also prone to errors and omissions. This method greatly increases the burden on developers and limits the efficiency and versatility of operator fusion. Summary of the invention

[0004] The present invention provides an operator fusion method, system, electronic device and storage medium, which are used to solve the defects of low operator fusion efficiency and poor versatility in related technologies.

[0005] The present invention provides an operator fusion method, which is applied to a device and includes: receiving the operator quantity and device function identifier of the operator to be fused sent by the host, where the device function identifier is determined by the host based on the operator identifier of the operator to be fused; Based on the device function identifier, load the device function corresponding to the operator to be fused, where the device function is pre-developed based on the operator to be fused, and is compiled and loaded into the memory of the device; The operators to be fused are fused based on the number of operators and the device function.

[0006] According to an operator fusion method provided by the present invention, the operator to be fused is fused based on the number of operators and the device function, including: Acquire data to be processed, where the data to be processed is divided into a plurality of data blocks; Each data block in the data to be processed is traversed, and based on the number of operators and the device function, the currently traversed data block is calculated until the calculation of each data block is completed, and the device function is used to execute the calculation task corresponding to the operator to be fused.

[0007] According to an operator fusion method provided by the present invention, the calculation of the currently traversed data block based on the operator quantity and the device function includes: Determine the coordinates of the currently traversed data block, and load the data block based on the coordinates; Based on the number of operators, the device function corresponding to each operator in the operators to be fused is executed in sequence to calculate the data block and obtain a calculation result.

[0008] According to an operator fusion method provided by the present invention, the loading of the device function corresponding to the operator to be fused based on the device function identifier includes: In the case where the device function identifier is a device function pointer, loading the device function corresponding to the operator to be fused based on the device function pointer; In the case that the device function identifier is not a device function pointer, the device function pointer corresponding to the operator to be fused is obtained based on the device function identifier, and the device function pointer is applied to load the device function corresponding to the operator to be fused.

[0009] According to an operator fusion method provided by the present invention, the step of determining the device function identifier includes: Receiving an operator identifier sent by the host; Based on the operator identifier, searching for a device function pointer corresponding to the operator identifier from an operator function mapping table, wherein the operator function mapping table is a mapping relationship table between the operator identifier and the device function pointer pre-constructed on the device; The device function pointer is returned to the host as a device function identifier corresponding to the operator to be fused.

[0010] The present invention also provides an operator fusion method, which is applied to a host and includes: Determine the number of operators and operator identifiers of operators to be fused, and determine the device function identifier based on the operator identifier; The device function identifier and the number of operators are sent to a device, and the device is used to load the device function corresponding to the operator to be fused based on the device function identifier, and fuse the operator to be fused based on the operator number and the device function. The device function is pre-developed based on the operator to be fused, and is compiled and loaded into the memory of the device.

[0011] The present invention also provides an operator fusion device, which is applied to a device and includes: A receiving unit, configured to receive the operator quantity and device function identifier of the operator to be merged sent by the host, where the device function identifier is obtained by the host based on the operator identifier of the operator to be merged; A loading unit, configured to load a device function corresponding to the operator to be fused based on the device function identifier, wherein the device function is pre-developed based on the operator to be fused, and is compiled and loaded into the memory of the device; A fusion unit is used to fuse the operators to be fused based on the number of operators and the device function.

[0012] The present invention also provides an operator fusion device, which is applied to a host and includes: A determination unit, used to determine the number of operators and operator identifiers of operators to be fused, and determine a device function identifier based on the operator identifier; A sending unit is used to send the device function identifier and the number of operators to a device, and the device is used to load the device function corresponding to the operator to be merged based on the device function identifier, and merge the operator to be merged based on the operator number and the device function. The device function is pre-developed based on the operator to be merged, and is compiled and loaded into the memory of the device.

[0013] The present invention also provides an operator fusion system, comprising a host and a device; The host is used to determine the operator quantity and operator identifier of the operator to be fused, determine the device function identifier based on the operator identifier, and send the device function identifier and the operator quantity to the device; The device is used to load the device function corresponding to the operator to be fused based on the device function identifier, and fuse the operator to be fused based on the operator quantity and the device function. The device function is pre-developed based on the operator to be fused and compiled and loaded into the memory of the device.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the operator fusion method described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the operator fusion method described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned operator fusion methods.

[0017] The operator fusion method, system, electronic device and storage medium provided by the present invention can realize effective transmission and reception of information by receiving the operator number and device function identifier of the operator to be fused sent by the host, thereby providing necessary data support for subsequent operator fusion; based on the received device function identifier, the device can load the corresponding device function from the memory. Since the device function is pre-developed based on the operator to be fused, it can better adapt to the hardware characteristics and optimize the computing performance. Moreover, the pre-compilation and loading of the device function helps to reduce the computing burden of the operator at runtime, thereby improving the efficiency of operator fusion; according to the received operator number and the loaded device function, the device can automatically fuse the operators to be fused, without the need for developers to manually write a large amount of fusion code for the operators to be fused, thereby reducing the possibility of human intervention and errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is one of the flow charts of the operator fusion method provided by the present invention; Figure 2 It is a schematic diagram of a process for determining a device function identifier provided by the present invention; Figure 3 This is the second flow chart of the operator fusion method provided by the present invention; Figure 4 It is a structural schematic diagram of the operator fusion system provided by the present invention; Figure 5 It is a flowchart of the kernel program execution provided by the present invention; Figure 6 It is one of the structural schematic diagrams of the operator fusion device provided by the present invention; Figure 7 This is the second structural schematic diagram of the operator fusion device provided by the present invention; Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] As AI application scenarios continue to expand, the types of operators are becoming increasingly diverse, from basic matrix operations to complex convolutions and pooling. The effective fusion of these operators has become an important way to optimize the performance of AI models. Operator fusion is an important optimization technology in deep learning and other computationally intensive tasks. Its core idea is to combine multiple consecutive operations or operators to reduce the storage and reloading of intermediate results, thereby reducing the need for data transmission from memory to computing units, reducing computing and memory overhead, and thus improving computing efficiency.

[0022] However, since there are a variety of different operators in the AI ​​field and the combinations between these operators are complex and diverse, manually writing fusion code for each possible operator combination is not only time-consuming and laborious, but also prone to errors and omissions. In order to reduce manual work to a certain extent, a method of automatic operator fusion at compile time is proposed. This method uses the compiler or optimization tool to identify and merge multiple operators in the calculation graph to achieve operator fusion, but the optimization capability of the compiler is limited and its applicable scenarios are limited. For example, it cannot handle operator combinations that change dynamically at runtime, and the dependencies and context information between operators are crucial for correct fusion. In the absence of complete running information, it is difficult for the compiler to accurately determine which operators can be safely fused without introducing errors or performance degradation.

[0023] In addition, whether you are manually writing fusion code to implement operator fusion or performing operator fusion through the compiler, a large number of fusion operators will be generated during compilation and stored in the dynamic library, which will take up a lot of storage space and even exceed the file size limit of the dynamic library, resulting in the inability to generate more fusion operator combinations, making it impossible to fully utilize some optimization potential, affecting the reasoning and training efficiency of the model.

[0024] In this regard, the present invention proposes an operator fusion method, which develops different pointwise operator operations into device functions, and transmits the operator number and device function identifier of the operator to be fused to the device through the host during operation, and the device executes the corresponding device functions in sequence according to the operator number and device function identifier, so as to achieve automatic fusion of operators, thereby overcoming the above defects. It should be understood that pointwise operators refer to operators that operate element by element or point by point, and different pointwise operators refer to various types of pointwise operators, such as ReLU (linear rectification function), Dropout (random dropout), GeLU (Gaussian error linear unit), etc. Here, ReLU and GeLU are both activation functions, and Dropout is an operator used to reduce overfitting.

[0025] It should be noted that the above-mentioned device refers to a hardware component that performs a specific computing task, which may include an artificial intelligence (AI) chip, such as a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a neural network processor (NPU), a tensor processing unit (TPU), etc. The above-mentioned host refers to a computer or server responsible for overall control and management in a system, which may include a main control chip, which refers to a chip responsible for overall control and management in a system, which can communicate with other components (including AI chips), coordinate their operations, and perform higher-level tasks. For example, the main control chip may be a central processing unit (CPU), a microcontroller unit (MCU), a microprocessor (MPU), etc., and the embodiments of the present invention do not specifically limit this.

[0026] In deep learning applications, the model is usually run on the host side, but the specific computing tasks (including the execution of operator instructions) are completed on the device side. Therefore, the present invention realizes operator fusion through the mutual cooperation between the host and the device. The host provides an interface for communicating with the device side, which is responsible for managing and scheduling the resources on the device side to achieve specific computing tasks; the device communicates with the host side through the interface and receives instructions and data from the host side to perform computing tasks.

[0027] Figure 1This is one of the flow charts of the operator fusion method provided by the present invention, such as Figure 1 As shown, the method is applied to a device, and the method includes: Step 110: receiving the operator quantity and device function identifier of the operator to be fused sent by the host, where the device function identifier is obtained by the host based on the operator identifier of the operator to be fused.

[0028] Specifically, the operator fusion method provided by the embodiment of the present invention can be applied to a device. When implementing operator fusion, the host first determines the number of operators and operator identifiers of the operators to be fused according to the actual needs of the model. Here, in deep learning or machine learning models, operators refer to functions that perform specific mathematical operations or logical operations. Operators to be fused refer to those that can be combined or merged to optimize computing efficiency and resource utilization. For example, the operators to be fused can be various types of pointwise operators. In neural networks, continuous linear transformations and activation functions can be fused into a single operator to reduce memory access and computational complexity.

[0029] The above operator number refers to the total number of operators to be fused. By sending the operator number of operators to be fused to the device, the device can know how many operators need to be processed so that it can make appropriate preparations for the fusion of these operators. The operator ID refers to the unique identifier of each operator in the operators to be fused, which is used to distinguish different operators. The operator ID is generated during the construction and compilation of the model and is used to reference a specific operator at runtime. For example, the operator ID can be the operator name, the operator's unique code or ID (Identity, identity number), etc.

[0030] It is understandable that the host can determine the number of operators and operator identifiers of the operators to be fused through the model framework. Here, the model framework is a high-level tool for building and training deep learning models. They provide a rich application programming interface (API) and library, allowing developers to easily define model structures, perform training and reasoning operations. Specifically, when developers define a model in the model framework, they will explicitly specify each layer and each operator contained in the model. These layers and operators are organized in a specific order and together constitute the overall structure of the model. When the model is compiled and ready to be deployed on the hardware, the model framework will send the number of operators and operator identifiers of the operators to be fused in the model to the host.

[0031] After the host determines the number of operators and operator identifiers of the operators to be fused, it can determine the device function identifier of each operator in the operators to be fused based on these operator identifiers, and transmit the determined device function identifier and number of operators to the device through the PCIe (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard) bus, so that the device can execute the corresponding operator instructions. Here, the device function identifier refers to a unique identifier of the device function determined according to the operator identifier, which is used to call a specific device function on the hardware. For example, the device function identifier can be a device function pointer, or a string used to uniquely identify the device function, or the operator identifier can be directly used as the device function identifier, and the embodiment of the present invention does not specifically limit this.

[0032] The above-mentioned device function refers to a function developed in advance for the operator in the model and running on the device. It will be pre-compiled and loaded into the memory of the device to perform the computing tasks corresponding to the operator. It should be understood that each different operator operation in the model can be developed into a corresponding device function in advance according to the model structure, and these device functions can be compiled and loaded into the memory of the device. The device function can make full use of the parallel computing capabilities of the hardware and efficiently perform complex mathematical and logical operations. By developing device functions for operators, optimization at the hardware level can be achieved to improve computing efficiency.

[0033] Exemplarily, a mapping table can be set between the model framework and the hardware to map the operator identifier to the corresponding device function identifier. The mapping table can be generated during the model compilation phase, and it records the device function corresponding to each operator on the hardware. When the host determines the operator identifier of the operator to be fused, the mapping table can be used to find the device function identifier corresponding to the operator identifier. After finding the corresponding device function identifier, the host will send the identifier to the device so that the device can load and execute the corresponding device function.

[0034] In an embodiment of the present invention, the host can accurately determine the number of operators and device function identifiers of the operators to be fused according to the actual needs of the model, ensuring the accuracy and real-time nature of the information, and enabling the device to dynamically fuse the operators according to the number of operators and device function identifiers sent by the host, which is more flexible and efficient.

[0035] Step 120: Based on the device function identifier, load the device function corresponding to the operator to be fused, wherein the device function is pre-developed based on the operator to be fused, and is compiled and loaded into the memory of the device.

[0036] Specifically, according to the received device function identifiers of the operators to be fused, the device can load the device functions corresponding to these operators from the memory. Specifically, when the device receives the device function identifiers of the operators to be fused, it will first parse these identifiers to determine which device functions need to be loaded. There are usually one or more memory areas inside the device for storing device functions, and the device function identifiers will be mapped to specific addresses in these memory areas. By looking up the memory mapping table, the device can find the memory address corresponding to the device function identifier. After determining the memory address, the device will start loading the device function from this address. Here, loading the device function means reading the function code and data from the memory on the device side and loading it into the computing unit for execution.

[0037] It is understandable that during the development phase, corresponding device functions will be developed for each operator in the model, not limited to the operators to be fused. Developers can design corresponding algorithms and write function codes based on the characteristics and computing requirements of each operator. These function codes are compiled and loaded into the memory of the device through the compiler to obtain the corresponding device functions. It should be understood that device functions are functions designed specifically for operators executed on devices. These functions can make full use of the device's parallel processing capabilities, cache, and other hardware features to achieve efficient computing performance. In addition, by loading device functions into the device memory, the device can directly execute these functions, thereby completing the computing tasks of the operators to be fused and realizing automatic fusion of operators.

[0038] Step 130: Fusing the operators to be fused based on the number of operators and the device function.

[0039] Specifically, the device can use a loop-based mechanism to perform operator fusion. According to the number of operators to be fused received, the device can control a loop, and the number of loops is equal to the number of operators to be fused. In each loop, the device will call the device function of the current operator in turn, and execute the device function to perform calculations on the data to be processed. In this way, through the iteration of the loop, the device can sequentially execute the calculation tasks of all operators to be fused, and realize the fusion of operators to be fused.

[0040] It is understandable that although the device does not directly generate new fusion operator code or instructions, through control loops and function calls, the device can continuously execute the computing tasks of multiple operators, logically realize the fusion of operators, reduce the storage and reloading of intermediate results, avoid unnecessary memory access and data transfer, and thus improve computing performance.

[0041] The method provided by the embodiment of the present invention can realize effective transmission and reception of information by receiving the operator number and device function identifier of the operator to be fused sent by the host through the device, so as to provide necessary data support for subsequent operator fusion; based on the received device function identifier, the device can load the corresponding device function from the memory. Since the device function is pre-developed based on the operator to be fused, it can better adapt to the hardware characteristics and optimize the computing performance. Moreover, the pre-compilation and loading of the device function helps to reduce the computing burden of the operator at runtime, thereby improving the efficiency of operator fusion; according to the received operator number and the loaded device function, the device can automatically fuse the operators to be fused, without the need for developers to manually write a large amount of fusion code for the operators to be fused, thereby reducing the possibility of manual intervention and errors.

[0042] Based on the above embodiments, Figure 2 is a schematic diagram of the process of determining the device function identifier provided by the present invention, such as Figure 2 As shown, the step of determining the device function identifier includes: Step 210: Receive the operator identifier sent by the host.

[0043] Specifically, after determining the operator identifier of the operator to be fused, the host can send a query request to the device through an application program interface, such as a runtime API (runtime application program interface), and the request includes the operator identifier of the operator to be fused. It should be understood that the host can generate a query request after uniformly encapsulating the operator identifier of each operator in the operator to be fused, or it can generate a query request based on the operator identifier of each operator in the operator to be fused, and send the query request to the device through the interface in turn, and the embodiment of the present invention does not specifically limit this. After receiving the query request sent by the host, the device can obtain the operator identifier of the operator to be fused by parsing the query request.

[0044] Step 220: Based on the operator identifier, search for a device function pointer corresponding to the operator identifier from an operator function mapping table, where the operator function mapping table is a mapping relationship table between operator identifiers and device function pointers pre-constructed on the device.

[0045] Specifically, according to the operator identifier, the device can search for the device function pointer corresponding to the representation from the operator function mapping table. Here, the device can quickly find the device function pointer that matches the operator identifier from the operator function mapping table by traversing or using an index mechanism, and the pointer points to the actual memory address of the device function used to execute the operator on the device. It should be understood that the device function pointer is a pointer to a specific device function (i.e., a function executed on hardware), which allows the device to call the corresponding device function to execute the computing task of the operator corresponding to the function.

[0046] The operator function mapping table is a data structure used to store the mapping relationship between operator identifiers and device function pointers on the device. The mapping table can be pre-built during the device configuration phase and stored in the non-volatile memory of the device so that it can be used directly after the device is powered on. Specifically, the operator function mapping table can be constructed by the following steps: first, obtain the operator identifiers of all operators in the model and develop corresponding device functions for each operator; then, establish a mapping relationship between the operator identifier of each operator and its corresponding device function pointer, and store these mapping relationships in the mapping table; finally, deploy the mapping table to the device.

[0047] Step 230: Return the device function pointer to the host as the device function identifier corresponding to the operator to be fused.

[0048] Specifically, after the device finds the device function pointer corresponding to the identifier of each operator in the operator to be fused, the pointer can be returned to the host as the device function identifier of each operator through the runtime API. It should be understood that in a heterogeneous system, the host is mainly responsible for the scheduling and allocation of tasks. It can directly determine the operators to be fused through the model framework, while the device focuses on executing computing tasks. The device function corresponding to each operator is stored in the dynamic memory of the device. During the compilation period, the device does not know which operators to be fused, but the device function is pre-compiled and loaded in the memory of the device. Therefore, when the program is running, the operator to be fused can be determined by the host first, so as to perform more global scheduling and optimization, and the operator identifier of each operator in the operator to be fused is sent to the device, and the device finds the device function pointer corresponding to each operator in turn and returns it to the host, and then the host uniformly transmits the device function pointers of all operators in the operator to be fused and the number of operators to the device, so that the device can load the device function from the dynamic memory and perform the corresponding operator operation. This batch transmission method reduces the number and overhead of data transmission and improves the efficiency of data transmission.

[0049] It should be noted that if the device function of the operator is stored in the static memory of the device, the device can directly find the device function pointer of the operator from the operator function mapping table according to the operator identifier, and load the corresponding device function from the static memory according to the pointer.

[0050] Based on any of the above embodiments, in step 120, loading the device function corresponding to the operator to be fused based on the device function identifier includes: Step 121, when the device function identifier is a device function pointer, loading the device function corresponding to the operator to be fused based on the device function pointer; Specifically, when the device function is identified as a device function pointer, after receiving the device function pointer of each operator in the operator to be fused transmitted by the host, the device can first parse the received device function pointers of these operators. For each operator, its device function pointer is a memory address that points to the device function of the operator pre-compiled and stored in the device memory. The device uses this memory address to access the corresponding memory area and obtain the binary code of the device function. After obtaining the binary code of the device function, the device can load it into the execution environment, for example, copy the code to the appropriate execution cache or memory, and prepare the execution environment (such as setting registers, etc.). When the device function is loaded into the execution environment, the device can execute the function to complete the calculation task of the operator to be fused.

[0051] Step 122, when the device function identifier is not a device function pointer, based on the device function identifier, obtain the device function pointer corresponding to the operator to be fused, and apply the device function pointer to load the device function corresponding to the operator to be fused.

[0052] Specifically, considering that the device function is pre-compiled and loaded into the memory of the device, and the host side does not have data about the device function pointer, in the above embodiment, the host sends the operator identifier of the operator to be fused to the device to obtain the memory address of the device function of these operators in the device (i.e., the function pointer in the device). The embodiment of the present invention provides another implementation method, in which the host can directly send the operator identifier as the device function identifier to the device. After receiving the operator identifier of the operator to be fused, the device can find the corresponding device function pointer from the operator function mapping table, so that the corresponding device function can be loaded according to the device function pointer.

[0053] Based on any of the above embodiments, step 130 specifically includes: Step 131, obtaining data to be processed, wherein the data to be processed is divided into a plurality of data blocks; Specifically, data to be processed refers to data that needs to be calculated or processed to generate results. In the fields of deep learning, machine learning, big data processing, etc., these data can come from various data sources, such as images, text, sensor data, etc., and they need to go through a series of calculation steps (such as convolution, pooling, activation, etc.) to extract features or generate prediction results.

[0054] It is understandable that when computing data to be processed on a device, the data to be processed is usually divided into multiple data blocks, so that it can be loaded and processed in batches, effectively managing memory usage. Moreover, dividing the data into multiple data blocks can be assigned to different computing units for simultaneous processing, thereby improving computing efficiency.

[0055] Step 132, traverse each data block in the data to be processed, and calculate the currently traversed data block based on the number of operators and the device function until the calculation of each data block is completed. The device function is used to execute the calculation task corresponding to the operator to be fused.

[0056] Specifically, since the data to be processed is divided into multiple data blocks, the device function corresponding to the operator to be fused is executed to perform calculation processing on the data to be processed, which is essentially to perform calculation processing on each data block, and the calculation operation for each data block is the same. For any of the data blocks, the device can use a loop control flow to perform calculation processing on the data block. According to the number of operators, the device can control a loop, and the number of loops is equal to the number of operators to be fused. In each loop, the device function of the current operator is called in turn, and the device function is executed to calculate the current data block until the calculation of the device functions of all operators in the operator to be fused is completed, and the calculation result of the current data block can be obtained. Subsequently, the calculation processing can be started on the next data block until the calculation of all data blocks is completed.

[0057] It can be understood that the embodiment of the present invention completes the calculation of each data block by executing the device function, which actually realizes the parallel execution of operators and the integration of computing tasks at the hardware level. By calling the corresponding device functions in sequence according to the execution order of the operators to process the data blocks, the data exchange and intermediate result storage between them can be reduced, thereby reducing the computing overhead. Therefore, by executing the device function to complete the calculation of each data block, it can be considered that operator fusion is achieved and the overall computing performance is improved.

[0058] Based on any of the above embodiments, in step 132, the calculation of the currently traversed data block based on the operator quantity and the device function includes: Step 1321, determining the coordinates of the currently traversed data block, and loading the data block based on the coordinates; Specifically, in the parallel computing framework of the device, the data to be processed is usually divided into multiple data blocks, which are assigned to multiple threads for parallel processing. Each thread is pre-assigned and mapped to which data block to process. Each thread has a unique number, so the coordinates of the data block that the current thread is responsible for processing can be determined based on the number. Here, the coordinates of the data block refer to the position identifier when the data is divided, usually including row coordinates and column coordinates, through which the current data block can be located in the data to be processed.

[0059] After determining the coordinates of the data block, the corresponding data block can be loaded from the global memory (or any other storage location) according to the coordinates. This process involves calculating the starting address of the data block in the memory and then reading the data of the data block from this address. The specific implementation depends on the storage layout and access mode of the data.

[0060] Step 1322: Based on the number of operators, execute the device function corresponding to each operator in the operators to be fused in sequence to calculate the data block and obtain a calculation result.

[0061] Specifically, according to the number of operators (i.e., the number of operators to be fused), the device function corresponding to each operator is executed in turn to perform different computing tasks on the data block, such as addition, multiplication, nonlinear transformation, etc., to obtain the calculation result. Here, the calculation result refers to the result obtained after the data block is processed by all the operators to be fused.

[0062] It is understandable that the device will loop through and call the corresponding device functions in sequence according to the number of operators to ensure that all operators that need to be fused can be executed correctly and sequentially. After receiving the number of operators to be fused, the device can know the number of operators to be executed, so as to allocate sufficient resources (such as cache, registers, etc.) in advance to reduce resource switching and waiting time during execution. In addition, during the execution process, the number of operators can also be used to check whether all expected device functions are called correctly. If the actual number of calls does not match the expectation, it may indicate a configuration error or an abnormality in execution, and the corresponding error handling mechanism can be triggered at this time.

[0063] Based on any of the above embodiments, Figure 3 This is the second flow chart of the operator fusion method provided by the present invention. The method is applied to a host and includes: Step 310, determining the number of operators and operator identifiers of operators to be fused, and determining a device function identifier based on the operator identifiers; Step 320, sending the device function identifier and the number of operators to the device, the device is used to load the device function corresponding to the operator to be fused based on the device function identifier, and fuse the operator to be fused based on the operator number and the device function, the device function is pre-developed based on the operator to be fused, and is compiled and loaded into the memory of the device.

[0064] Specifically, the operator fusion method provided by the embodiment of the present invention can be applied to the host. The host can accurately determine the number of operators and operator identifiers of the operators to be fused according to the actual needs of the model, and determine the device function identifier of each operator in the operators to be fused according to the operator identifier. Subsequently, the host can transmit the device function identifiers and the number of operators of these operators to the device through the PCIe bus. According to the number of operators received, the device can control a loop, and the number of loops is equal to the number of operators. In each loop, the device loads the corresponding device function from the memory according to the device function identifier of each operator received, and executes the device function to calculate the data to be processed. Through loop iteration, the device can sequentially execute the calculation tasks of all operators to be fused, thereby realizing the fusion of the operators to be fused.

[0065] The method provided by the embodiment of the present invention can realize effective transmission and reception of information by receiving the operator number and device function identifier of the operator to be fused sent by the host through the device, so as to provide necessary data support for subsequent operator fusion; based on the received device function identifier, the device can load the corresponding device function from the memory. Since the device function is pre-developed based on the operator to be fused, it can better adapt to the hardware characteristics and optimize the computing performance. Moreover, the pre-compilation and loading of the device function helps to reduce the computing burden of the operator at runtime, thereby improving the efficiency of operator fusion; according to the received operator number and the loaded device function, the device can automatically fuse the operators to be fused, without the need for developers to manually write a large amount of fusion code for the operators to be fused, thereby reducing the possibility of manual intervention and errors.

[0066] Based on any of the above embodiments, Figure 4 is a schematic diagram of the structure of the operator fusion system provided by the present invention, such as Figure 4 As shown, the system includes a host 410 and a device 420; The host 410 is used to determine the operator quantity and operator identifier of the operator to be fused, determine the device function identifier based on the operator identifier, and send the device function identifier and the operator quantity to the device 420; The device 420 is used to load the device function corresponding to the operator to be fused based on the device function identifier, and fuse the operator to be fused based on the operator quantity and the device function. The device function is pre-developed based on the operator to be fused and compiled and loaded into the memory of the device 420.

[0067] Specifically, the system can be applied to automatically fuse pointwise operators. For example, taking GPU as a device, in the parallel computing framework of GPU, kernel programs are usually used to process data. Here, kernel programs refer to functions that are executed in parallel on the GPU to accelerate large-scale data processing and machine learning tasks. Before executing the kernel program, different pointwise operator operations can be developed into device functions, compiled and loaded into the GPU's memory. When running the kernel program, the device function pointers corresponding to the pointwise operators to be fused are transferred from the host to the GPU through the PCIe bus. The GPU executes the device functions in sequence to achieve automatic fusion of pointwise operators. It should be noted that Figure 4 The func1, func2, func3, and func4 in the device memory are different device functions stored in the memory. The main process of the system to achieve automatic fusion of pointwise operators is as follows: Step S1, the host determines the number and identification of pointwise operators that need to be fused. For example, three operators need to be fused, namely ReLU, Dropout and GeLU; Step S2, the host obtains the memory addresses of the device functions corresponding to the operators to be fused in the device (that is, the device function pointers on the device side) from the device in sequence through the runtime API; Step S3, the host transmits the acquired device function pointer and operator quantity to the device through the PCIe bus; In step S4, the device executes corresponding device functions in sequence through the kernel program according to the transmitted device function pointer and the number of operators, thereby realizing automatic fusion of pointwise operators.

[0068] Figure 5 It is a flowchart of the kernel program execution provided by the present invention, such as Figure 5 As shown in the figure, the specific process of implementing operator fusion by executing device functions includes: Step S41, determining the coordinates of the data block currently being processed; Step S42, loading the corresponding data block according to the coordinates of the data block; Step S43, loading the corresponding device function according to the device function pointer transmitted from the host; Step S44, executing the loaded device function to calculate the data block; Step S45, repeating the above steps S43 to S44 until the calculation of all device functions is completed; Step S46, storing the calculation results of the processed data blocks; Step S47, repeat the above steps S41 to S46 until the calculation of all data blocks is completed.

[0069] The embodiment of the present invention realizes the automatic fusion of pointwise operators at runtime, which can greatly reduce the development workload of operator fusion. Moreover, the embodiment of the present invention does not need to manually write operator fusion code, does not generate new fusion operators, and thus does not increase the size of the dynamic library, thereby realizing the fusion of all operators to be fused, thereby optimizing model performance.

[0070] Based on any of the above embodiments, Figure 6 is one of the structural schematic diagrams of the operator fusion device provided by the present invention, such as Figure 6 As shown, the device is applied to a device, and the device includes: The receiving unit 610 is used to receive the operator quantity and device function identifier of the operator to be merged sent by the host, where the device function identifier is obtained by the host based on the operator identifier of the operator to be merged; A loading unit 620 is used to load the device function corresponding to the operator to be fused based on the device function identifier, where the device function is pre-developed based on the operator to be fused and is compiled and loaded into the memory of the device; The fusion unit 630 is used to fuse the operators to be fused based on the number of operators and the device function.

[0071] The apparatus provided by the embodiment of the present invention can realize effective transmission and reception of information by receiving the operator number and device function identifier of the operator to be fused sent by the host through the device, thereby providing necessary data support for subsequent operator fusion; based on the received device function identifier, the device can load the corresponding device function from the memory. Since the device function is pre-developed based on the operator to be fused, it can better adapt to the hardware characteristics and optimize the computing performance. In addition, the pre-compilation and loading of the device function helps to reduce the computing burden of the operator at runtime, thereby improving the efficiency of operator fusion; according to the received operator number and the loaded device function, the device can automatically fuse the operators to be fused, without the need for developers to manually write a large amount of fusion code for the operators to be fused, thereby reducing the possibility of manual intervention and errors.

[0072] Based on any of the above embodiments, the fusion unit 630 includes: An acquisition subunit, used for acquiring data to be processed, wherein the data to be processed is divided into a plurality of data blocks; A computing subunit is used to traverse each data block in the data to be processed, and calculate the currently traversed data block based on the number of operators and the device function until the calculation of each data block is completed. The device function is used to execute the computing task corresponding to the operator to be fused.

[0073] Based on any of the above embodiments, the computing subunit is specifically used for: Determine the coordinates of the currently traversed data block, and load the data block based on the coordinates; Based on the number of operators, the device function corresponding to each operator in the operators to be fused is executed in sequence to calculate the data block and obtain a calculation result.

[0074] Based on any of the above embodiments, the loading unit 620 is specifically used for: In the case where the device function identifier is a device function pointer, loading the device function corresponding to the operator to be fused based on the device function pointer; In the case that the device function identifier is not a device function pointer, the device function pointer corresponding to the operator to be fused is obtained based on the device function identifier, and the device function pointer is applied to load the device function corresponding to the operator to be fused.

[0075] Based on any of the above embodiments, the device further includes an identification determination unit, and the identification determination unit is used to: Receiving an operator identifier sent by the host; Based on the operator identifier, searching for a device function pointer corresponding to the operator identifier from an operator function mapping table, wherein the operator function mapping table is a mapping relationship table between the operator identifier and the device function pointer pre-constructed on the device; The device function pointer is returned to the host as a device function identifier corresponding to the operator to be fused.

[0076] Based on any of the above embodiments, Figure 7 This is the second structural schematic diagram of the operator fusion device provided by the present invention. The device is applied to a host and includes: A determination unit 710, configured to determine the number of operators and operator identifiers of operators to be fused, and determine a device function identifier based on the operator identifiers; The sending unit 720 is used to send the device function identifier and the number of operators to the device. The device is used to load the device function corresponding to the operator to be fused based on the device function identifier, and fuse the operator to be fused based on the number of operators and the device function. The device function is pre-developed based on the operator to be fused and compiled and loaded into the memory of the device.

[0077] The apparatus provided by the embodiment of the present invention can realize effective transmission and reception of information by receiving the operator number and device function identifier of the operator to be fused sent by the host through the device, thereby providing necessary data support for subsequent operator fusion; based on the received device function identifier, the device can load the corresponding device function from the memory. Since the device function is pre-developed based on the operator to be fused, it can better adapt to the hardware characteristics and optimize the computing performance. In addition, the pre-compilation and loading of the device function helps to reduce the computing burden of the operator at runtime, thereby improving the efficiency of operator fusion; according to the received operator number and the loaded device function, the device can automatically fuse the operators to be fused, without the need for developers to manually write a large amount of fusion code for the operators to be fused, thereby reducing the possibility of manual intervention and errors.

[0078] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the operator fusion method, which is applied to the device, and the method includes: receiving the operator quantity and device function identifier of the operator to be fused sent by the host, the device function identifier is obtained by the host based on the operator identifier of the operator to be fused; based on the device function identifier, loading the device function corresponding to the operator to be fused, the device function is pre-developed based on the operator to be fused, and is compiled and loaded into the memory of the device; based on the operator quantity and the device function, fusing the operator to be fused.

[0079] The processor 810 can also call the logic instructions in the memory 830 to execute the operator fusion method, which is applied to the host. The method includes: determining the number of operators and operator identifiers of the operators to be fused, and determining the device function identifier based on the operator identifier; sending the device function identifier and the number of operators to the device, the device is used to load the device function corresponding to the operator to be fused based on the device function identifier, and based on the operator number and the device function, fuse the operators to be fused, the device function is pre-developed based on the operator to be fused, and is compiled and loaded into the memory of the device.

[0080] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0081] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the operator fusion method provided by the above methods. The method is applied to a device, and the method includes: receiving the number of operators and device function identifiers of operators to be fused sent by a host, wherein the device function identifier is obtained by the host based on the operator identifier of the operator to be fused; based on the device function identifier, loading the device function corresponding to the operator to be fused, wherein the device function is pre-developed based on the operator to be fused and compiled and loaded into the memory of the device; based on the number of operators and the device function, fusing the operators to be fused.

[0082] When the computer program is executed by the processor, the computer can also execute the operator fusion method provided by the above methods, which is applied to a host, and includes: determining the operator quantity and operator identifier of the operator to be fused, and determining the device function identifier based on the operator identifier; sending the device function identifier and the operator quantity to the device, the device is used to load the device function corresponding to the operator to be fused based on the device function identifier, and fuse the operator to be fused based on the operator quantity and the device function, the device function is pre-developed based on the operator to be fused, and is compiled and loaded into the memory of the device.

[0083] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the operator fusion method provided by the above-mentioned methods, the method being applied to a device, the method comprising: receiving the number of operators to be fused and a device function identifier of the operator to be fused sent by a host, the device function identifier being obtained by the host based on the operator identifier of the operator to be fused; based on the device function identifier, loading the device function corresponding to the operator to be fused, the device function being pre-developed based on the operator to be fused, and compiled and loaded into the memory of the device; based on the number of operators and the device function, fusing the operators to be fused.

[0084] When the computer program is executed by the processor, it is implemented to execute the operator fusion method provided by the above methods. The method is applied to a host, and the method includes: determining the operator quantity and operator identifier of the operator to be fused, and determining the device function identifier based on the operator identifier; sending the device function identifier and the operator quantity to the device, the device is used to load the device function corresponding to the operator to be fused based on the device function identifier, and fuse the operator to be fused based on the operator quantity and the device function, the device function is pre-developed based on the operator to be fused, and is compiled and loaded into the memory of the device.

[0085] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0086] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An operator fusion method, characterized in that: The method is applied to a device, and the method comprises: receiving the operator quantity and device function identifier of the operator to be fused sent by the host, where the device function identifier is determined by the host based on the operator identifier of the operator to be fused; Based on the device function identifier, load the device function corresponding to the operator to be fused, where the device function is pre-developed based on the operator to be fused, and is compiled and loaded into the memory of the device; The operators to be fused are fused based on the number of operators and the device function.

2. The operator fusion method according to claim 1, characterized in that: The fusing the operators to be fused based on the number of operators and the device function includes: Acquire data to be processed, where the data to be processed is divided into a plurality of data blocks; Each data block in the data to be processed is traversed, and based on the number of operators and the device function, the currently traversed data block is calculated until the calculation of each data block is completed, and the device function is used to execute the calculation task corresponding to the operator to be fused.

3. The operator fusion method according to claim 2, characterized in that: The calculating of the currently traversed data block based on the operator quantity and the device function includes: Determine the coordinates of the currently traversed data block, and load the data block based on the coordinates; Based on the number of operators, the device function corresponding to each operator in the operators to be fused is executed in sequence to calculate the data block and obtain a calculation result.

4. The operator fusion method according to claim 1, characterized in that: The step of loading the device function corresponding to the operator to be fused based on the device function identifier includes: In the case where the device function identifier is a device function pointer, loading the device function corresponding to the operator to be fused based on the device function pointer; In the case that the device function identifier is not a device function pointer, the device function pointer corresponding to the operator to be fused is obtained based on the device function identifier, and the device function pointer is applied to load the device function corresponding to the operator to be fused.

5. The operator fusion method according to any one of claims 1 to 4, characterized in that: The step of determining the device function identifier includes: Receiving an operator identifier sent by the host; Based on the operator identifier, searching for a device function pointer corresponding to the operator identifier from an operator function mapping table, wherein the operator function mapping table is a mapping relationship table between the operator identifier and the device function pointer pre-constructed on the device; The device function pointer is returned to the host as a device function identifier corresponding to the operator to be fused.

6. An operator fusion method, characterized in that: The method is applied to a host, and the method comprises: Determine the number of operators and operator identifiers of operators to be fused, and determine the device function identifier based on the operator identifier; The device function identifier and the number of operators are sent to a device, and the device is used to load the device function corresponding to the operator to be fused based on the device function identifier, and fuse the operator to be fused based on the operator number and the device function. The device function is pre-developed based on the operator to be fused, and is compiled and loaded into the memory of the device.

7. An operator fusion system, characterized in that: Including host and equipment; The host is used to determine the operator quantity and operator identifier of the operator to be fused, determine the device function identifier based on the operator identifier, and send the device function identifier and the operator quantity to the device; The device is used to load the device function corresponding to the operator to be fused based on the device function identifier, and fuse the operator to be fused based on the operator quantity and the device function. The device function is pre-developed based on the operator to be fused and compiled and loaded into the memory of the device.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the operator fusion method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the operator fusion method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the operator fusion method according to any one of claims 1 to 6 is implemented.

Images