A method for accelerating AI data processing based on heterogeneous hardware

Abstract

The present invention discloses an AI data processing acceleration method based on heterogeneous hardware, which belongs to the field of data processing and computing technology, and specifically comprises: configuring an API interface, setting an operator information library for hardware, setting operator information of a MAP operation, loading the operator information library and a CPU data processing pipeline, extracting the MAP operation in the pipeline to establish an IR tree, traversing the IR tree, using the operator information library to search by a strategic search method to screen out migratable nodes, serializing the migratable nodes, establishing a migratable node operation sequence and transferring it into a tree adapter, migrating the migratable nodes to different heterogeneous acceleration hardware for execution according to a load balancing strategy and deserialization, and at the same time using the tree adapter to delete the migratable nodes in the IR tree, marking the updated IR tree as an execution tree, and the CPU executing the MAP operation in parallel according to the execution tree. The present invention can automatically and effectively integrate and utilize heterogeneous hardware resources for data processing acceleration.

Description

Technical Field

[0001] The present invention relates to the field of data processing and computing technology, and in particular to an AI data processing acceleration method based on heterogeneous hardware. Background Art

[0002] The AI ​​data processing process includes steps such as data collection, storage, preprocessing, model training, and reasoning optimization. Existing AI data processing acceleration solutions are mostly used in the model training stage, using methods such as parameter adjustment and improved algorithms to improve data processing efficiency. In the actual data processing process, the workload required for data preprocessing may often account for more than half of the total workload. In order to increase the data processing speed, heterogeneous hardware such as image processing units (GPUs) and neural processing units (NPUs) are currently used to accelerate AI operations.

[0003] In the prior art, the data processing part usually uses the CPU to load and preprocess the data set, and then inputs the processed data in batches into acceleration hardware such as GPU, NPU or TPU for network computing.

[0004] However, with the growth of data scale in multimodal fields, the CPU processing speed of existing technologies will be bottlenecked when facing large amounts of data, and some existing technologies, such as the tf.data.Dataset module provided by TensorFlow, only perform data preprocessing on the CPU, lacking inherent support for heterogeneous hardware. At the same time, existing technologies also have limitations in the coordination between multiple data operations and heterogeneous hardware. For example, the JAX library can realize accelerated computing on GPUs and TPUs, but only supports tensor operations of static shapes and only supports specific image processing operations. Therefore, the present invention proposes an AI data processing acceleration method based on heterogeneous hardware to solve the problems existing in the prior art. Summary of the invention

[0005] The purpose of the present invention is to provide an AI data processing acceleration method based on heterogeneous hardware to solve the following technical problems:

[0006] In the process of AI data processing, the existing technologies have obvious bottlenecks in CPU processing speed when facing large amounts of data. At the same time, some existing technologies only support data preprocessing on the CPU, but do not support the use of heterogeneous acceleration hardware, resulting in limited hardware resources available in the data preprocessing stage. In addition, the existing technologies also have limitations in the coordination between multiple data operations and heterogeneous hardware.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] A method for accelerating AI data processing based on heterogeneous hardware includes the following steps:

[0009] Configure an API interface, which includes a global API interface and a local API interface. The global API interface is used to uniformly process a single type of data set, and the local API interface is used to set the offload parameter of a specific MAP operation to True. The MAP operation represents a data preprocessing operation defined by a MAP function, including Decode, HWC2CHW, Normalize, and Resize.

[0010] Set corresponding operator information libraries for all hardware, including CPU and heterogeneous acceleration hardware, including GPU and NPU, set operator information of MAP operation, and complete loading of operator information library of corresponding hardware by writing operator information of MAP operation executable by hardware into IR file;

[0011] Load the CPU data processing pipeline, parse it, extract all MAP operations in the pipeline and build an IR tree, traverse all nodes in the IR tree, the nodes represent corresponding MAP operations, and use the operator information library and strategy search method to filter out migratable nodes, serialize the migratable nodes, build a migratable node operation sequence, and transfer it to the tree adapter in the form of a JSON file;

[0012] The load balancing strategy is used to dynamically allocate migratable nodes. The information of the migratable node operation sequence is extracted and deserialized according to the JSON file. The migratable nodes are migrated to different heterogeneous acceleration hardware, and a computational graph is established. The heterogeneous acceleration hardware executes the MAP operations corresponding to the migratable nodes according to the computational graph. At the same time, the tree adapter is used to delete the migratable nodes in the IR tree, complete the update of the IR tree, and mark the updated IR tree as the execution tree. The CPU executes the MAP operations corresponding to the remaining nodes according to the execution tree.

[0013] As a further solution of the present invention: the global API interface and the local API interface can be used alone or in combination. When used in combination, the priority of the local API interface is set higher than that of the global API interface.

[0014] As a further solution of the present invention: the loading process of the operator information library is performed synchronously with the loading process of the CPU data preprocessing pipeline.

[0015] As a further solution of the present invention: the specific process of establishing the migratable node operation sequence is:

[0016] The process of loading the CPU data processing pipeline is as follows: the original data enters the CPU through the API interface, and the CPU performs MAP operation processing on the original data to obtain a MAP operation sequence in the form of a pipeline;

[0017] Extract all MAP operations on the CPU data processing pipeline, build an IR tree, and store the MAP operations in the IR tree in the form of nodes;

[0018] Set Node_Offload_Pass to traverse the nodes in the IR tree. During the traversal, filter out all nodes that support migration operations according to the operator information loaded in the operator information library, mark them as nodes to be migrated, and use the strategy search method to filter all nodes to be migrated to obtain migratable nodes. Serialize the migratable nodes and establish a migratable node operation sequence.

[0019] As a further solution of the present invention: the specific process of obtaining the migratable nodes using the strategy search method is as follows:

[0020] Find the last node in the IR tree that does not support the migration operation, mark it as a non-migratable node, and prioritize the non-migratable node and all the nodes to be migrated before it. When the order between any node to be migrated and the non-migratable node is swapped without affecting the result of the original data preprocessing, the node to be migrated is marked as a migratable node.

[0021] As a further solution of the present invention, the specific process of dynamically allocating migratable nodes using the load balancing strategy is as follows:

[0022] By monitoring the key indicators of the operating status of the CPU data processing pipeline, it is determined whether to migrate the migratable node from the CPU to the heterogeneous acceleration hardware. The key indicators include the system CPU utilization and the average step time. When it is detected during the monitoring process that the system CPU utilization reaches 100%, and the average step time of the system processing is still greater than the set standard, the migratable node is migrated from the CPU to the heterogeneous acceleration hardware. Otherwise, the migration is not performed and the migratable node is still executed in the CPU.

[0023] As a further solution of the present invention: the serialization and deserialization processes are both executed in the CPU, and the serialization process is to convert the state information of the MAP operation into a binary byte stream.

[0024] As a further solution of the present invention: the process of executing MAP operation in heterogeneous acceleration hardware and the process of executing MAP operation in CPU are parallel processing processes. After the CPU executes the current MAP operation according to the execution tree, it loads the CPU data processing pipeline of subsequent original data.

[0025] Beneficial effects of the present invention:

[0026] The heterogeneous hardware acceleration method used in the present invention improves the efficiency of data preprocessing while achieving efficient use of hardware resources, solving the problem that traditional homogeneous computing platforms rely on CPUs to process data, which is difficult to meet the processing requirements of large-scale data sets, resulting in data bottlenecks and insufficient computing power. At the same time, the present invention is based on policy search and load balancing, and realizes automatic heterogeneity on the basis of heterogeneous hardware acceleration, avoiding the negative optimization problem caused by unreasonable migration in different environments, so that heterogeneous hardware acceleration has a high degree of automation and intelligence, and can work flexibly under different hardware configurations and different data set structures. At the same time, the present invention designs API interfaces for different scenarios, provides multi-domain data sets and column mapping support, and the relevant migration acceleration operations of the present invention will only be applied to specified data columns to ensure its accuracy and flexibility in actual application scenarios. Experimental data show that under the same hardware configuration conditions, the present invention achieves a data processing speed improvement of more than 77.6% compared to the optimal results of the traditional AI framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention will be further described below in conjunction with the accompanying drawings.

[0028] Figure 1 It is a schematic diagram of the process of the present invention;

[0029] Figure 2 It is the design framework of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] See also Figure 1-Figure 2 As shown, the present invention is an AI data processing acceleration method based on heterogeneous hardware, comprising the following steps:

[0032] According to the required related codes, the necessary libraries are imported into the system, and the ImageFolderDataset object is instantiated, which is used to read image data from the specified folder. Then a series of image preprocessing operations are determined according to the user definition, which will be applied to the dataset to complete the processing flow, thus completing the import of related presets.

[0033] Configure the API interface, which includes a global API interface and a local API interface. The global API interface is used to uniformly process a single type of data set, and the local API interface is used to set the offload parameter of a specific MAP operation to True. The MAP operation represents a data preprocessing operation defined by a MAP function, including Decode, HWC2CHW, Normalize, and Resiz.

[0034] In the field of artificial intelligence data processing, especially in tasks such as computer vision and natural language processing, data sets often have multiple column structures, and each column may contain different types of data such as images and text labels. This feature requires the data processing system to be able to flexibly process each column of data while ensuring efficiency and accuracy. Therefore, the present invention also provides a local API interface, which can be used to easily determine the range of data sets that support migration operations and the type of MAP operations based on user settings.

[0035] In the context of heterogeneous hardware acceleration design, the API of the MAP function has been updated and the "offload" flag has been added. Users can activate the migration operation between hardware by defining "offload=True". Fundamentally, the present invention improves parallelism by reasonably allocating MAP operations across hardware. At the API layer, when the migration operation is activated, the system will evaluate the various MAP operation requirements imported by the client to determine the feasibility of the migration. For example: in the example provided, Normalize and HWC2CHC are identified as migratable. And the global API interface and the local API interface support separate use or combined use. When used in combination, the local API interface is set with a higher priority than the global API interface.

[0036] After the API interface definition is completed, corresponding operator information libraries are set for all hardware respectively, and the hardware includes CPU and heterogeneous acceleration hardware, and the heterogeneous acceleration hardware includes GPU and NPU. The operator information of the MAP operation is set, and after the heterogeneous acceleration process is started, the operator information of the MAP operation executable by the hardware is written into the IR file to complete the loading of the operator information library of the corresponding hardware. The user can confirm whether the MAP operation can be migrated to other hardware by saving and checking the IR file. It should be clear that the framework of the present invention has realized a variety of MAP operations that support heterogeneous acceleration functions.

[0037] Load the CPU data processing pipeline. The loading process is as follows: the original data enters the CPU through the API interface, the CPU performs MAP operations on the original data, obtains a MAP operation sequence in the form of a pipeline, extracts all MAP operations on the CPU data processing pipeline, establishes an IR tree, and stores the MAP operations in the IR tree in the form of nodes. The loading of the CPU data processing pipeline and the loading of the operator information library are performed synchronously.

[0038] Set Node_Offload_Pass to traverse the nodes in the IR tree. During the traversal, filter out all nodes that support migration operations according to the operator information loaded in the operator information library, mark them as nodes to be migrated, use the policy search method to find the last node in the IR tree that does not support migration operations, mark it as a non-migratable node, and prioritize the non-migratable node and all the nodes to be migrated before it. When the order between any node to be migrated and the non-migratable node is swapped, and it does not affect the result of the original data preprocessing, the node to be migrated is marked as a migratable node.

[0039] Serialize the migratable nodes, establish the migratable node operation sequence, and transfer it to the tree adapter in the form of JSON file, waiting for subsequent data heterogeneous operations.

[0040] The present invention introduces a strategy search method to avoid negative optimization that may occur during heterogeneous hardware acceleration. The strategy search ensures that for the same batch of data, the MAP operation on the CPU is no longer performed after migration, because the cost of copying data between the CPU and other hardware is too high, thereby avoiding the negative improvement of preprocessing efficiency caused by heterogeneous hardware operations due to redundant data copy operations.

[0041] The load balancing strategy is used to dynamically allocate the migratable nodes. The dynamic allocation judgment process is as follows: by monitoring the key indicators of the operating status of the CPU data processing pipeline, it is judged whether to migrate the migratable nodes from the CPU to the heterogeneous acceleration hardware. The key indicators include the system CPU utilization and the average step time. When it is detected during the monitoring process that the system CPU utilization reaches 100%, and the average step time of the system processing is still greater than the set standard, the migratable nodes are migrated from the CPU to the heterogeneous acceleration hardware. Otherwise, the migration is not performed and the migratable nodes are still executed in the CPU.

[0042] The present invention introduces a load balancing strategy to achieve automatic heterogeneity while avoiding negative optimization. It uses the Profiler API to query operating information such as system CPU utilization and step time, dynamically allocates MAP operations on each hardware according to actual conditions to improve resource utilization, and uses automatic tuning to automate performance-based migration processes. The expected impact of such operations is: allowing the infrastructure to enable migration operations based on average step time and CPU utilization to achieve heterogeneous hardware acceleration, and using the data set automatic tuning function to automatically activate or disable the heterogeneous hardware acceleration function according to actual performance for specific workloads or hardware combinations.

[0043] The core design idea of ​​load balancing in the present invention is to realize performance-based adaptive migration decisions. The migration operation activated by load balancing is based on predefined strategies and real-time performance analysis results, aiming to maximize resource utilization and minimize processing delays. During the operation, according to the technical design architecture, the present invention first classifies data processing operations and identifies operations suitable for transfer to heterogeneous hardware. Then, according to the current system load and hardware status, the scheduling strategy is dynamically adjusted by adjusting the runtime parameters and the status of the migration activation flag.

[0044] In the present invention, users can adjust the operation transfer ratio by changing the migration activation flag for different batches. For example, for computationally intensive operations, the module may prioritize GPUs for operation migration; while for I / O intensive operations, it may select hardware with higher memory bandwidth. In addition, the design of the present invention takes data consistency and synchronization issues into consideration to ensure that data integrity and consistency can be guaranteed while completing migration acceleration.

[0045] When performing the migration operation, after determining the MAP operations that can be migrated, the current batch of data, such as Normalize and HWC2CHC operations, are marked as migratable and stored in the IR file. Based on the graph optimizer, its information is converted into a graph representation and sent from the CPU to the heterogeneous hardware acceleration device. The relevant nodes in the IR tree in the original pipeline are deleted to complete the reconstruction of the data graph. At the same time, the acceleration hardware extracts and deserializes the information of the migratable node operation sequence according to the JSON file to migrate the migratable nodes to different heterogeneous acceleration hardware and establish the relevant calculation graph. The migration of the MAP operations of Normalize and HWC2CHC is completed. The serialization and deserialization processes described above are all executed in the CPU. The serialization process is to convert the status information of the MAP operation into a binary byte stream, and deserialization is the reverse operation.

[0046] The heterogeneous acceleration hardware executes the MAP operations corresponding to the migratable nodes according to the computational graph, and uses the tree adapter to delete the migratable nodes in the IR tree, completes the update of the IR tree, and marks the updated IR tree as the execution tree. The CPU executes the MAP operations corresponding to the remaining nodes according to the execution tree.

[0047] The process of executing MAP operations in heterogeneous acceleration hardware and the process of executing MAP operations in the CPU are parallel processing processes. After the CPU completes the current MAP operation according to the execution tree, it immediately loads the CPU data processing pipeline of the subsequent original data. The automatic heterogeneous acceleration method provided by the present invention coordinates the migration of migratable operations from the CPU to other acceleration hardware to obtain additional computing resources. After executing the retained MAP operations on the CPU according to the reconstructed calculation graph, such as Decode and Resize, the data is transferred to the heterogeneous acceleration hardware in batches, and the relevant preprocessing tasks after migration are completed according to the calculation graph, and finally the network calculation is performed. This method allows the next batch of data to be preprocessed on the CPU, while the previous batch of data is processed and network calculated on the acceleration hardware, retaining the user-defined preprocessing standards and improving data processing efficiency.

[0048] The core of the heterogeneous technology in the present invention is to analyze the operation sequence in the data pipeline, find out the operations that can benefit from parallel execution, and move them to idle GPU, NPU and other acceleration hardware for execution, so as to reduce the CPU load and utilize the parallelism of data processing and network computing to improve the processing speed. During the IR tree update process, the tree adapter provides the necessary tools to handle the conversion of JSON representation and the positioning of migration operations, thereby separating the support for migration operations from the original data processing pipeline queue, and the data graph corresponding to the CPU's IR tree simultaneously completes the corresponding reconstruction.

[0049] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0050] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0051] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A method for accelerating AI data processing based on heterogeneous hardware, characterized in that: The following steps are involved: Configure an API interface, which includes a global API interface and a local API interface. The global API interface is used to uniformly process a single type of data set, and the local API interface is used to set the offload parameter of a specific MAP operation to True. The MAP operation represents a data preprocessing operation defined by a MAP function, including Decode, HWC2CHW, Normalize, and Resize. Set corresponding operator information libraries for all hardware, including CPU and heterogeneous acceleration hardware, including GPU and NPU, set operator information of MAP operation, and complete loading of operator information library of corresponding hardware by writing operator information of MAP operation executable by hardware into IR file; Load the CPU data processing pipeline, parse it, extract all MAP operations in the pipeline and build an IR tree, traverse all nodes in the IR tree, the nodes represent corresponding MAP operations, and use the operator information library and strategy search method to filter out migratable nodes, serialize the migratable nodes, build a migratable node operation sequence, and transfer it to the tree adapter in the form of a JSON file; The load balancing strategy is used to dynamically allocate migratable nodes. The information of the migratable node operation sequence is extracted and deserialized according to the JSON file. The migratable nodes are migrated to different heterogeneous acceleration hardware, and a computational graph is established. The heterogeneous acceleration hardware executes the MAP operations corresponding to the migratable nodes according to the computational graph. At the same time, the tree adapter is used to delete the migratable nodes in the IR tree, complete the update of the IR tree, and mark the updated IR tree as the execution tree. The CPU executes the MAP operations corresponding to the remaining nodes according to the execution tree.

2. The AI ​​data processing acceleration method based on heterogeneous hardware according to claim 1, characterized in that: The global API interface and the local API interface can be used individually or in combination. When used in combination, the priority of the local API interface is higher than that of the global API interface.

3. The AI ​​data processing acceleration method based on heterogeneous hardware according to claim 1, characterized in that: The loading process of the operator information library is carried out synchronously with the loading process of the CPU data preprocessing pipeline.

4. The AI ​​data processing acceleration method based on heterogeneous hardware according to claim 2, characterized in that: The specific process of establishing the migratable node operation sequence is as follows: The process of loading the CPU data processing pipeline is as follows: the original data enters the CPU through the API interface, and the CPU performs MAP operation processing on the original data to obtain a MAP operation sequence in the form of a pipeline; Extract all MAP operations on the CPU data processing pipeline, build an IR tree, and store the MAP operations in the IR tree in the form of nodes; Set Node_Offload_Pass to traverse the nodes in the IR tree. During the traversal, filter out all nodes that support migration operations according to the operator information loaded in the operator information library, mark them as nodes to be migrated, and use the strategy search method to filter all nodes to be migrated to obtain migratable nodes. Serialize the migratable nodes and establish a migratable node operation sequence.

5. The AI ​​data processing acceleration method based on heterogeneous hardware according to claim 4, characterized in that: The specific process of using the strategy search method to obtain migratable nodes is as follows: Find the last node in the IR tree that does not support the migration operation, mark it as a non-migratable node, and prioritize the non-migratable node and all the nodes to be migrated before it. When the order between any node to be migrated and the non-migratable node is swapped without affecting the result of the original data preprocessing, the node to be migrated is marked as a migratable node.

6. The AI ​​data processing acceleration method based on heterogeneous hardware according to claim 5, characterized in that: The specific process of dynamically allocating migratable nodes using the load balancing strategy is as follows: By monitoring the key indicators of the operating status of the CPU data processing pipeline, it is determined whether to migrate the migratable node from the CPU to the heterogeneous acceleration hardware. The key indicators include the system CPU utilization and the average step time. When it is detected during the monitoring process that the system CPU utilization reaches 100%, and the average step time of the system processing is still greater than the set standard, the migratable node is migrated from the CPU to the heterogeneous acceleration hardware. Otherwise, the migration is not performed and the migratable node is still executed in the CPU.

7. The AI ​​data processing acceleration method based on heterogeneous hardware according to claim 1, characterized in that: The serialization and deserialization processes are both executed in the CPU. The serialization process is to convert the state information of the MAP operation into a binary byte stream.

8. The AI ​​data processing acceleration method based on heterogeneous hardware according to claim 1, characterized in that: The process of executing the MAP operation in the heterogeneous acceleration hardware and the process of executing the MAP operation in the CPU are parallel processing processes. After the CPU completes the current MAP operation according to the execution tree, it loads the CPU data processing pipeline of the subsequent original data.

Images