Compatibility expansion system based on PyTorch framework

Abstract

The invention relates to the technical field of deep learning, in particular to a PyTorch framework-based compatibility expansion system, which comprises a cross-framework model converter, a heterogeneous hardware abstraction layer, a hybrid computational graph execution engine, an intelligent distributed trainer and a self-adaptive optimizer, the cross-frame model converter converts input model formats of different frames into PyTorch executable formats, the heterogeneous hardware abstraction layer supports rear ends of various hardware and automatically selects an optimal calculation path, and the hybrid calculation graph execution engine fuses dynamic graph flexibility and static subgraph optimization and supports dynamic control function execution flow. An intelligent distributed trainer automatically selects a parallel scheme and is compatible with multi-protocol communication, a self-adaptive optimizer performs dynamic optimization based on hardware characteristics and a PyTorch tensor, a cross-frame model converter supports model conversion of multiple deep learning frames, the cost of migration among different frames by a user is reduced, and the universality of the PyTorch frame is improved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of deep learning, and particularly relates to a compatibility expansion system based on a PyTorch framework. BACKGROUND

[0002] In the field of deep learning, with the continuous evolution of technology, various deep learning frameworks such as TensorFlow, Keras, ONNX and PyTorch have emerged, and model conversion between different frameworks has become a difficult problem. In practical applications, researchers and developers may need to switch between different frameworks according to different needs, but the existing model conversion technology is often not mature enough and has the following defects:

[0003] 1. Different frameworks have differences in model structure representation, operator naming, data format and type. Existing model conversion tools often cannot automatically adapt to all models, especially when new operators or custom operators are involved. The converted model is difficult to guarantee the consistency of the original function and performance, and problems such as numerical deviation and function loss frequently occur, which seriously restricts the model's portability and stability.

[0004] 2. Different codes or different parameters need to be written for model deployment on different hardware, which not only increases the development workload, but also easily causes configuration errors, resulting in the model being unable to normally run on different hardware. Hardware selection often depends on manual configuration, and the optimal calculation path cannot be automatically selected according to the actual situation of the model and the performance characteristics of the hardware, resulting in low utilization of hardware resources.

[0005] 3. Dynamic graphs are easy to use and debug, but each execution needs to re-interpret the code, and there is the overhead of Python interpreter, resulting in relatively low performance and being unable to meet the needs of high-performance computing scenarios. Static graphs have higher performance, but are not flexible enough to use. Modifying the model structure requires regenerating and compiling the computation graph, increasing the difficulty of development and debugging. SUMMARY

[0006] The application aims to overcome the above technical defects, provide a compatibility expansion system based on the PyTorch framework, which mainly comprises a cross-framework model converter, a heterogeneous hardware abstraction layer, a hybrid computing graph execution engine, an intelligent distributed trainer and an adaptive optimizer, the cross-framework model converter converts input model files of different frameworks such as TensorFlow, Keras and ONNX into a PyTorch executable format, which internally includes an IR generator, an operator mapping engine and a weight adapter, the IR generator parses different framework models into a unified UniIR, including operators, topologies and parameters, the operator mapping engine replaces the operators in the UniIR with PyTorch operators through a dynamic plugin mechanism, generates code and verifies numerical consistency, the weight adapter processes parameter format differences, performs tensor rearrangement and quantization calibration, the cross-framework model converter supports model conversion of multiple mainstream deep learning frameworks, reduces the cost of user migration between different frameworks, improves the universality and applicability of the PyTorch framework, the heterogeneous hardware abstraction layer supports CPU, GPU, NPU, FPGA and other hardware backends, automatically selects the optimal computing path, is composed of a device registry, a dynamic scheduler and a memory pooler, the device registry allows hardware manufacturers to register custom kernels, the dynamic scheduler selects the backend based on the cost of PyTorch operators, and the memory pooler uniformly manages hardware memory and memory allocation to avoid fragmentation, the heterogeneous hardware abstraction layer supports multiple hardware backends and can automatically select the optimal computing path to fully utilize the performance advantages of different hardware, improving the computing efficiency and scalability of the system, the hybrid computing graph execution engine combines the flexibility of dynamic graphs with the optimization of static subgraphs, supports dynamic control function execution flow, includes a graph tracer, a subgraph optimizer and an execution coordinator, the graph tracer records function execution flow to generate static subgraphs, the subgraph optimizer applies PyTorch operator fusion optimization to static subgraphs, and the execution coordinator dynamically schedules the original code and the optimized static subgraphs, the hybrid computing graph execution engine combines the flexibility of dynamic graphs with the optimization of static subgraphs, reduces Python interpreter overhead, improves code execution efficiency, at the same time, the subgraph optimizer optimizes the static subgraph by operator fusion, further improves the computing performance, the intelligent distributed trainer automatically selects a parallel scheme and is compatible with multiple protocol communications, is composed of a strategy analyzer, a communication manager and a gradient synchronization optimizer, the strategy analyzer recommends a parallel scheme according to the model structure, the communication manager supports multiple protocol communications such as NCCL, gRPC and RDMA, and the gradient synchronization optimizer compresses sparse gradients to reduce communication overhead, the intelligent distributed trainer automatically selects a parallel scheme and is compatible with multiple protocol communications, can efficiently utilize multiple machines for distributed training, improves the training speed and model accuracy, and the adaptive optimizer performs dynamic optimization based on hardware characteristics and PyTorch tensors, includes a performance analyzer, an automatic tuner and a dynamic shape analyzer, the performance analyzer monitors operator execution time and memory occupancy,The automatic tuner applies TVM and TensorRT for kernel replacement, and the dynamic shape resolver processes PyTorch tensors to generate a general kernel.

[0007] The application provides a compatibility expansion system based on a PyTorch framework, which comprises a cross-framework model converter, a heterogeneous hardware abstraction layer, a hybrid computing graph execution engine, an intelligent distributed trainer and an adaptive optimizer, the cross-framework model converter converts input model files of different frameworks such as TensorFlow, Keras and ONNX into a PyTorch executable format, the heterogeneous hardware abstraction layer supports backends of various hardware such as CPU, GPU, NPU and FPGA, and automatically selects an optimal computing path, the hybrid computing graph execution engine combines the flexibility of a dynamic graph and the optimization of a static subgraph, supports dynamic control of a function execution flow, the intelligent distributed trainer dynamically selects a parallel scheme and is compatible with multiple protocol communications, and the adaptive optimizer performs dynamic optimization based on hardware characteristics and PyTorch tensors;

[0008] Further, the cross-framework model converter comprises an IR generator, an operator mapping engine and a weight adapter, the heterogeneous hardware abstraction layer comprises a device registry, a dynamic scheduler and a memory pooler, the hybrid computing graph execution engine comprises a graph tracer, a subgraph optimizer and an execution coordinator, the intelligent distributed trainer comprises a strategy analyzer, a communication manager and a gradient synchronization optimizer, and the adaptive optimizer comprises a performance analyzer, an automatic tuner and a dynamic shape resolver.

[0009] Further, the IR generator parses input model files of different frameworks into a unified UniIR, the UniIR comprises operators, topologies and parameters, the operator mapping engine matches input model operators of different frameworks through a dynamic plugin mechanism, traverses the UniIR and replaces the UniIR with PyTorch operators, generates PyTorch code, verifies numerical consistency, supports user-defined mapping rules, the weight adapter processes parameter format differences, and calculates PyTorch tensors and quantization calibration by using tensor rearrangement.

[0010] Preferably, the IR generator parses input model files of different frameworks into a unified UniIR, and specifically comprises the following steps:

[0011] Step A1: input model parsing: the IR generator uses a model parser to read input model files of different frameworks, and the model parser constructs an abstract syntax tree of input models of different frameworks to represent the structures of input models of different frameworks;

[0012] Step A2: constructing an abstract syntax tree: for each operator in the input model of different frameworks, a corresponding abstract syntax tree node is created, each abstract syntax tree node contains the information of the input model operator of different frameworks, according to the execution order of the input model of different frameworks, the abstract syntax tree nodes are connected to form a directed acyclic graph, representing the topology of the input model of different frameworks, the parameters such as weights and biases of the input model of different frameworks are extracted and associated with the corresponding abstract syntax tree node;

[0013] Step A3: node normalization: uniformly converting the data types of the input model of different frameworks to the data types supported by UniIR;

[0014] Step A4: output UniIR: the final output UniIR of the IR generator, which can be further processed by the operator mapping engine to generate PyTorch code;

[0015] Further, the step A1 specifically comprises the following steps:

[0016] Step A11: initializing the parser: initializing the corresponding model parser according to the input model file of different frameworks;

[0017] Step A12: constructing an abstract syntax tree: the model parser traverses the input model file of different frameworks, extracts the input and output tensors in the input model operator of different frameworks, and creates an abstract syntax tree node for each input model operator of different frameworks;

[0018] Step A13: recording the operator formula: for each input model operator of different frameworks, record its mathematical formula, the formula is as follows:

[0019] Y = W * X + b;

[0020] Where Y is the output tensor in the input model operator of different frameworks, W is the convolution kernel weight of the input model operator of different frameworks, X is the input tensor in the input model operator of different frameworks, and b is the bias term of the input model operator of different frameworks.

[0021] Preferably, the operator mapping engine matches the input model operators of different frameworks through a dynamic plug-in mechanism, specifically comprising the following steps:

[0022] Step B1: initializing the mapping rule: loading the pre-defined mapping rule, which defines how to map the input model operators of different frameworks to PyTorch operators, the mapping rule can be hard-coded or user-defined;

[0023] Step B2: Traverse UniIR: The operator mapping engine traverses the UniIR output by the IR generator, visiting each IR node, which represents the input model operator of different frameworks;

[0024] Step B3: Operator matching: For each input model operator in UniIR from different frameworks, the corresponding PyTorch operator is found using the dynamic plugin mechanism to obtain the operator matching result;

[0025] Step B4: Apply mapping rules: Based on the operator matching results, apply mapping rules to replace the input model operators of different frameworks in UniIR with PyTorch operators;

[0026] Step B5: Record the mapping formula: For each input model operator mapped to a different frame of PyTorch operators, record its mathematical formula, as shown below:

[0027] Y PyTorch =nn.Conv2d(X PyTorch W PyTorch )+b PyTorch ;

[0028] Among them, Y PyTorch Let nn.Conv2d be the output tensor of the PyTorch operator, and X be the PyTorch operator. PyTorch W is the input tensor of the PyTorch operator. PYorch b represents the convolution kernel weights of the PyTorch operator. PyTorch For the bias terms of PyTorch operators;

[0029] Step B6: Generate code: Generate PyTorch code based on the mapped PyTorch operators;

[0030] Step B7: Verify numerical consistency: To ensure the correctness of the mapping, perform numerical consistency verification on the input model operators of different frameworks and the operators in the PyTorch framework.

[0031] Furthermore, step B7 specifically includes the following steps:

[0032] Step B71: Run the original frame operators: Execute the input model operators of different frames using the input models of different frames, and record the output results;

[0033] Step B72: Run PyTorch operators: Execute the mapped PyTorch operators using PyTorch and record the output results;

[0034] Step B73: Compare the output tensors: set a difference threshold, calculate the difference between the output tensors in the input model operators of different frameworks and the output tensors of the PyTorch operators, so that the difference is less than the difference threshold;

[0035] Step B74: Record the verification result: record the result of the numerical consistency verification.

[0036] Preferably, the device registry registers custom kernels using hardware vendors, the dynamic scheduler selects the backend based on the PyTorch operator cost, the memory pool manager uniformly manages the memory of different hardware and allocates memory to avoid fragmentation, and the heterogeneous hardware abstraction layer queries the device registry after the PyTorch initiates an interception request, selects the currently available and lowest cost backend, calls the kernel, and returns the result to the PyTorch tensor;

[0037] Preferably, the graph tracer records the function execution flow when fusing the dynamic graph, generates a static subgraph, the subgraph optimizer applies PyTorch operator fusion optimization to the static subgraph, the execution coordinator dynamically schedules the original PyTorch code and the optimized static subgraph, and the hybrid computation graph execution engine records the function execution flow when running for the first time, generates the optimized static subgraph, and directly executes the static subgraph in subsequent calls, skipping the Python interpreter overhead;

[0038] Further, the subgraph optimizer applies PyTorch operator fusion optimization to the static subgraph, and the specific process is as follows:

[0039] Step C1: Graph tracing and subgraph identification: In the process of fusing the dynamic graph, the graph tracer records the execution order and dependency relationship of the PyTorch operators, and identifies the continuous PyTorch operator sequence that can be converted into a static subgraph;

[0040] Step C2: Subgraph construction: Based on the information traced in step C1, a static subgraph representing continuous PyTorch operators is constructed;

[0041] Step C3: Operator fusion optimization: analyze the PyTorch operators in the static subgraph, and find PyTorch operator pairs that can be fused;

[0042] Step C4: Optimize the operator: optimize the fused PyTorch operator pair;

[0043] Step C5: Generate the optimized static subgraph: recombine the optimized PyTorch operator pair into a static subgraph;

[0044] Step C6: Verification and testing: perform numerical verification on the optimized static subgraph to ensure that the optimization does not change the output of the input model of different frameworks.

[0045] Preferably, the strategy analyzer recommends parallel schemes according to the input model structures of different frameworks, the communication manager supports multi-protocol communication such as NCCL, gRPC, RDMA, etc., and the gradient synchronization optimizer compresses the sparse gradients of the input models of different frameworks;

[0046] Further, the gradient synchronization optimizer compresses the sparse gradients of the input models of different frameworks, and the specific process is as follows:

[0047] Step D1: Sparse gradient identification: during the training process of the input models of different frameworks, monitor and record the gradient of each parameter, and extract the gradient matrix;

[0048] Step D2: Sparse processing: convert the gradient matrix to sparse representation, and only keep the non-zero elements;

[0049] Step D3: Compressing sparse gradient: compressing and reducing the communication overhead of sparse gradient;

[0050] Step D4: Asynchronous update: asynchronously update the parameters of the input models of different frameworks;

[0051] Step D5: Synchronization mechanism: synchronize parameters to ensure consistency of input models of different frameworks;

[0052] Further, the performance analyzer monitors the PyTorch operator execution time and memory occupation, the automatic tuner applies TVM and TensorRT for kernel replacement, and the dynamic shape resolver processes PyTorch tensors to generate general kernels.

[0053] The present application has the following beneficial effects:

[0054] (1) In the prior art, model conversion between different deep learning frameworks often requires complex custom scripts or tools, and the converted model may not completely retain the functions and performance of the original model. The present application parses the input models of different frameworks into a unified UniIR intermediate representation through an IR generator, realizing a unified conversion interface for cross-framework models. This not only simplifies the conversion process, but also improves the accuracy and efficiency of the conversion. In the prior art, when encountering new operators or custom operators, the entire conversion tool or script often needs to be modified. The present application supports incremental operator extension, and users can easily add new operator mappings through custom mapping rules or dynamic plugin mechanisms without modifying the core conversion logic.

[0055] (2) In the prior art, different codes or different parameters need to be written or configured for model deployment of different hardware, and the hardware details are abstracted by the heterogeneous hardware abstraction layer, so that the model can run seamlessly on different hardware backends, which greatly reduces the deployment cost and improves the development efficiency. In the prior art, hardware selection often depends on manual experience and manual configuration. The optimal calculation path is automatically selected based on the PyTorch operator cost by the dynamic scheduler, and the maximum utilization of hardware resources is realized.

[0056] (3) In the prior art, dynamic graphs are easy to use and debug, but have relatively low performance, and static graphs have high performance but are not flexible enough to use. The hybrid computation graph execution engine combines the flexibility of dynamic graphs and the performance advantages of static graphs. The function execution flow is recorded during the first run and an optimized static subgraph is generated. The static subgraph is directly executed during subsequent calls, skipping the overhead of the Python interpreter. In the prior art, the dynamic graph feature of PyTorch limits its application in some high-performance computing scenarios. The hybrid computation graph execution engine breaks through the original performance limitations of PyTorch, enabling PyTorch models to maintain ease of use while achieving performance close to that of static graphs.

[0057] (4) In the prior art, performance analysis often relies on offline tools or manual analysis, and cannot reflect the running situation of the model on the hardware in real time. The performance analyzer monitors the execution time and memory occupation of the PyTorch operator in real time, providing data support for adaptive optimization. In the prior art, automatic tuning often relies on complex algorithms and a large amount of computing resources. The automatic tuner applies TVM and TensorRT optimization tools for kernel replacement and parameter tuning, realizing offline automatic tuning, which not only improves the performance of the model, but also reduces the tuning cost. In the prior art, processing variable shape input data often requires writing complex code or configuring different kernels. The dynamic shape resolver processes input data and generates a general kernel, which not only improves the flexibility of the model, but also reduces the complexity of processing variable shape input data.

[0058] (5) In the prior art, distributed training often relies on manual configuration of parallel schemes. The strategy analyzer automatically recommends parallel schemes based on the model structure, improving the efficiency and accuracy of distributed training. In the prior art, gradient synchronization often becomes a bottleneck of distributed training. The gradient synchronization optimizer compresses sparse gradients and updates parameters asynchronously, reducing communication overhead and synchronization waiting time, and improving the efficiency of distributed training. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1is a module schematic diagram of a compatibility extension system based on a PyTorch framework of the present application;

[0060] Figure 2 is a unit structure schematic diagram of a cross-framework model converter;

[0061] Figure 3 is a unit structure schematic diagram of a heterogeneous hardware abstraction layer;

[0062] Figure 4 is a unit structure schematic diagram of a hybrid computation graph execution engine;

[0063] Figure 5 is a unit structure schematic diagram of an intelligent distributed trainer;

[0064] Figure 6 is a unit structure schematic diagram of an adaptive optimizer;

[0065] Figure 7 is a flowchart of an IR generator parsing input models of different frameworks into a unified UniIR;

[0066] Figure 8 is a flowchart of step A1;

[0067] Figure 9 is a workflow diagram of an operator mapping engine matching input model operators of different frameworks through a dynamic plugin mechanism;

[0068] Figure 10 is a flowchart of step B7;

[0069] Figure 11 is a workflow diagram of a subgraph optimizer applying PyTorch operator fusion optimization to static subgraphs;

[0070] Figure 12 is a workflow diagram of a gradient synchronization optimizer compressing sparse gradients of input models of different frameworks.

[0071] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of this specification, illustrate embodiments of the present application, and are used to explain the present application, and do not constitute a limitation on the present application. DETAILED DESCRIPTION

[0072] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all embodiments of the present application.

[0073] Embodiment one, as Figures 1-6As shown, in the present embodiment, the system includes a cross-framework model converter, a heterogeneous hardware abstraction layer, a hybrid computation graph execution engine, an intelligent distributed trainer, and an adaptive optimizer, the cross-framework model converter converts input model files of different frameworks such as TensorFlow, Keras, and ONNX into PyTorch executable format, the heterogeneous hardware abstraction layer supports backends of various hardware such as CPU, GPU, NPU, and FPGA, automatically selects the optimal computing path, the hybrid computation graph execution engine combines the flexibility of dynamic graph and the optimization of static subgraph, supports dynamic control function execution flow, the intelligent distributed trainer automatically selects parallel schemes, compatible with multiple protocol communication, the adaptive optimizer performs dynamic optimization based on hardware characteristics and PyTorch tensors;

[0074] The cross-framework model converter includes an IR generator, an operator mapping engine, and a weight adapter, the heterogeneous hardware abstraction layer includes a device registry, a dynamic scheduler, and a memory pooler, the hybrid computation graph execution engine includes a graph tracer, a subgraph optimizer, and an execution coordinator, the intelligent distributed trainer includes a strategy analyzer, a communication manager, and a gradient synchronization optimizer, and the adaptive optimizer includes a performance analyzer, an automatic tuner, and a dynamic shape resolver;

[0075] The cross-framework model converter can convert input model files of different frameworks such as TensorFlow, Keras, and ONNX into PyTorch executable format, greatly improving the compatibility of the PyTorch framework, making it easier for users to migrate models of other frameworks to the PyTorch environment for further development and application, the heterogeneous hardware abstraction layer supports multiple hardware backends such as CPU, GPU, NPU, and FPGA, and automatically selects the optimal computing path, which can fully utilize the performance advantages of different hardware, improve the computing efficiency and flexibility of the system, adapt to different hardware environments, the hybrid computation graph execution engine combines the flexibility of dynamic graph and the optimization of static subgraph, supports dynamic control function execution flow, reduces the overhead of Python interpreter, improves code execution efficiency, while static subgraph optimization can further improve computing performance, the intelligent distributed trainer automatically selects parallel schemes, compatible with multiple protocol communication, which can efficiently utilize multiple machines for distributed training, improve training speed and model accuracy, reduce training time and cost, the adaptive optimizer performs dynamic optimization based on hardware characteristics and PyTorch tensors, which can monitor performance in real time and perform kernel replacement and shape resolution, improving the adaptive ability and performance of the system.

[0076] Embodiment two, as Figure 2 and Figure 7As shown, the embodiment is based on the above-mentioned embodiment, the IR generator parses the input model files of different frameworks TensorFlow, ONNX, Keras into a unified UniIR, the UniIR includes operators, topologies and parameters, the operator mapping engine matches the input model operators of different frameworks through a dynamic plugin mechanism, traverses the UniIR and replaces the UniIR with PyTorch operators, generates PyTorch code, and verifies numerical consistency, supports user-defined mapping rules, the weight adapter processes parameter format differences, and calculates PyTorch tensors and quantization calibration using tensor rearrangement.

[0077] The IR generator parses the input model files of different frameworks into a unified UniIR, specifically including the following steps:

[0078] Step A1: input model parsing: the IR generator uses the model parser ONNXParser to read the ONNX file, the ONNXParser constructs an abstract syntax tree of the ONNX model to represent the structure of the ONNX, calls onnx.load(model_path) to obtain the graph structure of the ONNX model, extracts nodes, edges and attributes;

[0079] Step A2: building an abstract syntax tree: for each operator in the ONNX model, create a corresponding abstract syntax tree node, each abstract syntax tree node contains the information of the ONNX model operator, connect the abstract syntax tree nodes according to the execution order of the ONNX model to form a directed acyclic graph, representing the topology structure of the ONNX model, extract the weight and bias parameters of the ONNX model and bind them to the corresponding abstract syntax tree node;

[0080] Step A3: node normalization: uniformly convert the data type Conv of the ONNX model to the data type UniConv supported by the UniIR, the data type uniform conversion structure is: UniNode(op_type="UniConv", dtype="float32", params={...});

[0081] Step A4: output UniIR: the final output UniIR of the IR generator, the UniIR can be further processed by the operator mapping engine to generate PyTorch code, the specific process is shown in the following code:

[0082]

[0083] The IR generator parses the input models of different frameworks into a unified UniIR, simplifies the subsequent operator mapping and code generation processes, improves the scalability and maintainability of the system, the operator mapping engine matches the input model operators of different frameworks through a dynamic plugin mechanism, supports user-defined mapping rules, enables the system to handle more types of operators, improves the versatility of the system, the weight adapter processes parameter format differences, and uses tensor rearrangement calculation to process PyTorch tensors and quantization calibration, ensuring the correctness and consistency of model parameters during the conversion process.

[0084] Embodiment three, as shown in Figure 2 、 Figure 7 and Figure 8 , based on the above embodiment, the step A1 specifically comprises the following steps:

[0085] Step A11: Initialize the parser: initialize the corresponding model parser ONNXParser according to the ONNX model file;

[0086] Step A12: Build an abstract syntax tree: the model parser ONNXParser traverses the ONNX model file, extracts the input and output tensors in the ONNX model operator, and creates an abstract syntax tree node for each ONNX model operator;

[0087] Step A13: Record the operator formula: for each ONNX model operator, record its mathematical formula, the formula used is as follows:

[0088] Y = W * X + b;

[0089] Where Y is the output tensor in the ONNX model operator, W is the convolution kernel weight of the ONNX model operator, X is the input tensor in the ONNX model operator, and b is the bias term of the ONNX model operator.

[0090] Embodiment four, as shown in Figure 2 and Figure 9 , based on the above embodiment, the operator mapping engine matches the input model operators of different frameworks through a dynamic plugin mechanism, specifically comprising the following steps:

[0091] Step B1: Initialize the mapping rule: load the pre-defined mapping rule, which defines how to map the ONNX model operator to the PyTorch operator, the mapping rule can be hard-coded or user-defined, the hard-coded mapping is as shown in the following code:

[0092]

[0093] Users can customize mapping rules (such as custom_rules.yaml) through YAML files to override or extend default plugins, as shown in the following code:

[0094] -name: "CustomReLU"

[0095] source_op: "UniReLU"

[0096] target_op: "nn.ReLU(inplace=True)"

[0097] Step B2: Traverse UniIR: The operator mapping engine traverses the UniIR output by the IR generator, accessing each IR node, which represents an ONNX model operator;

[0098] Step B3: Operator matching: For each UniIR node UniConv, call the match() method of the plugin to filter out matching plugins and find the corresponding PyTorch operator, obtaining the operator matching result;

[0099] Step B4: Apply mapping rules: According to the operator matching result, call the apply() method of the plugin to convert the UniIR parameters into the initialization parameters of the PyTorch operator, as shown in the following code:

[0100]

[0101] pytorch_code = plugin.apply(uni_node, params) # generate "nn.Conv2d(3, 64, kernel_size=(3, 3))";

[0102] Step B5: Record mapping formulas: For each input model operator of a different framework that is mapped to a PyTorch operator, record its mathematical formula, which is as follows:

[0103] Y PyTorch = nn.Conv2d(X PyTorch , W PyTorch ) + b PyTorh ;

[0104] Where Y PyTorch is the output tensor of the PyTorch operator, nn.Conv2d is the PyTorch operator, X PyTorch is the input tensor of the PyTorch operator, W PyTorch is the convolution kernel weight of the PyTorch operator, and b PyTorch is the bias term of the PyTorch operator;

[0105] Step B6: Generating code: According to the mapped PyTorch operators, generate PyTorch code, the code is as follows:

[0106]

[0107]

[0108] Step B7: Verify numerical consistency: To ensure the correctness of the mapping, perform numerical consistency verification on the input model operators of different frameworks and the operators in the PyTorch framework, fix the random seed, generate unified test input test_input, use the original framework ONNX to forward inference test_input, get ref_output, load the generated PyTorch model, execute converted_model(test_input), get pytorch_output, calculate the maximum absolute error MAE and relative error L2 Norm between tensors, the code is as follows:

[0109]

[0110] The operator mapping engine can accurately map the input model operators of different frameworks to PyTorch operators by loading predefined mapping rules or using dynamic plugin mechanisms, ensuring the correctness of model conversion, generating PyTorch code and performing numerical consistency verification, and ensuring the correctness and usability of the converted model in the PyTorch environment.

[0111] As shown in Figure 2 , Figure 9 and Figure 10 , this embodiment is based on the above-mentioned embodiments, and the step B7 specifically includes the following steps:

[0112] Step B71: Run the original framework operator: execute the ONNX model operator using the ONNX model, and record the output result;

[0113] Step B72: Run PyTorch operator: execute the mapped PyTorch operator using PyTorch, and record the output result;

[0114] Step B73: Compare output tensors: set a difference threshold, calculate the difference between the output tensors in the ONNX model operator and the output tensors of the PyTorch operator, so that the difference is less than the difference threshold;

[0115] Step B74: Record verification results: record the results of numerical consistency verification.

[0116] Embodiment six, asFigures 3-4 As shown, based on the above embodiment, the device registry registers the custom kernel using the hardware vendor, in this embodiment, MatMul of Huawei NPU, and the vendor registers the kernel to the device registry through API (the NPU bottom layer driver needs to be connected), and the code used is as shown below:

[0117]

[0118] The dynamic scheduler selects the backend based on the PyTorch operator cost, selects the optimal backend according to the current hardware availability and cost rules, and the code used is as shown below:

[0119]

[0120]

[0121] The memory pooling manager uniformly manages the video memory and memory allocation of different hardware, encapsulates the memory allocation / release interfaces of different hardware, records the memory block information to avoid fragmentation, and the code used is as shown below:

[0122]

[0123] The heterogeneous hardware abstraction layer queries the device registry after the PyTorch initiates an interception request, selects the currently available and lowest cost backend, calls the kernel, and returns the result to the PyTorch tensor, replaces the PyTorch native operator through monkey patch, inserts the scheduling logic, and the code used is as shown below:

[0124]

[0125] The graph tracer records the function execution flow when fusing the dynamic graph, generates a static subgraph, uses torch.jit.trace of PyTorch to capture the dynamic graph execution flow, the subgraph optimizer applies PyTorch operator fusion optimization to the static subgraph, defines the optimization rule Conv+ReLU fusion as ConvReLU, the execution coordinator dynamically schedules the original PyTorch code and the optimized static subgraph, the hybrid computation graph execution engine records the function execution flow when first running, generates the optimized static subgraph, and directly executes the static subgraph when called subsequently, skipping the Python interpreter overhead, according to whether the subgraph has been optimized, selects the execution path, and the code used is as shown below:

[0126]

[0127] # Execute the optimized static subgraph

[0128] return self.static_subgraphs[input_signature](inputs)

[0129] else:

[0130] # First run: trace dynamic graph and optimize

[0131] traced = self.trace_subgraph(inputs)

[0132] optimized = fuse_conv_relu(traced)

[0133] self.static_subgraphs[input_signature] = optimized

[0134] return optimized(inputs)

[0135] The device registry registers custom kernels provided by hardware vendors, so that the system can make full use of the optimized kernels provided by hardware vendors to improve computing performance. The dynamic scheduler selects the backend based on the cost of PyTorch operators, and the memory pool manager uniformly manages the memory of different hardware to avoid fragmentation and improve resource utilization and computing efficiency of the system.

[0136] In an embodiment, the subgraph optimizer applies PyTorch operator fusion optimization to the static subgraph, and the specific process is as follows: Figure 4 Figure 11 In an embodiment, the subgraph optimizer applies PyTorch operator fusion optimization to the static subgraph, and the specific process is as follows:

[0137] Step C1: graph tracing and subgraph identification: in the process of fusing the dynamic graph, the graph tracer records the execution order and dependency relationship of the PyTorch operators, and identifies the continuous PyTorch operator sequence that can be converted into a static subgraph;

[0138] Step C2: subgraph construction: based on the information traced in step C1, a static subgraph representing continuous PyTorch operators is constructed;

[0139] Step C3: operator fusion optimization: analyze the PyTorch operators in the static subgraph, and find the PyTorch operator pairs that can be fused;

[0140] Step C4: optimize the operator: optimize the fused PyTorch operator pair;

[0141] Step C5: generate an optimized static subgraph: recombine the optimized PyTorch operator pair into a static subgraph;​

[0142] Step C6: Verification and testing: numerical verification of the optimized static subgraph;

[0143] The subgraph optimizer applies PyTorch operator fusion optimization to the static subgraph, generates an optimized static subgraph through graph tracking, subgraph identification, operator fusion optimization and other steps, improves the code execution efficiency, and performs numerical verification on the optimized static subgraph to ensure that the optimization does not change the output of the input model of different frameworks, ensuring the correctness and reliability of the optimization.

[0144] Embodiment eight, as shown in Figure 5 and Figure 12 Based on the above embodiment, the policy analyzer recommends a parallel scheme according to the input model structure of different frameworks, the communication manager supports multi-protocol communication such as NCCL, gRPC, and RDMA, and the gradient synchronization optimizer compresses the sparse gradient of the input model of different frameworks.

[0145] The gradient synchronization optimizer compresses the sparse gradient of the input model of different frameworks, and the specific process is as follows:

[0146] Step D1: Sparse gradient identification: monitor and record the gradient of each parameter during the training process of the input model of different frameworks, extract the gradient matrix, and the code used is as follows:

[0147] import torch

[0148] fromtorch.autograd importFunction

[0149] class GradientMonitor:

[0150] def__init__(self,model):

[0151] self.model=model

[0152] self.gradients={}

[0153] self._register_hooks()

[0154] def_register_hooks(self):

[0155] for name,paramin self.model.named_parameters():

[0156] ifparam.requires_grad:

[0157] param.register_hook(self._save_gradient(name))

[0158] def_save_gradient(self, name):

[0159] defhook(grad):

[0160] self.gradients[name]=grad.detach()

[0161] return hook

[0162] #Example: Monitoring the gradients of a ResNet18 model

[0163] model=torchvision.models.resnet18()

[0164] monitor=GradientMonitor(model)

[0165] #Forward and backward propagation trigger gradient capture

[0166] inputs=torch.randn(4,3,224,224)

[0167] outputs = model(inputs)

[0168] loss = outputs.sum()

[0169] loss.backward()

[0170] # View the captured gradients (example output shows the gradient of the first convolutional layer)

[0171] print("gradient shape:", monitor.gradients['conv1.weight'].shape);

[0172] Step D2: Sparsification: Convert the gradient matrix into a sparse representation, retaining only non-zero elements. The code used is shown below:

[0173] defsparsify_gradient(grad,threshold=1e-4):

[0174] #Method 1: Absolute value threshold filtering

[0175] mask = (grad.abs() > threshold)

[0176] sparse_grad = grad[mask]

[0177] indices = torch.nonzero(mask, as_tuple=True)

[0178] # Method 2: Top-K sparsification (keep top 10% elements)

[0179] # k = int(0.1 * grad.numel())

[0180] # values, indices = torch.topk(grad.abs().flatten(), k)

[0181] # sparse_grad = grad.flatten()[indices]

[0182] return sparse_grad, indices

[0183] # Example: Sparsify the gradients of the first convolutional layer

[0184] dense_grad = monitor.gradients['conv1.weight']

[0185] sparse_grad, indices = sparsify_gradient(dense_grad)

[0186] print(f"Dense gradient size: {dense_grad.numel()}, Sparse gradient size: {sparse_grad.numel()}");

[0187] Step D3: Compressing the sparse gradient: Compressing reduces the communication overhead of the sparse gradient, the code used is as follows:

[0188] def quantize_gradient(grad, bits=8):

[0189] # Dynamic range quantization

[0190] min_val = grad.min()

[0191] max_val = grad.max()

[0192] scale = (max_val - min_val) / (2 ** bits - 1)

[0193] quantized = torch.round((grad - min_val) / scale).to(torch.uint8)

[0194] return quantized, scale, min_val

[0195] # Example: Quantizing sparse gradients

[0196] quantized_grad, scale, min_val = quantize_gradient(sparse_grad)

[0197] print("Quantized gradient size (Bytes):", quantized_grad.element_size() * quantized_grad.numel())

[0198] Step D4: Asynchronous Update: Asynchronously update the parameters of the input model in different frameworks using the following code:

[0199] import grpc

[0200] from concurrent import futures

[0201] class ParameterClient:

[0202] def __init__(self, server_address):

[0203] self.channel = grpc.insecure_channel(server_address)

[0204] self.stub = ParameterService_pb2_grpc.ParameterServiceStub(self.channel)

[0205] def async_push(self, param_name, compressed_grad):

[0206] request = ParameterService_pb2.GradientUpdate(

[0207] param_name = param_name,

[0208] data = compressed_grad )

[0210] self.stub.PushGradient.future(request) # Asynchronous non-blocking call

[0211] #Example: Asynchronously pushing gradients to the parameter server

[0212] client=ParameterClient("localhost:50051")

[0213] client.async_push("conv1.weight",compressed_data);

[0214] Step D5: Synchronization Mechanism: Synchronize parameters to ensure consistency of input models across different frameworks.

[0215] Example 9, as Figure 6 The embodiment shown is based on the above embodiment. The performance analyzer monitors the execution time and memory usage of PyTorch operators, the automatic tuner applies TVM and TensorRT to replace the kernel, and the dynamic shape parser processes PyTorch tensors to generate a general kernel.

[0216] The performance analyzer monitors PyTorch operator execution time and memory usage, providing data support for system performance optimization. The automatic tuner uses TVM and TensorRT for kernel replacement, leveraging the optimization capabilities of these tools to improve system computational performance. The dynamic shape resolver processes PyTorch tensors to generate a general-purpose kernel, enhancing the system's versatility and flexibility.

[0217] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A compatible extension system based on the PyTorch framework, characterized in that: The system includes a cross-framework model converter, a heterogeneous hardware abstraction layer, a hybrid computation graph execution engine, an intelligent distributed trainer, and an adaptive optimizer. The cross-framework model converter converts input model files from different frameworks into PyTorch executable format. The heterogeneous hardware abstraction layer supports backends of various hardware and automatically selects the optimal computation path. The hybrid computation graph execution engine combines the flexibility of dynamic graphs with the optimization of static subgraphs and supports dynamic control of function execution flow. The intelligent distributed trainer automatically selects parallel schemes and is compatible with multi-protocol communication. The adaptive optimizer performs dynamic optimization based on hardware characteristics and PyTorch tensors. The cross-framework model converter includes an IR generator, an operator mapping engine, and a weight adapter; the heterogeneous hardware abstraction layer includes a device registry, a dynamic scheduler, and a memory pooler; the hybrid computation graph execution engine includes a graph tracer, a subgraph optimizer, and an execution coordinator; the intelligent distributed trainer includes a policy analyzer, a communication manager, and a gradient synchronization optimizer; and the adaptive optimizer includes a performance analyzer, an auto-tuner, and a dynamic shape resolver.

2. The compatibility extension system based on the PyTorch framework according to claim 1, characterized in that, The IR generator parses input model files from different frameworks into a unified UniIR, which contains operators, topology, and parameters. The operator mapping engine matches input model operators from different frameworks through a dynamic plugin mechanism, traverses the UniIR, replaces the UniIR with PyTorch operators, generates PyTorch code, and verifies numerical consistency. The weight adapter handles parameter format differences and uses tensor rearrangement to calculate PyTorch tensors and perform quantization calibration. The device registry uses hardware manufacturers to register custom kernels. The dynamic scheduler selects backends based on PyTorch operator costs. The memory pooler uniformly manages the video memory and memory allocation of different hardware. After PyTorch initiates an interception request, the heterogeneous hardware abstraction layer queries the device registry, selects the currently available and lowest-cost backend, calls the kernel, and returns the result to the PyTorch tensor. The graph tracer records the function execution flow and generates a static subgraph when fusing dynamic graphs. The subgraph optimizer applies PyTorch operators to the static subgraph for fusion optimization. The execution coordinator dynamically schedules the original PyTorch code and the optimized static subgraph. When the hybrid computation graph execution engine runs for the first time, the graph tracer records the function execution flow and generates an optimized static subgraph. In subsequent calls, the static subgraph is executed directly. The policy analyzer recommends parallel schemes based on the input model structure of different frameworks, the communication manager supports multi-protocol communication, and the gradient synchronization optimizer compresses the sparse gradients of the input models of different frameworks. The performance analyzer monitors PyTorch operator execution time and memory usage, the automatic tuner performs kernel replacement, and the dynamic shape parser processes PyTorch tensors to generate a generic kernel.

3. A compatible extension system based on the PyTorch framework according to claim 2, characterized in that, The IR generator parses input model files from different frameworks into a unified UniIR, specifically including the following steps: Step A1: Input Model Parsing: The IR generator uses a model parser to read input model files from different frameworks. The model parser constructs an abstract syntax tree for the input models from different frameworks. Step A2: Constructing Abstract Syntax Tree Nodes: For each operator in the input model of different frameworks, create a corresponding abstract syntax tree node. Each abstract syntax tree node contains information about the operators of the input model of different frameworks. According to the execution order of the input models of different frameworks, connect the abstract syntax tree nodes to form a directed acyclic graph, which represents the topological structure of the input models of different frameworks. Extract the input model parameters of different frameworks and associate them with the corresponding abstract syntax tree nodes. Step A3: Node Normalization: Convert the data types of input models from different frameworks to data types supported by UniIR; Step A4: Output UniIR: The final output UniIR of the IR generator.

4. A compatible extension system based on the PyTorch framework according to claim 3, characterized in that, Step A1 specifically includes the following steps: Step A11: Initialize the parser: Initialize the corresponding model parser according to the input model file of different frameworks; Step A12: Constructing an abstract syntax tree: The model parser traverses the input model files of different frameworks, extracts the input tensors and output tensors from the input model operators of different frameworks, and creates an abstract syntax tree node for each input model operator of different frameworks. Step A13: Record operator formulas: For each different frame's input model operator, record its mathematical formula, as shown below: Y = W*X + b; Where Y is the output tensor in the input model operator of different frameworks, W is the convolution kernel weight of the input model operator of different frameworks, X is the input tensor in the input model operator of different frameworks, and b is the bias term of the input model operator of different frameworks.

5. A compatibility extension system based on the PyTorch framework according to claim 2, characterized in that, The operator mapping engine matches input model operators from different frameworks through a dynamic plugin mechanism, specifically including the following steps: Step B1: Initialize mapping rules: Load predefined mapping rules that define how to map input model operators from different frameworks to PyTorch operators; Step B2: Traverse UniIR: The operator mapping engine traverses the UniIR output by the IR generator, visiting each IR node, which represents the input model operator of different frameworks; Step B3: Operator matching: For each input model operator in UniIR from different frameworks, the corresponding PyTorch operator is found using the dynamic plugin mechanism to obtain the operator matching result; Step B4: Apply mapping rules: Based on the operator matching results, apply mapping rules to replace the input model operators of different frameworks in UniIR with PyTorch operators; Step B5: Record the mapping formula: For each input model operator mapped to a different frame of PyTorch operators, record its mathematical formula, as shown below: Y PyTorch =nn.Conv2d(X PyTorch ,W Pyrorch )+b PyTorch ; Among them, Y PyTorch Let nn.Conv2d be the output tensor of the PyTorch operator, and X be the PyTorch operator. PyTorch W is the input tensor of the PyTorch operator. PyTorch b represents the convolution kernel weights of the PyTorch operator. PyTorch For the bias terms of PyTorch operators; Step B6: Generate code: Generate PyTorch code based on the mapped PyTorch operators; Step B7: Verify numerical consistency: Perform numerical consistency verification on the input model operators of different frameworks and the operators in the PyTorch framework.

6. A compatibility extension system based on the PyTorch framework according to claim 5, characterized in that, Step B7 specifically includes the following steps: Step B71: Run the original frame operators: Execute the input model operators of different frames using the input models of different frames, and record the output results; Step B72: Run PyTorch operators: Execute the mapped PyTorch operators using PyTorch and record the output results; Step B73: Compare output tensors: Set a difference threshold, calculate the difference between the output tensors in the input model operators of different frameworks and the output tensors of PyTorch operators, and make the difference less than the difference threshold; Step B74: Record the verification results: Record the results of the numerical consistency verification.

7. A compatible extension system based on the PyTorch framework according to claim 2, characterized in that, The subgraph optimizer applies PyTorch operator fusion optimization to the static subgraph, as detailed below: Step C1: Graph tracing and subgraph identification: During the fusion of dynamic graphs, the graph tracer records the execution order and dependencies of PyTorch operators and identifies continuous PyTorch operator sequences that can be converted into static subgraphs. Step C2: Subgraph Construction: Based on the information traced in Step C1, construct a static subgraph representing continuous PyTorch operators; Step C3: Operator fusion optimization: Analyze the PyTorch operators in the static subgraph and find PyTorch operator pairs that can be fused; Step C4: Operator optimization: Optimize the fused PyTorch operator pair; Step C5: Generate the optimized static subgraph: Recombine the optimized PyTorch operator pairs into a static subgraph; Step C6: Verification and Testing: Perform numerical verification on the optimized static subgraph to ensure that the optimization has not changed the output of the input model of different frameworks.

8. A compatibility extension system based on the PyTorch framework according to claim 2, characterized in that, The gradient synchronization optimizer compresses the sparse gradients of input models from different frameworks, and the specific process is as follows: Step D1: Sparse gradient identification: During the training process of input models in different frameworks, monitor and record the gradient of each parameter, and extract the gradient matrix; Step D2: Sparsification: Convert the gradient matrix into a sparse representation, retaining only the non-zero elements; Step D3: Compress sparse gradients: Compression reduces the communication overhead of sparse gradients; Step D4: Asynchronous Update: Asynchronously update the parameters of the input models of different frameworks; Step D5: Synchronization Mechanism: Synchronization Parameters.

Images