NPU model optimization method and device based on TensorFlow and storage medium

Abstract

The invention relates to an NPU model optimization method and device based on TensorFlow and a storage medium. The method comprises the steps that a model analysis module reads and extracts the structure and parameters of a TensorFlow model; the model conversion module is used for converting the TensorFlow model into an intermediate representation supported by the NPU; through operator fusion, memory optimization and quantization, the model optimization module optimizes the converted TensorFlow model, and deploys the optimized TensorFlow model to the NPU execution module; and the performance monitoring module monitors the operation performance of the TensorFlow model on the NPU and outputs performance data. According to the method, an efficient model conversion tool is provided, the TensorFlow model is quickly converted into the format supported by the NPU, and the model conversion efficiency is improved; the hardware characteristics of low-precision calculation, parallel calculation and the like of the NPU are fully utilized, the calculation efficiency is improved, and the conversion time and the development cost are reduced by means of operator fusion, memory optimization, quantization and the like; manual operation of developers is reduced, the model deployment process is simplified, and the deployment difficulty is reduced; large-scale model deployment is supported, and actual application requirements are met.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and particularly to an NPU model optimization method, device and storage medium based on TensorFlow. Background Art

[0002] With the rapid development of deep learning technology, TensorFlow, as a mainstream deep learning framework, is widely used in fields such as image processing and natural language processing. However, when deploying a TensorFlow model to an NPU, when converting the TensorFlow model into a format supported by the NPU, the model conversion efficiency is low; the performance of the converted model on the NPU is insufficiently optimized; moreover, the compatibility between the TensorFlow model and the NPU hardware is poor, resulting in difficult deployment.

[0003] Existing technical solutions require developers to manually convert the TensorFlow model into a format supported by the NPU. This processing is inefficient, time-consuming, error-prone, and has poor compatibility. In addition, using general model conversion tools (such as ONNX, TVM) to convert the TensorFlow model into an intermediate representation (IR), and then further converting it into a format supported by the NPU, this processing has a complex conversion process, limited performance optimization, insufficient support for NPU hardware, and poor compatibility. Additionally, it is also possible to perform model conversion and optimization by using a tool chain provided by the hardware manufacturer (such as MindSpore of Huawei, Cambricon of Cambricon), but the tool chain is usually designed for specific hardware, has poor generality, limited support for the TensorFlow model, and the optimization effect is not ideal. Summary of the Invention

[0004] The present invention provides an NPU model optimization method, device and storage medium based on TensorFlow, aiming to solve at least one of the technical problems existing in the prior art.

[0005] The technical solution of the present invention is an NPU model optimization method based on TensorFlow. The NPU model optimization method based on TensorFlow is applied to an NPU model optimization device based on TensorFlow. The NPU model optimization device based on TensorFlow includes: a model parsing module, a model conversion module, a model optimization module, an NPU execution module, and a performance monitoring module. The NPU model optimization method based on TensorFlow includes the following steps: S100, the model parsing module reads and extracts the structure and parameters of the TensorFlow model; S200. The model conversion module converts the TensorFlow model into an intermediate representation IR supported by the NPU; S300. Through operator fusion, memory optimization, and quantization, the model optimization module optimizes the converted TensorFlow model and deploys the optimized TensorFlow model to the NPU execution module; S400. The performance monitoring module monitors the running performance of the TensorFlow model on the NPU and outputs performance data.

[0006] Further, step S100 includes: S110. The model parsing module reads the TensorFlow model from storage to obtain the structure and parameters of the TensorFlow model; S120. Parse the structure of the TensorFlow model to extract the computation graph Graph and operations Operation; S130. Based on the custom mapping rules, map the TensorFlow model to NPU-compatible hardware operations.

[0007] Further, step S130 includes: S131. Optimize the TensorFlow computation graph Graph, including merging operations and deleting redundant operations; S132. Divide the computation graph Graph into parts to be executed on the NPU and parts to be executed on the CPU; S133. Compile the part of the computation graph Graph to be executed on the NPU into the NPU instruction set; S134. Execute the compiled instruction set on the NPU.

[0008] Further, step S200 includes: S210. Generate the intermediate representation IR supported by the NPU according to the result of mapping the TensorFlow model to NPU-compatible hardware mapping; S220. Save the generated intermediate representation to a preset location to prepare for optimizing the TensorFlow model.

[0009] Further, step S210 includes: S211. Execute custom operator registration and graph optimization by registering NPU-specific operators, creating an NPU-optimized Grappler configuration, running the model using an NPU-optimized session, and constructing a computation graph containing NPU operators; S212. Perform dynamic IR conversion and AI-driven optimization through dynamic intermediate representation IR conversion, AI-driven optimization strategy AI-Optimized IR, automatic mixed precision for NPU Auto Mixed Precision for NPU, and underlying implementation of NPU IR conversion.

[0010] Further, in step S210, the intermediate representation IR includes a computation graph Graph, operations Operation, and parameters.

[0011] Further, step S300 includes: S310. Read the intermediate representation IR supported by the NPU from storage; S320. Fuse multiple small operators into large operators to reduce computational overhead; S330. Improve memory efficiency by memory reuse and data prefetching, optimize the memory access pattern of the TensorFlow model, and reduce memory bandwidth occupancy; S340. Convert the floating-point model to a low-precision model based on quantization-aware training QAT and / or post-training quantization PTQ to improve computational efficiency; S350. Verify whether the optimized TensorFlow model meets the accuracy and performance requirements. If the verification fails, return to step S320 to step S340; S360. If the verification is successful, save the optimized TensorFlow model to a preset location.

[0012] Further, the present invention also proposes an NPU model optimization device based on TensorFlow for performing the NPU model optimization method based on TensorFlow. The NPU model optimization device based on TensorFlow includes: A model parsing module for parsing the input TensorFlow model using the tf.Graph and tf.Session tools of TensorFlow to extract the structure and parameters of the TensorFlow model; A model conversion module for converting the TensorFlow model into an intermediate representation IR supported by the NPU, and mapping the TensorFlow model operations to NPU operations through a custom conversion tool. The model conversion module is connected to the model parsing module; A model optimization module for optimizing the converted TensorFlow model, including operator fusion, memory optimization, and quantization, and improving the model performance by using the hardware characteristics of NPU low-precision computing; An NPU execution module for executing the optimized TensorFlow model on the NPU; A performance monitoring module for monitoring the running performance of a TensorFlow model on an NPU and outputting performance data such as the utilization rate of computing units, memory bandwidth, and power consumption.

[0013] Furthermore, the NPU execution module includes a model loading sub-module for loading the optimized TensorFlow model and an inference engine sub-module for performing model inference.

[0014] Furthermore, the present invention also proposes a computer-readable storage medium on which program instructions are stored, and when the program instructions are executed by a processor, the NPU model optimization method based on TensorFlow is implemented.

[0015] The beneficial effects of the present invention are as follows: The NPU model optimization method, device, and storage medium based on TensorFlow provide an efficient model conversion tool to quickly convert the TensorFlow model into a format supported by the NPU, improving the model conversion efficiency; making full use of the hardware characteristics of the NPU such as low-precision computing and parallel computing to enhance the computing efficiency, and reducing the conversion time and development cost through means such as operator fusion, memory optimization, and quantization; improving the comprehensive support for mainstream TensorFlow models and enhancing the compatibility between the model and the NPU hardware; providing a complete tool chain from model conversion to performance optimization to achieve end-to-end model optimization and deployment; reducing the manual operations of developers, simplifying the model deployment process, and reducing the deployment difficulty; supporting large-scale model deployment to meet the actual application requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 FIG. is the overall flowchart of the NPU model optimization method based on TensorFlow.

[0017] Figure 2 FIG. is a schematic diagram of the NPU model optimization device based on TensorFlow. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] The following will clearly and completely describe the concept, specific structure, and technical effects generated by the present invention in combination with the embodiments and the drawings to fully understand the purpose, solution, and effects of the present invention. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0019] It should be noted that, unless otherwise specified, when a feature is referred to as "fixed" or "connected" to another feature, it can be directly fixed or connected to the other feature, or indirectly fixed or connected to the other feature. In addition, the descriptions such as upper, lower, left, right, top, bottom, etc. used in the present invention are only relative to the mutual positional relationship of the components of the present invention in the drawings.

[0020] In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art of this technology. The terms used in the specification of this application are only for describing specific embodiments, rather than for limiting the present invention. The term "and / or" used herein includes any combination of one or more of the related listed items.

[0021] It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various elements, these elements should not be limited to these terms. These terms are only used to distinguish elements of the same type from each other. For example, without departing from the scope of this disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element.

[0022] Referring to Figures 1 to 2 , in some embodiments, the technical solution of the present invention is an NPU model optimization method based on TensorFlow. The NPU model optimization method based on TensorFlow is applied to an NPU model optimization device based on TensorFlow. The NPU model optimization device based on TensorFlow includes: a model parsing module, a model conversion module, a model optimization module, an NPU execution module, and a performance monitoring module. Referring to Figure 1 , the NPU model optimization method based on TensorFlow includes the following steps: S100. The model parsing module reads and extracts the structure and parameters of the TensorFlow model; S200. The model conversion module converts the TensorFlow model into an intermediate representation IR supported by the NPU; S300. Through operator fusion, memory optimization, and quantization, the model optimization module optimizes the converted TensorFlow model and deploys the optimized TensorFlow model to the NPU execution module; S400. The performance monitoring module monitors the running performance of the TensorFlow model on the NPU and outputs performance data.

[0023] The beneficial effects of the present invention are: The described NPU model optimization method, device, and storage medium based on TensorFlow provide an efficient model conversion tool to quickly convert TensorFlow models into a format supported by NPUs, improving the model conversion efficiency; make full use of the hardware features of NPUs such as low-precision computing and parallel computing to enhance the computing efficiency, and reduce the conversion time and development cost through means such as operator fusion, memory optimization, and quantization; improve the comprehensive support for mainstream TensorFlow models and enhance the compatibility between the model and NPU hardware; provide a complete tool chain from model conversion to performance optimization to achieve end-to-end model optimization and deployment; reduce the manual operations of developers, simplify the model deployment process, and lower the deployment difficulty; support large-scale model deployment to meet the actual application requirements.

[0024] Among them, NPU (Neural Processing Unit) is a processor specifically designed to accelerate artificial intelligence (AI) and machine learning (ML) tasks. It is optimized for the computing characteristics of neural networks and can efficiently execute matrix operations, convolution operations, and other deep learning-related computing tasks.

[0025] Further, step S100 includes: S110. The model parsing module reads the TensorFlow model from storage to obtain the structure and parameters of the TensorFlow model. S120. Parse the structure of the TensorFlow model to extract the computational graph Graph and operations Operation. S130. Based on custom mapping rules, map the TensorFlow model to NPU-compatible hardware operations.

[0026] Specifically, in TensorFlow, the parsing of model structure and parameters is usually achieved through graphs (Graphs) and tensors (Tensors). When TensorFlow runs on an NPU (Neural Processing Unit), the parsing process of the model structure and parameters will be different because NPUs usually have dedicated hardware accelerators and optimized computing libraries.

[0027] Further, step S130 includes: S131. Optimize the TensorFlow computational graph Graph, including merging operations and deleting redundant operations. S132. Divide the computational graph Graph into parts to be executed on the NPU and parts to be executed on the CPU. S133. Compile the computational graph Graph of the part to be executed on the NPU into an NPU instruction set. S134. Execute the compiled instruction set on the NPU.

[0028] In TensorFlow, the model structure is defined by a computation graph. Nodes in the computation graph represent operations (such as matrix multiplication, convolution, etc.), and edges represent tensors (data streams). Model parameters (such as weights and biases) are variables in the graph, and they are optimized during the training process.

[0029] When TensorFlow runs on the NPU, the model structure and parameters are parsed and converted into an instruction set that the NPU can understand. This includes the following steps: 1) Graph optimization: TensorFlow optimizes the computation graph, for example, merging operations, deleting redundant operations, etc.

[0030] 2) Graph partitioning: Divide the computation graph into parts that can be executed on the NPU and parts that need to be executed on the CPU.

[0031] 3) Graph compilation: Compile the NPU part of the computation graph into the NPU's instruction set.

[0032] 4) Execution: Execute the compiled instruction set on the NPU.

[0033] In a specific embodiment, for a convolutional neural network model, we want to run it on the NPU. The following is an example code: import tensorflow as tf # Define a simple convolutional neural network model def create_model(): model = tf.keras.Sequential([[ tf.keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)), tf.keras.layers.MaxPooling2D((2, 2)), tf.keras.layers.Flatten(), tf.keras.layers.Dense(128, activation='relu'), tf.keras.layers.Dense(10, activation='softmax')]) return model # Create a model model = create_model() # Compile the model model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) # An NPU device npu_device = ' / device:NPU:0' # Run the model on the NPU with tf.device(npu_device): # Assume we have some training data (x_train, y_train), (x_test, y_test) =tf.keras.datasets.mnist.load_data() x_train = x_train[..., tf.newaxis] / 255.0 x_test = x_test[..., tf.newaxis] / 255.0 # Train the model model.fit(x_train, y_train, epochs=5) # Evaluate the model model.evaluate(x_test, y_test) Furthermore, step S200 includes: S210. Generate an intermediate representation IR supported by the NPU according to the result of mapping the TensorFlow model to a hardware mapping compatible with the NPU; S220. Save the generated intermediate representation to a preset location to prepare for optimizing the TensorFlow model.

[0034] Specifically, the model conversion module converts the TensorFlow model into an intermediate representation IR supported by the NPU, including the following steps: 1) Read the TensorFlow model: Read the TensorFlow model from storage, including the model structure and parameters.

[0035] 2) Analyze the model structure: Analyze the structure of the TensorFlow model and extract the computational graph (Graph) and operations (Operation).

[0036] 3) Map TensorFlow operations to NPU operations: Map TensorFlow operations to the hardware operations supported by the NPU, using custom mapping rules to ensure compatibility.

[0037] 4) Generate the intermediate representation (IR): Generate the intermediate representation (IR) supported by the NPU based on the mapping results. The intermediate representation includes the computational graph, operations, and parameters.

[0038] 5) Save the intermediate representation: Save the generated intermediate representation to the specified location for model optimization.

[0039] Further, step S210 includes: S211. Perform custom operator registration and graph optimization by registering NPU-specific operators, creating a Grappler configuration optimized for the NPU, running the model using a session optimized for the NPU, and constructing a computational graph containing NPU operators; S212. Perform dynamic IR conversion and AI-driven optimization through dynamic intermediate representation IR conversion, AI-driven optimization strategy AI-Optimized IR, automatic mixed precision migration for NPU Auto Mixed Precision for NPU, and the underlying implementation of NPU IR conversion.

[0040] Further, in step S210, the intermediate representation IR includes the computational graph Graph, operations Operation, and parameters.

[0041] In a specific embodiment, taking Huawei Ascend as an example, the IR conversion process of TensorFlow on the NPU involves multi-level graph optimization and hardware adaptation. TensorFlow usually uses a computational graph to represent the model and optimizes this graph during runtime, which may be divided into different stages. For different hardware backends, such as CPU, GPU, or NPU, TensorFlow needs to convert the computational graph into an IR suitable for that hardware. For the NPU, this may involve specific graph optimization and operator conversion. The graph optimization process of TensorFlow usually uses Grappler for graph optimization and then converts the optimized graph into an NPU-specific IR through the NPU converter. For example, in the Huawei Ascend NPU, GE (Graph Engine) is used to handle graph conversion and execution.

[0042] The present invention mainly focuses on an AI-based automatic optimization strategy and uses reinforcement learning to dynamically adjust the graph optimization strategy.

[0043] For step S211, custom operator registration and graph optimization, the following is an example code: import tensorflow as tf from tensorflow.core.framework import node_def_pb2 from tensorflow.core.protobuf import rewriter_config_pb2 # Register NPU-specific operators (such as Huawei's AscendConv2D) def register_npu_ops(): npu_ops = ("AscendConv2D", "NpuConv2DOp", "float32"), ("AscendMatMul", "NpuMatMulOp", "float16") ] for op_name, op_class, dtype in npu_ops: tf.raw_ops.Register( f"{op_name}_{dtype}", op_class, dtype=dtype, device="NPU") # Create a Grappler configuration optimized for NPU def create_npu_grappler_config(): config = rewriter_config_pb2.RewriterConfig() config.meta_optimizer_iterations = rewriter_config_pb2.RewriterConfig.ONE config.add_optimizers("ascend_npu_optimizer") # Custom NPU optimizer return config # Run the model using a session optimized for NPU with tf.Session(config=tf.ConfigProto( graph_options = tf.GraphOptions( rewrite_options = create_npu_grappler_config()), device_count = {'NPU': 1} )) as sess: # Build a computation graph containing NPU operators register_npu_ops() x = tf.placeholder(tf.float32, shape=[1, 224, 224, 3]) w = tf.Variable(tf.truncated_normal([3, 3, 3, 64])) y = tf.raw_ops.AscendConv2D_float32(input = x, filter = w, strides=[1, 1, 1, 1]) sess.run(tf.global_variables_initializer()) print(sess.run(y, feed_dict={x: np.random.randn(1, 224, 224, 3)})) For step S212, dynamic IR conversion and AI-driven optimization, the following is an example code: Dynamic IR Adaptation A. Problem: Traditional static IR conversion cannot adapt to dynamic input shapes or conditional branches.

[0044] Solution: Introduce JIT (Just-In-Time) technology to dynamically generate NPU instructions: # Pseudocode: Dynamic JIT compilation def npu_jit_compile(compute_graph): if input_shape not in cached_ir: # Generate new IR according to the input shape new_ir = ascend_compiler.compile(compute_graph, input_shape) cached_ir[input_shape] = new_ir return cached_ir[input_shape] AI-driven Optimization Strategy (AI-Optimized IR) B. Problem: High cost in manually designing graph optimization rules.

[0045] Solution: Use reinforcement learning (RL) to automatically select the optimal optimization strategy: # Pseudo-code: Reinforcement learning selects the graph optimization order class IROptimizationAgent: def __init__(self): self.policy_network = build_rl_model() def select_optimization_sequence(self, graph): state = graph.feature_vector() action = self.policy_network.predict(state) # e.g., ["fuse_ops", "quantize"] return apply_actions(graph, action) C. Auto Mixed Precision for NPU Solution: Automatically identify operators suitable for low-precision calculations and insert type conversion nodes: # Pseudo-code: Auto mixed precision optimization def auto_mixed_precision(graph): for node in graph.nodes: if node.op_type in ["Conv2D", "MatMul"]: insert_cast(node.inputs[0], "float16") # Convert the input to FP16 insert_cast(node.outputs[0], "float32") # Restore the output to FP32 D. Underlying implementation of NPU IR conversion (taking Ascend as an example) The Ascend CANN library completes IR conversion through the following steps: 1) Graph segmentation: Split the TensorFlow graph into subgraphs supported and not supported by the NPU.

[0046] 2) Operator mapping: Map tf.Conv2D to AscendConv2D.

[0047] 3) IR serialization: Generate a *.om file (Ascend's binary IR format).

[0048] / / Pseudo C++ code: Operator mapping implementation void MapOpToNPU(const tensorflow::NodeDef& tf_node, ascend::Graph& ascend_graph) { if (tf_node.op() == "Conv2D") { auto* npu_conv = ascend_graph.add_node(); npu_conv->set_op("AscendConv2D"); npu_conv->add_input(tf_node.input(0)); npu_conv->add_input(tf_node.input(1)); / / Set Ascend-specific attributes npu_conv->set_attr("padding_mode", ascend::PADDING_SAME); } } Furthermore, step S300 includes: S310. Read the NPU-supported intermediate representation IR from storage; S320. Fuse multiple small operators into a large operator to reduce computational overhead; S330. Improve memory efficiency by memory reuse and data prefetching, optimize the memory access pattern of the TensorFlow model, and reduce memory bandwidth occupancy; S340. Convert the floating-point model to a low-precision model based on quantization-aware training QAT and / or post-training quantization PTQ to improve computational efficiency; S350. Verify whether the optimized TensorFlow model meets the accuracy and performance requirements. If the verification fails, return to steps S320 to S340; S360. If the verification is successful, save the optimized TensorFlow model to a preset location.

[0049] Specifically, through operator fusion, memory optimization, and quantization, the model optimization module optimizes the converted TensorFlow model. Deploying the optimized TensorFlow model to the NPU execution module includes the following steps: 1) Read the Intermediate Representation (IR): Read the Intermediate Representation (IR) supported by the NPU from storage, including the computation graph, operations, and parameters.

[0050] 2) Operator Fusion: Fuse multiple small operators into one large operator to reduce computational overhead. For example, fuse Convolution and the ReLU activation function into one operator.

[0051] 3) Memory Optimization: Optimize the memory access pattern of the model to reduce memory bandwidth occupancy. For example, improve memory efficiency through memory reuse and data prefetching techniques.

[0052] 4) Quantization: Convert the floating-point model to a low-precision model (such as INT8) to improve computational efficiency, using quantization-aware training (QAT) or post-training quantization (PTQ) techniques.

[0053] 5) Verify the Optimization Results: Verify whether the optimized model meets the accuracy and performance requirements. If the verification fails, return for re-optimization.

[0054] 6) Save the Optimized Model: Save the optimized model to the specified location, preparing for model deployment.

[0055] Among them, Operator Fusion is to combine multiple fine-grained computational operations into one larger operation to reduce the number of nodes in the computation graph, thereby reducing the overhead during execution. This optimization can reduce the number of memory accesses, improve cache utilization, and at the same time reduce the overhead of kernel launches, especially effective on dedicated hardware such as NPUs.

[0056] Operator Fusion includes: horizontal fusion, that is, merging adjacent operators with the same computational pattern (such as multiple Conv2D merges); vertical fusion, that is, merging consecutive front and back operators (such as Conv2D + BiasAdd + ReLU); and hybrid fusion, that is, combining horizontal and vertical (such as the QKV projection fusion in multi-head attention).

[0057] The implementation of operator fusion in TensorFlow includes using Grappler for automatic fusion, manually registering custom fusion operators, and NPU-specific fusion strategies, where: Using Grappler automatic fusion, the built-in Grappler optimizer in TensorFlow can automatically identify fusible operator patterns through the MetaOptimizer. For example, Conv2D + BiasAdd + ReLU can be merged into FusedConv2D.

[0058] Manually register custom fusion operators. For fusion patterns specific to the NPU (such as Conv2D+LayerNorm), it is necessary to manually define the fusion operator and register it with TensorFlow.

[0059] For example: Customize the Conv2D+LayerNorm fusion operator import tensorflow as tf from tensorflow.python.framework import ops # Forward calculation of the custom fusion operator def fused_conv2d_layernorm(inputs, filters, gamma, beta, strides,padding): conv = tf.nn.conv2d(inputs, filters, strides, padding) mean, var = tf.nn.moments(conv, axes=[1,2,3], keepdims=True) normalized = (conv - mean) / tf.sqrt(var + 1e-5) return gamma * normalized + beta # LayerNorm scaling and translation # Register the TensorFlow OP of the fusion operator @ops.RegisterGradient("FusedConv2DLayerNorm") def _fused_conv2d_layernorm_grad(op, grad): # Customize the backpropagation (need to implement the gradient calculation after fusion) x, w, gamma, beta = op.inputs # Omit the specific gradient calculation return [grad_x, grad_w, grad_gamma, grad_beta] # Use a custom fusion operator x = tf.placeholder(tf.float32, [1, 224, 224, 3]) w = tf.Variable(tf.truncated_normal([3,3,3,64])) gamma = tf.Variable(tf.ones(

[64] )) beta = tf.Variable(tf.zeros(

[64] )) output = fused_conv2d_layernorm(x, w, gamma, beta, strides=[1,1,1,1],padding='SAME') NPU-specific fusion strategy (taking Huawei Ascend as an example): NPU compilers (such as Huawei's CANN) usually provide more aggressive fusion strategies, which need to be implemented through hardware plugins.

[0060] AscendCL fusion configuration: On the Ascend platform, set the fusion rules through aclGenGraph and aclSetCompileOpt: / / Pseudo C code: Set Ascend fusion optimization options aclGraphHandle graph; aclCreateGraph(&graph); aclSetCompileOpt(graph, ACL_COMPILE_OPT_FUSION_ENABLE, true); / / Enable fusion aclSetCompileOpt(graph, ACL_COMPILE_OPT_FUSION_PATTERNS, "ConvBiasRelu,MatMulAdd"); / / Specify the fusion mode aclGenKernel(graph); / / Generate the fused NPU kernel TensorFlow and Ascend fusion docking Apply fusion automatically when converting the TensorFlow graph to Ascend IR through AscendGraphOptimizer: # Enable the Ascend Graph Optimizer in TensorFlow from npu_bridge.estimator import npu_ops config = tf.ConfigProto() config.graph_options.optimizer_options.global_jit_level =tf.OptimizerOptions.ON_1 config.graph_options.rewrite_options.ascend_graph_optimization =tf.AscendGraphOptimization(enable_fusion=True) Advanced Fusion Strategy: Dynamic Shape Adaptation For dynamic input shapes (such as variable-length sequences in NLP), a dynamic fusion strategy needs to be implemented: # Dynamic Shape Fusion Example (using TF conditional branches) def dynamic_fusion(x, dynamic_shape): if dynamic_shape > 1024: # Use block fusion for large inputs return tf.split_and_fuse_large_input(x) else: # Directly fully fuse small inputs return fused_conv2d_layernorm(x) For memory optimization, Quantization Aware Training (QAT): Simulate the quantization process during training to reduce quantization errors, so that the model can still maintain high accuracy after quantization. This is very important for reducing the memory footprint of the model and improving the inference speed on the NPU.

[0061] import tensorflow as tf # Assume model is your pre-trained model model =... # Create a quantization-aware model using the tf.quantization module quantize_model = tf.quantization.quantize_model(model) # Continue training with the quantization-aware model Custom operator optimization: Develop efficient custom operators for specific tasks and utilize the hardware characteristics of the NPU to accelerate calculations. This requires analyzing the key operations in the model and rewriting these operations according to the capabilities of the NPU.

[0062] / / Assume this is a simplified implementation of a convolutional layer optimized for the NPU void custom_convolution(const float* input, const float* weights,float* output, int input_height, int input_width,...) { / / Implementation details optimized for the NPU...} Mixed-precision training: Use 16-bit floating-point numbers instead of 32-bit floating-point numbers to represent weights and activation values, which can significantly reduce the memory footprint of the model and may speed up training and inference.

[0063] from tensorflow.keras import mixed_precision # Enable mixed precision mixed_precision.set_global_policy('mixed_float16') # Create and train your model model =... Dynamic batch size adjustment: Automatically adjust the batch size according to the real-time load situation to optimize resource utilization and response time.

[0064] Use TensorFlow for training or inference and achieve this through a custom dataset or iterator: class DynamicBatchDataset(tf.data.Dataset): def __init__(self, base_dataset, initial_batch_size=32): self.base_dataset = base_dataset self.current_batch_size = initial_batch_size def __iter__(self): iterator = iter(self.base_dataset) while True: batch = [] for _ in range(self.current_batch_size): try: element = next(iterator) batch.append(element) except StopIteration: break if not batch: break # Adjust the batch size according to the actual situation self.current_batch_size = adjust_batch_size(self.current_batch_size) yield tf.stack(batch) # Usage example dataset =... # Original dataset dynamic_batch_dataset = DynamicBatchDataset(dataset) for batch in dynamic_batch_dataset: # Process batches... For quantization, adopt the method of mixed-precision quantization: converter = tf.lite.TFLiteConverter.from_keras_model(model)converter.optimizations = [tf.lite.Optimize.DEFAULT] converter.target_spec.supported_types = [tf.float16] quantized_tflite_model = converter.convert() By specifying `supported_types` as `tf.float16`, the model can use half-precision floating-point numbers in part, which can provide good performance improvement and maintain high precision in many cases.

[0065] For the S400, monitoring the running performance of the TensorFlow model on the NPU is crucial for ensuring the efficient execution of the model and the effective utilization of resources. Through monitoring, the performance bottlenecks and resource usage of the model can be understood in real time, and optimization and adjustment can be made accordingly. Integrate a real-time monitoring tool to collect and visualize the performance metrics during model runtime: import tensorflow as tf import psutil def monitor_performance(): # Get the current device (assumed to be NPU) current_device = tf.config.list_physical_devices('NPU')[0] # Simulate getting NPU utilization (in reality, a specific NPU API may be required) npu_utilization = psutil.cpu_percent(interval=1) # Using CPU as an example in the print(f"NPU Utilization: {npu_utilization}%") return npu_utilization # Regularly call monitor_performance() in the inference loop to monitor performance To monitor the model performance more deeply, use Prometheus and Grafana: First, set up a Prometheus client to collect model performance metrics.

[0066] Then, use Grafana to create dashboards to visualize these metrics.

[0067] from prometheus_client import start_http_server, Gauge # Start the Prometheus HTTP server start_http_server(8000) # Define the performance metric npu_utilization_gauge = Gauge('npu_utilization', 'NPU Utilization Percentage') def monitor_and_report_performance(): utilization = monitor_performance() npu_utilization_gauge.set(utilization) Auto - tuning mechanism: Based on the collected performance data, use the auto - tuning mechanism to dynamically adjust model parameters: def auto_tune_model(batch_size): utilization = monitor_performance() if utilization < 30: batch_size += 16 # Increase the batch size elif utilization > 80: batch_size = max(16, batch_size - 16) # Decrease the batch size return batch_size Distributed tracing: Use OpenTelemetry to trace the model inference process: pip install opentelemetry - api opentelemetry - sdk opentelemetry - exporter - otlp from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter # Set the global TracerProvider trace.set_tracer_provider(TracerProvider()) tracer = trace.get_tracer(__name__) # Configure the exporter otlp_exporter = OTLPSpanExporter(endpoint="localhost:4317", insecure=True)span_processor = BatchSpanProcessor(ConsoleSpanExporter()) # Create a span during inference with tracer.start_as_current_span("model_inference"): # Perform inference Furthermore, referring to Figure 2 , the present invention also proposes an NPU model optimization device based on TensorFlow for performing the NPU model optimization method based on TensorFlow. The NPU model optimization device based on TensorFlow includes: A model parsing module for parsing the input TensorFlow model using the tf.Graph and tf.Session tools of TensorFlow to extract the structure and parameters of the TensorFlow model; A model conversion module for converting the TensorFlow model into an intermediate representation IR supported by the NPU, and mapping the TensorFlow model operations to NPU operations through a custom conversion tool. The model conversion module is connected to the model parsing module; A model optimization module for optimizing the converted TensorFlow model, including operator fusion, memory optimization, and quantization, and improving the model performance by using the hardware characteristics of NPU low-precision computing; An NPU execution module for executing the optimized TensorFlow model on the NPU; A performance monitoring module for monitoring the running performance of the TensorFlow model on the NPU and outputting performance data such as computing unit utilization, memory bandwidth, and power consumption.

[0068] Specifically, the input TensorFlow model includes a model structure and parameters; the model parsing module is used to parse the input TensorFlow model and extract the model structure and parameters; the TensorFlow API (such as tf.Graph and tf.Session) is used to read the TensorFlow model; the model conversion module converts the TensorFlow model into an intermediate representation (IR) supported by the NPU, and uses a custom conversion tool to map TensorFlow operations to NPU operations; the model optimization module optimizes the converted model, including operator fusion, memory optimization, and quantization, and uses the hardware characteristics of the NPU (such as low-precision computing) to improve the model performance; the NPU execution module (NPU runtime environment) is responsible for executing the optimized model on the NPU, and the NPU execution module (NPU runtime environment) includes the following model loading sub-module and inference engine sub-module; the performance monitoring module is used to monitor the running performance of the model on the NPU and output performance data (such as computing unit utilization, memory bandwidth, power consumption, etc.).

[0069] The data flow is described as follows: 1) Model parsing: The model parsing module reads the TensorFlow model and extracts the model structure and parameters.

[0070] 2) Model conversion: The model conversion module converts the TensorFlow model into an intermediate representation (IR) supported by the NPU.

[0071] 3) Model optimization: The model optimization module optimizes the converted model, including operator fusion, memory optimization, and quantization.

[0072] 4) Model deployment: Deploy the optimized model to the NPU runtime environment.

[0073] 5) Performance monitoring: The performance monitoring module monitors the running performance of the model on the NPU and provides performance data.

[0074] Furthermore, the NPU execution module includes a model loading sub-module for loading the optimized TensorFlow model and an inference engine sub-module for performing model inference.

[0075] Furthermore, the present invention also proposes a computer-readable storage medium, on which program instructions are stored, and when the program instructions are executed by a processor, the NPU model optimization method based on TensorFlow is implemented.

[0076] As described above, these are only the preferred embodiments of the present invention. The present invention is not limited to the above-described embodiments. As long as the same means are used to achieve the technical effects of the present invention, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure.

Claims

1. A NPU model optimization method based on TensorFlow, wherein the NPU model optimization method based on TensorFlow is applied to a NPU model optimization device based on TensorFlow, and the NPU model optimization device based on TensorFlow includes: The model parsing module, the model conversion module, the model optimization module, the NPU execution module and the performance monitoring module are characterized in that the NPU model optimization method based on TensorFlow comprises the following steps: S100, the model parsing module reads and extracts the structure and parameters of the TensorFlow model; S200, the model conversion module converts the TensorFlow model into an intermediate representation supported by the NPU; S300, the model optimization module optimizes the converted TensorFlow model through operator fusion, memory optimization and quantization, and deploys the optimized TensorFlow model to the NPU execution module; S400, the performance monitoring module monitors the running performance of the TensorFlow model on the NPU and outputs performance data.

2. The NPU model optimization method based on TensorFlow according to claim 1, characterized in that: Step S100 includes: S110, the model parsing module reads the TensorFlow model from the storage to obtain the structure and parameters of the TensorFlow model; S120, parse the structure of the TensorFlow model and extract the computational graph and operation; S130. Map the TensorFlow model to NPU-compatible hardware operations based on the custom mapping rule.

3. The NPU model optimization method based on TensorFlow according to claim 2, characterized in that: Step S130 includes: S131. Optimize the TensorFlow computation graph, including merging operations and removing redundant operations; S132, dividing the computation graph into a part to be executed on the NPU and a part to be executed on the CPU; S133, compiling the computation graph of the part executed on the NPU into an NPU instruction set; S134. Execute the compiled instruction set on the NPU.

4. The NPU model optimization method based on TensorFlow according to claim 1, characterized in that: Step S200 includes: S210, generating an intermediate representation supported by the NPU according to the result of mapping the TensorFlow model to an NPU-compatible hardware operation; S220, saving the generated intermediate representation to a preset location, preparing for TensorFlow model optimization.

5. The NPU model optimization method based on TensorFlow according to claim 4, characterized in that: Step S210 includes: S211, performing custom operator registration and graph optimization by registering NPU-specific operators, creating NPU-optimized Grappler configurations, running models using NPU-optimized sessions, and building computation graphs containing NPU operators; S212. Perform dynamic IR conversion and AI-driven optimization through dynamic intermediate representation IR conversion, AI-driven optimization strategy, mixed precision automatic migration and NPU IR conversion underlying implementation.

6. The NPU model optimization method based on TensorFlow according to claim 3, characterized in that: In step S210, the intermediate representation includes a computation graph, operations, and parameters.

7. The NPU model optimization method based on TensorFlow according to claim 1, characterized in that: Step S300 includes: S310, reading an intermediate representation supported by the NPU from storage; S320, merging multiple small operators into a large operator to reduce computing overhead; S330, improve memory efficiency through memory reuse and data prefetching, optimize the memory access mode of TensorFlow model, and reduce memory bandwidth usage; S340, based on quantization-aware training and / or post-training quantization, convert the floating-point model into a low-precision model to improve computing efficiency; S350, verify whether the optimized TensorFlow model meets the accuracy and performance requirements. If the verification fails, return to step S320 to step S340; S360. If the verification is successful, save the optimized TensorFlow model to the preset location.

8. A NPU model optimization device based on TensorFlow, used to execute the NPU model optimization method based on TensorFlow as described in any one of claims 1 to 7, characterized in that: The NPU model optimization device based on TensorFlow includes: Model parsing module, used to parse the input TensorFlow model using TensorFlow's tf.Graph and tf.Session tools, and extract the structure and parameters of the TensorFlow model; A model conversion module, used to convert the TensorFlow model into an intermediate representation supported by the NPU, and map the TensorFlow model operations to NPU operations through a custom conversion tool, wherein the model conversion module is connected to the model parsing module; Model optimization module, which is used to optimize the converted TensorFlow model, including operator fusion, memory optimization and quantization, and improve model performance by using the hardware characteristics of NPU low-precision computing; NPU execution module, used to execute the optimized TensorFlow model on the NPU; The performance monitoring module is used to monitor the running performance of the TensorFlow model on the NPU and output computing unit utilization, memory bandwidth, and power consumption performance data.

9. The NPU model optimization device based on TensorFlow according to claim 8, characterized in that: The NPU execution module includes a model loading submodule for loading the optimized TensorFlow model and an inference engine submodule for performing model inference.

10. A computer-readable storage medium, characterized in that: Program instructions are stored thereon, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images