Multi-modal basic model convolution operation optimization framework method suitable for GPU / DCU

Abstract

The invention provides a multi-modal basic model convolution operation optimization framework method suitable for a GPU / DCU. The method is used for solving the technical problems that the optimization process of an existing convolution operator optimization method is time-consuming and cross-layer operator fusion is difficult to capture. The method comprises the following steps: receiving a convolution operator through a parameterized input interface; collecting shape features of each convolution operator in the structured parameter set by using a shape analyzer to realize convolution feature analysis; judging whether batch normalization and nonlinear activation function fusion calculation is started or not; a heuristic optimizer dynamically selects a calculation graph optimization strategy based on convolution parameter characteristics; a calculation task is adapted to a physical architecture of the GPU / DCU, so that hardware resources are utilized to the maximum extent; a convolution calculation kernel function oriented to DCU / GPU architecture optimization is generated based on the convolution parameter features; kernel function assembly line execution is achieved through a DCU / GPU hardware task queue; gradient synchronization among multiple computing units is realized by adopting atomic operation at an equipment end. According to the method, the execution efficiency of the convolution operation in the multi-modal basic model can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and computer vision, and in particular to a multi-modal basic model convolution operation optimization framework method suitable for GPU / DCU. Background Art

[0002] In recent years, multimodal foundational models (such as CLIP, BLIP, and SAM) have achieved remarkable results in tasks such as vision-language alignment and image segmentation. These foundational models often use convolutional layers for visual feature extraction or mapping, and these convolutional operators often account for the majority of computational overhead during model runtime. However, the high complexity and diverse shapes of the convolutional operations contained in these large models pose challenges to inference and training performance. For example, the ResNet50 backbone network of the CLIP model contains numerous medium-sized convolutional operators, while the Patch Embedding layer of the SAM model is a single convolutional operator with a large spatial size. Because the number of channels, kernel size, and input / output sizes of convolutional layers vary significantly across models and tasks, traditional convolutional computing libraries (such as cuDNN, oneDNN, and MIOpen) typically implement acceleration in specific scenarios in a black-box manner, making it difficult to achieve optimal performance across a variety of convolutional modes. Furthermore, while automatic tuning compilers like TensorRT and TVM (Tensor Virtual Machine) can generate optimized code for a single convolution instance, the tuning process is time-consuming and makes it difficult to capture optimization opportunities for cross-layer operator fusion. Therefore, the industry urgently needs a universal convolution optimization framework that can automatically analyze the convolution shape characteristics in the model, select appropriate computational methods for different convolution types, and support efficient execution on multiple hardware platforms. Summary of the Invention

[0003] In response to the technical problems that the tuning process of existing convolution operator optimization methods is time-consuming and difficult to capture cross-layer operator fusion, the present invention proposes a multi-modal basic model convolution operation optimization framework method suitable for GPU / DCU. It can optimize the convolution operator by analyzing the convolution features and combining the GPU / DCU hardware environment to achieve efficient implementation of convolution operations, thereby improving the computing performance of the multimodal model.

[0004] To achieve the above objectives, the technical solution of the present invention is implemented as follows: a multi-modal basic model convolution operation optimization framework method suitable for GPU / DCU, the steps of which are as follows: Step 1: Receive the structured parameter set of the convolution operator through the parameterized input interface; Step 2: Use the shape analyzer to collect the shape features of each convolution operator in the structured parameter set to perform convolution feature analysis. Step 3: Determine whether to enable batch normalization and nonlinear activation function fusion calculation; the heuristic optimizer dynamically selects the calculation graph optimization strategy based on the convolution parameter characteristics; Step 4: Adapt the computing tasks to the physical architecture of the GPU / DCU to maximize the use of hardware resources; Step 5: Kernel function generation: Generate a convolution kernel function optimized for DCU / GPU architecture based on the convolution parameter features; Step 6: Implement kernel function pipeline execution through the DCU / GPU hardware task queue; use device-side atomic operations to achieve gradient synchronization between multiple computing units.

[0005] Preferably, the plurality of structured parameter sets include but are not limited to batch size, number of input / output channels, convolution kernel space size, sliding step size and padding strategy.

[0006] It traverses the computational graph of the model, intercepts and replaces the convolution nodes therein; when a convolution operator is detected, the shape analyzer extracts parameters such as the batch size, number of input channels, number of output channels, input / output dimensions, convolution kernel size, stride, etc. of the convolution operator, and evaluates the computational intensity and data access pattern of the convolution based on these shape features.

[0007] Preferably, the heuristic optimizer automatically selects the optimal convolution algorithm based on the shape features and hardware environment obtained by the shape analyzer, and identifies adjacent operators that can be fused through data dependency analysis and memory access pattern analysis, computing characteristic compatibility check and numerical equivalence verification.

[0008] Preferably, the heuristic optimizer applies predefined heuristic rules to select an optimization strategy based on the shape analysis results: when the convolution kernel size is 1×1, the im2col+GEMM scheme is used; when the number of input or output channels is less than 64 and the spatial size is greater than or equal to 128×128, FFT is considered; when the number of input or output channels is less than 128 and the spatial size is greater than or equal to 64×64, Winograd algorithm is considered; in other scenarios, direct convolution or manually tuned memory scheduling strategies are used; The heuristic optimizer scans the computational graph before and after the node where the convolution operator is located, identifies batch normalization and activation operators that can be fused with the convolution operator, and merges these operators with the convolution operator into an overall optimization task.

[0009] Preferably, the implementation method of the computational graph includes dynamic computational graph interception: intercepting the input computational graph flow, identifying the convolution operator and its adjacent operations to form a subgraph topology; multimodal operator analysis: parsing the data dependency between convolution operators by matching input tensors with output tensors, graph reachability analysis and control flow graph analysis; multimodal dependency modeling, establishing a cross-modal dependency matrix, using symbolic derivation verification, numerical stability verification and equivalence verification matrix to detect fusionable operator patterns; collecting DCU / GPU hardware feature parameters, fusion strategy decision engine: dynamically selecting optimization strategies based on cost models, generating device instruction sequences through AMD GCN and NVIDIA PTX instruction mapping, and activating DCU / GPU mixed precision computing units; heterogeneous kernel generation mechanism: calling optimized fusion operators in the ROCm math library or mapping virtual instructions to hardware instructions, and optimizing vector registers; computational graph rewriting and execution: completing subgraph matching, node replacement and dependency reconstruction to rewrite the original computational graph nodes into single fusion nodes; retaining the native execution path of unfused operators.

[0010] Preferably, the optimization strategy includes memory optimization, computation optimization, parallel optimization and precision optimization; By running a large number of operator benchmarks on GPU / DCU hardware, data on execution time, memory usage, and power consumption are collected to build a cost model.

[0011] Preferably, the method for adapting the computing task to the physical architecture of the GPU / DCU is: (1) Dynamic partitioning of computing units: Decomposing the computing load according to the dimensions of the convolution operator and mapping the computing tasks to the matrix computing core cluster of the DCU / GPU; (2) Distributed register scheduling: For large-scale convolution kernels, a multi-level register block strategy is adopted to reduce global memory access; (3) Collaborative data prefetching: Utilize the high-bandwidth memory characteristics of DCU or GPU to preload feature data through the asynchronous data transfer engine. For DCU, asynchronous transmission is triggered by the memcpy instruction. For GPU, NVIDIA GPUDirect RDMA is used to bypass the CPU and directly connect to NVMe. For scenarios with convolution step size > 1, invalid data prefetching and double buffering are skipped to hide data transfer delay and achieve parallel calculation and transmission.

[0012] Preferably, the multi-level registers include L0 private registers, L1 thread block shared registers and L2 on-chip cache registers; Use the convolution kernel function generator / selector to generate or select the execution kernel for the current convolution task according to the preset optimization decision, and hand it over to the lower-level execution engine for scheduling and execution; The convolution kernel generator / selector automatically adjusts parameters such as thread block size, memory layout, and parallelism based on runtime performance feedback.

[0013] Preferably, for GPU platforms, call pre-written CUDA templates or high-performance libraries to implement the optimized convolution calculation kernel function; for other GPU platforms, call the corresponding library or JIT compile to generate the kernel function of the platform; for CPU platforms, generate optimized code based on SIMD instructions and multi-threading; Tensor slice remapping technology is used to reorganize data intelligently, rearrange the data required for calculation in memory to make it conform to the hardware access mode, and integrate low-precision calculation instructions to accelerate matrix multiplication and accumulation operations.

[0014] Preferably, the input convolution operator is received by the shape analyzer and its parameter characteristics are analyzed. The heuristic optimizer determines the optimal algorithm and fusion scheme based on the analysis results. The convolution calculation kernel function generator / selector generates and outputs the kernel function code optimized for the convolution task according to the preset optimization strategy.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The optimizer of this invention can adaptively select algorithmic solutions for different convolution shapes, including direct convolution, matrix multiplication (im2col+GEMM), Winograd fast convolution, and FFT convolution. It supports fusing a convolution operator with its preceding and following operators, such as batch normalization (BN) and activation (ReLU), combining multiple operators into a single computational kernel for execution, thereby reducing memory read / write and scheduling overhead for intermediate results. This invention has excellent cross-platform adaptability and supports heterogeneous hardware backends. Users can integrate convolution implementations optimized for specific hardware platforms (such as NVIDIA GPUs, AMD GPUs, Sugon DCU accelerators, and CPUs) into the framework, and shape analysis automatically calls the appropriate kernel based on the platform at runtime. Relevant examples demonstrate that applying this framework on NVIDIA GPUs significantly improves the runtime efficiency of models such as CLIP and BLIP. Performance acceleration can also be achieved on the Sugon DCU platform through customized direct convolution kernels. Through the above technical solutions, this invention can significantly improve the execution efficiency of convolution operations in multimodal base models. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 Flowchart of the present invention.

[0018] Figure 2 Schematic diagram of convolution calculation of the multimodal basic model of the present invention.

[0019] Figure 3 Schematic diagram of the three-level convolutional framework of the multimodal basic model of the present invention. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0021] like Figure 1 As shown, a multi-modal basic model convolution operation optimization framework method suitable for GPU / DCU is provided. This embodiment is applicable to the multi-modal basic model convolution operation optimization of the Dawning supercomputer DCU architecture and the NVIDIA GPU architecture, and includes the following steps: Step 1: Receive the structured parameter set of the convolution operator through the parameterized input interface, including but not limited to batch size, number of input / output channels, convolution kernel spatial size, sliding step size and padding strategy.

[0022] The structured parameter set of the convolution operator is obtained through the convolution parameters input by the user. The structured parameter set serves as the basis for analyzing and extracting convolution features.

[0023] Step 2: Convolution feature extraction: Use the shape analyzer to collect the shape features of each convolution operator in the structured parameter set to implement convolution feature analysis.

[0024] The shape analyzer is used to collect information about the convolution operator, such as batch size, number of input and output channels, spatial dimensions, kernel size, and stride. By writing a convolution feature extraction algorithm in the shape analyzer, we can then perform convolution feature analysis based on the convolution parameters.

[0025] Reference Figure 1 and Figure 2The convolution optimization framework of this invention is integrated into the training or inference process as an intermediate layer module of a deep learning model. In specific implementation, the framework first traverses the model's computation graph, intercepting and replacing convolution nodes. When a convolution operator is detected, a shape analyzer extracts parameters such as the batch size, number of input channels, number of output channels, input / output dimensions, kernel size, and stride. Based on these shape features, the framework evaluates the computational intensity and data access pattern of the convolution. Interception involves the framework obtaining the complete computational graph through symbolic tracing or graph parsing, identifying convolution nodes (such as conv2d and depthwise_conv2d) within the graph. Replacement involves selecting the optimal implementation based on GPU / DCU hardware characteristics and convolution parameters. This includes matching small convolution kernels with the Winograd algorithm, matching large-size convolutions with FFT acceleration, and matching mobile devices with depthwise separable convolutions. Shape analysis guides algorithm selection based on computational intensity, while data access patterns determine memory optimization. Shape features determine data organization, while convolution features determine computational methods.

[0026] Step 3: Fusion of diversified convolution operators. Determine whether to enable batch normalization (BatchNorm) and nonlinear activation function fusion calculations; dynamically select the computational graph optimization strategy based on the convolution parameter characteristics.

[0027] The heuristic optimizer automatically selects the optimal convolution algorithm based on the shape features obtained by the shape analyzer and the hardware environment. It then identifies adjacent operators that can be fused through data dependency analysis, memory access pattern analysis, computational compatibility checks, and numerical equivalence verification. The hardware environment is a GPU / DCU computing environment. This identification of adjacent operators is done through data dependency analysis, memory access pattern analysis, computational compatibility checks, and numerical equivalence verification.

[0028] The heuristic optimizer applies predefined heuristic rules based on shape analysis results to select an optimization strategy. For example, when the convolution kernel size is 1×1, the im2col (image to column) + GEMM (general matrix multiplication) scheme is preferred, avoiding sliding window overhead and directly converting to fully connected matrix multiplication. When the number of input or output channels is less than 64 and the spatial size is greater than or equal to 128×128, FFT is preferred. When the number of input or output channels is less than 128 and the spatial size is greater than or equal to 64×64, the Winograd algorithm is considered, which can achieve a 3-7x speedup, a 2-3x improvement in end-to-end model inference, and a 30-50% reduction in memory usage. In other scenarios, direct convolution or manually tuned memory scheduling strategies can be used. The heuristic optimizer also scans the computational graph before and after the convolution operator node to identify operators such as batch normalization and activation that can be fused with the convolution operator. These operators are then combined with the convolution operator into a single optimization task, reducing memory read / write and kernel launch overhead.

[0029] Figure 2 This paper demonstrates an implementation method for the convolutional computation graph of a multimodal base model. The method includes dynamic computation graph interception: intercepting the input computation graph flow, identifying the convolution operator and its adjacent operations (such as BN, ReLU, and Pooling) to form a subgraph topology. Multimodal operator analysis: parsing the data dependencies between convolution operators through input tensor matching, graph reachability analysis, and control flow graph analysis. Multimodal dependency modeling is performed, and a cross-modal dependency matrix is ​​established. Symbolic derivation, numerical stability, and equivalence verification of the matrix are used to detect fusible operator patterns (such as Conv-BN-ReLU triples). Furthermore, DCU / GPU hardware characteristic parameters are collected. A fusion strategy decision engine: dynamically selects optimization strategies based on a cost model. Optimization strategies include memory optimization, computational optimization, parallel optimization, and precision optimization. A large number of operator benchmarks are run on GPU / DCU hardware to collect data such as execution time, memory usage, and power consumption, and then construct a cost model. Generate device-specific instruction sequences through AMD GCN (Graphics Core Next) and NVIDIA PTX (Parallel Thread Execution) instruction mappings, activating the DCU / GPU mixed-precision compute units. A heterogeneous kernel generation mechanism: Calls optimized fused operators from the ROCm math library or maps virtual instructions to hardware instructions, optimizing VGPR (vector register) usage. Instruction scheduling hides memory latency (AMD's L1 / L2 cache strategy) and outputs device-specific machine code (e.g., the AMD GCN ISA). Graph rewriting and execution: Completes subgraph matching, node replacement, and dependency reconstruction, rewriting original graph nodes into a single fused node. The native execution paths of unfused operators are preserved to ensure graph compatibility.

[0030] Step 4: DCU / GPU hardware-aware mapping. The core goal of hardware-aware mapping is to efficiently and intelligently adapt computing tasks to the physical architecture of specific hardware (such as the GPU / DCU) to maximize the utilization of hardware resources (computing units, memory bandwidth, cache, etc.). Its implementation is a multi-layered, cross-domain systems engineering project involving compilers, runtime systems, algorithm design, and performance modeling.

[0031] (1) Dynamic partitioning of computing units: Decomposing the computing load according to the dimensions of the convolution operator, mapping the computing tasks to the matrix computing core cluster of the DCU / GPU, thus improving computing power utilization; (2) Distributed register scheduling: For large-scale convolution kernels, a multi-level register block strategy is adopted: L0 private registers, L1 thread block shared registers and L2 on-chip cache registers, that is, a three-level L0, L1 and L2 block structure. The convolution input data retains adjacent areas in the registers, reducing repeated loading, avoiding spilling, and reducing global memory access; (3) Collaborative data prefetching: Utilize the high-bandwidth memory characteristics of DCU or GPU to preload feature data through the asynchronous data transfer engine. For DCU, asynchronous transmission is triggered by the memcpy instruction. For GPU, NVIDIA GPUDirect RDMA is used to bypass the CPU and directly connect to NVMe. For scenarios with convolution step size > 1, invalid data prefetching and double buffering are skipped to hide data transfer delay and achieve parallel calculation and transmission.

[0032] Step 5: Kernel function code generation: Generate a convolution kernel function optimized for DCU / GPU architecture based on the convolution parameter features.

[0033] The convolution kernel function generator / selector is responsible for generating or selecting efficient kernel code for the current convolution task and handing it over to the lower-level execution engine for scheduling and execution.

[0034] The convolution kernel generator / selector generates or selects specific execution kernels based on preset optimization decisions. A variety of optimization decisions are currently available, which can be implemented using existing libraries or custom-written. For GPUs, the framework can call pre-written CUDA templates or high-performance libraries (such as cuBLAS and cuDNN) to implement the optimized convolution kernel. For other GPU platforms, corresponding libraries can be called or JIT-compiled to generate platform-specific kernels. For CPU platforms, optimized code based on SIMD instructions and multithreading can be generated. The convolution kernel generator / selector can also automatically adjust parameters such as thread block size, memory layout, and parallelism based on runtime performance feedback to further improve execution efficiency.

[0035] Tensor slice remapping technology is used to intelligently reorganize data and rearrange the data required for calculation in memory to conform to hardware access patterns (such as continuous memory access and alignment requirements), thereby improving data locality, efficient stride access, and reducing memory discontinuity. Low-precision calculation instructions (FP16 / BF16) are integrated to accelerate matrix multiplication and accumulation operations.

[0036] Step 6: Heterogeneous task scheduling: Implement pipeline execution of kernel functions through DCU / GPU hardware task queues to improve computation-communication overlap; implement device-side atomic operations to synchronize gradients between multiple computing units and reduce execution gaps.

[0037] Figure 3The system structure diagram of the convolution optimization framework of the present invention is shown. Applicable to multimodal basic models, the framework includes modules such as input convolution operation, shape analyzer, heuristic optimizer, convolution kernel function generation / selector (KernelGenerator / Selector), and output optimized convolution kernel (Output Optimized Conv Kernel). Arrows between modules indicate the direction of data flow: the input convolution operator is first received by the shape analyzer and its parameter characteristics are analyzed. Then, the heuristic optimizer determines the optimal algorithm and fusion scheme based on the analysis results. Finally, the convolution calculation kernel function generation / selector generates and outputs the kernel function code optimized for the convolution task according to the preset optimization strategy. Figure 3 The main functional points of each module are marked respectively, reflecting the processing flow from input to output of the framework of the present invention.

[0038] Through the collaborative work of the above components, the framework of the present invention can replace the convolution operation with an efficient implementation optimized for the input shape and hardware environment, thereby improving the operational performance of the multimodal model.

[0039] For example, in the ResNet50 backbone network of the CLIP model, after a convolutional layer is identified by the shape analyzer, the heuristic optimizer discovers that the convolution operator in this layer is followed by BN and ReLU operators. The convolution operator is then fused with the BN and ReLU operators, and the fused kernel is generated using the im2col+GEMM algorithm. Field measurements show that this fused operator runs approximately 1.2 times faster on an NVIDIA A100 GPU. Similarly, in the Patch Embedding phase of the SAM model, a convolutional layer with a large spatial size is identified as suitable for the Winograd algorithm. The resulting Winograd convolution kernel improves the execution efficiency of this layer by approximately 1.3 times. Furthermore, when running on the Sugon DCU accelerator, the framework of the present invention can also prioritize the use of direct convolution kernels customized for the DCU, thereby improving the inference performance of the CLIP model's ResNet50 backbone network by approximately 1.8 times.

[0040] To sum up, through the above specific implementation methods, the present invention realizes a general optimization framework and method for convolution operations suitable for multimodal basic models, which can significantly improve the execution efficiency of various convolution operators, adapt to different hardware platforms, and meet the high performance requirements of multimodal deep learning models, and has important practical value.

[0041] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-modal basic model convolution operation optimization framework method suitable for GPU / DCU, characterized by: The steps are as follows: Step 1: Receive the structured parameter set of the convolution operator through the parameterized input interface; Step 2: Use the shape analyzer to collect the shape features of each convolution operator in the structured parameter set to perform convolution feature analysis. Step 3: Determine whether to enable batch normalization and nonlinear activation function fusion calculation; the heuristic optimizer dynamically selects the calculation graph optimization strategy based on the convolution parameter characteristics; Step 4: Adapt the computing tasks to the physical architecture of the GPU / DCU to maximize the use of hardware resources; Step 5: Kernel function generation: Generate a convolution kernel function optimized for DCU / GPU architecture based on the convolution parameter features; Step 6: Implement kernel function pipeline execution through DCU / GPU hardware task queue; use device-side atomic operations to synchronize gradients between multiple computing units.

2. The multimodal basic model convolution operation optimization framework method suitable for GPU / DCU according to claim 1 is characterized in that: The plurality of structured parameter sets include but are not limited to batch size, number of input / output channels, convolution kernel spatial size, sliding step size and padding strategy; It traverses the computational graph of the model, intercepts and replaces the convolution nodes therein; when a convolution operator is detected, the shape analyzer extracts parameters such as the batch size, number of input channels, number of output channels, input / output dimensions, convolution kernel size, stride, etc. of the convolution operator, and evaluates the computational intensity and data access pattern of the convolution based on these shape features.

3. The multimodal basic model convolution operation optimization framework method suitable for GPU / DCU according to claim 1 or 2, characterized in that: The heuristic optimizer automatically selects the optimal convolution algorithm based on the shape features and hardware environment obtained by the shape analyzer, and identifies adjacent operators that can be fused through data dependency analysis and memory access pattern analysis, computing feature compatibility check and numerical equivalence verification.

4. The multimodal basic model convolution operation optimization framework method suitable for GPU / DCU according to claim 3 is characterized in that: The heuristic optimizer applies predefined heuristic rules based on the shape analysis results to select an optimization strategy: when the convolution kernel size is 1×1, the im2col+GEMM scheme is used; when the number of input or output channels is less than 64 and the spatial size is greater than or equal to 128×128, FFT is considered; when the number of input or output channels is less than 128 and the spatial size is greater than or equal to 64×64, the Winograd algorithm is considered; in other scenarios, direct convolution or manually tuned memory scheduling strategies are used; The heuristic optimizer scans the computational graph before and after the node where the convolution operator is located, identifies batch normalization and activation operators that can be fused with the convolution operator, and merges these operators with the convolution operator into an overall optimization task.

5. The multi-modal basic model convolution operation optimization framework method suitable for GPU / DCU according to claim 2 is characterized in that: The computational graph implementation method includes dynamic computational graph interception: intercepting the input computational graph flow, identifying the convolution operator and its adjacent operations to form the subgraph topology; multimodal operator analysis: parsing the data dependencies between convolution operators by matching input tensors with output tensors, graph reachability analysis, and control flow graph analysis; Multimodal dependency modeling, establishing a cross-modal dependency matrix, using symbolic derivation verification, numerical stability verification, and equivalence verification matrix to detect fusible operator patterns; collecting DCU / GPU hardware characteristic parameters, fusion strategy decision engine: dynamically select optimization strategies based on cost models, generate device instruction sequences through AMD GCN and NVIDIA PTX instruction mapping, and activate DCU / GPU mixed-precision computing units; heterogeneous kernel generation mechanism: call optimized fusion operators in the ROCm math library or map virtual instructions to hardware instructions, and optimize vector registers; computational graph rewriting and execution: complete subgraph matching, node replacement, and dependency reconstruction to rewrite the original computational graph nodes into a single fused node; retain the native execution path of unfused operators.

6. The multi-modal basic model convolution operation optimization framework method suitable for GPU / DCU according to claim 5 is characterized in that: The optimization strategy includes memory optimization, computation optimization, parallel optimization and precision optimization; By running a large number of operator benchmarks on GPU / DCU hardware, data on execution time, memory usage, and power consumption are collected to build a cost model.

7. The multimodal basic model convolution operation optimization framework method applicable to GPU / DCU according to any one of claims 1-2, 4-6, characterized in that: The method for adapting computing tasks to the physical architecture of GPU / DCU is as follows: (1) Dynamic partitioning of computing units: Decomposing the computing load according to the dimensions of the convolution operator and mapping the computing tasks to the matrix computing core cluster of the DCU / GPU; (2) Distributed register scheduling: For large-scale convolution kernels, a multi-level register block strategy is adopted to reduce global memory access; (3) Collaborative data prefetching: Utilize the high-bandwidth memory characteristics of DCU or GPU to preload feature data through the asynchronous data transfer engine. For DCU, asynchronous transmission is triggered by the memcpy instruction. For GPU, NVIDIA GPUDirect RDMA is used to bypass the CPU and directly connect to NVMe. For scenarios with convolution step size > 1, invalid data prefetching and double buffering are skipped to hide data transfer delay and achieve parallel calculation and transmission.

8. The multi-modal basic model convolution operation optimization framework method suitable for GPU / DCU according to claim 7 is characterized in that: The multi-level registers include L0 private registers, L1 thread block shared registers and L2 on-chip cache registers; Use the convolution kernel function generator / selector to generate or select the execution kernel for the current convolution task according to the preset optimization decision, and hand it over to the lower-level execution engine for scheduling and execution; The convolution kernel generator / selector automatically adjusts parameters such as thread block size, memory layout, and parallelism based on runtime performance feedback.

9. The multi-modal basic model convolution operation optimization framework method suitable for GPU / DCU according to claim 8, characterized in that: For GPU platforms, call pre-written CUDA templates or high-performance libraries to implement optimized convolution calculation kernel functions; for other GPU platforms, call corresponding libraries or JIT-compiled kernel functions generated by the platform; For CPU platforms, generate optimized code based on SIMD instructions and multi-threading; Tensor slice remapping technology is used to reorganize data intelligently, rearrange the data required for calculation in memory to make it conform to the hardware access mode, and integrate low-precision calculation instructions to accelerate matrix multiplication and accumulation operations.

10. The multi-modal basic model convolution operation optimization framework method applicable to GPU / DCU according to claim 8 or 9, characterized in that: The input convolution operator is received by the shape analyzer and its parameter characteristics are analyzed. The heuristic optimizer determines the optimal algorithm and fusion scheme based on the analysis results, and the convolution calculation kernel function generator / selector generates and outputs the kernel function code optimized for the convolution task according to the preset optimization strategy.

Images