A model deployment method and device combined with hardware deployment

Abstract

This application discloses a model deployment method and device combined with hardware deployment. According to the device nodes in the target model deployment environment and the corresponding connection relationships of each device node, a hardware topology diagram corresponding to the device node is constructed; the dependency relationship of the target operator in the target model is parsed to generate a model calculation diagram corresponding to the target operator; based on the hardware topology diagram and the model calculation diagram, a preset evaluation algorithm is used to calculate the target matching value between the device node and the target operator; based on the target matching value and the preset grading strategy, the deployment plan corresponding to the target model is determined. Through the dynamic matching mechanism of real-time perception of hardware status and model calculation diagram, the optimal deployment strategy is automatically generated, and adaptive optimization is triggered when the hardware topology or resource conditions change, so as to achieve real-time collaborative adaptation of the model segmentation strategy and the heterogeneous environment, significantly improving the level of deployment automation and system elasticity.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a model deployment method and device combined with hardware deployment. Background Art

[0002] The purpose of artificial intelligence model deployment is to transform large-model technology into actual productivity. Model deployment promotes technology from the laboratory to industrial application by optimizing computing resources, improving data processing efficiency and enhancing scenario adaptability.

[0003] In current model deployment practices, technicians often use an experience-driven offline model partitioning strategy, deploying models to target devices in a single deployment based on manual pre-determination. This approach has three limitations: First, it relies too heavily on technicians' understanding of hardware topology, which can easily lead to deviations between the model parallelization strategy and the actual hardware architecture. Second, when computing node configurations or network topologies undergo dynamic changes, manual resource reassessment and strategy formulation are required, lacking real-time responsiveness. Third, the entire deployment process lacks automation, making it difficult to dynamically adapt the model partitioning strategy to the hardware environment. This static deployment model has become a key bottleneck restricting the elastic expansion of AI systems. Summary of the Invention

[0004] The present application provides a model deployment method and device combined with hardware deployment to solve the above-mentioned problem that reliance on the experience of technical personnel easily leads to a mismatch between the model parallel strategy and the physical connection topology, and when the hardware resources change, manual re-evaluation and adoption of a new model segmentation strategy are required, and the entire process cannot be automated.

[0005] In a first aspect, the present application provides a model deployment method combined with hardware deployment, comprising:

[0006] According to the device nodes in the target model deployment environment and the corresponding connection relationships of each device node, a hardware topology diagram corresponding to the device node is constructed;

[0007] Analyze the dependencies of the target operators in the target model to generate the model calculation graph corresponding to the target operators;

[0008] According to the hardware topology diagram and model calculation diagram, the target matching value between the device node and the target operator is calculated using the preset evaluation algorithm;

[0009] Determine the deployment plan corresponding to the target model based on the target matching value and the preset grading strategy.

[0010] In a second aspect, the present application provides a model deployment device combined with hardware deployment, comprising:

[0011] A hardware topology determination module is configured to construct a hardware topology corresponding to the device nodes according to the device nodes in the target model deployment environment and the connection relationships corresponding to the device nodes;

[0012] a model computation graph determination module, configured to parse the dependency relationship of the target operator in the target model to generate a model computation graph corresponding to the target operator;

[0013] a target matching value determination module configured to calculate a target matching value between a device node and a target operator using a preset evaluation algorithm based on the hardware topology graph and the model calculation graph;

[0014] The deployment scheme determination module is configured to determine the deployment scheme corresponding to the target model according to the target matching value and the preset classification strategy.

[0015] In a third aspect, the present application provides a readable medium comprising execution instructions. When a processor of an electronic device executes the execution instructions, the electronic device executes any method described in the first aspect.

[0016] In a fourth aspect, the present application provides an electronic device comprising a processor and a memory storing execution instructions. When the processor executes the execution instructions stored in the memory, the processor executes any method described in the first aspect.

[0017] This application provides a model deployment method combined with hardware deployment. According to the device nodes in the target model deployment environment and the corresponding connection relationships of each device node, a hardware topology diagram corresponding to the device node is constructed; the dependency relationship of the target operator in the target model is parsed to generate a model calculation graph corresponding to the target operator; based on the hardware topology diagram and the model calculation graph, a preset evaluation algorithm is used to calculate the target matching value between the device node and the target operator; based on the target matching value and the preset grading strategy, the deployment plan corresponding to the target model is determined. Through the dynamic matching mechanism of real-time perception of hardware status and model calculation graph, the optimal deployment strategy is automatically generated, and adaptive optimization is triggered when the hardware topology or resource conditions change, so as to achieve real-time collaborative adaptation of the model segmentation strategy and the heterogeneous environment, significantly improving the level of deployment automation and system elasticity.

[0018] The further effects of the above-mentioned non-conventional preferred embodiment will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the existing technical solutions, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 A flowchart of a model deployment method combined with hardware deployment provided in one embodiment of the present application;

[0021] Figure 2 A flowchart of another model deployment method combined with hardware deployment provided in one embodiment of the present application;

[0022] Figure 3 A flowchart of another model deployment method combined with hardware deployment provided in one embodiment of the present application;

[0023] Figure 4 This is a flowchart of another model deployment method combined with hardware deployment provided by an embodiment of the present application;

[0024] Figure 5 This is a structural diagram of a model deployment device combined with hardware deployment provided by an embodiment of the present application;

[0025] Figure 6 A schematic structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0027] The purpose of artificial intelligence model deployment is to transform large-model technology into actual productivity. Model deployment promotes technology from the laboratory to industrial application by optimizing computing resources, improving data processing efficiency and enhancing scenario adaptability.

[0028] In current model deployment practices, technicians often use an experience-driven offline model partitioning strategy, deploying models to target devices in a single deployment based on manual pre-determination. This approach has three limitations: First, it relies too heavily on technicians' understanding of hardware topology, which can easily lead to deviations between the model parallelization strategy and the actual hardware architecture. Second, when computing node configurations or network topologies undergo dynamic changes, manual resource reassessment and strategy formulation are required, lacking real-time responsiveness. Third, the entire deployment process lacks automation, making it difficult to dynamically adapt the model partitioning strategy to the hardware environment. This static deployment model has become a key bottleneck restricting the elastic expansion of AI systems.

[0029] To solve this problem, the present application proposes a model deployment method combined with hardware deployment, aiming to solve the above-mentioned reliance on technical personnel's experience, which easily leads to a mismatch between the model parallel strategy and the physical connection topology. Moreover, after the hardware resources change, manual re-evaluation and adoption of a new model segmentation strategy are required, and the entire process cannot be automated. In this embodiment, a model deployment method combined with hardware deployment includes:

[0030] Step 101: Construct a hardware topology diagram corresponding to the device nodes according to the device nodes in the target model deployment environment and the connection relationships corresponding to the device nodes.

[0031] In a model deployment environment, a device node represents a physical hardware unit with independent computing and storage capabilities, such as a GPU, CPU, TPU, or edge computing device. The core attributes of each device node include static performance indicators such as theoretical computing power, video memory capacity, and input and output bandwidth limits, as well as dynamic status information such as real-time available video memory, current computing power load, and actual bandwidth utilization. For example, in a GPU cluster, the device node must clearly specify the number of CUDA cores, video memory specifications, and interconnection topology. In edge computing scenarios, the device nodes may include heterogeneous CPUs and low-power accelerators, and their energy consumption constraints and other characteristics must be collected simultaneously.

[0032] The connection between devices is quantitatively modeled using communication costs, with key parameters including physical transmission latency and effective bandwidth. For example, if two GPUs are connected via PCIe 4.0 x16, the theoretical bidirectional bandwidth is 32 GB / s. However, the actual effective bandwidth may be affected by protocol overhead or link contention, so the minimum value measured in both directions should be used. Latency, on the other hand, includes both the start-up time and propagation time of data transmission. Communication costs can be further converted into the time cost per unit of data transmitted, thus characterizing the difference in data exchange efficiency between devices.

[0033] Based on this information, the hardware topology is abstractly represented as a graph structure, where vertex sets correspond to device nodes and edge weights map to communication costs. For example, in a multi-GPU training cluster, the edge weights between devices connected via NVLink are significantly lower than those between devices connected via PCIe. These high-bandwidth, low-latency links are prioritized for high-frequency parameter synchronization.

[0034] Through this structured modeling, the hardware topology diagram not only intuitively presents the resource distribution and connection characteristics, but also provides a global decision-making basis for dynamic optimization algorithms. For example, it automatically adjusts the task allocation path when resources fluctuate to achieve a global optimal balance between computing power and communication efficiency.

[0035] Step 102: Analyze the dependency relationship of the target operator in the target model to generate a model calculation graph corresponding to the target operator.

[0036] To analyze the dependencies of target operators within a target model, we must begin with the model's computational logic and deconstruct its execution process and data flow path layer by layer. Taking deep learning models as an example, we first analyze the model's structure statically (e.g., the connection order of neural network layers, branch conditions, or loop control flow) to extract the input-output relationships of all operators. For example, the output tensor of a convolutional layer may serve as the input of an activation function layer, while the addition operation in a residual connection must synchronously aggregate the outputs of multiple branches. These dependencies directly define the execution order of operators and the data transfer rules.

[0037] On this basis, the model computation graph is abstractly represented as a directed graph, where vertices represent operators and edges represent data flow dependencies. Each operator vertex must record its computational cost (such as floating-point operations) and resource requirements (such as video memory usage and bandwidth requirements), while data flow edges quantify the amount of data transferred between adjacent operators (such as tensor dimensions and data types). For example, in the Transformer model, the multi-head computation of the self-attention layer generates an intermediate key-value pair matrix, the transmission volume of which must be dynamically calculated based on the number of heads and sequence length. This information directly affects the subsequent evaluation of inter-device communication costs.

[0038] For complex models (such as graph structures with dynamic control flow or conditional branches), symbolic execution or dynamic tracing techniques must be further introduced to ensure the completeness and accuracy of dependency relationships. For example, in recurrent neural networks, the transfer of hidden states between time steps may form a cross-iteration data dependency chain, and its computational path must be clarified through the time dimension. The resulting model computation graph must not only reflect the explicit data flow between operators but also capture implicit resource competition relationships (such as the mutual exclusivity between operators in a shared memory buffer), thereby providing an optimization basis for dynamic deployment.

[0039] Step 103: Calculate the target matching value between the device node and the target operator using a preset evaluation algorithm based on the hardware topology diagram and the model calculation diagram.

[0040] During dynamic deployment, the target matching value is calculated by first extracting the multi-dimensional resource requirements of each operator from the model computation graph. These include computing power requirements (such as floating-point operations), video memory usage, bandwidth requirements, and quantization support requirements. For example, convolution operators may require high video memory capacity, while fully connected layers rely more on the parallel processing capabilities of high-performance devices.

[0041] The hardware topology map also provides real-time performance characteristics of device nodes, including theoretical computing power limits, currently available memory, measured bandwidth, and supported quantization precision. For example, the available memory of a GPU node may be dynamically reduced due to other tasks, and its actual bandwidth may be lower than the theoretical value due to network congestion.

[0042] The evaluation algorithm uses a multi-dimensional matching quantification model to analyze the differences between operator requirements and device capabilities. Specifically, for each resource dimension (such as video memory), the normalized difference between operator requirements and device capabilities is calculated. The parameters of each dimension can first be normalized by mean-variance to eliminate dimensional differences (for example, converting video memory requirements from GB to the same standardized unit as device video memory). Subsequently, the difference is mapped to a single-dimensional matching degree through an exponential function. Ultimately, the matching degrees of all dimensions are combined into a global target matching value in the form of a product. For example, if an operator has high requirements for video memory and bandwidth, and a device has sufficient video memory but limited bandwidth, its overall matching value will be significantly lower than that of a device with both sufficient video memory and bandwidth.

[0043] Step 104: Determine the deployment plan corresponding to the target model based on the target matching value and the preset classification strategy.

[0044] According to the grading threshold corresponding to the grading strategy, the target matching level corresponding to the target matching value is determined; the target matching level includes high matching level, low matching level and medium matching level; the deployment plan is determined based on the target matching level.

[0045] In the development of dynamic deployment plans, a tiered strategy is based on the quantitative results of target matching values. Device-operator compatibility is divided into three levels: high, medium, and low, according to tiered thresholds. Differentiated optimization methods are then applied to each level. The design of tiered thresholds must balance the continuity of resource adaptation with the discrete nature of policy execution. Threshold settings are typically based on historical deployment data statistics and scenario-based experimental verification.

[0046] For example, in a memory-intensive task scenario, if the target matching values ​​of 90% of efficient deployment cases are concentrated above 0.8, the threshold for a high matching level can be initially set to 0.8, while a target matching value below 0.3 may correspond to a low-level scenario with severe resource shortages.

[0047] When the target matching value belongs to a high matching level, the target operators are prioritized based on the numerical value of the target matching value to generate a deployment queue corresponding to the target operator; the target operator is deployed to the corresponding device node according to the deployment queue.

[0048] When the target match value is high, the system uses a priority sorting mechanism to achieve optimal resource allocation. First, highly adaptable operators are globally sorted based on the numerical value of the match value. For example, a convolution operator with a match value of 0.95 is prioritized over a fully connected layer operator with a match value of 0.88, forming a deployment queue.

[0049] This ranking not only reflects the absolute level of device-operator compatibility but also requires dynamic adjustment based on the dependencies within the model's computational graph. If a highly compatible operator has a predecessor dependency (e.g., the output of a data preprocessing layer is the input to a subsequent layer), its deployment order must be optimized while maintaining the dependency chain to avoid idle resources due to task blocking. For example, in an image recognition model, although the feature extraction operator's matching score is slightly lower than that of the classification layer, it still requires priority deployment as the starting point of the computational process to ensure the coherence of the overall task pipeline.

[0050] After generating a deployment queue, the system dynamically allocates tasks based on the real-time load status of device nodes. For example, if a GPU is already loaded with high-computing tasks, even if it has the highest match value for the target operator, the system may select a suboptimal but less loaded device to avoid performance degradation caused by local resource overload. Such decisions require a comprehensive assessment of multiple metrics, such as available device memory, bandwidth utilization, and task queue duration. For example, a weighted scoring model can be used to dynamically adjust the deployment order.

[0051] The execution of deployment queues must be deeply coupled with the communication costs of the hardware topology. For example, in distributed training, parameter servers with high matching scores are preferentially deployed to groups of devices with high interconnect bandwidth and low latency, while also ensuring that the data exchange paths between adjacent compute nodes are minimized. If a sudden change in device status (such as a sudden drop in graphics memory or network congestion) is detected during deployment, the system triggers a dynamic reconfiguration of the queue—pausing the current deployment, recalculating matching values, and generating an updated queue to maintain global resource utilization efficiency.

[0052] This closed-loop optimization mechanism ensures that high-matching deployment strategies not only have static priorities but also respond to environmental changes in real time. For example, in an autonomous driving scenario, when the on-board computing unit experiences a surge in load due to sudden tasks, the system automatically migrates some high-matching tasks to roadside edge nodes, ensuring that decision-making tasks with the highest real-time requirements continue to receive optimal resource support.

[0053] When the target matching value belongs to a low matching level, determine the original subgraph corresponding to the target operator in the model calculation graph; use the calculation path reorganization algorithm to reorganize the path of the original subgraph to generate an optimized subgraph; replace the original subgraph with the optimized subgraph to generate the corresponding optimized calculation graph; calculate the first matching value corresponding to the optimized calculation graph; when the first matching value does not belong to a low matching level, use the optimized subgraph to update the model calculation graph.

[0054] When the target match value falls into a low match level, the system initiates a computational path reorganization mechanism. First, the original subgraph associated with the low-match operator is located in the model computation graph, including the data flow dependencies between its predecessor and successor nodes. For example, if the self-attention layer in a Transformer model is judged to be a low match due to excessive video memory requirements, the subgraph consisting of this layer and its associated normalization, residual connection, and other operators must be extracted to determine the flow path and resource usage characteristics of its input and output tensors.

[0055] The path reorganization algorithm optimizes the structure of the original subgraph using various strategies. For example, in memory-constrained scenarios, operator fusion can be used to combine multiple consecutive small operators into a composite operator, reducing the need for temporary storage of intermediate results in the graphics memory. For very large operators in distributed scenarios, they can be broken down into multiple subtasks, with the load shared through collaborative computing between devices. For example, a large-scale matrix multiplication can be decomposed into block-like sub-matrices, which are deployed to multiple GPUs for parallel execution, with communication synchronization ensuring the consistency of the results.

[0056] After generating the optimized subgraph, the system replaces it with the original subgraph, forming a new optimized computation graph and recalculating the first matching value between the device and the operator. If the new matching value moves beyond the low matching level (e.g., to a medium or high level), a dynamic update of the model computation graph is triggered, formally incorporating the optimized subgraph into the deployment process. For example, if an edge device is unable to execute the original convolution operator due to insufficient computing power, after path reorganization, it is replaced with a fusion subgraph of depthwise separable convolution and grouped convolution, and its matching value is improved from 0.3 (low level) to 0.8 (high level), successfully moving out of the low matching range.

[0057] At this point, the system synchronizes the updated computational graph to all relevant devices and redistributes tasks. If the matching value remains low after optimization, an iterative optimization process is initiated, attempting other restructuring strategies (such as introducing sparse computation or dynamic pruning) until deployment conditions are met or a degradation fault tolerance mechanism is triggered.

[0058] When the target matching value belongs to the medium matching level, the resource demand characteristics are analyzed to determine the bottleneck dimension characteristics that cause the target matching value to be the medium matching level; based on the bottleneck dimension characteristics, the connection relationship is adjusted from the initial connection to the optimized connection; the second matching value corresponding to the optimized connection is determined; when the second matching value belongs to the high matching level, the connection relationship in the hardware topology diagram is updated using the optimized connection.

[0059] When the target match value reaches a medium match level, the system initiates bottleneck dimension diagnosis and topology optimization to overcome local resource constraints. By analyzing the operator's multi-dimensional resource requirements (such as memory, bandwidth, and computing power), the core bottleneck that causes the insufficient match value is identified.

[0060] For example, a TPU node's matching value might be classified as medium because its memory utilization is close to a threshold (e.g., 90%), while computing power and bandwidth are well-matched. In this case, memory becomes the limiting factor. Or, an edge device's bandwidth utilization is consistently above 80%, causing a sharp increase in data transmission latency and dragging down the overall matching value. This type of diagnosis requires combining real-time monitoring data with historical trend analysis, such as tracking the temporal correlation of bandwidth utilization, to ensure the accuracy of bottleneck identification.

[0061] Based on the identified bottleneck dimensions, the system dynamically adjusts the connection relationships in the hardware topology. For example, in a bandwidth-constrained scenario, if the original connection bandwidth between device A and device B is insufficient, a relay device C can be introduced to build an A→C→B transmission path, distributing the pressure of a single path across multiple links; or switching to a backup high-bandwidth link can be used.

[0062] After completing the connection optimization, the system recalculates the second matching value between the device and the operator to verify the adjustment effect. If the new matching value reaches a high level, it triggers a real-time update of the hardware topology map, synchronizing the optimized connection relationship to the global view and regenerating the deployment strategy based on the new topology.

[0063] For example, the original deployment of a video encoding operator resulted in a matching value of 0.7 (medium) due to insufficient bandwidth on edge devices. After switching to a 5G millimeter wave link, the bandwidth increased from 50 Mbps to 1 Gbps, and the matching value jumped to 0.92. The system immediately marked the new link as the preferred path and migrated the operator to the edge device for execution, fully utilizing local computing power and reducing cloud reliance. If the matching value remains medium or degrades after optimization, the system will review the bottleneck diagnosis results and try other optimization strategies (such as adjusting the task allocation ratio or introducing redundant computing nodes).

[0064] Through the above technical solution, it can be seen that the beneficial effects of this embodiment are:

[0065] Based on the device nodes in the target model deployment environment and the corresponding connection relationships between each device node, a hardware topology diagram corresponding to the device node is constructed; the dependency relationships of the target operators in the target model are analyzed to generate the model calculation graph corresponding to the target operator; based on the hardware topology diagram and the model calculation graph, a preset evaluation algorithm is used to calculate the target matching value between the device node and the target operator; based on the target matching value and the preset grading strategy, the deployment plan corresponding to the target model is determined. Through a dynamic matching mechanism that perceives the hardware status in real time and the model calculation graph, the optimal deployment strategy is automatically generated, and adaptive optimization is triggered when the hardware topology or resource conditions change. This enables real-time collaborative adaptation of the model partitioning strategy to the heterogeneous environment, significantly improving the level of deployment automation and system elasticity.

[0066] Figure 1 What is shown is only a basic embodiment of a model deployment method combined with hardware deployment of the present application. By performing certain optimization and expansion on this basis, other preferred embodiments of a model deployment method combined with hardware deployment can be obtained.

[0067] like Figure 2 As shown, it is another specific embodiment of the model deployment method combined with hardware deployment of the present application.

[0068] In this embodiment, a model deployment method combined with hardware deployment includes the following steps:

[0069] Step 201: Construct a hardware topology diagram corresponding to the device nodes according to the device nodes in the target model deployment environment and the connection relationships corresponding to the device nodes.

[0070] Step 202: Determine the device information corresponding to each device node.

[0071] Device information is a core set of data describing the performance and status of physical hardware units in a computing environment, encompassing their inherent capabilities, real-time operational status, and connectivity characteristics. In a model deployment environment, determining the device information of each device node requires systematic collection and modeling from both static capabilities and dynamic status dimensions.

[0072] Static capability characteristics reflect the inherent performance parameters of the hardware, such as the number of CUDA cores, theoretical floating-point computing power, video memory capacity, supported interconnect protocols, and their theoretical bandwidth limits. For heterogeneous devices (such as TPUs or FPGAs), the characteristics of specialized compute units (such as the TPU's matrix multiplication acceleration capability) or the size of programmable logic resources must also be documented. This information is typically obtained through hardware specification documents or system-level query tools, and a device capability database is established to provide a benchmark for resource allocation.

[0073] Dynamic status information requires real-time monitoring of device resource usage during runtime, including the current available video memory ratio, computing load, actual bandwidth utilization, device temperature, and energy consumption. Furthermore, the connection status between devices (such as link failures or bandwidth fluctuations) must also be dynamically updated. For example, this can be achieved through a heartbeat detection mechanism to verify the availability of the NVLink channel or by using network probes to measure inter-node latency and packet loss rates.

[0074] Step 203: Determine the communication cost corresponding to the device node based on the device information and connection relationship.

[0075] Communication cost is a core metric for measuring the efficiency of data transmission between devices. Its calculation requires a comprehensive consideration of the physical characteristics and real-time status of the device connection. Specifically, communication cost is determined by both transmission delay and effective bandwidth. The effective bandwidth is the smaller of the bidirectional bandwidth between devices, reflecting the actual transmission capacity limit of the link. For example, if the bandwidth from device A to device B is 10 Gbps and the reverse bandwidth is 8 Gbps, the effective bandwidth is 8 Gbps, avoiding misjudgments of transmission efficiency due to link asymmetry.

[0076] In a dynamic environment, communication costs need to be updated in real time to adapt to fluctuations in network status. For example, in distributed training, if the actual bandwidth of an NVLink link decreases due to a surge in load, the system needs to remeasure its effective bandwidth and update the communication cost. In edge computing scenarios, the bandwidth of a 5G link may drop sharply from 1 Gbps to 200Mbps due to changes in signal strength. At this time, the cost calculation needs to be dynamically adjusted to avoid assigning high-data-volume tasks to inefficient links. In addition, changes in the physical topology (such as the addition of relay devices or switching of faulty links) will also trigger the recalculation of communication costs. For example, by introducing cache nodes to build multi-hop paths, a single high-latency link can be split into multiple low-latency sub-paths, thereby reducing the overall transmission cost.

[0077] In practical applications, optimizing communication costs directly impacts task deployment strategies. For example, in GPU clusters, high-bandwidth, low-latency NVLink devices are prioritized for adjacent compute nodes that frequently synchronize parameters. In cloud-edge collaborative inference scenarios, preprocessing tasks are deployed to edge devices to reduce the need to transmit raw data to the cloud, thereby avoiding the impact of high public network latency. By dynamically sensing and calculating communication costs in real time, the system can achieve optimal allocation of data flows and computing tasks in complex heterogeneous environments, maximizing overall hardware resource utilization.

[0078] Step 204: Build a hardware topology graph with device nodes as vertices and communication costs as edge weights.

[0079] In the specific process, the device node serves as the vertex, and its attributes must fully record static capabilities (such as computing power, memory capacity, and protocol support) and dynamic status (such as real-time available memory, bandwidth utilization, and temperature). The connection relationship between devices is quantified through edge weights, and the weight value is dynamically calculated by the communication cost formula.

[0080] For example, in a GPU cluster, if GPU0 and GPU1 are directly connected via NVLink 3.0 (bandwidth 600 GB / s, latency 0.1 μs), their edge weight is significantly lower than that of GPU2 and GPU3 connected via PCIe 4.0 x16 (bandwidth 32 GB / s, latency 1 μs). This difference directly affects the task allocation strategy - operators with high-frequency data exchange (such as parameter synchronization) will be preferentially deployed to the NVLink directly connected device group to avoid communication bottlenecks.

[0081] The construction of the hardware topology graph requires a balance between static initialization and dynamic updates. During the initialization phase, basic device properties are obtained through system detection tools, and initial latency and bandwidth are measured through network performance testing to generate a weighted graph structure.

[0082] In dynamic scenarios, the topology map needs to be updated periodically: if a device's actual bandwidth drops due to a surge in load (for example, from 10 Gbps to 5 Gbps), or a link failure triggers a switch to an alternative path (for example, from optical fiber to a 5G redundant link), the system will recalculate edge weights and update the graph structure to ensure the real-time nature of the resource view.

[0083] Step 205: Analyze the dependency relationship of the target operator in the target model to generate the model calculation corresponding to the target operator.

[0084] Step 206: Calculate the target matching value between the device node and the target operator using a preset evaluation algorithm based on the hardware topology diagram and the model calculation diagram.

[0085] Step 207: Determine the deployment plan corresponding to the target model based on the target matching value and the preset classification strategy.

[0086] Through the above technical solutions, it can be seen that the beneficial effects of this embodiment are: the dynamic update mechanism of the hardware topology map ensures its real-time and accuracy in heterogeneous environments, and provides underlying support for the elasticity and adaptability of model deployment.

[0087] like Figure 3 FIG. 1 is another specific embodiment of a model deployment method combined with hardware deployment in the present application. This embodiment further describes the above embodiment.

[0088] In this embodiment, a model deployment method combined with hardware deployment includes the following steps:

[0089] Step 301: Construct a hardware topology diagram corresponding to the device nodes according to the device nodes in the target model deployment environment and the connection relationships corresponding to the device nodes.

[0090] Step 302: Analyze the dependency relationship of the target operator in the target model to generate a model calculation graph corresponding to the target operator.

[0091] Step 303: According to the calculation process corresponding to the target model, the dependency relationship between the target operators is analyzed to generate data flow edges corresponding to the target operators.

[0092] During model deployment, analyzing the dependencies of target operators requires starting from the model's computational logic. Through a combination of static analysis and dynamic tracing, the input-output dependencies and execution order of each operator are clearly defined. Taking a typical deep learning model as an example, the connection topology of the operators is first extracted by parsing the model definition. For example, the output tensor of a convolutional layer serves as the input of an activation function layer, while the addition operation in a residual connection requires synchronously receiving the output results of both the main path and the bypass branch. These explicit dependencies directly determine the execution timing of the operators.

[0093] For dynamic graph models or structures containing conditional branches (such as the dynamic masking mechanism in the Transformer), symbolic execution or runtime instrumentation techniques must be combined to capture the data flow path during the actual reasoning process. For example, in a reinforcement learning policy network, action selection may depend on real-time feedback from the environment state. Multiple sampling or dynamic graph expansion is required to convert implicit dependencies into explicit data flow edges.

[0094] In complex model scenarios, dependency resolution requires handling multimodal inputs and heterogeneous data streams. For example, in multi-task learning models, the output of the same feature extraction layer may flow to both the classification and regression heads, forming branching dependencies. In time series models, the propagation of hidden states across time steps can introduce cyclic dependencies. For such scenarios, virtual operators or time step markers are needed to enhance the expressiveness of computational graphs.

[0095] For example, in recurrent neural networks, by expanding time steps into independent computational nodes and explicitly constructing data flow edges between adjacent time steps, the accuracy of gradient propagation paths is ensured. Furthermore, for control flow, placeholder operators are used to represent conditional logic, and corresponding subgraph paths are dynamically activated based on runtime decisions. For example, in model compression scenarios, different subnetwork branches can be selected based on input complexity.

[0096] The generation of data flow edges requires quantifying the scale and type of data transmission between operators. For example, in an object detection model, a feature pyramid network must pass feature maps of different scales to the fusion layer. Each data flow edge must record the dimension, data type, and batch size of the feature map to accurately calculate the communication cost.

[0097] Step 304: Build a model calculation graph with the target operator as the vertex and the data flow edge as the directed edge.

[0098] The key to building a model computation graph lies in abstracting target operators as vertices and characterizing their dependencies and data flow logic through data flow edges. First, the model's computational flow must be deconstructed into atomic operator units, such as convolution, matrix multiplication, or activation functions. Each operator is represented as a vertex in the graph, and its attributes must record key parameters such as computational cost, peak memory usage, input and output tensor dimensions, and bandwidth requirements.

[0099] For example, in the Transformer model, the vertices of the self-attention layer need to be labeled with the memory capacity required for their multi-head computations and quantify the transmission scale of the key-value matrix, while the vertices of the fully connected layer need to clearly specify the dimensions and computational intensity of the weight matrix.

[0100] Data flow edges, as directed edges connecting operators, must accurately describe the data transfer relationship and transmission characteristics between operators. The direction of the edge is determined by computational dependencies: if the output tensor of operator A is the input of operator B, then there is a directed edge from A to B. The edge weight must quantify the amount of data transferred (such as the byte size of the tensor), the data type (such as float32 or int8), and bandwidth sensitivity (such as high throughput requirements or low latency priority).

[0101] For dynamic models (such as recurrent neural networks), the transmission of hidden states between time steps requires time expansion to generate stride data flow edges. Each edge corresponds to the output of a specific time step and the dependency of the input of the next time step, while recording the dimension and update frequency of the hidden state.

[0102] Building model computation graphs also requires handling complex control flows and heterogeneous data paths. For example, in models with conditional branches (such as multi-task learning networks), dataflow edges may fork into different subgraph branches, requiring the introduction of virtual decision operators to represent branch selection logic. In real-time stream processing scenarios, if the model supports dynamic batching, dataflow edges must be accompanied by batch size ranges and elastic scaling policies. For example, when edge device resources fluctuate, the batch size must be automatically adjusted to adapt to available video memory.

[0103] In addition, for parameter synchronization in distributed training, the computational graph needs to explicitly mark gradient aggregation edges, whose data flow edge transmission volume is determined by the parameter scale and update frequency, and is associated with the high-bandwidth links in the hardware topology graph.

[0104] Step 305: Calculate the target matching value between the device node and the target operator using a preset evaluation algorithm based on the hardware topology diagram and the model calculation diagram.

[0105] Step 306: Determine the deployment plan corresponding to the target model based on the target matching value and the preset classification strategy.

[0106] The above technical solution demonstrates the beneficial effects of this embodiment: for dynamic or unstructured models, the introduction of virtual operators and dynamic edges enhances the expressiveness of computation graphs. This refined modeling enables model computation graphs to fully reflect computational logic, resource requirements, and performance bottlenecks, providing a precise optimization basis for subsequent dynamic deployment.

[0107] like Figure 4 FIG. 1 is another specific embodiment of a model deployment method combined with hardware deployment in the present application. This embodiment further describes the above embodiment.

[0108] Step 401: Construct a hardware topology diagram corresponding to the device nodes according to the device nodes in the target model deployment environment and the connection relationships corresponding to the device nodes.

[0109] Step 402: Analyze the dependency relationship of the target operator in the target model to generate a model calculation graph corresponding to the target operator.

[0110] Step 403: Calculate the target matching value between the device node and the target operator using a preset evaluation algorithm based on the hardware topology diagram and the model calculation diagram.

[0111] Step 404: Determine the resource demand characteristics corresponding to each target operator based on the model calculation graph.

[0112] Resource demand characteristics are multi-dimensional quantitative indicators that describe the hardware resource requirements of model operators (such as convolutional layers and fully connected layers) during operation. Resource demand characteristics are derived by analyzing the operator's computational logic, data flow, and hardware compatibility.

[0113] First, analyze the computational characteristics of operators based on their type (e.g., convolution, matrix multiplication, activation function). Convolutional layers typically require high parallel computing power (e.g., CUDA cores on GPUs) and video memory capacity to store weights and feature maps. Attention operators are bandwidth-sensitive due to the frequent transmission of key-value matrices. Quantized operators (e.g., INT8 inference) may rely on specific hardware acceleration units (e.g., Tensor Cores). For example, the self-attention layer in a Transformer model requires quantifying its multi-head computational memory usage and clarifying the bandwidth requirements for transmitting key-value matrices across devices.

[0114] The transmission characteristics of data stream edge records directly affect bandwidth requirements. For example, in a feature pyramid network, the amount of data in cross-layer fusion edges is determined by the feature map resolution and number of channels. If sparsification or quantization compression is used, the bandwidth requirement must be adjusted to 25% of the original value.

[0115] For dynamic models (such as recurrent neural networks), the amount of data transferred between hidden state steps must be dynamically calculated based on the sequence length and hidden layer dimensions. Resource requirements must be aligned with hardware capabilities to form multi-dimensional quantitative indicators.

[0116] Step 405: Determine the device capability characteristics corresponding to each device node according to the hardware topology diagram.

[0117] Device capability characteristics are a multidimensional collection of attributes that describe the comprehensive performance and real-time status of hardware units in a computing environment. These attributes encompass inherent capabilities, runtime resource status, and connection efficiency. In heterogeneous computing environments, determining device capability characteristics requires global modeling across three dimensions: static performance baseline, dynamic resource status, and connection efficiency. These capability characteristics are structured and stored using a unified data model and integrated into the vertex attributes of the hardware topology graph.

[0118] Static performance baselines reflect the inherent hardware properties of the device, such as the GPU's CUDA core count, theoretical floating-point computing power, video memory capacity, supported interconnect protocols, and their theoretical bandwidth limits. For specialized accelerators (such as the TPU or NPU), the characteristics of their dedicated compute units (such as the TPU's matrix multiplication engine throughput) or instruction set support (such as INT8 quantization acceleration capabilities) must be documented.

[0119] Dynamic resource status is periodically collected through a real-time monitoring system, including the current available video memory ratio (e.g., 18GB remaining), computing load (e.g., streaming multiprocessor occupancy rate of 70%), actual bandwidth utilization (e.g., NVLink bandwidth drops to 400 GB / s due to network congestion), device temperature, and power consumption.

[0120] For example, during distributed training, a GPU may experience memory fragmentation due to long-term operation, with available memory dropping from 30GB to 22GB. In this case, its dynamic memory capacity must be updated in real time to prevent memory overflows during subsequent task allocation. The frequency of dynamic state acquisition must balance accuracy and overhead. For example, computing load may be sampled once per second in real-time inference scenarios, while memory usage may be updated every 5 seconds during offline training.

[0121] Connection efficiency is indirectly measured by the communication cost between devices (such as latency and effective bandwidth). For example, the communication cost (latency + data bandwidth latency + bandwidth data volume) of two GPU devices directly connected via NVLink is significantly lower than that of a group of devices connected via PCIe. However, in edge-cloud collaboration scenarios, the public network link between the edge node and the cloud TPU may introduce additional latency (e.g., from 1 ms to 10 ms) due to the increased number of routing hops. This latency must be dynamically measured and updated in the topology map.

[0122] Step 406: Calculate the difference between the resource requirement characteristics and the device capability characteristics.

[0123] In a dynamic deployment system, the calculation of difference values ​​aims to quantify the degree of adaptation between the resource requirements of model operators and the device capabilities. Through multi-dimensional normalization and nonlinear mapping, the differences in heterogeneous resource dimensions are converted into comparable values.

[0124] Specifically, we first need to align the operator's resource requirements with the device's capabilities. For example, in the video memory dimension, if a convolution operator requires 20GB of video memory, and the target device has 25GB of available video memory, the original difference is 20-25 = -5GB. A negative value indicates that the device's capabilities can meet the requirement. However, if the device has only 15GB of available video memory, the difference is +5GB, indicating insufficient resources.

[0125] Step 407: Input the difference value into a preset mathematical model so that the mathematical model outputs a target matching value.

[0126] The process of inputting the difference value into the preset mathematical model to generate the target matching value is achieved through nonlinear mapping of multi-dimensional resource adaptability, and its formula is:

[0127] ,

[0128] in, This value represents the degree of compatibility between device node i and operator j. It quantifies the degree of compatibility between device capabilities and operator resource requirements, reflecting the optimization potential of their collaboration. A higher value indicates a more suitable device for deploying operator j; a lower value indicates a poorer compatibility.

[0129] k represents different resource dimensions. The parameters of each dimension are normalized using the mean and variance of its own dimension to avoid calculation bias caused by dimensional differences.

[0130] is the sensitivity adjustment factor used to control the k The impact of differences in resource dimensions on matching strength.

[0131] It is the difference between resource demand characteristics and equipment capability characteristics.

[0132] The dimension between operator requirements and device capabilities k The difference value on .

[0133] The parameters of each dimension are normalized using the mean variance of their own dimension to avoid calculation deviations caused by dimensional differences.

[0134] The target matching value is calculated and updated in real time to drive the generation of deployment strategies. For example, when a GPU's video memory drops from 25GB to 10GB, the normalized difference in its video memory dimension changes from −1.25 to +2.5, and the matching degree drops sharply from 0.8 to 0.05. The system immediately marks the device as having a low matching level.

[0135] Step 408: Determine the deployment plan corresponding to the target model based on the target matching value and the preset classification strategy.

[0136] It can be seen from the above technical solutions that the beneficial effects of this embodiment are: the use of mathematical models not only quantifies the global optimality of resource adaptation, but also ensures elasticity and robustness in heterogeneous environments through a dynamic response mechanism.

[0137] like Figure 5 As shown, this is a specific embodiment of a model deployment device combined with hardware deployment in this application. This embodiment is a model deployment device combined with hardware deployment, that is, for executing Figures 1 to 4 A physical device for a model deployment method combined with hardware deployment. Its technical solution is essentially consistent with the above embodiment, and the corresponding descriptions in the above embodiment are also applicable to this embodiment. In this embodiment, a model deployment device combined with hardware deployment includes:

[0138] The hardware topology map determining module 501 is configured to construct a hardware topology map corresponding to the device nodes according to the device nodes and the connection relationships corresponding to the device nodes in the target model deployment environment;

[0139] A model calculation graph determination module 502 is configured to parse the dependency relationship of the target operator in the target model to generate a model calculation graph corresponding to the target operator;

[0140] The target matching value determination module 503 is configured to calculate the target matching value between the device node and the target operator using a preset evaluation algorithm based on the hardware topology diagram and the model calculation diagram;

[0141] The deployment solution determination module 504 is configured to determine a deployment solution corresponding to the target model according to the target matching value and a preset classification strategy.

[0142] Figure 6: This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include internal memory, such as high-speed random access memory (RAM), and may also include non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0143] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0144] Memory is used to store execution instructions. Specifically, execution instructions are computer programs that can be executed. Memory can include internal memory and non-volatile memory, and provides execution instructions and data to the processor.

[0145] In one possible implementation, a processor reads corresponding execution instructions from a non-volatile memory into a memory and then executes them. The processor may also obtain corresponding execution instructions from other devices to form a model deployment device combined with hardware deployment at a logical level. The processor executes the execution instructions stored in the memory to implement a model deployment method combined with hardware deployment provided in any embodiment of the present application through the execution of the execution instructions.

[0146] The above application Figure 5The method provided in the illustrated embodiment, performed by a model deployment device in conjunction with hardware deployment, can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor or by software instructions. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor.

[0147] The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0148] The embodiment of the present application also proposes a readable medium, which stores execution instructions. When the stored execution instructions are executed by the processor of the electronic device, the electronic device can execute a model deployment method combined with hardware deployment provided in any embodiment of the present application, and is specifically used to execute the following: Figure 1 or Figure 2 or Figure 3 or Figure 4 The method shown.

[0149] The electronic device in each of the aforementioned embodiments may be a computer.

[0150] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods or computer program products. Therefore, the present application may adopt a completely hardware embodiment, a completely software embodiment, or a combination of software and hardware.

[0151] The various embodiments in this application are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the partial description of the method embodiments.

[0152] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0153] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A model deployment method combined with hardware deployment, characterized in that: include: Constructing a hardware topology diagram corresponding to the device nodes according to the device nodes in the target model deployment environment and the connection relationships corresponding to the device nodes; Analyzing the dependencies of the target operators in the target model to generate a model calculation graph corresponding to the target operators; Calculating a target matching value between the device node and the target operator using a preset evaluation algorithm according to the hardware topology diagram and the model calculation diagram; Determining a deployment plan corresponding to the target model based on the target matching value and a preset grading strategy; The step of calculating the target matching value between the device node and the target operator using a preset evaluation algorithm based on the hardware topology diagram and the model calculation diagram includes: determining resource requirement characteristics corresponding to each target operator based on the model calculation diagram; determining device capability characteristics corresponding to each device node based on the hardware topology diagram; calculating a difference between the resource requirement characteristics and the device capability characteristics; and inputting the difference into a preset mathematical model so that the mathematical model outputs the target matching value. Determining the deployment plan corresponding to the target model according to the target matching value and a preset grading strategy includes: determining a target matching level corresponding to the target matching value according to a grading threshold corresponding to the grading strategy; the target matching level includes a high matching level, a low matching level, and a medium matching level; and determining the deployment plan based on the target matching level; Determining the deployment solution based on the target matching level includes: When the target matching value belongs to the high matching level, the target operators are prioritized based on the numerical values ​​of the target matching values ​​to generate a deployment queue corresponding to the target operators; Deploying the target operator to the corresponding device node according to the deployment queue; and / or, When the target matching value belongs to the low matching level, determining the original subgraph corresponding to the target operator in the model calculation graph; Using a computational path reorganization algorithm, the original subgraph is reorganized to generate an optimized subgraph; Replacing the original subgraph with the optimized subgraph to generate a corresponding optimized computation graph; Calculating a first matching value corresponding to the optimized calculation graph; When the first matching value does not belong to the low matching level, the model calculation graph is updated using the optimized subgraph.

2. The method according to claim 1, characterized in that The constructing of a hardware topology diagram corresponding to the device nodes according to the device nodes in the target model deployment environment and the connection relationships corresponding to the device nodes includes: Determine the device information corresponding to each of the device nodes; Determining a communication cost corresponding to the device node according to the device information and the connection relationship; The hardware topology graph is constructed with the device nodes as vertices and the communication costs as edge weights.

3. The method according to claim 1, characterized in that The step of analyzing the dependency relationship of the target operators in the target model and the resource requirements corresponding to the target operators to generate a model calculation graph corresponding to the target operators includes: According to the calculation process corresponding to the target model, the dependency relationship between the target operators is analyzed to generate data flow edges corresponding to the target operators; The model calculation graph is constructed with the target operator as a vertex and the data flow edge as a directed edge.

4. The method according to claim 1, wherein Determining the deployment solution based on the target matching level includes: When the target matching value belongs to the medium matching level, analyzing the resource demand characteristics to determine the bottleneck dimension characteristics that cause the target matching value to be the medium matching level; According to the bottleneck dimension feature, adjusting the connection relationship from an initial connection to an optimized connection; Determining a second matching value corresponding to the optimized connection; When the second matching value belongs to the high matching level, the connection relationship in the hardware topology map is updated using the optimized connection.

5. A model deployment device combined with hardware deployment, used to execute the model deployment method combined with hardware deployment described in any one of claims 1 to 4, characterized in that: include: A hardware topology map determining module is configured to construct a hardware topology map corresponding to the device nodes according to the device nodes in the target model deployment environment and the connection relationships corresponding to the device nodes; a model calculation graph determination module, configured to parse the dependency relationship of the target operator in the target model to generate a model calculation graph corresponding to the target operator; a target matching value determination module, configured to calculate a target matching value between the device node and the target operator using a preset evaluation algorithm according to the hardware topology diagram and the model calculation diagram; The deployment scheme determination module is configured to determine the deployment scheme corresponding to the target model according to the target matching value and a preset classification strategy.

6. A computer-readable storage medium storing a computer program, characterized in that: The computer program is used to execute the model deployment method combined with hardware deployment as described in any one of claims 1 to 4 above.

7. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the model deployment method combined with hardware deployment as described in any one of claims 1-4 above.

Images