Adaptive model partitioning collaborative inference optimization method based on reinforcement learning

Abstract

The application discloses an adaptive model segmentation collaborative reasoning optimization method based on reinforcement learning, relates to the technical field of machine learning, and acquires current computing power load characteristics, current network transmission state characteristics and data complexity characteristics, splices an output state characteristic vector, inputs the state characteristic vector into a reinforcement learning decision network, outputs a segmentation layer index and a feature quantization compression ratio, divides a target model into a head sub-model and a tail sub-model, inputs current input data into the head sub-model to obtain an intermediate feature tensor, acquires quantization scaling parameters, quantizes to obtain compressed feature data, sends the compressed feature data and the quantization scaling parameters to a second computing node for dequantization, outputs a reasoning result through the tail sub-model, performs back propagation calculation on the head sub-model by receiving compressed feature gradient data generated by the second computing node based on the reasoning result to obtain a parameter gradient tensor, and updates network parameters of the head sub-model based on the parameter gradient tensor by using a parameter optimizer.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to an adaptive model segmentation collaborative reasoning optimization method based on reinforcement learning. Background Technology

[0002] With the rapid development of the Internet of Things (IoT) and edge computing, the demand for deploying deep neural network models on smart set-top boxes and edge computing boxes is increasing. However, due to limitations in storage space, computing power, and battery life of terminal devices, it is difficult to independently support the efficient operation of large models. To address this, existing collaborative inference technologies typically employ a model segmentation strategy, dividing the model into a head sub-model deployed on the terminal and a tail sub-model deployed in the cloud, in order to balance the load of local computing and cloud offloading.

[0003] Currently, Chinese patent CN120409669A discloses an edge computing collaborative inference method, medium, and device for adaptive model segmentation. It utilizes directed acyclic graphs and network flow graph theory to screen feasible model segmentation strategies and combines a pre-trained quantized model with a deep network to search for the optimal segmentation point in the discrete action space, thereby reducing inference latency. To alleviate storage pressure on terminal devices, existing technologies mostly employ static model compression methods after training and quantization, fixing model parameters into a low-bit format. However, these technologies have the following shortcomings: First, existing collaborative inference schemes mostly adopt a static deployment mode with fixed parameters, lacking a mechanism for lightweight online adaptation of the edge feature extraction layer using feedback from cloud inference results. Especially when the tail model in the cloud remains fixed, existing technologies cannot perform unilateral fine-tuning of the head sub-model on the edge using only the feature gradient backpropagated to the segmentation point. Second, the optimization objectives of existing technologies are relatively singular, making it difficult to consider the multi-dimensional constraints in actual business. The reward function design of existing technologies mainly focuses on computation latency, transmission latency, and numerical loss caused by quantization. In the actual implementation of edge computing, the terminal device... Energy consumption and the final inference confidence are often more critical indicators. The neglect of energy consumption and confidence in existing technologies leads to poor overall service performance in real-world high-load or high-dynamic network environments. Third, the data transmission compression strategies of existing technologies lack dynamic flexibility. Existing technologies usually adopt fixed static quantization strategies, performing compression only once during the model training phase. When faced with drastic fluctuations in network bandwidth, this fixed compression ratio cannot dynamically adjust the amount of data transmitted. Jointly optimizing the segmentation position and feature quantization compression ratio based on the real-time network status during the inference phase can more effectively utilize network resources and ensure the real-time performance and stability of inference. Summary of the Invention

[0004] The technical problem addressed by this invention is that existing technologies struggle to achieve a balance between low inference latency, low power consumption, and high accuracy in dynamically changing edge collaborative computing environments. Traditional strategies often employ fixed model partitioning or single-index optimization, making it difficult to perceive and adapt in real time to the multidimensional dynamic changes in the computing load of the first computing node, network transmission status, and input data complexity. This results in an inability to flexibly adjust the partitioning position and data compression intensity when faced with computing power fluctuations or network jitter, leading to high transmission overhead, high inference latency, or low resource utilization.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: an adaptive model segmentation collaborative reasoning optimization method based on reinforcement learning, comprising the following steps:

[0006] Step S1: Obtain the current computing load characteristics, current network transmission status characteristics, and data complexity characteristics, and concatenate them to output the status feature vector;

[0007] Step S2: Input the state feature vector into the reinforcement learning decision network, output the layering index and feature quantization compression ratio, and divide the target model into a head sub-model configured on the first computing node and a tail sub-model configured on the second computing node based on the layering index.

[0008] Step S3: Input the current input data into the head sub-model to obtain the intermediate feature tensor, obtain the quantization scaling parameter based on the feature quantization compression ratio, quantize to obtain compressed feature data, send the compressed feature data and quantization scaling parameter to the second computing node for dequantization, and output the inference result through the tail sub-model.

[0009] Step S4: Receive compressed feature gradient data generated by the second computing node based on the inference results, use the compressed feature gradient data to perform backpropagation calculation on the head sub-model to obtain the parameter gradient tensor, and use the parameter optimizer to update the network parameters of the head sub-model based on the parameter gradient tensor.

[0010] Preferably, step S1 includes the following sub-steps:

[0011] Step S101: Collect the processor utilization and remaining memory capacity of the first computing node, normalize the processor utilization and remaining memory capacity, and output the current computing load characteristics.

[0012] Step S102: Probe the communication link of the first computing node, obtain the effective transmission bandwidth value and round-trip delay value at the current moment, and combine them to generate the current network transmission status characteristics.

[0013] Parse the current input data, extract the tensor dimension information and tensor size information of the current input data, traverse the directed acyclic computation graph of the target model and accumulate the forward propagation floating-point operation volume of each network layer, and calculate the estimated total floating-point operation volume of the current input data in the target model;

[0014] Traverse the directed acyclic computation graph to count the peak memory usage of the output tensors of each network layer, and obtain the estimated peak memory usage.

[0015] The estimated total floating-point operations and the estimated peak video memory usage are combined to generate data complexity features.

[0016] Preferably, step S1 further includes the following sub-steps:

[0017] Step S103: Map the current computing load characteristics, current network transmission status characteristics, and data complexity characteristics to a unified numerical range, and perform tensor concatenation operation on the mapped features along a preset feature dimension to output a state feature vector.

[0018] Preferably, step S2 includes the following sub-steps:

[0019] Step S201: Input the state feature vector into the reinforcement learning decision network for forward computation, and output the policy probability distribution tensor corresponding to the preset joint action space;

[0020] Step S202: Based on the policy probability distribution tensor, select the action, determine the target joint action, map the target joint action to the preset physical parameter space, obtain the layer index and feature quantization compression ratio of the target model, and store the state feature vector, policy probability distribution tensor and target joint action in the preset empirical trajectory cache area.

[0021] Preferably, step S2 further includes the following sub-steps:

[0022] Step S203: Determine the target logic execution breakpoint in the directed acyclic computation graph of the target model based on the layered index;

[0023] The forward propagation execution boundary of the first computing node is set as the target logic execution breakpoint, and the target logic execution breakpoint and the network layer on the predecessor path of the target logic execution breakpoint are determined as the head sub-model.

[0024] The layer index is sent to the second computing node, which instructs the second computing node to set the forward propagation start boundary as the set of direct successor network layers of the target logic execution breakpoint, and to determine the set of direct successor network layers and the network layers on the subsequent path of the set of direct successor network layers as the tail sub-model.

[0025] Preferably, step S3 includes the following sub-steps:

[0026] Step S301: In the first computing node, the current input data is input into the head sub-model, and forward propagation calculation is performed sequentially along the predecessor path of the directed acyclic computation graph until the target logic execution breakpoint is reached, and the intermediate feature tensor is output.

[0027] Preferably, step S3 further includes the following sub-steps:

[0028] Step S302: Determine the target quantization bit width based on the feature quantization compression ratio, statistically analyze the numerical distribution of the intermediate feature tensor, and calculate the quantization scaling parameter and zero point parameter matching the target quantization bit width based on the numerical distribution.

[0029] The intermediate feature tensor is subjected to numerical type conversion and linear mapping operations using the quantization scaling parameters and zero-point parameters to output compressed feature data.

[0030] Preferably, step S3 further includes the following sub-steps:

[0031] Step S303: Send the compressed feature data, quantization scaling parameters, and zero-point parameters to the second computing node;

[0032] The second computing node uses the quantization scaling parameters and zero-point parameters to perform an inverse linear mapping on the compressed feature data, generating an inverse quantization feature tensor. The inverse quantization feature tensor is then input into the set of direct successor network layers in the tail sub-model to perform forward propagation computation and output the inference result.

[0033] Preferably, step S4 includes the following sub-steps:

[0034] Step S401: The second computing node obtains the real labels corresponding to the current input data through a preset sample label library, calculates the model prediction error based on the inference results and real labels, and constructs a loss function.

[0035] Along the directed acyclic computation graph of the tail sub-model, the gradient of the loss function is backpropagated to the breakpoint of the target logic execution to obtain the feature gradient tensor. The feature gradient tensor is then compressed using a preset gradient compression algorithm to generate compressed feature gradient data, which is then distributed to the first computation node.

[0036] In step S402, in the first computing node, a decompression operation is performed on the compressed feature gradient data to reconstruct the feature gradient tensor, and the quantization step is approximated as an identity mapping in backpropagation. Based on the pass-through estimator mechanism, the reconstructed feature gradient tensor is used as the backpropagation input gradient of the head sub-model at the execution breakpoint of the target logic.

[0037] Preferably, step S4 further includes the following sub-steps:

[0038] Step S403: Using the backpropagation input gradient, backpropagation calculation is performed sequentially along the directed acyclic computation graph of the head sub-model to each network layer contained in the head sub-model, and the parameter gradient tensor corresponding to the network parameters in the head sub-model is output.

[0039] The parameter gradient tensor is input into a preset parameter optimizer, and the network parameters of the head sub-model are updated by performing gradient descent iterative calculation on the network parameters of the head sub-model through the preset parameter optimizer.

[0040] Step S404: Collect the network transmission delay, node computing power consumption, and model prediction error within the current execution cycle as state evaluation variables;

[0041] After normalizing each state evaluation variable, a weighted sum is obtained to get the comprehensive cost, and the negative value of the comprehensive cost is used as the immediate reward.

[0042] The state feature vector, policy probability distribution tensor, and target joint action are extracted from the preset experience trajectory buffer. The policy gradient is calculated in combination with the immediate reward, and the network parameters of the reinforcement learning decision network are updated using the policy gradient.

[0043] The beneficial effects of this invention are as follows: By receiving current computing load characteristics, current network transmission status characteristics, and data complexity characteristics through a reinforcement learning decision network, and dynamically outputting the segmentation index and feature quantization compression ratio, the joint adaptive adjustment of model segmentation position and feature quantization intensity is achieved. This enables the first computing node to flexibly set the target logic execution breakpoint and target quantization bit width based on processor utilization, remaining memory capacity, effective transmission bandwidth value, and round-trip latency value. This overcomes the resource waste caused by the fixed strategy of existing technologies. By using quantization scaling parameters to perform numerical type conversion and linear mapping on intermediate feature tensors to generate compressed feature data, the accuracy of the dequantized feature tensor is ensured while reducing the amount of transmitted data. This achieves coordinated optimization of inference latency, energy consumption, and accuracy in a dynamic computing power and network environment. Attached Figure Description

[0044] Figure 1 The flowchart illustrates the steps of an adaptive model segmentation and collaborative reasoning optimization method based on reinforcement learning, as provided in one embodiment of the present invention. Detailed Implementation

[0045] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0048] Example, refer to Figure 1 This paper presents an adaptive model segmentation collaborative reasoning optimization method based on reinforcement learning, which includes the following steps:

[0049] Step S1: Obtain the current computing load characteristics, current network transmission status characteristics, and data complexity characteristics, and concatenate them to output the status feature vector.

[0050] Step S2: Input the state feature vector into the reinforcement learning decision network, output the layering index and feature quantization compression ratio, and divide the target model into a head sub-model configured on the first computing node and a tail sub-model configured on the second computing node based on the layering index.

[0051] Step S3: Input the current input data into the head sub-model to obtain the intermediate feature tensor, obtain the quantization scaling parameter based on the feature quantization compression ratio, quantize to obtain compressed feature data, send the compressed feature data and quantization scaling parameter to the second computing node for dequantization, and output the inference result through the tail sub-model.

[0052] Step S4: Receive compressed feature gradient data generated by the second computing node based on the inference results, use the compressed feature gradient data to perform backpropagation calculation on the head sub-model to obtain the parameter gradient tensor, and use the parameter optimizer to update the network parameters of the head sub-model based on the parameter gradient tensor.

[0053] This invention utilizes a reinforcement learning decision network to receive current computing load characteristics, current network transmission status characteristics, and data complexity characteristics. It dynamically outputs the segmentation index and feature quantization compression ratio, achieving joint adaptive adjustment of model segmentation position and feature quantization intensity. This allows the first computing node to flexibly set the target logic execution breakpoint and target quantization bit width based on processor utilization, remaining memory capacity, effective transmission bandwidth, and round-trip latency. This overcomes the resource waste caused by fixed strategies in existing technologies. By using quantization scaling parameters to perform numerical type conversion and linear mapping on intermediate feature tensors to generate compressed feature data, it reduces the amount of transmitted data while ensuring the accuracy of the dequantized feature tensors. This achieves coordinated optimization of inference latency, energy consumption, and accuracy in dynamic computing power and network environments.

[0054] The collaborative inference optimization method of this invention differs fundamentally from existing edge-cloud joint training or federated learning techniques. Existing joint training techniques typically require the cloud to aggregate gradients from multiple endpoints and update the global model, then synchronously update each endpoint node, involving complex version synchronization and model redeployment overhead. This invention, however, adopts an architecture design where the endpoint adapts unidirectionally while the cloud remains unchanged. Specifically, the tail model in the cloud acts as a fixed global anchor point, and only the feature gradients backpropagated to the breakpoint are used to unidirectionally drive the fine-tuning of the head sub-models on the endpoint. This design solves the feature mismatch problem caused by data drift on the endpoint with extremely low communication and computational costs, without changing the cloud model version, and avoids the burden of version management for the cloud model.

[0055] In the technical solution of this invention, the first computing node and the second computing node constitute an edge-cloud collaborative computing architecture. The first computing node is a terminal-side / edge computing box, specifically a resource-constrained local computing device, such as a smart home set-top box, edge computing box, smart TV, or edge AI gateway. It integrates a limited processor (such as CPU, NPU) and local memory, and is responsible for directly connecting to user input and undertaking the front-end computing (i.e., the head sub-model) of the target model.

[0056] The second computing node is a cloud / edge server, specifically a remote server with abundant computing resources, such as a central GPU server, a cloud computing center cluster, or a high-performance edge server. It is equipped with high-performance acceleration hardware (such as multi-GPUs and large-capacity video memory) and is responsible for carrying out the backend computation of the target model (i.e., the tail sub-model), the output of global inference results, and the inverse calculation of the loss function.

[0057] In a specific embodiment, step S1 includes the following sub-steps:

[0058] Step S101: Collect the processor utilization and remaining memory capacity of the first computing node, normalize the processor utilization and remaining memory capacity, and output the current computing load characteristics.

[0059] Step S102: Detect the communication link accessed by the first computing node, obtain the effective transmission bandwidth value and round-trip delay value at the current moment, and combine them to generate the current network transmission status characteristics.

[0060] The current input data is parsed, extracting its tensor dimension and size information. The directed acyclic graph (DAG) of the target model is traversed, and the forward propagation floating-point operations of all network layers are accumulated to calculate the estimated total floating-point operations of the current input data in the target model. Simultaneously, the peak memory usage of the output tensors of each network layer is statistically analyzed by traversing the DAG, yielding the estimated peak memory usage. The estimated total floating-point operations and the estimated peak memory usage are combined to generate data complexity features.

[0061] In deep learning frameworks, the raw size (resolution) of tensors represents discrete spatial geometric values, which cannot be directly aligned with the physical state of computation nodes. This application transforms the raw size of tensors into estimated floating-point operations and estimated peak memory usage, giving data features a true physical computational dimension. This processing method can accurately reflect the actual computational load and memory consumption caused by input data of different dimensions during the forward inference process of the target model, thereby assisting reinforcement learning decision networks in more accurately predicting resource bottlenecks.

[0062] Data complexity features reflect the computational load of the input data. For example, the higher the resolution of the input image, the larger the tensor size, and the computational cost of subsequent convolutional layers typically increases quadratically. The tensor dimension determines the feature richness processed by the model. Extracting data complexity features helps decision networks predict computational costs in advance.

[0063] The target model is a pre-trained deep neural network model. Its topology is represented by a directed acyclic computation graph, consisting of multiple network layers combined in a serial or parallel manner, with each layer having a unique layer index. The structural description information of the target model and the weight parameters of each layer are pre-loaded and stored in the local storage media of the first and second computing nodes.

[0064] In this embodiment, the estimated floating-point operation volume and the estimated peak memory usage are calculated based on the target model structure description information stored locally on the first computing node. Specifically, the first computing node has loaded the target model's structure metadata during the deployment phase. The metadata includes the operator type and convolution kernel size for each network layer. , ), sliding step size, fill quantity, number of input channels and number of output channels After parsing the current input data to obtain its spatial dimension, the directed acyclic computation graph of the target model is traversed to calculate the spatial size of the output tensor of each network layer. , The floating-point computation of each network layer is calculated by accumulating the floating-point computation of each layer to obtain the estimated total floating-point computation of the entire target model network.

[0065] It should be noted that in actual inference, the first computation node typically performs real-time streaming processing on a single sample (i.e., batch size of 1). Therefore, the following formula for the single-layer floating-point computation complexity is based on a single sample input (batch size x=1). If the current input data contains a batch of samples (i.e., batch size x), the computation results of each layer need to be multiplied by N to obtain the total floating-point computation complexity for that batch.

[0066] As a specific implementation of this embodiment, the floating-point operation cost of the convolutional layer is accumulated to the total amount according to the following formula:

[0067] ;

[0068] This represents the floating-point computation cost of the convolutional layer.

[0069] For a fully connected layer, if its input feature dimension is The output feature dimension is The mathematical expression for its floating-point operation complexity is:

[0070] ;

[0071] This represents the floating-point computation cost of the fully connected layer.

[0072] The estimated peak memory usage is the maximum memory space required for the output tensors of all network layers during the traversal of the computation graph. The memory usage of a single layer is calculated as follows: Sure.

[0073] It should be further clarified that the estimated total floating-point operations are the total floating-point operations of the entire network of the target model. Specifically, after obtaining the tensor dimension information of the current input data, the first computing node traverses each network layer of the directed acyclic computation graph of the target model in sequence according to the topological order of the graph:

[0074] If the current layer is a convolutional layer, then apply the formula. Calculate the floating-point computation cost of this layer. If the current layer is a fully connected layer, apply the formula... Calculate the floating-point computation cost of this layer. For pooling layers or activation function layers, the floating-point computation cost is accumulated based on the number of output elements.

[0075] The total number of floating-point operations performed by all network layers during the traversal is summed to obtain the estimated total number of floating-point operations. This sum, together with the estimated peak memory usage, constitutes the data complexity feature, which represents the overall resource requirements of the current input data on the first computing node.

[0076] Step S103: Map the current computing load characteristics, current network transmission status characteristics, and data complexity characteristics to a unified numerical range, and perform tensor concatenation operation on each mapped feature along the preset feature dimension to output the state feature vector.

[0077] In actual operation, the computational load, network status, and input data of the first computing node (home smart set-top box) have different physical dimensions. In order to eliminate the adverse effects of the difference in dimensions on the convergence speed of the reinforcement learning network, the following processing method is adopted in this embodiment:

[0078] The normalized current computing load characteristics, current network transmission status characteristics, and data complexity characteristics are mapped to a unified numerical range [-1, 1]. A tensor concatenation operation is then performed along the preset feature dimensions. If the current computing load characteristics are... The current network transmission status characteristics are: The data complexity characteristics are The final output state feature vector dimension is The current state feature vector is output to the decision network.

[0079] In a specific embodiment, step S2 includes the following sub-steps:

[0080] Step S201: Input the state feature vector into the reinforcement learning decision network, perform forward computation through the network layers of the reinforcement learning decision network, and output the policy probability distribution tensor corresponding to the preset joint action space.

[0081] The action space of reinforcement learning decision networks is designed as a joint action space. Joint Action Space By dividing the hierarchical index space and feature quantization compression ratio space The Cartesian product is formed, that is .in, It contains a set of indexes for all feasible network layers of the target model. It includes multiple preset compression ratios, including 8-bit and 4-bit.

[0082] The output layer of the decision network uses the Softmax normalization function to transform continuous Logits values ​​into a policy probability distribution tensor that satisfies the probability axiom. Each element in the policy probability distribution tensor represents the probability of selecting a specific joint action.

[0083] Specifically, let N be the total number of network layers in the target model that can serve as segmentation points, and let the feature quantization compression ratio space be... The total number of compression ratio levels included is M. In this embodiment, M equals 2, corresponding to two quantization bit widths: 4-bit and 8-bit.

[0084] The total size of the joint action space A is N*M. Therefore, the number of neurons in the output layer of the reinforcement learning decision network is set to N*M.

[0085] The output policy probability distribution tensor is a one-dimensional probability vector of length N*M. After Softmax normalization, the value at each position represents the probability of selecting the corresponding joint action.

[0086] To establish the correspondence between the output vector positions and specific physical configurations, the preset physical parameter space adopts a sequential index encoding method:

[0087] Arrange the layer indexes from 0 to N minus 1, and under each layer index, arrange M quantization options in order of compression ratio.

[0088] For example, layer 0 corresponds to positions 0 to M-1 of the output vector, layer 1 corresponds to positions M to 2M-1, and so on.

[0089] The i-th position in the output vector can be decoded by integer division and modulo operation as the layer index equal to the quotient of i divided by M, and the feature quantization compression ratio index equal to the remainder of i modulo M.

[0090] Step S202: Based on the policy probability distribution tensor, select the action, determine the target joint action, map the target joint action to the preset physical parameter space, obtain the layer index and feature quantization compression ratio of the target model, and store the state feature vector, policy probability distribution tensor and target joint action in the preset empirical trajectory cache area.

[0091] In this embodiment of the invention, the preset physical parameter space refers to a structured mapping table configured in system memory, preferably a key-value pair data dictionary, used as a decoder between abstract algorithmic actions and underlying hardware execution instructions. In the discrete action space architecture of reinforcement learning, the joint action of the target determined by the decision network is usually represented as an abstract integer index value, such as action index ID=12. The preset physical parameter space predefines the specific physical configuration tuple corresponding to each action index. For example, after a specific action index is mapped through this parameter space, it will be accurately parsed into the underlying execution tuple: [target logic execution breakpoint= Feature quantization compression bit width = By using this physical parameter space for direct table lookup, the abstract mathematical output of the decision network is deterministically transformed into physical instructions that can directly drive the first and second computing nodes to execute model segmentation and tensor quantization.

[0092] The pre-defined experience trajectory cache refers to a structured storage medium, such as a replay buffer or rollout cache, allocated in the memory or video memory of the first computing node specifically for storing the interactive time-series data of the Markov decision process. In the edge-cloud collaborative inference system, after the decision network makes a splitting decision in step S2, it cannot immediately know the merits of the decision. It must wait for the second computing node to complete the forward propagation and calculate the prediction error before generating a reward signal for evaluation, leading to a typical delayed feedback phenomenon. Therefore, this invention sets up an experience trajectory cache area to persistently store the observed state feature vector, the policy probability distribution tensor of the original network output, and the actual sampled target joint action as a decision snapshot at the current decision moment. When the collaborative inference in the subsequent step S4 is completed and a reward signal is generated, the cache area can be used to accurately backtrack the historical decision trajectory corresponding to the current reward signal, thereby extracting tensor data for calculating the policy gradient, thus establishing a closed loop of reinforcement learning computing power for state observation, action decision, delayed feedback, and gradient update.

[0093] In practical applications, to balance the exploration and utilization of reinforcement learning, random sampling or maximization of the policy probability distribution tensor can be performed on the discrete joint action space. An index for a target joint action is determined. Using methods such as table lookup, the discrete joint action indexes are precisely mapped to a preset physical parameter space, and the corresponding layer index and feature quantization compression ratio are decoded.

[0094] Simultaneously, the input of the current step (state feature vector), the original network output (policy probability distribution tensor), and the final decision result (joint target action) are stored as a complete interactive Markov tuple in a pre-defined empirical trajectory buffer. The data in the empirical trajectory buffer retains all contextual information at the time of the decision, constructing a complete gradient backpropagation data stream for subsequent steps, ensuring that the reinforcement learning decision network can achieve closed-loop parameter updates.

[0095] Step S203: Determine the target logic execution breakpoint in the directed acyclic computation graph of the target model based on the layered index.

[0096] The forward propagation execution boundary of the first computing node is set as the target logic execution breakpoint, and the network layers on the target logic execution breakpoint and the preceding path of the target logic execution breakpoint are determined as the head sub-model.

[0097] The layer index is sent to the second computing node, which is instructed to set the forward propagation start boundary as the set of direct successor network layers of the target logic execution breakpoint, and to determine the set of direct successor network layers and the network layers on the subsequent path of the set of direct successor network layers as the tail sub-model.

[0098] In this embodiment of the invention, the target model refers to a deep neural network model pre-trained and deployed in a collaborative computing architecture. At the topological level, the target model is abstractly represented as a directed acyclic computation graph, internally composed of multiple network layers performing specific mathematical operations, linked sequentially or through multi-branch parallel paths. These network layers include convolutional layers, self-attention layers, pooling layers, and fully connected layers. By abstracting the target model as a directed acyclic computation graph, the topological ambiguity arising from traditional chain-like network partitioning in multi-branch structures can be overcome.

[0099] In terms of physical representation, the target model consists of a fixed set of network structure description files and tensor data files containing massive weight parameters. In specific industrial applications, the target model can encompass data processing networks of multiple modalities. For example, in the field of high-precision manufacturing, the target model can be a machine vision network for defect detection in watch micro-motor components, or an interdisciplinary coupled algorithm model for simultaneously processing multimodal physical data streams to predict industrial faults.

[0100] This invention does not restrict the specific network category of the target model. As long as the forward inference process of the model can be parsed into a directed graph structure that is interruptible, divisible, and has no loops in the data flow, it falls within the protection scope of the target model in this invention.

[0101] Specifically, the head sub-model includes not only the breakpoint itself, but also all network layers along the preceding data flow paths pointing to that breakpoint. More importantly, in the tail sub-model partitioning, when the output tensor of the target logic execution breakpoint is simultaneously connected to multiple downstream branches, the first computation node sets the starting boundary of the forward propagation to a set of direct successor network layers—that is, all network layer nodes that directly receive the breakpoint's output tensor. This partitioning method, based on sets and subsequent paths, ensures that regardless of the complexity of the nonlinear connection architecture within the target model, the partitioned head and tail sub-models maintain absolute integrity and topological continuity in their physical computation chains.

[0102] In a specific embodiment, step S3 includes the following sub-steps:

[0103] Step S301: In the first computation node, the current input data is input into the head sub-model, and forward propagation computation is performed sequentially along the predecessor path of the directed acyclic computation graph until the target logic execution breakpoint is reached, and the intermediate feature tensor in floating-point numerical format is output.

[0104] The output intermediate feature tensors are represented in a high-precision floating-point numerical format, such as FP32. By performing computations along the predecessor paths of the directed acyclic computation graph, it is ensured that no topological structural omissions occur in the data flow before reaching the logical breakpoint.

[0105] Step S302: Determine the target quantization bit width based on the feature quantization compression ratio, statistically analyze the numerical distribution of the intermediate feature tensor, and calculate the quantization scaling parameter and zero point parameter matching the target quantization bit width in real time based on the numerical distribution.

[0106] The intermediate feature tensor is subjected to numerical type conversion and linear mapping operations using quantization scaling parameters and zero-point parameters, and the output is compressed feature data in low-precision format.

[0107] Numeric type conversion and linear mapping operations compress intermediate feature tensors in high-precision floating-point format into low-precision format, such as compressed feature data in INT8 discrete integer format, significantly reducing network bandwidth consumption for cross-node communication.

[0108] In this embodiment, the quantization scaling parameter and the zero-point parameter are calculated in real time based on the actual numerical distribution of the intermediate feature tensor, without the need for pre-calibration. The specific calculation configuration method is as follows:

[0109] After outputting the intermediate feature tensor, the first computation node iterates through the intermediate feature tensors in real time to extract their numerical dynamic range, i.e., the maximum and minimum values ​​of the current intermediate feature tensor. Based on the numerical representation interval corresponding to the determined target quantization bit width and the extracted numerical dynamic range, the corresponding scaling parameters and zero-point parameters are calculated immediately. Since the network parameters of the head sub-model are continuously updated in step S4, the quantization parameters calculated in real time can always accurately match the current model state, thereby avoiding quantization overflow or accuracy loss caused by parameter fixation.

[0110] As a further preferred embodiment of the present invention, in order to suppress quantization jitter caused by outliers in a single frame, the first computing node can also maintain a lightweight historical feature statistical sliding window, using the mean and variance within the window to replace the extreme values ​​of a single frame. When the rate of change of the feature gradient tensor returned in step S4 is drastic, a rapid reset of the window data is triggered to ensure that the quantization parameters transition smoothly during model convergence.

[0111] Step S303: Send the compressed feature data, quantization scaling parameters, and zero-point parameters to the second computing node.

[0112] The second computation node performs an inverse linear mapping on the compressed feature data using quantization scaling parameters and zero-point parameters to generate an inverse quantization feature tensor. The inverse quantization feature tensor is then input into the set of direct successor network layers in the tail sub-model to perform forward propagation computation and output the inference result.

[0113] In this context, inverse linear mapping refers to the second computation node using the received zero-point parameters and scaling parameters to restore the low-precision compressed feature data into an inversely quantized feature tensor with approximately the original floating-point precision. Specifically, since the target model is abstracted as a directed acyclic computation graph, when multiple parallel branches exist at the breakpoint of the target logic execution, the generated inversely quantized feature tensor is synchronously fed into each network layer node in the set of direct successor network layers. This drives the tail sub-model to continue executing the subsequent complete inference process without disrupting the topological integrity of the original model, until the final inference result is output.

[0114] In a specific embodiment, step S4 includes the following sub-steps:

[0115] Step S401: The second computing node obtains the real labels and calculates the backpropagation gradient. The specific implementation method is as follows:

[0116] The second computing node obtains the real labels corresponding to the current input data through a pre-set sample label library or a business feedback interface. Based on the inference results and the real labels, it calculates the model prediction error and constructs a loss function. The pre-set sample label library is an offline historical standard sample database deployed in the cloud. The business feedback interface is the actual user interaction correction signal received by the terminal device.

[0117] In the backpropagation mechanism of deep learning, the dimension of the feature gradient tensor is exactly the same as that of the intermediate feature tensor in the forward propagation. Direct transmission across nodes would cause huge network bandwidth overhead and even lead to communication link congestion. To address this, this embodiment backpropagates the gradient of the loss function to the breakpoint of the target logic execution along the directed acyclic computation graph of the tail sub-model. After obtaining a high-precision feature gradient tensor, a preset gradient compression algorithm is used to perform gradient compression on the feature gradient tensor. This operation generates compressed feature gradient data with a significantly reduced data volume and distributes the compressed feature gradient data to the first computing node, thereby ensuring that the backward communication overhead and the forward feature compression are matched in magnitude on the physical link. The preset gradient compression algorithm is preferably a gradient sparsity algorithm or a low-bit quantization algorithm.

[0118] Step S402: In the first computing node, a reverse decompression operation is performed on the received compressed feature gradient data to reconstruct an approximate feature gradient tensor, specifically:

[0119] In the first computation node, the compressed feature gradient data is decompressed to reconstruct the feature gradient tensor. In the backpropagation, the quantization step is approximated as an identity mapping. Based on the pass-through estimator mechanism, the reconstructed feature gradient tensor is used as the backpropagation input gradient of the head sub-model at the breakpoint of the target logic execution.

[0120] Specifically, since the feature quantization operations (rounding and truncation) in step S3 are step functions in mathematical differentiation, their derivatives are almost everywhere zero. If the derivative is calculated using the conventional chain rule, the gradient during backpropagation will be truncated at the quantization layer, causing the gradient to vanish and preventing the head sub-model from being updated. To overcome this obstacle, this embodiment explicitly introduces a pass-through estimator mechanism during backpropagation, which approximates the partial derivatives of the quantization stage as an identity mapping, i.e., sets the derivative to 1. Based on this mechanism, the reconstructed feature gradient tensor can smoothly penetrate the non-differentiable quantization stage. The specific processing logic is as follows:

[0121] In forward propagation, the quantization operation consists of three sub-steps:

[0122] First, the floating-point intermediate features are divided by the quantization scaling parameter to perform numerical scaling;

[0123] Second, zero-point offset is performed by adding zero-point parameters;

[0124] Third, perform the round-to-round and clip-to-truncate operations.

[0125] Dequantization is the reverse process: first subtract the zero-point parameter, then multiply by the quantization scaling parameter to restore the floating-point value range.

[0126] When constructing the gradient flow through backpropagation, the zero-point parameter is used as a constant offset term. According to the differentiation rule, the gradient contribution of the constant is zero, so it can be safely ignored.

[0127] Although the quantization scaling parameter participates in division and multiplication operations, since quantization and dequantization use the same quantization scaling parameter value, in a complete forward-backward closed loop, the effect of the quantization scaling parameter is a symmetrical operation of division followed by multiplication, and their scaling effects on the gradient cancel each other out.

[0128] Therefore, the gradient propagation strategy adopted in this embodiment is as follows:

[0129] The feature gradient tensor obtained by returning and decompressing the second computing node is directly used as the gradient of the intermediate feature tensor relative to the loss function. There is no need to perform additional gradient correction on the quantization scaling parameters, nor is there a need to consider the gradient term of the zero-point parameters.

[0130] This invention is intentionally designed this way because when the quantization bit width is not less than 4 bits, the gradient estimation deviation is controllable and will not significantly affect the convergence performance of the head sub-model. This balances maintaining computational efficiency while effectively transmitting error signals, enabling efficient parameter updates for the user's head sub-model.

[0131] Step S403: Using the backpropagation input gradient, backpropagation calculations are performed sequentially along the directed acyclic computation graph of the head sub-model to each network layer contained in the head sub-model, and the parameter gradient tensors corresponding to the network parameters in the head sub-model are output.

[0132] The parameter gradient tensor is input into a preset parameter optimizer. The optimizer performs gradient descent iterative calculations on the network parameters of the head sub-model, updating the network parameters of the head sub-model. At this point, the online fine-tuning loop of the target model's business execution chain is complete. The preset parameter optimizer is the Adam optimization algorithm.

[0133] It should be noted that continuous parameter updates of the head sub-model will cause the distribution of its output intermediate feature tensor values ​​to gradually shift, which may lead to a distribution mismatch problem between it and the fixed tail sub-model.

[0134] This embodiment also performs feature normalization processing after the target logic execution breakpoint, including:

[0135] After outputting the intermediate feature tensor in step S301 and before performing the quantization operation, the intermediate feature tensor is normalized to map the feature values ​​to a stable distribution range. Normalization adjusts the features by calculating the statistical properties of the current intermediate feature tensor, ensuring the output features maintain a relatively stable numerical range. This normalization module serves as the final network layer of the head sub-model, and its internal parameters are updated along with the head sub-model in step S403, ensuring that the normalization process can adaptively track changes in the head sub-model's parameters.

[0136] This invention is configured such that even if the parameters of the head network drift, the output features after normalization can still maintain statistical properties that match the expected input of the tail sub-model. Subsequent quantization operations are performed on the normalized features, thereby ensuring that the compressed feature data transmitted to the second computing node can be stably processed by the tail sub-model after dequantization.

[0137] The cloud-based tail sub-model, serving as a global anchor, requires no updates or redeployment. All computational burden for distribution alignment is handled by the lightweight normalization module on the edge, reducing the overhead costs of cloud-based model version management and multi-device synchronization in traditional joint training.

[0138] Step S404: Collect the network transmission delay, node computing power consumption, and model prediction error within the current execution cycle as state evaluation variables, and use them together as state evaluation variables to evaluate the quality of the current segmentation and quantization actions.

[0139] When constructing the immediate reward function for reinforcement learning, since latency, energy consumption, and error are independent indicators with different physical dimensions, this application normalizes each state evaluation variable and then sums them by weight to obtain the comprehensive cost, and takes the negative value of the comprehensive cost as the immediate reward.

[0140] The weighting coefficients are pre-calibrated on the offline validation set using either the analytic hierarchy process (AHP) or the grid search method, specifically as follows:

[0141] When constructing the immediate reward function for reinforcement learning, network transmission latency is a significant factor. Node computing power consumption and model prediction error As independent indices with different physical dimensions, each state evaluation variable is first subjected to Min-Max normalization, mapped to the [0,1] interval to eliminate dimensional differences. The mathematical expression for the comprehensive cost is:

[0142] ;

[0143] in, For the overall cost, , , This represents the normalized network transmission delay, node computing power consumption, and model prediction error. Preferably, Take 0.4, Take 0.3, Take 0.3.

[0144] The immediate reward R is defined as the negative of the total cost. This means that a higher reward value indicates better overall system performance.

[0145] The system extracts state feature vectors, policy probability distribution tensors, and joint target actions from a pre-defined experience trajectory buffer. It then calculates the policy gradient using the policy gradient theorem, combined with immediate rewards, and updates the network parameters of the reinforcement learning decision network using this gradient. By continuously interacting with the environment and receiving immediate rewards, the decision network gradually converges, enabling it to adaptively output the optimal model segmentation and quantization configuration scheme in future inference tasks.

[0146] In this invention, step S4 breaks the open-loop limitation of traditional collaborative reasoning systems. By constructing a physical communication link equivalent to forward data compression and backward gradient compression, and combining it with a policy feedback closed loop, the online fine-tuning of the target model and the adaptive evolution of the decision network are realized.

[0147] The specific execution timing of steps S403 and S404 satisfies the asynchronous scheduling constraint, as follows:

[0148] The first computing node divides the execution cycle into multiple inference time slots. In any inference time slot, the first computing node only performs the forward computation of the reinforcement learning decision network in step S2 and the forward inference of the head sub-model in step S3. The update of the head sub-model network parameters in step S403 and the update of the reinforcement learning decision network parameters in step S404 are only performed asynchronously during the idle time of the first computing node after the accumulated input data of the current processing reaches the preset batch threshold. The preset batch threshold is set to be updated once every 64 inference samples. The updates of the head sub-model network parameters and the updates of the reinforcement learning decision network parameters are performed in alternating rounds and do not trigger backpropagation computation at the same time.

[0149] In practical edge computing scenarios, such as IoT devices or low-power terminals, the memory and instantaneous computing power of edge devices are usually extremely limited. Simultaneously executing forward inference, RL decision network forward propagation, model backpropagation, and policy backpropagation within a single cycle would overwhelm the device's computing power, resulting in extremely high latency overhead. Therefore, this embodiment introduces asynchronous decoupling and frequency-layered scheduling constraints in the execution mechanisms of steps S403 and S404. Specifically, the first computing node divides the execution cycle into multiple inference time slots. In the regular high-frequency inference time slots, the edge device is only responsible for one forward inference of the lightweight RL decision network (equivalent to passing through a small MLP network) and the forward propagation of the head sub-model, while the computationally intensive backpropagation and parameter update operations are offloaded to idle computing periods.

[0150] In this embodiment, the network parameter update frequency of the head sub-model is much lower than the inference frequency, and the update frequency of the reinforcement learning decision network is even lower than that of the head sub-model. One decision network update is performed every five alternating model update cycles. Through this staggered scheduling and batch processing in the time dimension, the additional latency introduced by a single backpropagation is amortized over multiple inference cycles, making the instantaneous peak computing power of the edge device controllable. While ensuring the system has continuous learning and adaptive evolution capabilities, it also ensures that its actual end-to-end latency remains significantly lower than that of pure local inference.

[0151] The network parameters of the reinforcement learning decision network are updated using policy gradients, specifically as follows:

[0152] Considering the computing power limitations of edge devices, this embodiment preferably uses the Monte Carlo policy gradient algorithm as the core reinforcement learning framework. The Monte Carlo policy gradient algorithm does not require the construction of a complex value network, making it the lightest and most suitable policy search method for edge scenarios. Specifically, the reinforcement learning decision network uses two fully connected hidden layers with dimensions of 256 and 128, respectively. The hidden layer activation function is ReLU, and the output layer is Softmax to map the policy probability distribution. A preset learning rate is used. Preferred The discount factor is 0.99, and the capacity of the experience trajectory buffer is set to [value missing]. A series of interactive trajectories are used. The Monte Carlo policy gradient algorithm is employed to update the reinforcement learning decision network. The first computation node extracts the state feature vector from the experience trajectory buffer. Policy probability distribution tensor and joint action with targets Combined with instant rewards Calculate the policy gradient :

[0153] ;

[0154] The decision network parameters are updated using the stochastic gradient ascent method. ,in The preset learning rate is preferably between 0.0001 and 0.001, and a linear warm-up strategy is used in the early stages of training to gradually increase it to the target value, preventing policy oscillations caused by unstable reward signals in the early stages of parameter updates. Network parameter update formula. The addition operation in the algorithm reflects the gradient ascent direction, that is, iteratively optimizing the decision network in the direction of maximizing the expected reward.

[0155] In this embodiment, policy gradient mathematical expectation symbol in This is not the theoretical overall expectation, but rather the arithmetic mean of the gradients of all samples within a small batch randomly selected by the first computation node from the empirical trajectory buffer, with a batch size of B=64. Specifically, assume that the historical interaction trajectories stored in the buffer are... The policy gradient that actually participates in parameter updates is approximately:

[0156] ;

[0157] in, For the first The state feature vectors recorded in the trajectory For the corresponding selected target joint action, This is the instantaneous reward scalar obtained and calculated by step S404 after the action is performed.

[0158] It should be noted that the REINFORCE algorithm variant used in this application relies solely on the immediate reward following a single-step decision. Instead of relying on multi-step cumulative discount rewards as in traditional reinforcement learning, this approach is used in edge-cloud collaborative inference scenarios. This is because after each segmentation and quantization decision, the second computing node immediately returns the inference result and calculates the energy consumption, latency, and accuracy costs of that decision. The decision's effectiveness is instantly evaluable, eliminating the need for backtracking across time steps. This single-step, instant reward design significantly reduces the memory overhead of storing historical reward sequences on edge devices, aligning with the engineering constraints of limited resources on the first computing node.

[0159] Log probability gradient in the formula The physical meaning is that in the current policy network parameters The gradient direction is sensitive to changes in the probability of an action occurring with respect to parameters. Multiplying the immediate reward by this gradient direction increases the probability of that action being chosen again in the future if the immediate reward from a decision is positive (low latency, low energy consumption, low error). Conversely, if the reward is negative, the update decreases the probability of that action, allowing the reinforcement learning decision network to maximize benefits and minimize risks in continuous online interactions, ultimately converging to the optimal joint splitting and quantization strategy.

[0160] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0161] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this invention should be included within the scope of protection of this invention.

Claims

1. An adaptive model segmentation and collaborative reasoning optimization method based on reinforcement learning, characterized in that, Includes the following steps: Step S1: Obtain the current computing load characteristics, current network transmission status characteristics, and data complexity characteristics, and concatenate them to output the status feature vector; Step S2: Input the state feature vector into the reinforcement learning decision network, output the layering index and feature quantization compression ratio, and divide the target model into a head sub-model configured on the first computing node and a tail sub-model configured on the second computing node based on the layering index. Step S3: Input the current input data into the head sub-model to obtain the intermediate feature tensor, obtain the quantization scaling parameter based on the feature quantization compression ratio, quantize to obtain compressed feature data, send the compressed feature data and quantization scaling parameter to the second computing node for dequantization, and output the inference result through the tail sub-model. Step S4: Receive compressed feature gradient data generated by the second computing node based on the inference results, use the compressed feature gradient data to perform backpropagation calculation on the head sub-model to obtain the parameter gradient tensor, and use the parameter optimizer to update the network parameters of the head sub-model based on the parameter gradient tensor.

2. The adaptive model segmentation and collaborative reasoning optimization method based on reinforcement learning as described in claim 1, characterized in that, Step S1 includes the following sub-steps: Step S101: Collect the processor utilization and remaining memory capacity of the first computing node, normalize the processor utilization and remaining memory capacity, and output the current computing load characteristics. Step S102: Probe the communication link of the first computing node, obtain the effective transmission bandwidth value and round-trip delay value at the current moment, and combine them to generate the current network transmission status characteristics. Parse the current input data, extract the tensor dimension information and tensor size information of the current input data, traverse the directed acyclic computation graph of the target model and accumulate the forward propagation floating-point operation volume of each network layer, and calculate the estimated total floating-point operation volume of the current input data in the target model; Traverse the directed acyclic computation graph to count the peak memory usage of the output tensors of each network layer, and obtain the estimated peak memory usage. The estimated total floating-point operations and the estimated peak video memory usage are combined to generate data complexity features.

3. The adaptive model segmentation and collaborative reasoning optimization method based on reinforcement learning as described in claim 2, characterized in that, Step S1 further includes the following sub-steps: Step S103: Map the current computing load characteristics, current network transmission status characteristics, and data complexity characteristics to a unified numerical range, and perform tensor concatenation operation on the mapped features along a preset feature dimension to output a state feature vector.

4. The adaptive model segmentation and collaborative reasoning optimization method based on reinforcement learning as described in claim 3, characterized in that, Step S2 includes the following sub-steps: Step S201: Input the state feature vector into the reinforcement learning decision network for forward computation, and output the policy probability distribution tensor corresponding to the preset joint action space; Step S202: Based on the policy probability distribution tensor, select the action, determine the target joint action, map the target joint action to the preset physical parameter space, obtain the layer index and feature quantization compression ratio of the target model, and store the state feature vector, policy probability distribution tensor and target joint action in the preset empirical trajectory cache area.

5. The adaptive model segmentation and collaborative reasoning optimization method based on reinforcement learning as described in claim 4, characterized in that, Step S2 further includes the following sub-steps: Step S203: Determine the target logic execution breakpoint in the directed acyclic computation graph of the target model based on the layered index; The forward propagation execution boundary of the first computing node is set as the target logic execution breakpoint, and the target logic execution breakpoint and the network layer on the predecessor path of the target logic execution breakpoint are determined as the head sub-model. The layer index is sent to the second computing node, which instructs the second computing node to set the forward propagation start boundary as the set of direct successor network layers of the target logic execution breakpoint, and to determine the set of direct successor network layers and the network layers on the subsequent path of the set of direct successor network layers as the tail sub-model.

6. The adaptive model segmentation and collaborative reasoning optimization method based on reinforcement learning as described in claim 5, characterized in that, Step S3 includes the following sub-steps: Step S301: In the first computing node, the current input data is input into the head sub-model, and forward propagation calculation is performed sequentially along the predecessor path of the directed acyclic computation graph until the target logic execution breakpoint is reached, and the intermediate feature tensor is output.

7. The adaptive model segmentation and collaborative reasoning optimization method based on reinforcement learning as described in claim 6, characterized in that, Step S3 further includes the following sub-steps: Step S302: Determine the target quantization bit width based on the feature quantization compression ratio, statistically analyze the numerical distribution of the intermediate feature tensor, and calculate the quantization scaling parameter and zero point parameter matching the target quantization bit width based on the numerical distribution. The intermediate feature tensor is subjected to numerical type conversion and linear mapping operations using the quantization scaling parameters and zero-point parameters to output compressed feature data.

8. The adaptive model segmentation and collaborative reasoning optimization method based on reinforcement learning as described in claim 3, characterized in that, Step S3 further includes the following sub-steps: Step S303: Send the compressed feature data, quantization scaling parameters, and zero-point parameters to the second computing node; The second computing node uses the quantization scaling parameters and zero-point parameters to perform an inverse linear mapping on the compressed feature data, generating an inverse quantization feature tensor. The inverse quantization feature tensor is then input into the set of direct successor network layers in the tail sub-model to perform forward propagation computation and output the inference result.

9. The adaptive model segmentation and collaborative reasoning optimization method based on reinforcement learning as described in claim 8, characterized in that, Step S4 includes the following sub-steps: Step S401: The second computing node obtains the real labels corresponding to the current input data through a preset sample label library, calculates the model prediction error based on the inference results and real labels, and constructs a loss function. Along the directed acyclic computation graph of the tail sub-model, the gradient of the loss function is backpropagated to the breakpoint of the target logic execution to obtain the feature gradient tensor. The feature gradient tensor is then compressed using a preset gradient compression algorithm to generate compressed feature gradient data, which is then distributed to the first computation node. In step S402, in the first computing node, a decompression operation is performed on the compressed feature gradient data to reconstruct the feature gradient tensor, and the quantization step is approximated as an identity mapping in backpropagation. Based on the pass-through estimator mechanism, the reconstructed feature gradient tensor is used as the backpropagation input gradient of the head sub-model at the execution breakpoint of the target logic.

10. The adaptive model segmentation and collaborative reasoning optimization method based on reinforcement learning as described in claim 9, characterized in that, Step S4 further includes the following sub-steps: Step S403: Using the backpropagation input gradient, backpropagation calculation is performed sequentially along the directed acyclic computation graph of the head sub-model to each network layer contained in the head sub-model, and the parameter gradient tensor corresponding to the network parameters in the head sub-model is output. The parameter gradient tensor is input into a preset parameter optimizer, and the network parameters of the head sub-model are updated by performing gradient descent iterative calculation on the network parameters of the head sub-model through the preset parameter optimizer. Step S404: Collect the network transmission delay, node computing power consumption, and model prediction error within the current execution cycle as state evaluation variables; After normalizing each state evaluation variable, a weighted sum is obtained to get the comprehensive cost, and the negative value of the comprehensive cost is used as the immediate reward. The state feature vector, policy probability distribution tensor, and target joint action are extracted from the preset experience trajectory buffer. The policy gradient is calculated in combination with the immediate reward, and the network parameters of the reinforcement learning decision network are updated using the policy gradient.

Images