Learning-based collaborative computing method for multi-branch models in D2D networks

Abstract

The present application discloses a learning-based collaborative computing method for a multi-branch model in a D2D network. The method comprises: obtaining an inference model submitted by a terminal device in the D2D network and computing resource information for each layer in the inference model; the inference model is a multi-branch structure, i.e., each layer contains multiple sub-inference tasks; the computing resource information includes at least the number of sub-inference tasks that need to be calculated in parallel in each layer, the transmission data size of each sub-inference task, and the number of floating-point operations required; obtaining computing power information for each network node in the D2D network; and using a reinforcement learning model to output a target collaborative scheduling strategy based on the computing resource information for each layer and the computing power information for each network node, so that each network node in the D2D network can perform collaborative computing on the sub-inference tasks of each layer according to the target collaborative scheduling strategy. The present application can solve the inference problem of complex multi-branch models, minimize inference latency, and improve resource utilization.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of edge computing, and in particular to a learning-based multi-branch model collaborative computing method in a D2D network. Background Art

[0002] In recent years, the number of mobile devices and connected devices has exploded, bringing more opportunities for edge computing. By combining mobile edge computing with deep learning, powerful computing power and low-latency processing are provided at the edge of the network, greatly promoting the development of real-time applications.

[0003] When processing simple deep model inference, a split point is typically selected based on the system environment and model structure, dividing the model into two parts: one part computed locally on the mobile device, and the other offloaded to an edge server for computation. While this task collaboration approach reduces inference latency to a certain extent, it mostly only considers simple linear deep models. In these schemes, inference computations on the mobile device and edge server are performed serially, meaning the output of the sub-model on the mobile device serves as the input to the edge sub-model.

[0004] However, the structures of widely used deep models are becoming increasingly complex, such as multi-branch structures that can be executed in parallel. For such complex models, relying solely on traditional mobile edge computing technologies has certain limitations and high inference latency.

[0005] Furthermore, device-to-device (D2D) communication technology allows devices to communicate directly with each other without going through base stations or repeaters. This technology offers significant advantages over other technologies in terms of improving spectrum efficiency, reducing latency, and reducing power consumption. With the development of the Internet of Things, mobile internet, and 5G networks, D2D communication technology has been widely researched and applied. D2D technology is used to offload computing tasks to neighboring mobile devices, providing real-time computing services for compute-intensive applications. However, existing D2D technologies still have limitations when handling complex multi-branch models and lack efficient collaborative computing strategies. Summary of the Invention

[0006] Based on the above problems, the embodiments of the present application provide a learning-based multi-branch model collaborative computing method in a D2D network, with the aim of reducing inference latency and improving resource utilization.

[0007] In a first aspect, an embodiment of the present application provides a learning-based multi-branch model collaborative computing method in a D2D network, which is applied to a device-to-device D2D network, wherein the D2D network is distributed with a plurality of network nodes, and a direct communication path exists between the plurality of network nodes, wherein the network nodes include a plurality of terminal devices and a plurality of edge servers, and the method includes:

[0008] Obtaining the inference model submitted by the terminal device in the D2D network and computing resource information for each layer in the inference model; wherein the computing resource information includes at least the number of sub-inference tasks that need to be calculated in parallel at each layer in the inference model, the transmission data size of each sub-inference task, and the number of floating-point operations required for each sub-inference task; the inference model adopts a multi-branch model structure and includes multiple sub-inference tasks;

[0009] Obtaining computing power information of each of the network nodes in the D2D network;

[0010] According to the computing resource information of each layer in the reasoning model and the computing power information of each network node, the reinforcement learning model is used to output the target collaborative scheduling strategy so that each network node under the D2D network can perform collaborative calculations on the sub-reasoning tasks of each layer according to the target collaborative scheduling strategy.

[0011] In one embodiment, the outputting of the target collaborative scheduling strategy using a reinforcement learning model based on the computing resource information of each layer in the inference model and the computing power information of each network node includes:

[0012] In combination with the Markov decision process, the state space, action space, and action reward of the reinforcement learning model are determined;

[0013] Wherein, for each layer in the inference model, the state space includes computing power information of each network node, actions to be executed determined by the previous layer, and computing resource information of the current layer;

[0014] The action space includes the action selection state of each of the network nodes;

[0015] The action reward includes the negative of the inference latency of the current layer;

[0016] Determining, based on the state space, an action to be performed for each of the network nodes having a maximum state value;

[0017] Taking the to-be-executed actions of each of the network nodes having the maximum state value as the collaborative scheduling strategy of the current layer;

[0018] The target collaborative scheduling strategy is constructed and output according to the collaborative scheduling strategies of each layer.

[0019] In one embodiment, the state space also includes: network parameters, which include an adjacency matrix that characterizes the communication connection relationship between network nodes at each layer under the D2D network; the adjacency matrix is ​​used to input into a convolutional network model to learn and output the network topology characteristics under the current D2D network.

[0020] In one embodiment, the reinforcement learning model further includes a policy model, the policy model including an actor network and a critic network, the actor network being configured to output a pending action for each of the network nodes, the critic network being configured to output a state value based on the pending action output by the actor network and a current state space, and determining the pending action for each of the network nodes having the maximum state value based on the state space, including:

[0021] Determining an action to be performed by each of the network nodes based on the state space;

[0022] In the actor network, updating actor network parameters based on a first update expression;

[0023] In the critic network, updating critic network parameters based on a second update expression;

[0024] Based on the updated actor network parameters and commentator network parameters, the to-be-executed actions of each of the network nodes having the maximum state value are determined.

[0025] In one embodiment, in the actor network, updating the actor network parameters based on the first update expression includes:

[0026] Using the proximal policy optimization method, the actor network parameters are updated by the first update expression:

[0027] The first update expression is:

[0028]

[0029] in, Indicates the The generalized advantage estimate of the step, is the discount factor, Generalized advantage estimate The smoothing parameter, represents the time step offset, For the moment Time difference;

[0030] In the critic network, updating the critic network parameters based on the second update expression includes:

[0031] Based on the second update expression, the stochastic gradient descent method is used to calculate the minimum time difference to update the critic network parameters;

[0032] The second update expression is:

[0033]

[0034] in, Indicates the The time difference of the step, For the time step Action rewards obtained; is the discount factor used to calculate the current value of future rewards; In state The state value of Status The status value.

[0035] In one embodiment, the method further comprises:

[0036] When updating the actor network parameters and the critic network parameters, limiting the ratio of the policy distribution between the new policy model and the old policy model by updating the loss function;

[0037] The updated loss function is:

[0038]

[0039] in, is the clipping loss function; Under the new strategy, Take action probability;

[0040] For the old policy in the state Take action probability; For the The advantage function estimate of the step, is the shear limit function; is a preset value used to control the step size of the strategy update. is the weight of the actor network.

[0041] In one embodiment, the actor network is configured to have multiple outputs, each output representing a collaborative scheduling strategy for the sub-reasoning task.

[0042] In a second aspect, an embodiment of the present application further provides a learning-based multi-branch model collaborative computing device in a D2D network, which is applied to a device-to-device D2D network, wherein the D2D network is distributed with a plurality of network nodes, and a direct communication path exists between the plurality of network nodes, wherein the network nodes include a plurality of terminal devices and a plurality of edge servers, and the device includes:

[0043] An acquisition unit is configured to acquire the inference model submitted by the terminal device in the D2D network and computing resource information of each layer in the inference model; wherein the computing resource information includes at least the number of sub-inference tasks that need to be calculated in parallel at each layer in the inference model, the transmission data size of each sub-inference task, and the number of floating-point operations required for each sub-inference task; the inference model adopts a multi-branch model structure and includes multiple sub-inference tasks;

[0044] The acquiring unit is further configured to acquire computing power information of each of the network nodes in the D2D network;

[0045] A processing unit is used to output a target collaborative scheduling strategy using a reinforcement learning model based on the computing resource information of each layer in the inference model and the computing power information of each network node, so that each network node under the D2D network can collaboratively calculate the sub-inference tasks of each layer according to the target collaborative scheduling strategy.

[0046] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0047] CPU, memory, input and output interfaces;

[0048] The memory is a transient storage memory or a persistent storage memory;

[0049] The central processing unit is configured to communicate with the memory and execute instruction operations in the memory to perform any one of the above-mentioned learning-based multi-branch model collaborative computing methods in a D2D network.

[0050] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is run on a computer, it executes any one of the above-mentioned learning-based multi-branch model collaborative computing methods in a D2D network.

[0051] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages: by obtaining the inference model submitted by the terminal device under the D2D network and the computing resource information of each layer thereof; obtaining the computing power information of each network node under the D2D network; according to the computing resource information of each layer in the inference model and the computing power information of the network node, the reinforcement learning model is used to output the target collaborative scheduling strategy, so that each network node under the D2D network can perform collaborative calculations on the sub-inference tasks of each layer according to the target collaborative scheduling strategy. By considering the computing power information of each network node and the computing resource information of each layer in the inference model, the reinforcement learning model is used to make decisions to maximize the state value, optimize the allocation of network node resources, and enable multiple devices to execute the sub-inference tasks in the inference model, i.e., the multi-branch model, in parallel, thereby reducing the total time required to complete the model inference and improving resource utilization and system efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0053] Figure 1 A flowchart of a learning-based multi-branch model collaborative computing method in a D2D network provided by an embodiment of the present application;

[0054] Figure 2 A schematic diagram of a reinforcement learning actor network architecture provided in an embodiment of the present application;

[0055] Figure 3 A flowchart of a multi-branch model collaborative computing method based on learning in a D2D network provided by an embodiment of the present application;

[0056] Figure 4 A schematic diagram of the structure of a multi-branch model collaborative computing device based on learning in a D2D network provided by an embodiment of the present application;

[0057] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0059] With the continuous development of artificial intelligence (AI) technology, deep learning-based applications are proliferating. While these applications have brought significant convenience, they also place higher demands on computing device performance. Latency-sensitive applications, in particular, urgently require computing devices with faster deep learning model inference capabilities. However, current mobile devices have relatively limited computing resources. Constrained by processor power constraints, they can only run simple, lightweight deep learning models. While these lightweight models can reduce the burden on mobile devices, they suffer from low inference accuracy, limited support for certain application scenarios, and relatively poor quality of service.

[0060] For example, the recently popular generative large language model, ChatGPT, requires enormous computing power, making it impossible for current mobile devices to run such complex models. While deploying deep learning applications on cloud servers can address the issue of limited computing resources, cloud servers are typically located far from mobile devices, resulting in significant transmission latency and failing to meet the needs of latency-sensitive applications.

[0061] To address these issues, Mobile Edge Computing (MEC) technology has emerged. This technology reduces model inference latency by offloading some computational tasks to edge servers close to mobile devices, effectively meeting application latency requirements. When processing simple deep model inference tasks, an appropriate model split point can be selected based on the system environment and model structure, splitting the model into two parts: one part computed locally on the mobile device, and the other offloaded to the edge server for processing. However, while this collaborative computing approach of splitting the inference model into two parts can reduce inference latency to a certain extent, it has mostly only considered deep models with simple linear structures, such as AlexNet and VGGNet. In these solutions, inference computations on the mobile device and edge server are performed serially, meaning the output of the sub-model on the mobile device serves as the input to the sub-model on the edge. However, the architectures of widely used deep models are becoming increasingly complex, particularly in fields such as computer vision, natural language processing, and multi-task learning, where multi-branch model structures are common. For example, the Inception architecture of Google LeNet is a multi-branch model that can be executed in parallel. A multi-branch model is a neural network structure containing multiple parallel branches, each of which can independently perform different computational tasks or process different information flows. Unlike traditional linear models (such as fully connected networks or convolutional neural networks, where each layer is connected layer by layer), the multi-branch model design allows multiple operations to be performed simultaneously through different paths in the network, thereby improving the network's expressiveness and computational efficiency. This is particularly true when processing complex data, allowing for more flexible capture of diverse features. However, currently, relying solely on traditional mobile edge computing technologies for such complex models has certain limitations and also results in high inference latency.

[0062] Furthermore, with the development of the Internet of Things (IoT), mobile internet, and 5G networks, D2D communication technology has been widely researched and applied. D2D communication technology refers to direct communication between two terminal devices within an allowable close range. In a D2D network, terminal devices can communicate directly with each other without going through intermediate facilities such as base stations, thereby improving spectrum utilization, reducing base station pressure, reducing end-to-end transmission latency, and enhancing system network performance. The application of D2D technology can offload computing tasks to adjacent terminal devices, providing real-time computing services for compute-intensive applications. However, existing D2D technologies and mobile edge computing still have limitations when processing complex multi-branch models, and they lack efficient collaborative computing strategies. As deep learning models become increasingly complex, traditional collaborative computing technologies can no longer meet current complex computing needs, and a new approach is urgently needed to complete the inference calculations of multi-branch models.

[0063] Based on this, in each embodiment of the present application, by obtaining the computing power information of each network node and the computing resource information of the inference model, the reinforcement learning model is used to make decisions to maximize the state value, optimize the allocation of network node resources, and enable sub-inference tasks to be executed in parallel among multiple network nodes, thereby reducing the total time required to complete task inference and improving resource utilization.

[0064] The following is a further detailed description of the various embodiments of the present application in conjunction with the accompanying drawings.

[0065] The embodiment of the present application provides a learning-based multi-branch model collaborative computing method in a D2D network, which is applied to a device-to-device D2D network. There are several network nodes distributed on the D2D network, and there are direct communication paths between the several network nodes. The network nodes include multiple terminal devices and multiple edge servers, such as Figure 1 As shown, the method includes steps S101-S103.

[0066] S101: Obtaining the inference model submitted by the terminal device in the D2D network and the computing resource information of each layer in the inference model.

[0067] S102: Obtain computing power information of each network node in the D2D network.

[0068] In the embodiments of the present application, a D2D network is composed of multiple network nodes, including several terminal devices (such as smartphones, sensors, and industrial controllers) at the device layer and several edge servers at the network layer. These network nodes are located at the edge of the network, close to the data source, and are used to process data and perform computing tasks. In a D2D network, one of the multiple terminal devices will publish an inference model to be calculated and transmit information about the computing resources required for the inference model to an edge network node.

[0069] An inference model can be understood as the process of reasoning about input data using a trained model. Based on its internal parameters and structure, the model processes the input data and generates output. Inference models are widely used in fields such as image classification, object detection, speech recognition, and natural language processing. Their core goal is to predict and analyze input data using the knowledge learned during model training. More complex inference models typically contain millions to billions of parameters. For example, inference models can adopt a multi-branch model structure, achieving efficient processing through a multi-branch structure that can be executed in parallel. This multi-branch model has multiple layers of computing structure, and each layer also includes sub-inference tasks that require parallel processing. This allows multiple branches to process data simultaneously, and finally, the processing results of each branch are aggregated to generate the final output.

[0070] Therefore, in the embodiment of the present application, the computing resource information of each layer includes at least the number of sub-inference tasks that need to be calculated in parallel at each layer in the inference model, the transmission data size of each sub-inference task, and the number of floating-point operations required for each sub-inference task. The purpose of obtaining this computing resource information is to better decompose and reasonably allocate complex deep learning models, especially multi-branch models, to ensure that each sub-inference task can be executed efficiently and quickly in the D2D network, which helps to optimize task scheduling, reduce latency, and improve the response speed and performance of the entire collaborative computing system.

[0071] Computing power information refers to the ability of network nodes to complete computing tasks, including but not limited to: the number of floating point operations per second (FLOPS) of each network node, performance indicators of the central processing unit (CPU) or graphics processing unit (GPU) (such as the number of cores, clock frequency, etc.), network bandwidth (the speed at which network nodes upload and download data), energy consumption, the number of basic computing units (such as media access control layer (MAC) computing units, computing cores), utilization rate, received signal strength indication (RSSI), etc.

[0072] Obtaining computing power information can help the reinforcement learning model determine the ability of each network node to perform computing tasks, so that the edge server can perform task allocation and resource scheduling according to the computing power of each network node, ensuring that the sub-inference task can be executed on the most appropriate network node and balancing the load of the network node. It is understandable that traditional mobile edge computing usually relies on the edge server as the main computing node, and the embodiment of the present application introduces a D2D network, so that each terminal device can directly collaborate to perform reasoning calculations. Furthermore, it can also combine the computing power information of each terminal device and edge server to dynamically adapt to changes in the network environment and task requirements to adjust the task allocation strategy. Regardless of how the sub-inference tasks are allocated, they ultimately achieve the same optimization goal, thereby adapting to changes in the network scale. This can not only reduce dependence on centralized edge servers, but also flexibly utilize the computing resources of other terminal devices or edge servers when some network nodes are heavily loaded or have limited resources, thereby improving the robustness and flexibility of the system.

[0073] S103: Based on the computing resource information of each layer in the inference model and the computing power information of each network node, the reinforcement learning model is used to output the target collaborative scheduling strategy, so that each network node in the D2D network can collaboratively calculate the sub-inference tasks of each layer according to the target collaborative scheduling strategy.

[0074] In reinforcement learning, a targeted collaborative scheduling strategy can be implemented by selecting actions that maximize expected rewards. This maximum expected reward can be evaluated using state value. State value refers to the cumulative reward expected from taking a specific action in a given state. The maximum state value guides the selection of actions that maximize expected rewards. This targeted collaborative scheduling strategy guides how to allocate sub-inference tasks to different network nodes in the network so that they can collaborate to complete the task.

[0075] The embodiments of the present application obtain computing resource information for inference models submitted by terminal devices in a D2D network, as well as computing power information for each network node in the D2D network. Based on the computing resource information for the sub-inference tasks and the computing power information for the network nodes, a reinforcement learning model is then used to output a target collaborative scheduling strategy, enabling each network node in the D2D network to collaboratively compute the sub-inference tasks according to the collaborative scheduling strategy. By utilizing multiple computing devices in the D2D network and taking into account the computing power information for each network node and the computing resource information for the sub-inference tasks, the reinforcement learning model is used to output the target collaborative scheduling strategy. This optimizes network node resource allocation, enabling the pending sub-inference tasks at each layer of the multi-branch model to be computed and processed simultaneously by multiple devices in the D2D network. Parallel execution of sub-inference tasks across multiple devices reduces the total time required to complete deep learning model inference, improving resource utilization and system efficiency. Furthermore, by solving the inference problem of complex multi-branch models, the application scope of mobile terminal devices in practical applications is expanded, enabling support for more high-precision, high-computation artificial intelligence (AI) applications.

[0076] In one embodiment, based on the computing resource information of each layer in the inference model and the computing power information of each network node, a reinforcement learning model is used to output a target collaborative scheduling strategy, including:

[0077] Combined with the Markov decision process, the state space, action space, and action reward of the reinforcement learning model are determined;

[0078] Among them, for each layer in the inference model, the state space includes the computing power information of each network node, the pending actions determined by the previous layer and the computing resource information of the current layer; the action space includes the action selection state of each network node; the action reward includes the negative of the inference delay of the current layer; based on the state space, the pending actions of each network node with the maximum state value are determined; the pending actions of each network node with the maximum state value are used as the collaborative scheduling strategy of the current layer; and the target collaborative scheduling strategy is constructed and output according to the collaborative scheduling strategies of each layer.

[0079] In an embodiment of the present application, in order to address the limitations of traditional mobile edge computing technology and the high inference latency, a collaborative scheduling strategy for the output of the reinforcement learning model can be guided by the goal of minimizing the inference latency.

[0080] This embodiment of the application defines a state space, an action space, and action rewards. Each layer's actions, or collaborative computing strategies, are generated by a policy model based on the current state. The state space can be composed of computing power information for each network node, task parameters (i.e., information about the computing resources available at the current layer of the inference task), and the collaborative scheduling strategy of the previous layer (i.e., the actions to be executed determined by the previous layer).

[0081] The task parameters include: the number of sub-inference tasks that need to be calculated in parallel at each layer in the inference model, the transmission data size of each sub-inference task, and the number of floating-point operations required for each sub-inference task.

[0082] The action space represents the network node to which each sub-reasoning task is offloaded, i.e., the action selection state of each network node, which is a multi-dimensional action. It is important to understand that multi-dimensional actions represent the multiple dimensions or aspects that the agent needs to consider when making decisions. For example, it can be based on task allocation, CPU, memory, storage, channel status, and other resource allocations.

[0083] Action Reward: Since the goal of this embodiment of the application is to minimize the inference latency, the action reward is specified as the negative of the current layer's inference latency: In this way, the learning direction of the strategy model is to maximize the reward when outputting the collaborative scheduling strategy for each layer, minimize the inference delay of each layer as much as possible, and make the strategy model learn and train towards the goal set by the present invention.

[0084] In the above embodiments, it is mentioned that the present invention takes minimizing the inference delay as the optimization goal. Therefore, the optimization problem required by the embodiments of the present application is:

[0085]

[0086] in, The inference delay of the embodiment of this application; is the cooperative scheduling strategy for the nth layer; represents the mth collaborative computing device of the first sub-reasoning task at the nth layer; For the nth layer The latency of collaborative computing of a sub-inference task on the mth device.

[0087] Because each task can only be calculated by one device, , all possible devices The sum must equal 1:

[0088]

[0089] if The value of is 1, which means the nth layer The task is assigned to the mth device for computation.

[0090] if The value of is 0, which means the nth layer Tasks are not assigned to the mth device for computation.

[0091] Through the above method, it can be ensured that the policy model can quickly adapt to the dynamic changes of the D2D network, while minimizing the reasoning delay and improving the efficiency of task collaborative reasoning.

[0092] In one embodiment, the state space also includes: network parameters, which represent the adjacency matrix of the communication connection relationship between network nodes at each layer under the D2D network; the adjacency matrix is ​​used to input into the convolutional network model to learn and output the network topology characteristics under the current D2D network.

[0093] Assuming there are n network nodes, the adjacency matrix can be understood as an n×n matrix, where each element A[i][j] represents the connection relationship between the i-th network node and the j-th network node. If the i-th network node and the j-th network node have a communication connection, the matrix value is 1; if there is no connection, the matrix value is 0.

[0094] The adjacency matrix can provide complete information about the D2D network structure, including the connection relationship between nodes and the characteristic information of nodes. In a D2D network, the network topology changes dynamically. Therefore, the embodiment of the present application proposes an improved state extraction model, which is designed in the actor network of the reinforcement learning model, such as Figure 2As shown, the state extraction model combines convolutional networks and fully connected networks. Convolutional networks can be used to extract network topology information from adjacency matrices (such as those for D2D), while fully connected networks are used to process other task parameters. It's important to understand that because convolutional networks excel at processing spatially structured information, such as images or graphs, they can be used to extract node relationships and network structure features. Task parameters processed by fully connected networks can include traffic load, latency, computing resource information, and computing power information in edge computing scenarios. Unlike adjacency matrices, these task data lack spatial structure, making fully connected networks more suitable for processing such unstructured data. Combining convolutional and fully connected networks can fully leverage the structural characteristics of different data to extract network topology features, improving the accuracy and efficiency of state extraction.

[0095] Furthermore, embodiments of the present application may consider replacing convolutional networks with graph neural networks (GNNs) when processing D2D adjacency matrices to more effectively capture network topology and node features. GNNs aggregate information about neighboring nodes to update node feature representations, thereby learning the position and role of network nodes in the D2D network, as well as the topology of the entire network. GNNs can effectively extract the network topology from these adjacency matrices, which can then be used for various downstream tasks such as node classification and graph classification.

[0096] In addition, in addition to using fully connected networks, other deep learning methods, such as self-attention mechanisms, can also be used to extract task parameter information to improve the accuracy of state representation.

[0097] In one embodiment, the reinforcement learning model also includes a strategy model, which includes an actor network and a critic network. The actor network is used to output the pending actions of each network node, and the critic network is used to output the state value according to the pending actions output by the actor network and the current state space. Based on the state space, the pending actions of each network node with the maximum state value are determined, including: determining the pending actions of each network node based on the state space; in the actor network, updating the actor network parameters based on the first update expression; in the critic network, updating the critic network parameters based on the second update expression; based on the updated actor network parameters and critic network parameters, determining the pending actions of each network node with the maximum state value.

[0098] Reinforcement learning models also include a policy model, which consists of an actor network and a critic network. The actor network predicts the action the agent should take based on the current state of the environment. The actor network outputs a probability distribution over actions, that is, the probability of executing each possible action in the current state. The critic network, with the same structure as the actor network, is used to evaluate the value of the current state and thus outputs the state value, that is, the expected reward of the agent in the current state.

[0099] In practice, the actor network updates its parameters using the first update expression, aiming to gradually optimize its ability to predict individual actions. Its update direction is guided by feedback from the critic network. The critic network updates its own parameters based on the second update expression, outputting the state value at the current state. By evaluating the actor network's performance, the critic network provides feedback on the value of each action in each state, guiding the actor network's update direction.

[0100] After updating the parameters of the actor and critic networks, the policy model calculates and evaluates the state value of each network node to select the best action that maximizes reward. Therefore, feedback from the critic network is crucial to optimizing the actor network, ensuring that the actor network can be updated and improved in the direction that maximizes reward.

[0101] In one embodiment, in an actor network, updating actor network parameters based on a first update expression includes: using a proximal policy optimization method to update the actor network parameters through the first update expression; the first update expression is:

[0102]

[0103] in, Indicates the The generalized advantage estimate of the step, is the discount factor, Generalized advantage estimate The smoothing parameter, represents the time step offset, For the moment time difference.

[0104] The first update expression calculates the generalized advantage estimate GAE, which is used to measure the advantage of the action and update the parameters of the actor network. By minimizing the difference between this advantage estimate and the actual reward, the output cooperative scheduling policy can be adjusted to select better actions.

[0105] In the critic network, based on the second update expression, the critic network parameters are updated, including:

[0106] Based on the second update expression, the stochastic gradient descent method is used to calculate the minimum time difference to update the critic network parameters;

[0107] The second update expression is:

[0108]

[0109] in, Indicates the The time difference of the step, For the time step Action rewards obtained; is the discount factor used to calculate the current value of future rewards; In state The state value of Status The status value.

[0110] The second update expression evaluates the error between the critic network's value prediction and actual reward by calculating the temporal difference (TD). The critic network updates its parameters using the TD error to learn a more accurate value function. If the TD error is positive, indicating that the actual action reward is better than expected, the critic network will increase the state value of the current state; if the TD error is negative, indicating that the actual action reward is worse than expected, the critic network will decrease the state value of the current state.

[0111] Assume that an agent is in state An action was chosen and gained =10 reward, and then transferred to the new state If the critic network predicts The long-term value is =20, and The prediction is =30, and the discount factor =0.99, then the time difference Calculated as: =10+0.99×30−20=10+29.7−20=19.7.

[0112] In one embodiment, the method further comprises: when updating the actor network parameters and updating the critic network parameters, limiting the ratio of the policy distribution between the new policy model and the old policy model by updating the loss function;

[0113] Update the loss function to:

[0114]

[0115] in, is the clipping loss function; Under the new strategy, Take action probability; For the old policy in the state Take action probability; For the The advantage function estimate of the step, is the shear limit function; is a preset value used to control the step size of the strategy update. is the weight of the actor network.

[0116] The update loss function defines the loss function used to update the policy network parameters in Proximal Policy Optimization (PPO), with the goal of optimizing the policy while maintaining the stability of the policy update. For example, assuming that the new and old policies are and , the probability of taking a certain action in a certain state is 0.6 and 0.4 respectively, and the advantage function is estimated =5, and =0.1, then the strategy ratio is = 1.5, which exceeds the limit range [1−0.1, 1+0.1]=[0.9, 1.1][1−0.1, 1+0.1]=[0.9, 1.1], so the loss function is calculated using the limited value 1.1. This ensures that the policy update improves performance without deviating too far from the original policy, thus avoiding performance instability caused by large updates.

[0117] In one embodiment, the actor network is configured to output multiple heads, each output representing a collaborative scheduling strategy for a sub-reasoning task.

[0118] like Figure 2As shown, considering the special structure of the state space and the multi-dimensional characteristics of the action space, the present invention customizes the network architecture for the actor network. At the input end, the state space is divided into two parts. Due to the two-dimensional structure of the connection matrix of the network nodes, the embodiment of the present application can extract the network topology features by using a convolutional network, and the remaining state parameters are extracted by a fully connected network. A multi-head output design is adopted at the output end to improve the PPO algorithm in the traditional deep reinforcement learning algorithm. The original PPO algorithm only has one output head, that is, only one sub-reasoning task is input each time to obtain the result corresponding to the sub-reasoning task. In this embodiment, multiple output heads are added, which can simultaneously input the computing resource information of multiple sub-reasoning tasks in a layer and the computing power information of each network node, and then through one calculation, the collaborative scheduling strategy corresponding to each sub-reasoning task in the layer can be output. Finally, the collaborative scheduling strategies of the sub-reasoning tasks of each layer are combined to generate the final target collaborative scheduling strategy.

[0119] Through this improvement, the embodiment of the present application can more efficiently handle the collaborative scheduling problem of multiple inference tasks, thereby improving network node resource utilization and decision-making efficiency.

[0120] The following will be Figure 1 and Figure 2 Based on the embodiment shown, the embodiment of the present application is further described in detail. Please refer to Figure 3 , Figure 3 A flowchart of a collaborative computing method is shown in FIG.

[0121] In In a D2D network with multiple devices, a device publishes the computing resource information of the model inference task to be calculated to the edge server. layer, each layer The number of tasks that need to be computed in parallel is For the Tier A task defines two important parameters: Calculate the size of the data to be transferred and the number of floating-point operations required In addition, let Indicates the collaborative computing device for each task. If the value is It represents the Tier The task is carried out by Device calculation, otherwise if The value of is 0, which means the nth layer The task is not assigned to the mth device for computation.

[0122] For the The task of the layer is to minimize the inference delay as the optimization goal, using Characterizes the collaborative scheduling strategy of each layer. When the parameters of each layer are passed to the edge server, the edge server will give the collaborative scheduling strategy of the layer, i.e. After that, the same steps are performed on the next layer in the inference model until all layers have obtained corresponding collaborative scheduling strategies. The collaborative scheduling strategies of each layer are then combined into the final target collaborative scheduling strategy. In this way, all network nodes in the network can follow the target collaborative scheduling strategy formulated by the edge server and collaboratively complete the inference model calculation.

[0123] In one feasible embodiment, in addition to minimizing inference latency as the optimization objective, inference latency and other performance indicators (such as energy consumption and bandwidth utilization) can also be jointly optimized, using a multi-objective optimization approach to balance the relationships between different performance indicators. The core concept of multi-objective optimization is to simultaneously consider multiple performance indicators and find an optimal solution set, rather than a single optimal solution. In practice, it is necessary to define an optimization model that can simultaneously reflect inference latency, energy consumption, and bandwidth utilization, and design a corresponding optimization algorithm to solve this model. In one feasible embodiment, a multi-objective optimization model can be established, with inference latency, energy consumption, and bandwidth utilization as optimization objectives, and weights assigned to each objective. By adjusting the weights, trade-offs can be achieved between different objectives. For example, if energy consumption is more important than inference latency, a higher weight can be assigned to energy consumption.

[0124] Based on the aforementioned embodiment, in addition to using the PPO algorithm, other reinforcement learning algorithms, such as Deep Deterministic Policy Gradient (DDPG) or Soft Actor Critic (SAC), can also be used to perform policy optimization to obtain the target collaborative scheduling strategy.

[0125] In order to implement the multi-branch model collaborative computing method based on learning in the D2D network of the embodiment of the present application, the embodiment of the present application also provides a multi-branch model collaborative computing device based on learning in the D2D network, which is applied to a device-to-device D2D network, wherein the D2D network is distributed with a plurality of network nodes, and there are direct communication paths between the plurality of network nodes, and the network nodes include a plurality of terminal devices and a plurality of edge servers, such as Figure 4 As shown, the device includes:

[0126] An acquisition unit 401 is configured to acquire the inference model submitted by the terminal device in the D2D network and computing resource information for each layer in the inference model; wherein the computing resource information includes at least the number of sub-inference tasks that require parallel computing at each layer in the inference model, the transmission data size of each sub-inference task, and the number of floating-point operations required for each sub-inference task; the inference model adopts a multi-branch model structure and includes multiple sub-inference tasks;

[0127] The acquiring unit 401 is further configured to acquire computing power information of each of the network nodes in the D2D network;

[0128] Processing unit 402 is used to output a target collaborative scheduling strategy using a reinforcement learning model based on the computing resource information of each layer in the inference model and the computing power information of each network node, so that each network node under the D2D network can collaboratively calculate the sub-inference tasks of each layer according to the target collaborative scheduling strategy.

[0129] In one embodiment, the processing unit 402 is further configured to determine the state space, action space, and action reward of the reinforcement learning model in combination with a Markov decision process;

[0130] For each layer in the inference model, the state space includes computing power information and topological structure of each network node, actions to be executed determined by the previous layer, and computing resource information of the current layer;

[0131] The action space includes the action selection state of each of the network nodes;

[0132] The action reward includes the negative of the inference latency of the current layer;

[0133] The apparatus further includes: a determining unit; the determining unit is configured to determine, based on the state space, an action to be executed for each of the network nodes having the maximum state value;

[0134] The determining unit is further configured to use the to-be-executed action of each of the network nodes having the maximum state value as the collaborative scheduling strategy of the current layer;

[0135] The processing unit 402 is further configured to construct and output a target collaborative scheduling strategy based on the collaborative scheduling strategies of each layer.

[0136] In one embodiment, the state space also includes: network parameters, which include an adjacency matrix that characterizes the communication connection relationship between network nodes at each layer under the D2D network; the adjacency matrix is ​​used to input into a convolutional network model to learn and output the network topology characteristics under the current D2D network.

[0137] In one embodiment, the determining unit is further configured to determine an action to be performed by each of the network nodes based on the state space;

[0138] The processing unit 402 is further configured to update actor network parameters in the actor network based on a first update expression;

[0139] The processing unit 402 is further configured to update the commentator network parameters in the commentator network based on a second update expression;

[0140] The determining unit is further configured to determine, based on the updated actor network parameters and commentator network parameters, the to-be-executed actions of each of the network nodes having the maximum state value.

[0141] In one embodiment, the processing unit 402 is further configured to use a proximal policy optimization method to update the actor network parameters using a first update expression:

[0142] The first update expression is:

[0143]

[0144] in, Indicates the The generalized advantage estimate of the step, is the discount factor, Generalized advantage estimate The smoothing parameter, represents the time step offset, For the moment Time difference;

[0145] The processing unit 402 is further configured to update the commentator network parameters by minimizing the time difference based on the second update expression using a stochastic gradient descent method;

[0146] The second update expression is:

[0147]

[0148] in, Indicates the The time difference of the step, For the time step Action rewards obtained; is the discount factor used to calculate the current value of future rewards; In state The state value of Status The status value.

[0149] In one embodiment, the processing unit 402 is further configured to limit the ratio of the policy distribution between the new policy model and the old policy model by updating the loss function when updating the actor network parameters and the critic network parameters;

[0150] The updated loss function is:

[0151]

[0152] in, is the clipping loss function; Under the new strategy, Take action probability; For the old policy in the state Take action probability; is the advantage function estimate for the nth step, is the shear limit function; is a preset value used to control the step size of the strategy update. is the weight of the actor network.

[0153] In one embodiment, the actor network is configured to have multiple outputs, each output representing a collaborative scheduling strategy for the sub-reasoning task.

[0154] In actual application, the processing unit 402 can be implemented by a processor in an electronic device in combination with a communication interface, and the acquisition unit and the determination unit can be implemented by a communication interface in a learning-based multi-branch model collaborative computing device in a D2D network.

[0155] It should be noted that: the above-mentioned embodiment provides a multi-branch model collaborative computing device based on learning in a D2D network. When performing collaborative computing, it only uses the division of the above-mentioned program modules as an example. In actual applications, the above-mentioned processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the multi-branch model collaborative computing device based on learning in a D2D network provided by the above-mentioned embodiment and the multi-branch model collaborative computing method embodiment based on learning in a D2D network belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0156] Based on the hardware implementation of the above program modules, and in order to implement the multi-branch model collaborative computing method based on learning in the D2D network provided by the embodiment of the present application, the embodiment of the present application also provides an electronic device, such as Figure 5 As shown, the electronic device 500 includes:

[0157] CPU 501, memory 502 and input / output interface 503;

[0158] The memory 502 is a temporary storage memory or a permanent storage memory;

[0159] The central processing unit 501 is configured to communicate with the memory 502 and execute instructions in the memory 502 to perform any one of the above-mentioned learning-based multi-branch model collaborative computing methods in a D2D network.

[0160] Of course, in actual application, the various components in the electronic device 500 are coupled together through the bus system 504. It can be understood that the bus system 504 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 504 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 5 Various buses are labeled as bus system 504 .

[0161] The memory 502 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device 500. Examples of such data include: any computer program used to operate on the electronic device 500.

[0162] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is run on a computer, any one of the above-mentioned learning-based multi-branch model collaborative computing methods in a D2D network is executed.

[0163] An embodiment of the present application further provides a computer program product having a computer program / instruction stored thereon, which, when executed by a processor, is used to implement any of the above-mentioned learning-based multi-branch model collaborative computing methods in a D2D network.

[0164] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0165] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0166] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0167] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0168] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

Claims

1. A learning-based multi-branch model collaborative computing method in a D2D network, characterized by: Applied to a device-to-device (D2D) network, the D2D network has a plurality of network nodes distributed thereon, a direct communication path exists between the plurality of network nodes, the network nodes including a plurality of terminal devices and a plurality of edge servers, the method comprising: Obtaining the inference model submitted by the terminal device in the D2D network and computing resource information for each layer in the inference model; wherein the computing resource information includes at least the number of sub-inference tasks that need to be calculated in parallel at each layer in the inference model, the transmission data size of each sub-inference task, and the number of floating-point operations required for each sub-inference task; the inference model adopts a multi-branch model structure and includes multiple sub-inference tasks; Obtaining computing power information of each of the network nodes in the D2D network; Outputting a target collaborative scheduling strategy using a reinforcement learning model based on the computing resource information of each layer in the inference model and the computing power information of each network node, so that each network node in the D2D network performs collaborative computing on the sub-inference tasks of each layer according to the target collaborative scheduling strategy; Outputting a target collaborative scheduling strategy using a reinforcement learning model based on the computing resource information of each layer in the inference model and the computing power information of each network node includes: In combination with the Markov decision process, the state space, action space, and action reward of the reinforcement learning model are determined; Wherein, for each layer in the inference model, the state space includes computing power information of each network node, actions to be executed determined by the previous layer, and computing resource information of the current layer; The action space includes the action selection state of each of the network nodes; The action reward includes the negative of the inference latency of the current layer; Determining, based on the state space, an action to be performed for each of the network nodes having a maximum state value; Taking the to-be-executed actions of each of the network nodes having the maximum state value as the collaborative scheduling strategy of the current layer; The target collaborative scheduling strategy is constructed and output according to the collaborative scheduling strategies of each layer.

2. The learning-based multi-branch model collaborative computing method in a D2D network according to claim 1, characterized in that: The state space also includes: network parameters, which include an adjacency matrix that characterizes the communication connection relationship between network nodes in each layer of the D2D network; the adjacency matrix is ​​used to input into the convolutional network model to learn and output the network topology characteristics of the current D2D network.

3. The learning-based multi-branch model collaborative computing method in a D2D network according to claim 1, characterized in that: The reinforcement learning model further includes a strategy model, which includes an actor network and a critic network. The actor network is used to output the pending actions of each of the network nodes. The critic network is used to output a state value based on the pending actions output by the actor network and the current state space. Determining the pending actions of each of the network nodes with the maximum state value based on the state space includes: Determining an action to be performed by each of the network nodes based on the state space; In the actor network, updating actor network parameters based on a first update expression; In the critic network, updating critic network parameters based on a second update expression; Based on the updated actor network parameters and commentator network parameters, the to-be-executed actions of each of the network nodes having the maximum state value are determined.

4. The learning-based multi-branch model collaborative computing method in a D2D network according to claim 3, characterized in that: In the actor network, updating the actor network parameters based on the first update expression includes: Using the proximal policy optimization method, the actor network parameters are updated by the first update expression: The first update expression is: in, Indicates the The generalized advantage estimate of the step, is the discount factor, Generalized advantage estimate The smoothing parameter, represents the time step offset, For the moment Time difference; In the critic network, updating the critic network parameters based on the second update expression includes: Based on the second update expression, the stochastic gradient descent method is used to calculate the minimum time difference to update the critic network parameters; The second update expression is: in, Indicates the The time difference of the step, For the time step Action rewards obtained; is the discount factor used to calculate the current value of future rewards; In state The state value of Status The status value.

5. The learning-based multi-branch model collaborative computing method in a D2D network according to claim 4, characterized in that: The method further comprises: When updating the actor network parameters and the critic network parameters, limiting the ratio of the policy distribution between the new policy model and the old policy model by updating the loss function; The updated loss function is: in, is the clipping loss function; Under the new strategy, Take action probability; For the old policy in the state Take action probability; For the The advantage function estimate of the step, is the shear limit function; is a preset value used to control the step size of the strategy update. is the weight of the actor network.

6. The learning-based multi-branch model collaborative computing method in a D2D network according to claim 3, characterized in that: The actor network is configured with multiple outputs, each output representing a collaborative scheduling strategy for the sub-reasoning task.

7. A learning-based multi-branch model collaborative computing device in a D2D network, characterized in that: Applied to a device-to-device (D2D) network, the D2D network has a plurality of network nodes distributed thereon, a direct communication path exists between the plurality of network nodes, the network nodes include a plurality of terminal devices and a plurality of edge servers, and the apparatus includes: An acquisition unit is configured to acquire the inference model submitted by the terminal device in the D2D network and computing resource information of each layer in the inference model; wherein the computing resource information includes at least the number of sub-inference tasks that need to be calculated in parallel at each layer in the inference model, the transmission data size of each sub-inference task, and the number of floating-point operations required for each sub-inference task; the inference model adopts a multi-branch model structure and includes multiple sub-inference tasks; The acquiring unit is further configured to acquire computing power information of each of the network nodes in the D2D network; a processing unit, configured to output a target collaborative scheduling strategy using a reinforcement learning model based on the computing resource information of each layer in the inference model and the computing power information of each network node, so that each network node in the D2D network performs collaborative computing on the sub-inference tasks of each layer according to the target collaborative scheduling strategy; The processing unit is specifically configured to: In combination with the Markov decision process, the state space, action space, and action reward of the reinforcement learning model are determined; Wherein, for each layer in the inference model, the state space includes computing power information of each network node, actions to be executed determined by the previous layer, and computing resource information of the current layer; The action space includes the action selection state of each of the network nodes; The action reward includes the negative of the inference latency of the current layer; Determining, based on the state space, an action to be performed for each of the network nodes having a maximum state value; Taking the to-be-executed actions of each of the network nodes having the maximum state value as the collaborative scheduling strategy of the current layer; The target collaborative scheduling strategy is constructed and output according to the collaborative scheduling strategies of each layer.

8. An electronic device, characterized in that: include: CPU, memory and input / output interfaces; The memory is a transient storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instruction operations in the memory to perform the learning-based multi-branch model collaborative computing method in a D2D network according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is run on a computer, the learning-based multi-branch model collaborative computing method in a D2D network according to any one of claims 1 to 6 is executed.

10. A computer program product having a computer program / instructions stored thereon, characterized in that: When the computer program / instruction is executed by a processor, it is used to implement the learning-based multi-branch model collaborative computing method in a D2D network as described in any one of claims 1 to 6.

Images