Learning-based multi-branch model cooperative calculation method under D2D network

Abstract

The embodiment of the invention discloses a learning-based multi-branch model cooperative calculation method under a D2D network. The method comprises the following steps: acquiring an inference model submitted by terminal equipment under the D2D network and calculation resource quantity information of each layer in the inference model; the reasoning model is of a multi-branch structure, that is, each layer comprises a plurality of sub reasoning tasks; the calculation resource quantity information at least comprises the number of sub-reasoning tasks needing parallel calculation in each layer, the transmission data size of each sub-reasoning task and the number of required floating point calculation times; obtaining computing power information of each network node in the D2D network; and outputting a target cooperative scheduling strategy by using a reinforcement learning model according to the calculation resource quantity information of each layer and the calculation power information of each network node, so that each network node in the D2D network performs cooperative calculation on the sub-reasoning task of each layer according to the target cooperative scheduling strategy. According to the embodiment of the invention, the reasoning problem of a complex multi-branch model can be solved, the reasoning time delay is minimized, and the resource utilization rate is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of edge computing, and in particular to a learning-based multi-branch model collaborative computing method in a D2D network. Background Art

[0002] In recent years, the number of mobile devices and connected devices has exploded, bringing more opportunities for edge computing. By combining mobile edge computing with deep learning, powerful computing power and low-latency processing are provided at the edge of the network, greatly promoting the development of real-time applications.

[0003] When processing some simple deep model inferences, the model split point is usually selected according to the system environment and model structure, and the model is divided into two parts, one part is calculated locally on the mobile device, and the other part is offloaded to the edge server for calculation. Although this task collaboration method reduces the inference latency to a certain extent, most of them only consider deep models with simple linear structures. In these schemes, the inference calculations on the mobile device and the edge server are executed in serial order, that is, the output of the sub-model on the mobile device is the input of the edge sub-model.

[0004] However, the structures of the widely used deep models are becoming more and more complex, such as multi-branch structures that can be executed in parallel. For such complex structured models, relying solely on traditional mobile edge computing technology has certain limitations and high inference latency.

[0005] In addition, device-to-device (D2D) communication technology is a communication technology that allows devices to communicate directly with each other without going through base stations or repeaters. This technology has significant advantages over other technologies in terms of improving spectrum efficiency, reducing latency and power consumption. With the development of the Internet of Things, mobile Internet and 5G networks, D2D communication technology has been widely studied and applied. People use D2D technology to offload computing tasks to adjacent mobile devices and provide real-time computing services for computing-intensive applications. However, existing D2D technology still has certain limitations when dealing with complex multi-branch models, and it lacks efficient collaborative computing strategies. Summary of the invention

[0006] Based on the above problems, an embodiment of the present application provides a learning-based multi-branch model collaborative computing method in a D2D network, with the aim of reducing inference latency and improving resource utilization.

[0007] In a first aspect, an embodiment of the present application provides a learning-based multi-branch model collaborative computing method in a D2D network, which is applied to a device-to-device D2D network, wherein a plurality of network nodes are distributed on the D2D network, and a direct communication path exists between the plurality of network nodes, wherein the network nodes include a plurality of terminal devices and a plurality of edge servers, and the method includes:

[0008] Obtaining the inference model submitted by the terminal device under the D2D network and the computing resource information of each layer in the inference model; wherein the computing resource information at least includes the number of sub-inference tasks that need to be calculated in parallel at each layer in the inference model, the transmission data size of each sub-inference task and the number of floating-point operations required for each sub-inference task; the inference model adopts a multi-branch model structure and includes multiple sub-inference tasks;

[0009] Obtaining computing power information of each of the network nodes in the D2D network;

[0010] According to the computing resource information of each layer in the reasoning model and the computing power information of each network node, the reinforcement learning model is used to output the target collaborative scheduling strategy so that each network node under the D2D network can perform collaborative calculations on the sub-reasoning tasks of each layer according to the target collaborative scheduling strategy.

[0011] In one embodiment, the outputting of the target collaborative scheduling strategy using a reinforcement learning model according to the computing resource amount information of each layer in the reasoning model and the computing power information of each of the network nodes includes:

[0012] In combination with the Markov decision process, the state space, action space, and action reward of the reinforcement learning model are determined;

[0013] Wherein, for each layer in the reasoning model, the state space includes computing power information of each of the network nodes, actions to be executed determined by the previous layer, and computing resource information of the current layer;

[0014] The action space includes the action selection state of each of the network nodes;

[0015] The action reward includes the negative of the inference latency of the current layer;

[0016] Based on the state space, determining an action to be performed for each of the network nodes having a maximum state value;

[0017] Taking the to-be-executed actions of each of the network nodes having the maximum state value as the collaborative scheduling strategy of the current layer;

[0018] The target collaborative scheduling strategy is constructed and output according to the collaborative scheduling strategies of each layer.

[0019] In one embodiment, the state space also includes: network parameters, the network parameters include an adjacency matrix that characterizes the communication connection relationship between network nodes at each layer under the D2D network; the adjacency matrix is ​​used to input into a convolutional network model to learn and output network topology features under the current D2D network.

[0020] In one embodiment, the reinforcement learning model further includes a strategy model, the strategy model includes an actor network and a critic network, the actor network is used to output the to-be-executed actions of each of the network nodes, the critic network is used to output the state value according to the to-be-executed actions output by the actor network and the current state space, and the determining of the to-be-executed actions of each of the network nodes with the maximum state value based on the state space includes:

[0021] Based on the state space, determining actions to be performed by each of the network nodes;

[0022] In the actor network, updating actor network parameters based on a first update expression;

[0023] In the critic network, updating critic network parameters based on a second update expression;

[0024] Based on the updated actor network parameters and commentator network parameters, the actions to be executed of each of the network nodes having the maximum state value are determined.

[0025] In one embodiment, in the actor network, updating the actor network parameters based on the first update expression includes:

[0026] Using the proximal policy optimization method, the actor network parameters are updated by the first update expression:

[0027] The first update expression is:

[0028]

[0029] in, represents the generalized advantage estimate at the nth step, γ is the discount factor, λ is the smoothing parameter of the generalized advantage estimate GAE, l represents the time step offset, δ n+l is the time difference at time n+1;

[0030] In the critic network, updating the critic network parameters based on the second update expression includes:

[0031] Based on the second update expression, the critic network parameters are updated using the stochastic gradient descent method to minimize the time difference;

[0032] The second update expression is:

[0033] δ n =r n +γV θ (s n+1 )-V θ (s r )

[0034] Among them, δ n represents the time difference of step n, r n is the action reward obtained at time step n; γ is the discount factor used to calculate the current value of future rewards; For state s n+1 The state value of For state s n The status value.

[0035] In one embodiment, the method further comprises:

[0036] When updating the actor network parameters and updating the critic network parameters, limiting the ratio of the policy distribution between the new policy model and the old policy model by updating the loss function;

[0037] The updated loss function is:

[0038]

[0039] Among them, L CLIP (θ) is the shear loss function; π θ (a n |s n ) is the state s under the new strategy n Take action a n The probability of Under the new strategy, in state s n Take action a n The probability of is the advantage function estimate of the nth step, clip() is the clipping limit function; τ is the preset value used to control the step size of the policy update, and θ is the weight of the actor network.

[0040] In one embodiment, the actor network is configured to have multiple outputs, each output representing a collaborative scheduling strategy for the sub-reasoning task.

[0041] In a second aspect, the embodiment of the present application further provides a multi-branch model collaborative computing device based on learning in a D2D network, which is applied to a device-to-device D2D network, wherein a plurality of network nodes are distributed on the D2D network, and a direct communication path exists between the plurality of network nodes, wherein the network nodes include a plurality of terminal devices and a plurality of edge servers, and the device includes:

[0042] An acquisition unit is used to acquire the inference model submitted by the terminal device under the D2D network and the computing resource information of each layer in the inference model; wherein the computing resource information at least includes the number of sub-inference tasks that need to be calculated in parallel at each layer in the inference model, the transmission data size of each sub-inference task and the number of floating-point operations required for each sub-inference task; the inference model adopts a multi-branch model structure and includes multiple sub-inference tasks;

[0043] The acquisition unit is further used to acquire the computing power information of each of the network nodes in the D2D network;

[0044] A processing unit is used to output a target collaborative scheduling strategy using a reinforcement learning model based on the computing resource information of each layer in the reasoning model and the computing power information of each network node, so that each network node under the D2D network can collaboratively calculate the sub-reasoning tasks of each layer according to the target collaborative scheduling strategy.

[0045] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0046] CPU, memory, input and output interfaces;

[0047] The memory is a short-term storage memory or a persistent storage memory;

[0048] The central processing unit is configured to communicate with the memory and execute instruction operations in the memory to perform any one of the above-mentioned learning-based multi-branch model collaborative computing methods in a D2D network.

[0049] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is run on a computer, the learning-based multi-branch model collaborative computing method in a D2D network is executed as described in any one of the above.

[0050] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages: by obtaining the inference model submitted by the terminal device under the D2D network and the computing resource information of each layer thereof; obtaining the computing power information of each network node under the D2D network; according to the computing resource information of each layer in the inference model and the computing power information of the network node, the reinforcement learning model is used to output the target collaborative scheduling strategy, so that each network node under the D2D network can perform collaborative calculations on the sub-inference tasks of each layer according to the target collaborative scheduling strategy. By considering the computing power information of each network node and the computing resource information of each layer in the inference model, the reinforcement learning model is used to make decisions to maximize the state value, optimize the allocation of network node resources, and enable multiple devices to execute the sub-inference tasks in the inference model, i.e., the multi-branch model, in parallel, reducing the total time required to complete the model inference, and improving resource utilization and system efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0052] Figure 1 A flowchart of a multi-branch model collaborative computing method based on learning in a D2D network provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of a reinforcement learning actor network architecture provided in an embodiment of the present application;

[0054] Figure 3 A flowchart of a multi-branch model collaborative computing method based on learning in a D2D network provided in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of the structure of a multi-branch model collaborative computing device based on learning in a D2D network provided in an embodiment of the present application;

[0056] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] With the continuous development of artificial intelligence technology, applications based on deep learning are emerging in an endless stream. Although these applications have brought great convenience to people, they have also put forward higher requirements on the performance of computing devices. In particular, some applications that are sensitive to latency urgently need computing devices to have faster deep learning model reasoning capabilities. However, the computing resources of mobile devices are relatively limited at present. Due to the power limitations of the processor, they can only run some lightweight simple deep models. Although these lightweight models can reduce the burden on mobile devices, their reasoning accuracy is low, the application scenarios they support are limited, and the service quality is relatively poor.

[0059] Take ChatGPT, a popular generative language model, for example. Because it requires huge computing power, current mobile devices cannot run such complex models. Although deploying deep learning applications on cloud servers can solve the problem of limited computing resources, cloud servers are usually far away from mobile devices, which will lead to large transmission delays and cannot meet the needs of delay-sensitive applications.

[0060] To solve these problems, Mobile Edge Computing (MEC) technology has emerged. This technology reduces the latency of model inference by offloading some computing tasks to edge servers close to mobile devices to meet the latency requirements of applications. When processing simple deep model inference tasks, the appropriate model cutting point can be selected according to the system environment and model structure to divide the model into two parts: one part is calculated locally on the mobile device, and the other part is offloaded to the edge server for processing. However, although this collaborative computing method of cutting the inference model into two parts can reduce the latency of inference to a certain extent, most of them only consider deep models with simple linear structures, such as AlexNet and VGGNet. In these schemes, the inference calculations on the mobile device and the edge server are executed in serial order, that is, the output of the sub-model on the mobile device is the input of the edge sub-model. However, the widely used deep model structures are becoming more and more complex, especially in the fields of computer vision, natural language processing, and multi-task learning, where there are multi-branch model structures. For example, the Inception structure of GoogLeNet is a multi-branch model that can be executed in parallel. A multi-branch model refers to a neural network structure that contains multiple parallel branches, each of which can independently perform different computing tasks or process different information flows. Unlike traditional linear structure models (such as fully connected networks or convolutional neural networks where each layer is connected layer by layer), the design of multi-branch models allows multiple operations to be performed simultaneously through different paths in the network, thereby improving the network's expressiveness and computational efficiency, especially when processing complex data, and being able to capture different features more flexibly. However, for such complex structure models, relying solely on traditional mobile edge computing technology has certain limitations, and its inference latency is also high.

[0061] In addition, with the development of the Internet of Things, mobile Internet and 5G networks, D2D communication technology has been widely studied and applied. D2D communication technology refers to direct communication technology between two terminal devices within an allowable close range. In a D2D network, each terminal device can communicate with each other directly without going through intermediate facilities such as base stations, thereby improving spectrum utilization, reducing base station pressure, reducing end-to-end transmission delay, and improving system network performance. The application of D2D technology can offload computing tasks to adjacent terminal devices and provide real-time computing services for computing-intensive applications. However, existing D2D technology and mobile edge computing still have limitations in processing complex multi-branch models, and they lack efficient collaborative computing strategies. As deep learning models become more and more complex, traditional collaborative computing technologies can no longer meet the current complex computing needs, and a new method is urgently needed to complete the reasoning calculation of multi-branch models.

[0062] Based on this, in each embodiment of the present application, by obtaining the computing power information of each network node and the computing resource information of the reasoning model, a reinforcement learning model is used to make decisions to maximize the state value, optimize the allocation of network node resources, and enable sub-reasoning tasks to be executed in parallel among multiple network nodes, thereby reducing the total time required to complete task reasoning and improving resource utilization.

[0063] The various embodiments of the present application are described in further detail below in conjunction with the accompanying drawings.

[0064] The embodiment of the present application provides a multi-branch model collaborative computing method based on learning in a D2D network, which is applied to a device-to-device D2D network. A plurality of network nodes are distributed on the D2D network, and a direct communication path exists between the plurality of network nodes. The network nodes include a plurality of terminal devices and a plurality of edge servers, such as Figure 1 As shown, the method includes steps S101-S103.

[0065] S101: Obtaining the inference model submitted by the terminal device in the D2D network and the computing resource amount information of each layer in the inference model.

[0066] S102: Obtain computing power information of each network node in the D2D network.

[0067] In an embodiment of the present application, a D2D network is a network composed of multiple network nodes, which include several terminal devices (such as smart phones, sensors, industrial controllers, etc.) at the device layer and several edge servers at the network layer. These network nodes are located at the edge of the network, that is, close to the data source, for processing data and performing computing tasks. In a D2D network, one of the multiple terminal devices will publish an inference model that needs to be calculated, and pass the computing resource information of the inference model to the edge network node.

[0068] An inference model can be understood as the process of inferring input data using a trained model. The model processes the input data and generates output results based on its internal parameters and structure. Inference models are widely used in image classification, object detection, speech recognition, natural language processing and other fields. The core goal is to predict and analyze input data based on the knowledge learned by the model during training. More complex inference models usually contain millions to billions of parameters. For example, an inference model can adopt the structure of a multi-branch model to achieve efficient processing through a multi-branch structure that can be executed in parallel. There are multiple layers of computing structure in this multi-branch model, and each layer also contains sub-inference tasks that need to be processed in parallel, so that multiple branches can process data at the same time, and finally summarize the processing results of each branch to generate the final output.

[0069] Therefore, in the embodiment of the present application, the computing resource information of each layer includes at least the number of sub-reasoning tasks that need to be calculated in parallel at each layer in the reasoning model, the transmission data size of each sub-reasoning task, and the number of floating-point operations required for each sub-reasoning task. The purpose of obtaining this computing resource information is to better target complex deep learning models, especially the disassembly and reasonable allocation of multi-branch models, to ensure that each sub-reasoning task can be executed efficiently and quickly in the D2D network, which helps to optimize task scheduling, reduce latency, and improve the response speed and performance of the entire collaborative computing system.

[0070] Computing power information refers to the ability of network nodes to complete computing tasks, including but not limited to: the number of floating point operations per second (FLOPS) of each network node, performance indicators of the central processing unit (CPU) or graphics processing unit (GPU) (such as the number of cores, clock frequency, etc.), network bandwidth (the speed at which network nodes upload and download data), energy consumption, the number of basic computing units (such as media access control layer (MAC) computing units, computing cores), utilization, received signal strength indication (RSSI), etc.

[0071] Obtaining computing power information can help the reinforcement learning model determine the ability of each network node to perform computing tasks, so that the edge server can perform task allocation and resource scheduling according to the computing power of each network node, ensuring that the sub-inference task can be executed on the most suitable network node and balancing the load of the network node. It is understandable that traditional mobile edge computing usually relies on edge servers as the main computing nodes, and the embodiment of the present application introduces a D2D network, so that each terminal device can directly collaborate to perform reasoning calculations. Furthermore, it can also combine the computing power information of each terminal device and edge server to dynamically adapt to changes in the network environment and task requirements to adjust the task allocation strategy. No matter how the sub-inference tasks are allocated, they ultimately achieve the same optimization goal, thereby adapting to changes in the network scale. This can not only reduce the dependence on centralized edge servers, but also flexibly utilize the computing resources of other terminal devices or edge servers when some network nodes are heavily loaded or have limited resources, thereby improving the robustness and flexibility of the system.

[0072] S103: According to the computing resource information of each layer in the reasoning model and the computing power information of each network node, the reinforcement learning model is used to output the target collaborative scheduling strategy, so that each network node in the D2D network can perform collaborative calculations on the sub-reasoning tasks of each layer according to the target collaborative scheduling strategy.

[0073] In reinforcement learning, the target collaborative scheduling strategy can be selected by selecting the action that brings the maximum expected reward, and the maximum expected reward can be evaluated by the state value. The state value refers to the cumulative reward expected from taking a specific action in a given state, and the maximum state value can be used to guide the selection of actions that can bring the maximum expected reward. The target collaborative scheduling strategy is used to guide how to assign sub-reasoning tasks to different network nodes in the network so that they can collaborate to complete the task.

[0074] The embodiment of the present application obtains the computing resource information of the reasoning model submitted by the terminal device under the D2D network, and obtains the computing power information of each network node under the D2D network, and then uses the reinforcement learning model to output a target collaborative scheduling strategy based on the computing resource information of the sub-reasoning task and the computing power information of the network node, so that each network node under the D2D network can perform collaborative calculations on the sub-reasoning task according to the collaborative scheduling strategy. By utilizing multiple computing devices in the D2D network, and considering the computing power information of each network node and the computing resource information of the sub-reasoning task, the reinforcement learning model is used to output the target collaborative scheduling strategy, which can optimize the allocation of network node resources, so that the sub-reasoning tasks to be processed at each layer in the multi-branch model can be calculated and processed by multiple devices under the D2D network at the same time. Multiple devices execute sub-reasoning tasks in parallel, reducing the total time required to complete the deep learning model reasoning, and improving resource utilization and system efficiency. Furthermore, by solving the reasoning problem of complex multi-branch models, the scope of use of mobile terminal devices in practical applications is expanded, and more high-precision, high-computing artificial intelligence (AI, Artificial Intelligence) applications can be supported.

[0075] In one embodiment, according to the computing resource information of each layer in the inference model and the computing power information of each network node, a reinforcement learning model is used to output a target collaborative scheduling strategy, including:

[0076] Combined with the Markov decision process, the state space, action space, and action reward of the reinforcement learning model are determined;

[0077] Among them, for each layer in the reasoning model, the state space includes the computing power information of each network node, the actions to be executed determined by the previous layer and the computing resource information of the current layer; the action space includes the action selection state of each network node; the action reward includes the negative of the reasoning delay of the current layer; based on the state space, the actions to be executed of each network node with the maximum state value are determined; the actions to be executed of each network node with the maximum state value are used as the collaborative scheduling strategy of the current layer; according to the collaborative scheduling strategies of each layer, the target collaborative scheduling strategy is constructed and output.

[0078] In an embodiment of the present application, in order to solve the limitations of traditional mobile edge computing technology and the high inference latency, a collaborative scheduling strategy for the output of the reinforcement learning model can be guided by minimizing the inference latency.

[0079] The embodiment of the present application specifies the state space, action space and action reward. The action of each layer, i.e., the strategy of collaborative computing, is generated by the strategy model according to the current state. The state space can be composed of the computing power information of each network node, the task parameters (i.e., the computing resource information of the current layer of the reasoning task) and the collaborative scheduling strategy of the previous layer (i.e., the action to be executed determined by the previous layer).

[0080] The task parameters include: the number of sub-inference tasks that need to be calculated in parallel at each layer in the inference model, the transmission data size of each sub-inference task, and the number of floating-point operations required for each sub-inference task.

[0081] The action space is the network node to be unloaded for each sub-reasoning task, that is, the action selection state of each network node, which is a multi-dimensional action. It should be understood that the multi-dimensional action represents that the agent needs to consider multiple dimensions or aspects when making decisions, such as task allocation, CPU, memory, storage, channel status, and other resource allocations.

[0082] Action reward: Since the embodiment of this application aims to minimize the inference delay, the action reward is specified as the negative of the current layer inference delay: -Φ n In this way, the learning direction of the strategy model is to maximize the reward when outputting the collaborative scheduling strategy for each layer, minimize the reasoning delay of each layer as much as possible, and make the strategy model learn and train towards the goal set by the present invention.

[0083] In the above embodiments, it is mentioned that the present invention takes minimizing the inference delay as the optimization goal, so the optimization problem required by the embodiments of the present application is:

[0084]

[0085] Among them, Φ n A is the inference delay of the embodiment of the present application; nis the cooperative scheduling strategy of the nth layer; a n,1,m represents the mth collaborative computing device of the lth sub-reasoning task at the nth layer; It is the latency of collaborative computing of the lth sub-inference task of the nth layer on the mth device.

[0086] Since each task can be computed by only one device, for each task l, the number of a n,1,m The sum must equal 1:

[0087]

[0088] If a n,l,m If the value of is 1, it means that the lth task of the nth layer is assigned to the mth device for calculation.

[0089] If a n,l,m If the value of is 0, it means that the lth task of the nth layer is not assigned to the mth device for calculation.

[0090] Through the above method, it can be ensured that the policy model can quickly adapt to the dynamic changes of the D2D network, while minimizing the reasoning delay and improving the efficiency of task collaborative reasoning.

[0091] In one embodiment, the state space also includes: network parameters, which represent the adjacency matrix of the communication connection relationship between network nodes at each layer in the D2D network; the adjacency matrix is ​​used to input into the convolutional network model to learn and output the network topology characteristics of the current D2D network.

[0092] Assuming there are n network nodes, the adjacency matrix can be understood as an n×n matrix, where each element A[i][j] represents the connection relationship between the ith network node and the jth network node. If the ith network node has a communication connection with the jth network node, the matrix value is 1; if there is no connection, it is 0.

[0093] The adjacency matrix can provide complete information about the D2D network structure, including the connection relationship between nodes and the feature information of nodes. In a D2D network, the network topology changes dynamically. Therefore, the embodiment of the present application proposes an improved state extraction model, which is designed in the actor network of the reinforcement learning model, such as Figure 2As shown, in the state extraction model, the convolutional network and the fully connected network are combined. The convolutional network can be used to extract network topology information for the adjacency matrix (such as the adjacency matrix of D2D), and the fully connected network can be used to process other task parameters. It should be understood that since the convolutional network is good at processing information with spatial structure, such as data with spatial structure such as images or graph structures, the convolutional network can be used to extract the relationship between nodes and the characteristics of the network structure. The task parameters processed by the fully connected network can specifically be the traffic load, delay, computing resource information, computing power information, etc. in the edge computing scenario. Unlike the adjacency matrix, these task data do not have a spatial structure. Therefore, the fully connected network is more suitable for the processing of such unstructured data. In this way, the combination of the convolutional network and the fully connected network can make full use of the characteristics of the data structure of different data to extract network topology features and improve the accuracy and efficiency of state extraction.

[0094] Furthermore, the embodiments of the present application may also consider replacing the convolutional network with a graph neural network (GNN) when processing the adjacency matrix of D2D to more effectively capture the network topology and node features. GNN updates the feature representation of nodes by aggregating the information of neighboring nodes, so that it can learn the position and role of network nodes in the D2D network, as well as the topology of the entire network. Through GNN, the network topology in these adjacency matrices can be effectively extracted and then used for various downstream tasks, such as node classification, graph classification, etc.

[0095] In addition, in addition to using fully connected networks, other deep learning methods, such as self-attention mechanisms, can also be used to extract task parameter information to improve the accuracy of state representation.

[0096] In one embodiment, the reinforcement learning model also includes a strategy model, which includes an actor network and a critic network. The actor network is used to output the pending actions of each network node, and the critic network is used to output the state value according to the pending actions output by the actor network and the current state space. Based on the state space, the pending actions of each network node with the maximum state value are determined, including: based on the state space, determining the pending actions of each network node; in the actor network, updating the actor network parameters based on the first update expression; in the critic network, updating the critic network parameters based on the second update expression; based on the updated actor network parameters and critic network parameters, determining the pending actions of each network node with the maximum state value.

[0097] The reinforcement learning model also includes a policy model, which consists of an actor network and a critic network. The actor network predicts the action that the agent should take based on the current state of the environment. The actor network outputs the probability distribution of the action, that is, the execution probability of each possible action in the current state. The structure of the critic network is the same as that of the actor network. The critic network is used to evaluate the value of the current state, and thus outputs the state value, that is, the expected return of the agent in the current state.

[0098] In practical applications, the actor network updates its parameters through the first update expression, aiming to gradually optimize the network's ability to predict each action, and its update direction is guided by the feedback from the critic network. The critic network updates its own parameters based on the second update expression, and its output is the state value in the current state. By evaluating the action performance of the actor network, the critic network provides feedback on the value of each action in each state, providing guidance for the update direction of the actor network.

[0099] After updating the parameters of the actor and critic networks, the policy model calculates and evaluates the state value of each network node to select the best action that brings the maximum reward. Therefore, the feedback from the critic network is crucial to the optimization of the actor network, ensuring that the actor network can be updated and improved in the direction that can achieve the maximum reward.

[0100] In one embodiment, in an actor network, updating an actor network parameter based on a first update expression includes: using a proximal policy optimization method to update the actor network parameter through the first update expression; the first update expression is:

[0101]

[0102] in, represents the generalized advantage estimate at the nth step, γ is the discount factor, λ is the smoothing parameter of the generalized advantage estimate GAE, l represents the time step offset, δ n+l is the time difference at time n+1.

[0103] The first update expression calculates the generalized advantage estimate GAE, which is used to measure the advantage of the action and update the parameters of the actor network. By minimizing the difference between this advantage estimate and the actual reward, the output cooperative scheduling policy can be adjusted to select better actions.

[0104] In the critic network, based on the second update expression, the critic network parameters are updated, including:

[0105] Based on the second update expression, the critic network parameters are updated using the stochastic gradient descent method to minimize the time difference;

[0106] The second update expression is:

[0107]

[0108] Among them, δ n represents the time difference of step n, r n is the action reward obtained at time step n; γ is the discount factor used to calculate the current value of future rewards; For state s n+1 The state value of For state s n The status value.

[0109] The second update expression is used to evaluate the error between the critic network's value prediction and the actual reward by calculating the temporal difference. The critic network updates its parameters through the TD error in order to learn a more accurate value function. If the TD error is positive, it means that the actual action reward is better than expected, and the critic network will increase the state value valuation of the current state; if the TD error is negative, it means that the actual action reward is worse than expected, and the critic network will reduce the state value valuation of the current state.

[0110] Assume that an agent is in state s n An action is chosen and r is obtained n = 10 reward, and then transferred to the new state s n+1 If the critic network predicts n The long-term value is And in s n+1 The prediction is And the discount factor γ = 0.99, then the time difference δ n Calculated as: n =10+0.99×30-20=10+29.7-20=19.7.

[0111] In one embodiment, the method further comprises: when updating the actor network parameters and updating the critic network parameters, limiting the ratio of the policy distribution between the new policy model and the old policy model by updating the loss function;

[0112] Update the loss function to:

[0113]

[0114] Among them, L CLIP (θ) is the shear loss function; π θ (a n |s n ) is the state s under the new strategy n Take action an The probability of Under the new strategy, in state s n Take action a n The probability of is the advantage function estimate of the nth step, clip() is the clipping limit function; ∈ is the preset value used to control the step size of the policy update, and θ is the weight of the actor network.

[0115] The update loss function defines the loss function used to update the policy network parameters in Proximal Policy Optimization (PPO), with the goal of optimizing the policy while maintaining the stability of the policy update. For example, assume that the new and old policies are π θ and The probability of taking a certain action in a certain state is 0.6 and 0.4 respectively, and the advantage function estimates And ∈=0.1, then the strategy ratio is The limit range [1-0.1, 1+0.1] = [0.9, 1.1] [1-0.1, 1+0.1] = [0.9, 1.1] is exceeded, so the limited value 1.1 is used to calculate the loss function. This ensures that the policy update can improve performance while not deviating too far from the original policy, thus avoiding performance instability caused by large updates.

[0116] In one embodiment, the actor network is configured to output multiple heads, each output representing a collaborative scheduling strategy for a sub-reasoning task.

[0117] like Figure 2 As shown, considering the special structure of the state space and the multi-dimensional characteristics of the action space, the present invention customizes the network architecture for the actor network. At the input end, the state space is divided into two parts. Due to the two-dimensional structure of the connection matrix of the network nodes, the embodiment of the present application can extract the network topology features by using a convolutional network, and the remaining state parameters are extracted by a fully connected network. A multi-head output design is adopted at the output end to improve the PPO algorithm in the traditional deep reinforcement learning algorithm. The original PPO algorithm only sets one output head, that is, only one sub-reasoning task is input each time to obtain the result corresponding to the sub-reasoning task. In this embodiment, multiple output heads are added, which can simultaneously input the computing resource information of multiple sub-reasoning tasks in a layer and the computing power information of each network node, and then through one calculation, the collaborative scheduling strategy corresponding to each sub-reasoning task in the layer can be output. Finally, the collaborative scheduling strategy of the sub-reasoning tasks of each layer is combined to generate the final target collaborative scheduling strategy.

[0118] Through this improvement, the embodiment of the present application can more efficiently handle the problem of collaborative scheduling of multiple reasoning tasks, thereby improving network node resource utilization and decision-making efficiency.

[0119] The following will be Figure 1 and Figure 2 Based on the embodiment shown, the embodiment of the present application is further described in detail. Please refer to Figure 3 , Figure 3 A flowchart of a collaborative computing method is shown in FIG.

[0120] In In a D2D network with multiple devices, a device publishes the computing resource information of the model reasoning task to be calculated to the edge server. Layer, the number of tasks that need to be calculated in parallel for each layer n is For the nth layer and the lth task, two important parameters are defined: Calculate the size of data to be transmitted D n,l And the number of floating point operations required R n,l In addition, let a n,l,m Indicates the collaborative computing device for each task. If the value is a n,l,m = 1 means that the lth task of the nth layer is calculated by the mth device. n,l,m If the value of is 0, it means that the lth task of the nth layer is not assigned to the mth device for calculation.

[0121] For the tasks at the nth layer, minimizing the inference latency is used as the optimization goal. Characterizes the collaborative scheduling strategy of each layer. When the parameters of each layer are transmitted to the edge server, the edge server will give the collaborative scheduling strategy of the layer, i.e., A n After that, the same steps as above are performed on the next layer in the inference model until all layers obtain the corresponding collaborative scheduling strategies, and then the collaborative scheduling strategies of each layer are combined into the final target collaborative scheduling strategy, so that all network nodes in the network can follow the target collaborative scheduling strategy formulated by the edge server and collaboratively complete the inference model calculation.

[0122] In a feasible embodiment, in addition to minimizing the inference delay as the optimization goal, the inference delay and other performance indicators (such as energy consumption and bandwidth utilization) can also be used as joint optimization goals, and a multi-objective optimization method can be used to balance the relationship between different performance indicators. The core idea of ​​the multi-objective optimization method is to consider multiple performance indicators at the same time and find an optimal solution set rather than a single optimal solution. In practice, it is necessary to define an optimization model that can simultaneously reflect the inference delay, energy consumption and bandwidth utilization, and design a corresponding optimization algorithm to solve the model. In a feasible embodiment, a multi-objective optimization model can be established, with the inference delay, energy consumption and bandwidth utilization as optimization goals, and weights assigned to each goal. By adjusting the weights, a trade-off can be achieved between different goals. For example, if energy consumption is more important than inference delay, a higher weight can be assigned to energy consumption.

[0123] On the basis of the aforementioned embodiments, in addition to using the PPO algorithm, other reinforcement learning algorithms, such as Deep Deterministic Policy Gradient (DDPG) or Soft Actor Critic (SAC), can also be used to perform policy optimization to obtain the target collaborative scheduling strategy.

[0124] In order to implement the multi-branch model collaborative computing method based on learning in a D2D network of the embodiment of the present application, the embodiment of the present application also provides a multi-branch model collaborative computing device based on learning in a D2D network, which is applied to a device-to-device D2D network, wherein a plurality of network nodes are distributed on the D2D network, and a direct communication path exists between the plurality of network nodes, wherein the network nodes include a plurality of terminal devices and a plurality of edge servers, such as Figure 4 As shown, the device comprises:

[0125] The acquisition unit 401 is used to acquire the inference model submitted by the terminal device under the D2D network and the computing resource amount information of each layer in the inference model; wherein the computing resource amount information at least includes the number of sub-inference tasks that need to be calculated in parallel at each layer in the inference model, the transmission data size of each sub-inference task and the number of floating-point operations required for each sub-inference task; the inference model adopts a multi-branch model structure and includes multiple sub-inference tasks;

[0126] The acquisition unit 401 is further configured to acquire computing power information of each of the network nodes in the D2D network;

[0127] Processing unit 402 is used to output a target collaborative scheduling strategy using a reinforcement learning model based on the computing resource information of each layer in the reasoning model and the computing power information of each network node, so that each network node under the D2D network can collaboratively calculate the sub-inference tasks of each layer according to the target collaborative scheduling strategy.

[0128] In one embodiment, the processing unit 402 is further used to determine the state space, action space, and action reward of the reinforcement learning model in combination with the Markov decision process;

[0129] Wherein, for each layer in the reasoning model, the state space includes computing power information and topological structure of each of the network nodes, actions to be executed determined by the previous layer, and computing resource information of the current layer;

[0130] The action space includes the action selection state of each of the network nodes;

[0131] The action reward includes the negative of the inference latency of the current layer;

[0132] The device further includes: a determination unit; the determination unit is used to determine the to-be-executed action of each of the network nodes having the maximum state value based on the state space;

[0133] The determining unit is further configured to use the to-be-executed actions of each of the network nodes having the maximum state value as the collaborative scheduling strategy of the current layer;

[0134] The processing unit 402 is further configured to construct and output a target collaborative scheduling strategy according to the collaborative scheduling strategies of each layer.

[0135] In one embodiment, the state space also includes: network parameters, the network parameters include an adjacency matrix that characterizes the communication connection relationship between network nodes at each layer under the D2D network; the adjacency matrix is ​​used to input into a convolutional network model to learn and output network topology features under the current D2D network.

[0136] In one embodiment, the determining unit is further configured to determine an action to be performed by each of the network nodes based on the state space;

[0137] The processing unit 402 is further configured to update actor network parameters in the actor network based on a first update expression;

[0138] The processing unit 402 is further configured to update the commentator network parameters in the commentator network based on the second update expression;

[0139] The determining unit is further configured to determine the to-be-executed action of each of the network nodes having the maximum state value based on the updated actor network parameters and the commentator network parameters.

[0140] In one embodiment, the processing unit 402 is further configured to use a proximal policy optimization method to update the actor network parameters through a first update expression:

[0141] The first update expression is:

[0142]

[0143] in, represents the generalized advantage estimate at the nth step, γ is the discount factor, λ is the smoothing parameter of the generalized advantage estimate GAE, l represents the time step offset, δ n+l is the time difference at time n+1;

[0144] The processing unit 402 is further configured to update the critic network parameters by calculating the minimum time difference based on the second update expression using a stochastic gradient descent method;

[0145] The second update expression is:

[0146]

[0147] Among them, δ n represents the time difference of step n, r n is the action reward obtained at time step n; γ is the discount factor used to calculate the current value of future rewards; For state s n+1 The state value of For state s n The status value.

[0148] In one embodiment, the processing unit 402 is further configured to limit the ratio of the policy distribution between the new policy model and the old policy model by updating the loss function when updating the actor network parameters and updating the critic network parameters;

[0149] The updated loss function is:

[0150]

[0151] Among them, L CLIP (θ) is the shear loss function; π θ (a n |s n ) is the state s under the new strategy n Take action a n The probability of Under the new strategy, in state s n Take action a n The probability of is the advantage function estimate of the nth step, clip() is the clipping limit function; ∈ is the preset value used to control the step size of the policy update, and θ is the weight of the actor network.

[0152] In one embodiment, the actor network is configured to have multiple outputs, each output representing a collaborative scheduling strategy for the sub-reasoning task.

[0153] In actual application, the processing unit 402 can be implemented by a processor in an electronic device in combination with a communication interface, and the acquisition unit and the determination unit can be implemented by a communication interface in a learning-based multi-branch model collaborative computing device in a D2D network.

[0154] It should be noted that: the multi-branch model collaborative computing device based on learning in the D2D network provided in the above embodiment only uses the division of the above-mentioned program modules as an example when performing collaborative computing. In actual applications, the above-mentioned processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the multi-branch model collaborative computing device based on learning in the D2D network provided in the above embodiment and the multi-branch model collaborative computing method embodiment based on learning in the D2D network belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0155] Based on the hardware implementation of the above program modules, and in order to implement the multi-branch model collaborative computing method based on learning in a D2D network provided in the embodiment of the present application, the embodiment of the present application also provides an electronic device, such as Figure 5 As shown, the electronic device 500 includes:

[0156] CPU 501, memory 502 and input / output interface 503;

[0157] The memory 502 is a temporary storage memory or a permanent storage memory;

[0158] The central processor 501 is configured to communicate with the memory 502 and execute the instruction operation in the memory 502 to perform any one of the above-mentioned learning-based multi-branch model collaborative computing methods in the D2D network.

[0159] Of course, in actual application, the various components in the electronic device 500 are coupled together through the bus system 504. It can be understood that the bus system 504 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 504 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 5 Various buses are labeled as bus system 504 .

[0160] The memory 502 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device 500. Examples of such data include: any computer program used to operate on the electronic device 500.

[0161] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is run on a computer, any one of the above-mentioned learning-based multi-branch model collaborative computing methods in a D2D network is executed.

[0162] An embodiment of the present application also provides a computer program product having a computer program / instruction stored thereon, which, when executed by a processor, is used to implement any of the above-mentioned learning-based multi-branch model collaborative computing methods in a D2D network.

[0163] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0164] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0165] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0166] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0167] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A learning-based multi-branch model collaborative computing method in a D2D network, characterized in that: Applied to a device-to-device D2D network, the D2D network has a plurality of network nodes distributed thereon, a direct communication path exists between the plurality of network nodes, the network nodes include a plurality of terminal devices and a plurality of edge servers, the method includes: Obtaining the inference model submitted by the terminal device under the D2D network and the computing resource information of each layer in the inference model; wherein the computing resource information at least includes the number of sub-inference tasks that need to be calculated in parallel at each layer in the inference model, the transmission data size of each sub-inference task and the number of floating-point operations required for each sub-inference task; the inference model adopts a multi-branch model structure and includes multiple sub-inference tasks; Obtaining computing power information of each of the network nodes in the D2D network; According to the computing resource information of each layer in the reasoning model and the computing power information of each network node, the reinforcement learning model is used to output the target collaborative scheduling strategy so that each network node under the D2D network can perform collaborative calculations on the sub-reasoning tasks of each layer according to the target collaborative scheduling strategy.

2. The collaborative computing method of a multi-branch model based on learning in a D2D network according to claim 1, characterized in that: The method of outputting a target collaborative scheduling strategy using a reinforcement learning model according to the computing resource information of each layer in the inference model and the computing power information of each network node includes: In combination with the Markov decision process, the state space, action space, and action reward of the reinforcement learning model are determined; Wherein, for each layer in the reasoning model, the state space includes computing power information of each of the network nodes, actions to be executed determined by the previous layer, and computing resource information of the current layer; The action space includes the action selection state of each of the network nodes; The action reward includes the negative of the inference latency of the current layer; Based on the state space, determining an action to be performed for each of the network nodes having a maximum state value; Taking the to-be-executed actions of each of the network nodes having the maximum state value as the collaborative scheduling strategy of the current layer; The target collaborative scheduling strategy is constructed and output according to the collaborative scheduling strategies of each layer.

3. The learning-based multi-branch model collaborative computing method in a D2D network according to claim 2, characterized in that: The state space also includes: network parameters, which include an adjacency matrix that characterizes the communication connection relationship between network nodes at each layer under the D2D network; the adjacency matrix is ​​used to input into a convolutional network model to learn and output network topology features under the current D2D network.

4. The learning-based multi-branch model collaborative computing method in a D2D network according to claim 2, characterized in that: The reinforcement learning model also includes a strategy model, which includes an actor network and a critic network. The actor network is used to output the actions to be executed of each of the network nodes. The critic network is used to output the state value according to the actions to be executed output by the actor network and the current state space. The determining of the actions to be executed of each of the network nodes with the maximum state value based on the state space includes: Based on the state space, determining actions to be performed by each of the network nodes; In the actor network, updating actor network parameters based on a first update expression; In the critic network, updating critic network parameters based on a second update expression; Based on the updated actor network parameters and commentator network parameters, the actions to be executed of each of the network nodes having the maximum state value are determined.

5. The method for collaborative computing based on multi-branch model in D2D network according to claim 4, characterized in that: In the actor network, updating the actor network parameters based on the first update expression includes: Using the proximal policy optimization method, the actor network parameters are updated by the first update expression: The first update expression is: in, represents the generalized advantage estimate at the nth step, γ is the discount factor, λ is the smoothing parameter of the generalized advantage estimate GAE, l represents the time step offset, δ n+l is the time difference at time n+1; In the critic network, updating the critic network parameters based on the second update expression includes: Based on the second update expression, the critic network parameters are updated using the stochastic gradient descent method to minimize the time difference; The second update expression is: Among them, δ n represents the time difference of step n, r n is the action reward obtained at time step n; γ is the discount factor used to calculate the current value of future rewards; For state s n+1 The state value of For state s n The status value.

6. The learning-based multi-branch model collaborative computing method in a D2D network according to claim 5, characterized in that: The method further comprises: When updating the actor network parameters and updating the critic network parameters, limiting the ratio of the policy distribution between the new policy model and the old policy model by updating the loss function; The updated loss function is: Among them, L CLIP (θ) is the shear loss function; π θ (a n |s n ) is the state s under the new strategy n Take action a n probability; Under the new strategy, in state s n Take action a n probability; is the advantage function estimate of the nth step, clip() is the clipping limit function; ∈ is the preset value used to control the step size of the policy update, and θ is the weight of the actor network.

7. The method for collaborative computing based on multi-branch model in D2D network according to claim 4, characterized in that: The actor network is configured with multiple outputs, each output representing a collaborative scheduling strategy for the sub-reasoning task.

8. A multi-branch model collaborative computing device based on learning in a D2D network, characterized in that: Applied to a device-to-device D2D network, the D2D network is distributed with a plurality of network nodes, a direct communication path exists between the plurality of network nodes, the network nodes include a plurality of terminal devices and a plurality of edge servers, and the apparatus includes: An acquisition unit is used to acquire the inference model submitted by the terminal device under the D2D network and the computing resource information of each layer in the inference model; wherein the computing resource information at least includes the number of sub-inference tasks that need to be calculated in parallel at each layer in the inference model, the transmission data size of each sub-inference task and the number of floating-point operations required for each sub-inference task; the inference model adopts a multi-branch model structure and includes multiple sub-inference tasks; The acquisition unit is further used to acquire the computing power information of each of the network nodes in the D2D network; A processing unit is used to output a target collaborative scheduling strategy using a reinforcement learning model based on the computing resource information of each layer in the reasoning model and the computing power information of each network node, so that each network node under the D2D network can collaboratively calculate the sub-reasoning tasks of each layer according to the target collaborative scheduling strategy.

9. An electronic device, characterized in that: include: CPU, memory and input / output interface; The memory is a transient storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instruction operations in the memory to perform the learning-based multi-branch model collaborative computing method in a D2D network as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed on a computer, the learning-based multi-branch model collaborative computing method in a D2D network as claimed in any one of claims 1 to 7 is executed.

Images