A service network scheduling method based on deep reinforcement learning

Abstract

The present application provides a service network scheduling method based on deep reinforcement learning, the method comprising obtaining a service request of a target user node; inputting the service request into a trained network service resource scheduling model to obtain a resource scheduling strategy; obtaining the network service resources of the target service node based on the resource scheduling strategy and the service request; and sending the network service resources of the target service node to the target user node according to the service quality level and the transmission path. The present invention utilizes the group characteristics of the network to divide it into a dynamic service network composed of communities. Based on the network structure, a reasonable resource allocation can be performed through a network service resource scheduling model, including service node selection, path planning, and service level selection, thereby effectively reducing service delays, reducing resource transportation costs, and improving service quality and the efficiency of network service resource scheduling.

Description

Technical Field

[0001] The present invention relates to the field of communication technologies, and in particular to a service network scheduling method based on deep reinforcement learning. Background Art

[0002] Network service scheduling refers to the process of managing and allocating service requests in a network environment. Most existing network service scheduling adopts an end-to-end resource scheduling method, that is, based on a static network architecture, resources are directly obtained from the network architecture and then output. User demand and service resources are regarded as a simple input-output relationship, ignoring the inherent hierarchical structure, group characteristics, and user behavior information in the virtual layer (such as online shopping) in the service network. This leads to low service efficiency, untimely response, and unreasonable resource allocation.

[0003] Therefore, there is an urgent need to study an intelligent service scheduling and resource optimization method to meet dynamic and diverse needs and achieve reasonable allocation and optimal utilization of network service resources. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a service network scheduling method based on deep reinforcement learning to improve network service efficiency and the rationality of network service resource allocation.

[0005] In a first aspect, a service network scheduling method based on deep reinforcement learning is provided, which is applied to a pre-built service network, wherein the service network includes a plurality of communities, each community includes a service node, a user node, and a path between the service node and the user node; the method includes:

[0006] Get the service request of the target user node;

[0007] Input the service request into the trained network service resource scheduling model to obtain the resource scheduling strategy; the resource scheduling strategy includes the service node selection strategy, the service quality level selection strategy and the path selection strategy; among which, the service node selection strategy is used to select the target community and the target service node in the target community; the service quality level selection strategy is used to select the service quality level provided by the target service node for the target user node; the path selection strategy is used to select the transmission path between the target service node and the target user node;

[0008] Obtain network service resources of the target service node based on resource scheduling strategy and service request;

[0009] The network service resources of the target service node are sent to the target user node according to the service quality level and transmission path.

[0010] Optionally, the service network construction process includes:

[0011] Traverse each service node and user node, divide all nodes according to the preset division rule, and obtain preliminary communities; the preset division rule is to regard each node as an independent community; before dividing each node into an adjacent community, calculate the modularity change of the adjacent community after joining the node; if the modularity change is greater than 0, divide the node into the adjacent community; the expression of modularity change is:

[0012]

[0013] in, is the sum of the weights of the edges within the current community, For Node The degree of For Node The weight of the edge between the nodes in the adjacent community, is the total degree of all nodes in the adjacent community, Represents the total number of edges in the static service network;

[0014] Each preliminary community is further divided according to the preset division rules to obtain the final community.

[0015] Optionally, the network service resource scheduling model is composed of a main Q network and a target Q network; the training process of the network service resource scheduling model includes:

[0016] Randomly extract some experience data from the pre-built experience pool as training samples. The experience data is a four-tuple, including the current state of the dynamic service network, the resource scheduling strategy executed in the current state, the reward value obtained by executing the resource scheduling strategy, and the next state of the dynamic service network after executing the resource scheduling strategy;

[0017] The current state of the dynamic service network in the training sample and the resource scheduling strategy executed in the current state are input into the main Q network to obtain the main Q value;

[0018] Input the next state of the dynamic service network in the training sample into the target Q network to obtain the target Q value;

[0019] The error between the target Q value and the main Q value is calculated based on the preset loss function and reward value; the parameters of the main Q network are updated based on the error, and the parameters of the main Q network are updated to the target Q network every preset time step; the above training process is repeated for iterative training until the error is minimized; wherein the preset loss function is:

[0020]

[0021] in, It’s an instant reward. is the discount factor for future rewards, are the parameters of the main Q network; It is the main Q network; for Dynamically serve the status of the network at all times; for Always The resource scheduling strategy for execution in the state; is the target Q network; for Dynamically serve the status of the network at all times; for All possible resource scheduling strategies in the state; is the total number of iterative training; The number of the current training session.

[0022] Optionally, the method further comprises:

[0023] During each training, the resource adjustment strategy corresponding to the maximum Q value output by the main Q network is applied to the dynamic service network;

[0024] In response to the resource adjustment strategy corresponding to the maximum Q value, a new state and a new reward value are obtained;

[0025] Generate a new quadruple based on the new state and the new reward value;

[0026] Put the new quadruple into the experience pool to update the experience pool.

[0027] Optionally, putting the new quadruple into the experience pool to update the experience pool includes:

[0028] Compare the new reward value in the new quadruple with a preset threshold;

[0029] If it is greater than the preset threshold, it is placed in the important experience pool; otherwise, it is placed in the ordinary experience pool.

[0030] Optionally, the experience pool construction process includes:

[0031] Obtain the current state of the dynamic service network and several resource scheduling strategies for user requests under the current state;

[0032] Input the current state and several resource scheduling strategies into the main Q network to obtain the Q values ​​of several resource scheduling strategies;

[0033] In response to the resource scheduling strategy corresponding to the maximum Q value, obtaining the next state of the dynamic service network and the reward value of the resource scheduling strategy corresponding to the maximum Q value;

[0034] The initial resource scheduling strategy corresponding to the maximum Q value, the current state of the dynamic service network, the reward value of the resource scheduling strategy corresponding to the maximum Q value, and the next state of the dynamic service network are stored in the experience pool as a four-tuple, and the above construction process is repeated until the amount of data in the experience pool meets the preset demand.

[0035] Optionally, obtaining the reward value of the resource scheduling strategy corresponding to the maximum Q value includes:

[0036] Obtain the delay, revenue, and transportation cost incurred by the service node when executing the resource scheduling strategy corresponding to the maximum Q value;

[0037] The reward value is calculated based on the delay, income and transportation cost through the preset reward function, which is:

[0038]

[0039] in, For time delay; for income; For transportation costs, , , are the weight coefficients of delay, revenue and transportation cost respectively.

[0040] In a second aspect, a service network scheduling device based on deep reinforcement learning is provided, which is applied to a pre-built service network, wherein the service network includes a plurality of communities, each of which includes a service node, a user node, and a path between the service node and the user node; the device includes:

[0041] A first acquisition unit, used to acquire a service request of a target user node;

[0042] An input unit is used to input a service request into a trained network service resource scheduling model to obtain a resource scheduling strategy; the resource scheduling strategy includes a service node selection strategy, a service quality level selection strategy and a path selection strategy; wherein the service node selection strategy is used to select a target community and a target service node in the target community; the service quality level selection strategy is used to select a service quality level provided by the target service node for the target user node; and the path selection strategy is used to select a transmission path between the target service node and the target user node;

[0043] A second acquisition unit, configured to acquire network service resources of a target service node based on a resource scheduling strategy and a service request;

[0044] The sending unit is used to send the network service resources of the target service node to the target user node according to the service quality level and the transmission path.

[0045] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0046] Memory, used to store computer programs;

[0047] The processor is used to implement any method step described in the first aspect when executing a program stored in the memory.

[0048] According to a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any method step described in the first aspect is implemented.

[0049] The embodiment of the present invention provides a service network scheduling method based on deep reinforcement learning, which obtains the service request of the target user node; inputs the service request into the trained network service resource scheduling model to obtain the resource scheduling strategy; obtains the network service resources of the target service node based on the resource scheduling strategy and the service request; and sends the network service resources of the target service node to the target user node according to the service quality level and the transmission path. The present invention divides the network into a dynamic service network composed of communities by utilizing the group characteristics of the network, and each community is further layered, including a double-layer structure of service nodes and user nodes. Based on the network structure, reasonable resource allocation can be performed through the network service resource scheduling model, including service node selection, path planning and service level selection, thereby effectively reducing service delay, reducing resource transportation costs, and improving service quality and network service resource scheduling efficiency.

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 A schematic diagram showing a hierarchical structure of service nodes and user nodes provided in an embodiment of the present invention is shown;

[0053] Figure 2 A schematic diagram of a community structure of a service network provided by an embodiment of the present invention is shown;

[0054] Figure 3 A flow chart of a service network scheduling method based on deep reinforcement learning provided by an embodiment of the present invention is shown;

[0055] Figure 4 A schematic diagram of the training process of the network service resource scheduling model provided by an embodiment of the present invention is shown;

[0056] Figure 5 A schematic diagram of the structure of a service network scheduling device based on deep reinforcement learning provided by an embodiment of the present invention is shown;

[0057] Figure 6 A schematic structural diagram of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0059] Considering that the current network resource scheduling is based on a static network architecture, resources are directly obtained from the network architecture and then output. User demand and service resources are regarded as a simple input-output relationship, ignoring the inherent hierarchical structure, group characteristics and user behavior information in the virtual layer (such as online shopping) in the service network. This leads to low service efficiency, untimely response and unreasonable resource allocation. Based on this, an embodiment of the present invention provides a service network scheduling method and device based on deep reinforcement learning, which is described below through an embodiment.

[0060] An embodiment of the present invention provides a service network scheduling method based on deep reinforcement learning, which is applied to a pre-constructed service network. The service network includes several communities, each of which includes a service node, a user node, and a path between the service node and the user node.

[0061] In a specific implementation, the process of constructing the service network includes:

[0062] Step 1: Traverse each service node and user node, divide all nodes according to the preset division rules, and obtain preliminary communities.

[0063] In this step, if Figure 1 As shown, first establish a dynamic service network ;in, Is a collection of nodes, including a collection of service nodes and user nodes . Service Node Responsible for providing resources or services, while user nodes Indicates users who have demand for services. It is a set of edges, which represents the interaction relationship between nodes at time t, and reflects the interaction and transportation relationship between users and service nodes.

[0064] Determine the connection relationship of each node based on the established dynamic service network and establish an adjacency matrix : adjacency matrix Represents the connection relationship between nodes:

[0065] (1);

[0066] in, Represents the adjacency matrix Any element in ; Representation Node and nodes There is a connection between Representation Node and nodes There is no connection between them.

[0067] In a feasible implementation, the preset division rule is to regard each node as an independent community; try to divide each node into an adjacent community; before dividing each node into an adjacent community, calculate the modularity change of the adjacent community after the node is added; if the modularity change is greater than 0, divide the node into the adjacent community;

[0068] Among them, the modularity of each community is an indicator for evaluating the quality of community division, and the expression is:

[0069] (2);

[0070] in, represents the total number of edges in the network; is an element in the adjacency matrix, representing the node and nodes The connection relationship between them; and Node and nodes The degree of a node (i.e., the total number of edges connected to the node); is the Kronecker delta function, Is a node The associations you belong to; Is a node The society you belong to; hour , otherwise 0.

[0071] The expression of the modularity change is:

[0072] (3);

[0073] in, is the sum of the weights of the edges within the current community, For Node The degree of For Node The weight of the edge between the nodes in the adjacent community, is the total degree of all nodes in the adjacent community, Represents the total number of edges in the static service network.

[0074] In this step, each node is divided into a corresponding community by performing local optimization on each node to obtain a preliminary community structure.

[0075] Step 2: Continue to divide each preliminary community according to the preset division rules to obtain the final community.

[0076] In this step, each preliminary community is first regarded as a super node, and a network is rebuilt based on each super node. In the new network, each super node is further divided into corresponding communities according to the above preset division rules to optimize the community division in the first step. In an example, the divided community structure is as follows Figure 2 After the division, each community can be regarded as a subgraph ; Indicates The set of nodes within a community; Indicates The set of edges within a community.

[0077] Through two-stage iterative optimization, a high degree of modularity can be achieved quickly and it can also run efficiently on large-scale networks.

[0078] Based on the above-divided service network structure, an embodiment of the present invention provides a service network scheduling method based on deep reinforcement learning, such as Figure 3 As shown, the method comprises the following steps:

[0079] Step S301: Obtain a service request from a target user node.

[0080] Step S302: input the service request into the trained network service resource scheduling model to obtain a resource scheduling strategy.

[0081] In an embodiment of the present invention, the resource scheduling strategy includes a service node selection strategy, a service quality level selection strategy and a path selection strategy; wherein the service node selection strategy is used to select a target community and a target service node in the target community; the service quality level selection strategy is used to select the service quality level provided by the target service node for the target user node; and the path selection strategy is used to select a transmission path between the target service node and the target user node.

[0082] In an example, the resource scheduling policy is represented by a vector:

[0083] (5);

[0084] Among them, the service node selection strategy ; For the selected community label, For the community The index of the target service node in;

[0085] Quality of Service Level ; Indicates a service request Corresponding service quality level The service quality level can be determined based on the user's service request. After receiving the user's service request, the requirements in the service request are analyzed and the service quality level is determined based on the requirements. For example, the express delivery method provided by the e-commerce platform for shoppers is standard express or express.

[0086] Path selection strategy ;in, Indicates the target service node selects the path time delay; Indicates the path Transportation costs; is the weight of delay; is the weight of transportation cost.

[0087] Step S303: Acquire network service resources of the target service node based on the resource scheduling strategy and the service request.

[0088] In this step, the requirements in the user's service request are first parsed, such as the name and type of the product to be purchased; and network service resources are acquired according to the requirements and the determined resource scheduling strategy.

[0089] Step S304: Send the network service resources of the target service node to the target user node according to the service quality level and the transmission path.

[0090] In order to make the above embodiment clearer, a practical application scenario is given below to illustrate the embodiment of the present invention:

[0091] In the virtual e-commerce scenario, users are distributed in different regions, and their purchasing needs include ordering, payment, logistics and delivery services. When users submit service requests to the e-commerce platform, the e-commerce platform selects the appropriate storage center (service node) for delivery based on the network service resource scheduling model, and plans the delivery route and service level to minimize latency, reduce costs and increase platform revenue.

[0092] In this embodiment, users have different preferences and purchasing tendencies for different services in the virtual e-commerce network. These user characteristics and users' immediate decisions will affect the resource scheduling strategy of the e-commerce platform. At the same time, the resource scheduling strategy of the e-commerce platform will also affect users' choice of services.

[0093] The present invention utilizes the group characteristics of the network to divide it into a dynamic service network composed of communities. Each community is further layered, including a double-layer structure of upper-layer service nodes and lower-layer user nodes. Based on this network structure, reasonable resource allocation can be carried out through a network service resource scheduling model, including service node selection, path planning and service level selection, thereby effectively reducing service delays, reducing resource transportation costs, and improving service quality and the efficiency of network service resource scheduling.

[0094] Based on the above embodiment, the network service resource scheduling can adopt a deep reinforcement learning model, and the network service resource scheduling model is composed of a main Q network and a target Q network; Figure 4 As shown, the main Q network is the current Q network in the figure. The training process of the network service resource scheduling model includes:

[0095] Step A: Randomly select some experience data from the pre-built experience pool as training samples.

[0096] In this step, the empirical data is a four-tuple , including the current state of the dynamic service network , the resource scheduling strategy executed in the current state , execute the resource scheduling strategy Reward value obtained And the next state of the dynamic service network after executing the resource scheduling strategy .

[0097] The experience pool includes an important experience pool and an ordinary experience pool, and a number of training samples are randomly sampled from the important experience pool and the ordinary experience pool respectively.

[0098] Step B: Input the current state of the dynamic service network in the training sample and the resource scheduling strategy executed in the current state into the main Q network to obtain the main Q value.

[0099] In this step, the main Q network uses a multi-layer feedforward neural network that receives the state of the environment as input and outputs the expected return (Q value) for each possible action in that state. The Q value represents the expected value of the cumulative reward that can be obtained from the current state to the end after taking a certain action. The goal of the main Q network is to maximize the expected return in a given state.

[0100] Traditional Q learning is based on the value iteration method to construct a Q value table , recording the value of each state-action pair. The embodiment of the present invention uses a deep neural network The Q function is approximated to solve the problem of excessive computation when the state space and action space are large, so it can cope with the problem of large state space of node environment in complex service networks.

[0101] Step C: Input the next state of the dynamic service network in the training sample into the target Q network to obtain the target Q value.

[0102] The Target Q-Network is an auxiliary network introduced to stabilize the training process. It has the same structure as the Online Q-Network, but its parameters are copied from the Online Q-Network periodically instead of being updated frequently. This can reduce the fluctuation of the target Q value and improve the stability of training. The role of the Target Q-Network is to provide a relatively stable benchmark for calculating the target Q value in the loss function. This reduces the fluctuation of value estimates during training, making the training process more stable.

[0103] Step D: Calculate the error between the target Q value and the main Q value based on the preset loss function and reward value; and update the parameters of the main Q network based on the error, and update the parameters of the main Q network to the target Q network every preset time step.

[0104] In this step, the update of the main Q network parameters is realized through the Bellman equation of Q-learning, and its update formula is:

[0105] (6);

[0106] in is the learning rate; It’s an instant reward; is the discount factor for future rewards; are the parameters of the network; for Dynamically serve the status of the network at all times; for Always The resource scheduling strategy for execution in the state; for Dynamically serve the status of the network at all times; for All possible resource scheduling strategies in the state.

[0107] Repeat the above steps AD to perform iterative training until the error is minimized, which means that the error no longer decreases significantly. It should be noted that there are two cases for repeating the above steps. The first case is that some samples can be drawn from the experience pool, and then re-drawn from the experience pool after a preset number of training times and re-trained several times; then continue to repeat the extraction until the error is minimized. The second case is that samples can be re-drawn from the experience pool after each training. The specific training method can be selected according to needs.

[0108] Among them, the preset loss function is:

[0109] (7);

[0110] in, It’s an instant reward. is the discount factor for future rewards, are the parameters of the main Q network; It is the main Q network; for Dynamically serve the status of the network at all times; for Always The resource scheduling strategy for execution in the state; is the target Q network; for Dynamically serve the status of the network at all times; for All possible resource scheduling strategies in the state; is the total number of iterative training; The number of the current training session.

[0111] In another implementation, a fixed number of iterative training times may be set, and after the training is completed, the model corresponding to the number with the smallest error is selected as the final network service resource scheduling model.

[0112] The embodiment of the present invention effectively decomposes the action space in a complex network by introducing a multi-layer action design, thereby solving the problem of an overly large action search space. Specifically, the actions of the main Q network model are decomposed into three dimensions: service node selection, service level selection, and path selection. Actions at each level only need to make decisions in a relatively limited subspace. This multi-layer design greatly reduces the size of the overall action space, allowing the model to achieve a globally optimal resource allocation strategy through combination based on efficient exploration of local optimal solutions. This design not only improves the efficiency and convergence speed of training, but also enhances the adaptability and stability of the model in complex networks.

[0113] Based on the above embodiment, the method further includes:

[0114] Step E: During each training, the resource adjustment strategy corresponding to the maximum Q value output by the main Q network is applied to the dynamic service network.

[0115] like Figure 4 As shown, the resource adjustment strategy corresponding to the maximum Q value output by the current Q network each time Interaction occurs in the dynamic service network, so that the state of the service network environment of the dynamic service network changes.

[0116] Step F: In response to the resource adjustment strategy corresponding to the maximum Q value, a new state and a new reward value are obtained.

[0117] Step G: Generate a new quadruple based on the new state and the new reward value.

[0118] Step H: Put the new quadruple into the experience pool to update the experience pool.

[0119] In a feasible implementation, putting the new quadruple into the experience pool to update the experience pool includes:

[0120] Compare the new reward value in the new quadruple with a preset threshold;

[0121] If it is greater than the preset threshold, it is placed in the important experience pool; otherwise, it is placed in the ordinary experience pool.

[0122] The embodiment of the present invention helps to accumulate new experience and improve the diversity of training samples by continuously updating the experience pool, thereby improving the accuracy of main Q network training.

[0123] Based on the above embodiment, the process of constructing the experience pool includes:

[0124] Step A1: Obtain the current state of the dynamic service network and several resource scheduling strategies for user requests in the current state.

[0125] In this step, the current state of the dynamic service network is used to describe the global status of the network at the current moment, including node status and edge status.

[0126] Among them, the node state matrix ,in Indicates the total number of nodes; Represents the number of attributes of each node. The node state matrix is ​​expressed as:

[0127] (8);

[0128] in, Represents a service node availability of services; Represents a service node The service benefits Represents a service node The service level, Represents a user node The time tolerance of the other columns can represent attributes such as load and resource requirements.

[0129] Edge state matrix ,in is the number of edges, is the number of attributes of each edge. The attributes of all edges in the network can be represented by the edge state matrix:

[0130] (9);

[0131] in, represents the transportation time of the edge; It represents the transportation cost of the edge, and other columns can represent attributes such as the load limit of the edge.

[0132] In this step, the current state of the dynamic service network consists of node state and edge state, expressed as .

[0133] Step A2: Input the current state and several resource scheduling strategies into the main Q network to obtain the Q values ​​of several resource scheduling strategies.

[0134] In this step, if Figure 4 As shown, a set of all resource scheduling strategies is stored in the action space. Given a user service request, a set of all possible resource scheduling strategies for the request can be obtained. Several resource scheduling strategies are randomly selected from the set and input into the main Q network to obtain experience data.

[0135] Step A3: In response to the resource scheduling strategy corresponding to the maximum Q value, the next state of the dynamic service network and the reward value of the resource scheduling strategy corresponding to the maximum Q value are obtained.

[0136] Step A4: The initial resource scheduling strategy corresponding to the maximum Q value, the current state of the dynamic service network, the reward value of the resource scheduling strategy corresponding to the maximum Q value, and the next state of the dynamic service network are stored in the experience pool as a four-tuple, and all the above processes are repeated until the amount of data in the experience pool meets the preset demand.

[0137] The embodiment of the present invention constructs an experience pool and uses the experience pool to randomly obtain training samples, which effectively breaks the correlation of data and makes the training process more stable.

[0138] Based on the above embodiment, obtaining the reward value of the resource scheduling strategy corresponding to the maximum Q value includes:

[0139] Obtain the delay, revenue, and transportation cost incurred by the service node when executing the resource scheduling strategy corresponding to the maximum Q value;

[0140] The reward value is calculated based on the delay, income and transportation cost through the preset reward function, which is:

[0141] (10);

[0142] in, For time delay; for income; For transportation costs, , , are the weight coefficients of delay, revenue and transportation cost respectively.

[0143] In another embodiment, after obtaining the resource scheduling strategy output by the network service resource scheduling model, the quality of the resource scheduling strategy may also be evaluated.

[0144] In a feasible implementation, the quality of the resource scheduling strategy can be evaluated by evaluating the utility value of the target service node. Specifically, the utility of the target service node can be calculated by a utility function, and the utility function expression is:

[0145] (11);

[0146] in, Select a strategy for the service node, Select strategies for user nodes, such as user behavior information for selecting products; For dynamic income, For the delay cost, For transportation costs, , , are the weight coefficients of delay, revenue and transportation cost respectively; For service request.

[0147] in, As the number of users and service quality And the delay variation:

[0148] (12);

[0149] For service request The demand, For service request The price, For service cost, is the service level impact coefficient; Service Request The corresponding service level k.

[0150] Service demand (13);

[0151] in, For service request The corresponding service market potential; is the influence coefficient of service level on service demand.

[0152] In addition, the delay cost (14);

[0153] Represents the delay cost within the community where the target service node is located; represents the cross-community delay cost; The cost within the community where the user node is located.

[0154] Same transportation cost (15);

[0155] Indicates the transportation cost within the community where the target service node is located; represents the inter-community transportation cost; Represents the transportation cost within the community where the user node is located.

[0156] The embodiment of the present invention calculates the utility of the service node by using the utility function of the service node, and maximizes its utility from three aspects: revenue, delay and resource consumption. This enables the service node to reduce unnecessary resource consumption while meeting user needs. The user demand is also modeled in the utility function, simulating the situation where the user is affected by the service provider's strategy, which can better optimize the resource scheduling strategy.

[0157] Based on the same inventive concept, a service network scheduling device based on deep reinforcement learning is provided, which is applied to a pre-built service network. The service network includes several communities, each of which includes a service node, a user node, and a path between the service node and the user node; Figure 5 As shown, the device comprises:

[0158] A first acquisition unit 501 is used to acquire a service request of a target user node;

[0159] The input unit 502 is used to input the service request into the trained network service resource scheduling model to obtain a resource scheduling strategy; the resource scheduling strategy includes a service node selection strategy, a service quality level selection strategy and a path selection strategy; wherein the service node selection strategy is used to select a target community and a target service node in the target community; the service quality level selection strategy is used to select a service quality level provided by the target service node for the target user node; and the path selection strategy is used to select a transmission path between the target service node and the target user node;

[0160] The second acquisition unit 503 is used to acquire the network service resources of the target service node based on the resource scheduling strategy and the service request;

[0161] The sending unit 504 is used to send the network service resources of the target service node to the target user node according to the service quality level and the transmission path.

[0162] Based on the same technical concept, an embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a processor 601 , a communication interface 602 , a memory 603 and a communication bus 604 , wherein the processor 601 , the communication interface 602 , and the memory 603 communicate with each other via the communication bus 604 .

[0163] Memory 603, used for storing computer programs;

[0164] The processor 601 is used to implement the steps of the service network scheduling method based on deep reinforcement learning when executing the program stored in the memory 603.

[0165] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0166] The communication interface is used for communication between the above electronic device and other devices.

[0167] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0168] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0169] The computer program product for performing service network scheduling based on deep reinforcement learning provided in an embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0170] The device for service network scheduling based on deep reinforcement learning provided in the embodiment of the present invention can be specific hardware on the device or software or firmware installed on the device. The device provided in the embodiment of the present invention has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brief description, for the parts not mentioned in the device embodiment, reference can be made to the corresponding contents in the aforementioned method embodiment. Technical personnel in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0171] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0172] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] In addition, each functional unit in the embodiment provided by the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0174] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0175] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0176] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A service network scheduling method based on deep reinforcement learning, characterized in that: Applied to a pre-built service network, the service network includes a number of communities, each community includes a service node, a user node and a path between the service node and the user node; The construction process of the service network includes: Traverse each service node and user node, divide all nodes according to the preset division rule, and obtain preliminary communities; the preset division rule is to regard each node as an independent community; before dividing each node into an adjacent community, calculate the modularity change of the adjacent community after joining the node; if the modularity change is greater than 0, divide the node into the adjacent community; the expression of the modularity change is: in, is the sum of the weights of the edges within the current community, For Node The degree of For Node The weight of the edge between the nodes in the adjacent community, is the total degree of all nodes in the adjacent community, Represents the total number of edges in the static service network; Continue to divide each of the preliminary communities according to the preset division rule to obtain a final community; The method comprises: Get the service request of the target user node; The service request is input into the trained network service resource scheduling model to obtain a resource scheduling strategy; the resource scheduling strategy includes a service node selection strategy, a service quality level selection strategy and a path selection strategy; wherein the service node selection strategy is used to select a target community and a target service node in the target community; the service quality level selection strategy is used to select a service quality level provided by the target service node for the target user node; and the path selection strategy is used to select a transmission path between the target service node and the target user node; Acquire network service resources of a target service node based on the resource scheduling strategy and the service request; The network service resources of the target service node are sent to the target user node according to the service quality level and the transmission path.

2. The method according to claim 1, characterized in that The network service resource scheduling model is composed of a main Q network and a target Q network; the training process of the network service resource scheduling model includes: Randomly extract a number of experience data from the pre-built experience pool as training samples, wherein the experience data is a four-tuple, including the current state of the dynamic service network, the resource scheduling strategy executed in the current state, the reward value obtained by executing the resource scheduling strategy, and the next state of the dynamic service network after executing the resource scheduling strategy; Inputting the current state of the dynamic service network in the training sample and the resource scheduling strategy executed in the current state into the main Q network to obtain a main Q value; Inputting the next state of the dynamic service network in the training sample into the target Q network to obtain a target Q value; The error between the target Q value and the main Q value is calculated based on the preset loss function and the reward value; and the parameters of the main Q network are updated based on the error, and the parameters of the main Q network are updated to the target Q network every preset time step; the above training process is repeated for iterative training until the error is minimized; wherein the preset loss function is: in, It’s an instant reward. is the discount factor for future rewards, are the parameters of the main Q network; It is the main Q network; for Dynamically serve the status of the network at all times; for Always The resource scheduling strategy for execution in the state; is the target Q network; for Dynamically serve the status of the network at all times; for All possible resource scheduling strategies in the state; The target network is The resource scheduling strategy corresponding to the maximum Q value output in the state; is the total number of iterative training; The number of the current training session.

3. The method according to claim 2, characterized in that The method further comprises: During each training, the resource adjustment strategy corresponding to the maximum Q value output by the main Q network is applied to the dynamic service network; In response to the resource adjustment strategy corresponding to the maximum Q value, a new state and a new reward value are obtained; Generate a new quadruple based on the new state and the new reward value; The new quadruple is put into the experience pool to update the experience pool.

4. The method according to claim 3, characterized in that Putting the new quadruple into the experience pool to update the experience pool comprises: Comparing the new reward value in the new quadruple with a preset threshold; If it is greater than the preset threshold, it is placed in the important experience pool; otherwise, it is placed in the ordinary experience pool.

5. The method according to claim 2, characterized in that: The process of constructing the experience pool includes: Acquire the current state of the dynamic service network and several resource scheduling strategies for user requests under the current state; Inputting the current state and several resource scheduling strategies into the main Q network to obtain Q values ​​of several resource scheduling strategies; In response to the resource scheduling strategy corresponding to the maximum Q value, obtaining the next state of the dynamic service network and the reward value of the resource scheduling strategy corresponding to the maximum Q value; The initial resource scheduling strategy corresponding to the maximum Q value, the current state of the dynamic service network, the reward value of the resource scheduling strategy corresponding to the maximum Q value, and the next state of the dynamic service network are stored in the experience pool as a four-tuple, and the above construction process is repeated until the amount of data in the experience pool meets the preset demand.

6. The method according to claim 1, characterized in that The reward values ​​of the resource scheduling strategy corresponding to the maximum Q value include: Obtain the delay, revenue, and transportation cost incurred by the service node when executing the resource scheduling strategy corresponding to the maximum Q value; The reward value is calculated based on the delay, income and transportation cost through a preset reward function, and the reward function is: in, For time delay; for income; For transportation costs, , , are the weight coefficients of delay, revenue and transportation cost respectively.

7. A service network scheduling device based on deep reinforcement learning, characterized in that: Applied to a pre-built service network, the service network includes a number of communities, each community includes a service node, a user node and a path between the service node and the user node; The construction process of the service network includes: Traverse each service node and user node, divide all nodes according to the preset division rule, and obtain preliminary communities; the preset division rule is to regard each node as an independent community; before dividing each node into an adjacent community, calculate the modularity change of the adjacent community after joining the node; if the modularity change is greater than 0, divide the node into the adjacent community; the expression of the modularity change is: in, is the sum of the weights of the edges within the current community, For Node The degree of For Node The weight of the edge between the nodes in the adjacent community, is the total degree of all nodes in adjacent communities, Represents the total number of edges in the static service network; Continue to divide each of the preliminary communities according to the preset division rule to obtain a final community; The device comprises: A first acquisition unit, used to acquire a service request of a target user node; An input unit is used to input the service request into the trained network service resource scheduling model to obtain a resource scheduling strategy; the resource scheduling strategy includes a service node selection strategy, a service quality level selection strategy and a path selection strategy; wherein the service node selection strategy is used to select a target community and a target service node in the target community; the service quality level selection strategy is used to select a service quality level provided by the target service node for the target user node; and the path selection strategy is used to select a transmission path between the target service node and the target user node; A second acquisition unit, configured to acquire network service resources of a target service node based on the resource scheduling strategy and the service request; The sending unit is used to send the network service resources of the target service node to the target user node according to the service quality level and the transmission path.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 6 when executing a program stored in a memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 6 are implemented.

Images