Wireless mobile communication network resource scheduling method and system based on dgqn

Abstract

The application discloses a wireless mobile communication network resource scheduling method and system based on a DGQN, relates to the technical field of communication network resource scheduling, and comprises the following steps: firstly, mapping the resource scheduling process of the wireless mobile communication network into a Markov decision process; then, adopting a double-path architecture to process concave metric indexes and additive metric indexes of the resource scheduling process respectively; then, performing weighted summation on the q values of the two types of indexes, and obtaining a comprehensive model q value by comprehensively representing the q values of the two types of indexes; finally, determining the final strategy and the final routing path of the service flow according to the comprehensive model q value, and thereby realizing the resource scheduling of the wireless mobile communication network. The application considers the real-time dynamic change of the network environment and the differentiated demands of various service flows in the network on different performance indexes, and proposes the above-mentioned improved scheme for the wireless mobile communication network with high-frequency changes in the topological structure.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication network resource scheduling, in particular to a wireless mobile communication network resource scheduling method and system based on DGQN. BACKGROUND

[0002] Communication network resource scheduling technology refers to a technology of optimizing network resource configuration through scheduling strategies and algorithms under the condition of limited network resources. In wireless mobile networks, it usually involves multiple service modes such as voice, video, and message. Since different types of service flows have significant differences in the demand for network bandwidth, delay, packet loss rate, and other performance indicators, the resource scheduling strategy must be able to dynamically adapt to the needs of various service flows, and from the perspective of global network resource scheduling, various types of parameter indicators need to be classified and processed. At the same time, the wireless mobile network environment is easily affected by external environment and electromagnetic environment and other factors and changes in real time, so the scheduling strategy must also be able to adapt to these environmental changes. In addition, due to the limited computing power of devices in wireless mobile networks, the design of the scheduling algorithm should fully consider the situation of limited computing power.

[0003] At present, a variety of intelligent algorithms have been applied to solve network resource scheduling problems, including neural network algorithms, ant colony algorithms, and reinforcement learning algorithms. However, the parameter adjustment of neural network algorithms is relatively complex, and usually requires a large amount of historical and real-time data to train the model to achieve ideal performance. Not only does the training process require a large amount of computing resources and time cost, but the neural network also has less interaction with the environment during the training process, which may lead to a significant decline in algorithm effectiveness when facing frequently changing network environments. The ant colony algorithm takes a long time in the search process and is easy to fall into a local optimal solution. The reinforcement learning algorithm interacts with the environment in real time through a reward mechanism, but the setting of its learning rules requires a high requirement. Although the combination of reinforcement learning algorithm and deep learning such as convolutional neural network has been more widely applied, it also brings more computing power loss, so it is not suitable for direct application in wireless mobile networks with limited resources. In the existing research on network resource scheduling using reinforcement learning, most of them use additive metrics such as delay and hop count as input parameters, and pay insufficient attention to other important metrics such as remaining bandwidth. In fact, the q value of the remaining bandwidth is related to the entire transmission path of the service flow, and the reward cannot be directly extracted from each action. Therefore, the traditional deep reinforcement learning method such as DQN (Deep Q Learning) cannot be directly applied to the resource scheduling of wireless mobile networks. SUMMARY

[0004] The purpose of the present application is to provide a wireless mobile communication network resource scheduling method and system based on DGQN, which can efficiently and intelligently schedule communication network resources in diversified service scenarios.

[0005] To achieve the above object, the application provides the following scheme:

[0006] In a first aspect, the application provides a wireless mobile communication network resource scheduling method based on DGQN, comprising the following steps:

[0007] Obtaining the topology structure of the wireless mobile communication network and the index data of each node in the wireless mobile communication network; the index data includes the delay parameters and bandwidth parameters of each node through different links.

[0008] Pretreating the index data of each node in the wireless mobile communication network to obtain pretreated index data.

[0009] According to the topology structure of the wireless mobile communication network, the resource scheduling process of the wireless mobile communication network is mapped into a Markov decision process in the reinforcement learning algorithm.

[0010] The concave metric index and the additive metric index of the resource scheduling process are processed respectively by adopting a double-path architecture to obtain the concave metric index q value and the additive metric index q value; the concave metric index is the residual bandwidth on the routing path of the service flow, and the additive metric index is the transmission delay on the routing path of the service flow; the improved reinforcement learning algorithm DGQN is used to process the additive metric index to obtain the additive metric index q value; the improved reinforcement learning algorithm DGQN is a network obtained by improving the traditional reinforcement learning algorithm DQN based on GhostNet.

[0011] The concave metric index q value and the additive metric index q value are weighted and summed to obtain a comprehensive model q value; the comprehensive model q value is a q value representation that comprehensively considers the two types of indexes.

[0012] The final strategy is determined according to the comprehensive model q value, and the final routing path of any service flow is determined based on the final strategy.

[0013] The final routing path of the service flow is issued to each node of the wireless mobile communication network to perform resource scheduling of the wireless mobile communication network.

[0014] Optionally, the index data of each node in the wireless mobile communication network is pretreated to obtain pretreated index data, specifically comprising the following steps:

[0015] For the missing values in the index data of any node, the same index data of the previous time frame is filled according to the missing values.

[0016] All the index data after filling the missing values are normalized to obtain the pretreated index data.

[0017] Optionally, a dual-path architecture is adopted to process the concavity metric and the additive metric of the resource scheduling process respectively to obtain the concavity metric q value and the additive metric q value, which specifically includes the following steps:

[0018] The reinforcement learning algorithm Q-Learning is used to process the concavity metric and obtain the concavity metric q value.

[0019] The improved reinforcement learning algorithm DGQN is used to process the additive metric to obtain the additive metric q value; the improved reinforcement learning algorithm DGQN is based on GhostNet, and the convolution kernel in the traditional reinforcement learning algorithm DQN is replaced by the Ghost bottleneck structure to obtain a network.

[0020] Optionally, the comprehensive model q value is calculated according to the following formula:

[0021]

[0022] Among them, q(s,a * ) is the comprehensive model q value, s is the state, a * is the action taken, q w (s,a * ) is the concavity metric q value, w thr is the bandwidth requirement of the service flow, q d (s,a * ) is the additive metric q value, qMin d =min(q d (s,a * )), μ is the weight of the additive metric q value, and A is the action set.

[0023] Alternatively, the concavity metric q-value can be calculated as follows:

[0024]

[0025] Among them, q w (s,a) is the concavity metric q value of taking action a in the current state s, q w (s -1 ,a * ) is the concavity metric q value of taking action a* in the state s at the previous moment, A is the action set, w ss' for…….

[0026] Optionally, the remaining bandwidth on the routing path of the service flow can be expressed by the following formula:

[0027] M W (E ρ )=min(w ij), ij E E ρ .

[0028] wherein M W (E ρ ) is the residual bandwidth on the routing path of the traffic flow p, w ij is the bandwidth parameter of the link ij, E ρ is the routing path of the traffic flow p.

[0029] Optionally, the transmission delay on the routing path of the traffic flow can be represented by the following formula:

[0030]

[0031] wherein M D (E ρ ) is the transmission delay on the routing path of the traffic flow p, d ij is the delay parameter of the link ij, E ρ is the routing path of the traffic flow p.

[0032] Optionally, the delay parameter of the link ij can be determined by the following formula:

[0033]

[0034] wherein d ij is the delay parameter of the link ij, a T-1 ,..., a are forgetting factors, T is a preset time length, is the initial delay parameter of the link ij at the current time, is the delay parameter of the link ij at the last time.

[0035] Optionally, the forgetting factors a T-1 ,..., a increase, so as to reduce the influence of the delay parameters far away from the current time.

[0036] In a second aspect, the present application provides a wireless mobile communication network resource scheduling system based on DGQN, comprising the following modules:

[0037] A communication network parameter acquisition module is configured to acquire the topological structure of the wireless mobile communication network and the index data of each node in the wireless mobile communication network; the index data includes the delay parameters and bandwidth parameters of each node through different links.

[0038] An index data preprocessing module is configured to preprocess the index data of each node in the wireless mobile communication network to obtain preprocessed index data.

[0039] A decision scheduling process mapping module is configured to map a resource scheduling process of the wireless mobile communication network into a Markov decision process in the reinforcement learning algorithm according to a topology of the wireless mobile communication network.

[0040] A double-path index processing module is configured to process concave metric indexes and additive metric indexes of the resource scheduling process respectively in a double-path architecture to obtain q values of the concave metric indexes and q values of the additive metric indexes; the concave metric index is a residual bandwidth on a routing path of a service flow, and the additive metric index is a transmission delay on the routing path of the service flow; the improved reinforcement learning algorithm DGQN is used to process the additive metric index to obtain the q values of the additive metric index; the improved reinforcement learning algorithm DGQN is a network obtained by improving a traditional reinforcement learning algorithm DQN based on a GhostNet.

[0041] A comprehensive model q value calculation module is configured to perform weighted summation on the q values of the concave metric indexes and the q values of the additive metric indexes to obtain a comprehensive model q value; the comprehensive model q value is a q value representation that comprehensively combines the two types of indexes.

[0042] A final routing path determination module is configured to determine a final strategy according to the comprehensive model q value, and determine a final routing path of any one service flow based on the final strategy.

[0043] A final routing path issuing module is configured to issue the final routing path of the service flow to each node of the wireless mobile communication network to perform resource scheduling of the wireless mobile communication network.

[0044] According to the embodiments provided in the present application, the following technical effects are disclosed:

[0045] The application provides a wireless mobile communication network resource scheduling method and system based on a DGQN. In the method, after parameters in the wireless mobile communication network are acquired and preprocessed, the resource scheduling process of the wireless mobile communication network is first mapped into a Markov decision process in a reinforcement learning algorithm according to the topology structure of the wireless mobile communication network, then a double-path architecture is adopted to process concave metric indicators and additive metric indicators of the resource scheduling process, wherein a network based on a GhostNet and improved from a traditional reinforcement learning algorithm DQN is used to process the additive metric indicators, then the processed concave metric indicators q values and additive metric indicators q values are weighted and summed to obtain a comprehensive model q value, which comprehensively represents the q values of the two types of indicators; finally, the final strategy is determined according to the comprehensive model q value, the final routing path of the service flow is determined, and the final routing path is issued to each node of the wireless mobile communication network to realize the resource scheduling of the wireless mobile communication network. The application considers the real-time dynamic changes of the network environment and the differentiated requirements of various service flows in the network on different performance indicators for the wireless mobile communication network with high-frequency changes in the topology structure, and comprehensively considers the concave metric and additive metric of the communication network. In order to apply both types of indicators to network resource scheduling calculation, a double-path architecture is designed according to the characteristics of the two types of data, so that there is constraint and connection between each other, and the traditional reinforcement learning algorithm DQN is improved, a lightweight improved reinforcement learning algorithm DGQN is designed, which dynamically adapts to differentiated services and network environment, and to some extent, solves the communication network resource scheduling problem under the condition of limited computing power. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0047] Figure 1 A flowchart of a wireless mobile communication network resource scheduling method based on a DGQN provided by an embodiment of the present application.

[0048] Figure 2 A schematic diagram of a double-path architecture in a wireless mobile communication network resource scheduling method based on a DGQN provided by an embodiment of the present application.

[0049] Figure 3 A schematic diagram of an improved DGQN algorithm in a wireless mobile communication network resource scheduling method based on a DGQN provided by an embodiment of the present application.

[0050] Figure 4A functional module schematic diagram of a wireless mobile communication network resource scheduling system based on a DGQN is provided for an embodiment of the present application.

[0051] Figure 5 A structural schematic diagram of a computer device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0053] The above purposes, features and advantages of the present application can be more obvious and easy to understand. The present application will be described in further detail below with reference to the drawings and specific embodiments.

[0054] In an exemplary embodiment, as shown in Figure 1 , a wireless mobile communication network resource scheduling method based on a DGQN is provided, which can be executed by a computer device and includes the following steps:

[0055] S1, acquiring a topology structure of a wireless mobile communication network and index data of each node in the wireless mobile communication network; the index data includes time delay parameters and bandwidth parameters of each node through different links. In the embodiment, the index data of each node is acquired through reporting on a network monitoring software.

[0056] The topology structure of the communication network is represented by a graph G(V, E), where V and E are respectively a point set and an edge set in the communication network, i∈V represents a Mesh node in the network, and N(i) is defined as a neighbor node set of node i, and ij∈E represents a wireless link from node i to node j.

[0057] In the present application, it is assumed that a service flow is indivisible. To meet the differentiated QoS (Quality of Service) requirement of the service flow, two types of QoS indexes of a routing path are mainly considered, and M D (E ρ ) represents a transmission time delay on a routing path of the service flow ρ, and M W (E ρ ) represents a residual bandwidth on the routing path of the service flow ρ. The bandwidth parameter is a concave metric, which is a minimum value of residual bandwidths w ij on all links on the routing path. The residual bandwidth on the routing path of the service flow can be represented by the following formula:

[0058] MW (E ρ )=min(w ij ),ij∈E ρ 。

[0059] where M W (E ρ ) is the residual bandwidth on the routing path of traffic flow p, w ij is the bandwidth parameter of link ij, and E ρ is the routing path of traffic flow p.

[0060] The delay parameter is an additive metric, which is the sum of the delay d ij of all links on the routing path. The transmission delay on the routing path of traffic flow p can be expressed as:

[0061]

[0062] where M D (E ρ ) is the transmission delay on the routing path of traffic flow p, d ij is the delay parameter of link ij, and E ρ is the routing path of traffic flow p.

[0063] In a dynamic network environment, to ensure that the routing path has certain anti-volatility and stability, the parameters of the links need to maintain high performance for a certain period of time, so the average value of the delay parameter of the link within the preset time length is taken as the representation of the delay at the current time. The delay parameter d ij of link ij can be determined by:

[0064]

[0065] where d ij is the delay parameter of link ij, a T-1 ,…,a are forgetting factors, T is the preset time length, is the initial delay parameter of link ij at the current time, is the delay parameter of link ij at the previous time.

[0066] Since the information far from the current time has lower reference value, the forgetting factor a is introduced to reduce its interference. The forgetting factors a T-1 ,…,a increase to reduce the influence of the delay parameters far from the current time.

[0067] The resource scheduling problem of the communication network is modeled as a multi-objective optimization problem. The bandwidth parameter is expressed in the form of constraint, and when the residual bandwidth of the routing path is greater than the bandwidth demand of the traffic flow, the link with lower delay and higher residual bandwidth is selected; otherwise, the service request is rejected. Let x ij= 1 represents that the path ij is in the routing path E of the service ρ x = 1, otherwise x ij = 0. The mathematical model is expressed as follows:

[0068]

[0069] maximize M W (E ρ )-w thr

[0070] subject to M W (E ρ )≥w thr

[0071]

[0072] wherein t represents a service life cycle, E [epsilon] t represents an expectation at time t. The first objective in the model represents the routing path E ρ with the minimum expected transmission delay, and the second objective represents the maximum residual bandwidth.

[0073] The second objective is converted into a minimization form, and one is added to the denominator to avoid being 0:

[0074]

[0075] The weight mu combines the two objectives and optimizes them simultaneously.

[0076]

[0077] S2, pre-process the index data of each node in the wireless mobile communication network to obtain pre-processed index data. Specifically, in this embodiment, step S2 includes the following steps:

[0078] S21, for the missing values in the index data of any node, fill them according to the same index data of the previous time frame.

[0079] S22, normalize all the index data after the missing value filling to obtain the pre-processed index data. In this step, the index data is normalized to facilitate subsequent calculation.

[0080] S3. According to the topology of the wireless mobile communication network, the resource scheduling process of the wireless mobile communication network is mapped into a Markov decision process in the reinforcement learning algorithm. Specifically, in this embodiment, the topology graph G(V, E) is mapped into a Markov decision process E = (S, A, P, R), where S represents a state set, A represents an action set, P represents a probability transition matrix, and R represents a reward function. The correspondence between the routing problem of the communication network and the Markov decision process is shown in Table 1:

[0081] Table 1 Correspondence between the routing problem of the communication network and the Markov decision process

[0082] Markov decision process Routing problem of a communication network State set S Each node in the graph G(V, E) Dynamic set A Selecting which neighbor node to use as the next hop node Transition probability P Transition probability p = 1 for the selected action Reward R Link parameter indicator d ij ,w ij Correlation

[0083] In the reinforcement learning algorithm, the agent and the environment interact with each other in a series of discrete times to complete a total target task. The interaction between the agent and the environment includes the following processes:

[0084] (a) At time t, the agent observes the state s i ∈ S of the environment, and the agent selects an action a i according to the policy π(s i ).

[0085] (b) The action a i acts on the environment, and the state of the environment changes to s i+1 , and the environment gives the agent a corresponding reward r i .

[0086] (c) Then the agent will make a new decision according to the new environment state and the policy.

[0087] (d) The above interaction process is repeated until the agent completes the corresponding target task.

[0088] In the reinforcement learning algorithm, the goal of the agent is to maximize the cumulative reward. The Q under T-step cumulative reward can be defined as:

[0089]

[0090] The Q π (s, a) under T-step cumulative reward can be defined as:

[0091]

[0092] The T-step cumulative reward under Q

[0093]

[0094]

[0095] S4. A dual-path architecture is adopted to process the concave metric and additive metric of the resource scheduling process respectively, and obtain the concave metric q value and the additive metric q value; the concave metric is the remaining bandwidth on the routing path of the business flow, and the additive metric is the transmission delay on the routing path of the business flow; the improved reinforcement learning algorithm DGQN is used to process the additive metric to obtain the additive metric q value; the improved reinforcement learning algorithm DGQN is a network obtained by improving the traditional reinforcement learning algorithm DQN based on GhostNet.

[0096] Considering that the remaining bandwidth is a concave metric and the transmission delay is an additive metric, the two types of indicators are updated in different ways. Therefore, this application adopts different calculation methods for the two types of indicators when designing the Q network. A dual-path architecture is adopted to process the concave metric and the additive metric respectively, and there are certain constraints between the two, such as Figure 2 Schematic diagram of the dual-path architecture shown.

[0097] In this embodiment, step S4 specifically includes the following steps:

[0098] S41. The concavity metric is processed using the reinforcement learning algorithm Q-Learning to obtain the concavity metric q-value. Assume that the action set is A. In state s, action a selects s' as the next hop node. For a concavity metric using residual bandwidth, its reward r has no fixed pattern and cannot be directly obtained. Because the q-value is the minimum value of the residual bandwidth of the entire transmission link, as an example, the concavity metric q-value can be calculated using the following formula:

[0099]

[0100] Among them, q w (s,a) is the concavity metric q value of taking action a in the current state s, q w (s -1 ,a * ) is the concavity metric q value of taking action a* in the state s at the previous moment, A is the action set, w ss' for…….

[0101] Existence constraint: When q w (s,a)>w thr Then, for additive metrics such as link delay and hop count, the reward r is calculated based on the delay from s to s', taking -d ij , when s' is the target node (s is the current node, s' is the next hop node state), r is 0.

[0102] S42, the improved reinforcement learning algorithm DGQN is used to process the additive metric index to obtain the additive metric index q value; the improved reinforcement learning algorithm DGQN is based on Ghost Net, and a network obtained by replacing the convolution kernel in the traditional reinforcement learning algorithm DQN with a Ghost bottleneck structure.

[0103] In the Ghostnet, linear mapping is used to replace part of the convolution calculation, which effectively reduces the computing power burden. In the embodiment, the DQN based on Ghost Net is improved, the traditional convolution kernel is replaced with a Ghost bottleneck structure, a lightweight DGQN (Deep Ghost Q Network) algorithm is designed, and a DGQN diagram obtained by improvement is as shown in Figure 3 The specific algorithm process of the DGQN is as follows:

[0104] (I). Input state space S, action space A, and target network parameter update frequency F.

[0105] (II). Initialize the experience replay library D with a capacity of N.

[0106] (III). Randomly initialize the actual Q network parameters and the target network parameters, which are equal in initialization, wherein the Q network adopts the ghost bottleneck structure and is designed to be lightweight, thereby effectively saving computing power.

[0107] (IV). For the starting routing node i, the starting routing node state is s i , the action a = π(s i ) is selected, the action a is executed, the agent interacts with the environment to obtain a single-step immediate reward r and a new state s'.

[0108] (V). Put s i , a, r, s' into the experience replay library D.

[0109] (VI). Sample new ss, aa, rr, ss' data groups from D for training.

[0110] (VII). Train the Q network with as the loss function; wherein γ is the discount rate.

[0111] (VIII). State update: s←s'.

[0112] (VIIII). Update the target network parameters every F steps,

[0113] (X). When the state s is the destination node, return to step IV.​​

[0114] (XI) Training ends when the Q network parameters converge or other termination conditions are met.

[0115] S5. Perform a weighted summation of the concavity metric q value and the additive metric q value to obtain a comprehensive model q value; the comprehensive model q value is a q value representation that combines the two types of metrics. In an exemplary embodiment of the present application, the comprehensive model q value is calculated according to the following formula:

[0116]

[0117] Among them, q(s,α * ) is the comprehensive model q value, s is the state, a * is the action taken, q w (s,a * ) is the concavity metric q value, w thr is the bandwidth requirement of the service flow, q d (s,a * ) is the additive metric q value, qMin d =min(q d (s,a * )), μ is the weight of the additive metric q value, and A is the action set.

[0118] S6. Determine the final strategy based on the comprehensive model q value, and determine the final routing path of any business flow based on the final strategy. * ) converges or reaches the set number of iterations, a complete q-value representation of the two metrics is obtained. Each value represents the optimal reward for selecting a specific action in the current state. q(s,a) is the normalized weighted sum of the two objectives. Selecting the minimum q-value along the q(s,a) scale yields the final strategy π. Based on π, the optimal transmission path for each service is determined, achieving efficient scheduling of communication network resources.

[0119] S7. The final routing path of the service flow is sent to each node of the wireless mobile communication network to perform resource scheduling of the wireless mobile communication network. The final routing path calculated based on the above process is sent to the relevant nodes according to the signaling to achieve efficient transmission of diversified services and intelligent scheduling of communication network resources.

[0120] The application scheme applies the reinforcement learning technology to the frequently changing wireless mobile communication network topology, fully considers the real-time dynamic change characteristics of the wireless mobile communication network environment and the differentiated needs of services for different performance indicators. By modeling the wireless mobile communication network resource scheduling problem as a multi-objective optimization problem, both types of indicators, i.e., the concave metric and the additive metric in the communication network, are comprehensively considered. In order to enable both types of indicators to be effectively applied to the calculation process of network resource scheduling, a double-path architecture is designed for different data characteristics, and the two paths in the architecture are mutually constrained and kept in contact. In addition, the traditional reinforcement learning algorithm DQN is improved, and a lightweight reinforcement learning algorithm DGQN is proposed to dynamically adapt to changes in different services and network environments, thereby solving the problem of communication network resource scheduling under limited computing power to some extent.

[0121] Based on the same inventive concept, the embodiments of the application also provide a system for implementing the above-mentioned DGQN-based wireless mobile communication network resource scheduling method. The implementation scheme of the system for solving the problem is similar to the implementation scheme described in the above method. In one exemplary embodiment, as shown in Figure 4 Fig. 1, a DGQN-based wireless mobile communication network resource scheduling system is provided, which includes the following modules:

[0122] A communication network parameter acquisition module is configured to acquire the topology of the wireless mobile communication network and the indicator data of each node in the wireless mobile communication network. The indicator data includes the delay parameters and bandwidth parameters of each node through different links.

[0123] An indicator data preprocessing module is configured to preprocess the indicator data of each node in the wireless mobile communication network to obtain preprocessed indicator data.

[0124] A decision scheduling process mapping module is configured to map the resource scheduling process of the wireless mobile communication network to a Markov decision process in the reinforcement learning algorithm according to the topology of the wireless mobile communication network.

[0125] A double-path indicator processing module is configured to adopt a double-path architecture to process the concave metric indicators and the additive metric indicators of the resource scheduling process respectively to obtain the q values of the concave metric indicators and the q values of the additive metric indicators. The concave metric indicators are the residual bandwidths on the routing paths of the service flows, and the additive metric indicators are the transmission delays on the routing paths of the service flows. The improved reinforcement learning algorithm DGQN is used to process the additive metric indicators to obtain the q values of the additive metric indicators. The improved reinforcement learning algorithm DGQN is a network obtained by improving the traditional reinforcement learning algorithm DQN based on GhostNet.

[0126] The comprehensive model q value calculation module is used to perform weighted summation of the concavity measurement index q value and the additive measurement index q value to obtain the comprehensive model q value; the comprehensive model q value is a q value representation that combines the two types of indicators.

[0127] The final routing path determination module is used to determine the final strategy according to the q value of the comprehensive model, and determine the final routing path of any business flow based on the final strategy.

[0128] The final routing path sending module is used to send the final routing path of the service flow to each node of the wireless mobile communication network to perform resource scheduling of the wireless mobile communication network.

[0129] certainly, Figure 4 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different functions. Figure 4 One or at least two components of the system shown.

[0130] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a wireless mobile communication network resource scheduling method based on DGQN provided in the above embodiment can be implemented.

[0131] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0132] In an exemplary embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0133] In an example embodiment, a computer readable storage medium storing a computer program is provided, the computer program, when executed by a processor, implements the steps of any of the above method embodiments.

[0134] In an example embodiment, a computer program product is provided, comprising a computer program, the computer program, when executed by a processor, implements the steps of any of the above method embodiments.

[0135] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0136] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, database or other medium used in each embodiment provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0137] The database involved in each of the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, and the like, without being limited thereto. The processor involved in each of the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, and the like, without being limited thereto.

[0138] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but it should be considered that any combination of the technical features is within the scope of the present disclosure, as long as there is no contradiction.

[0139] The principles and implementation modes of the present application are described by applying specific examples herein, and the above embodiments are only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, the specific implementation modes and application ranges will be changed according to the idea of the present application. In summary, the content of the present description should not be understood as a limitation of the present application.

Claims

1. A wireless mobile communication network resource scheduling method based on DGQN, characterized in that: include: Acquire the topology of the wireless mobile communication network and index data of each node in the wireless mobile communication network; the index data includes delay parameters and bandwidth parameters of each node passing through different links; Preprocessing the index data of each node in the wireless mobile communication network to obtain preprocessed index data; According to the topological structure of the wireless mobile communication network, the resource scheduling process of the wireless mobile communication network is mapped into a Markov decision process in the reinforcement learning algorithm; A dual-path architecture is adopted to process the concave metric and additive metric of the resource scheduling process respectively, obtaining a concave metric q value and an additive metric q value; the concave metric is the remaining bandwidth on the routing path of the service flow, and the additive metric is the transmission delay on the routing path of the service flow; an improved reinforcement learning algorithm DGQN is used to process the additive metric to obtain the additive metric q value; the improved reinforcement learning algorithm DGQN is a network obtained by improving the traditional reinforcement learning algorithm DQN based on Ghost Net; Performing a weighted summation on the concavity metric q value and the additive metric q value to obtain a comprehensive model q value; The q value of the comprehensive model is a q value representation that combines two types of indicators; Determine a final strategy according to the q value of the comprehensive model, and determine a final routing path of any service flow based on the final strategy; Sending the final routing path of the service flow to each node of the wireless mobile communication network to perform resource scheduling of the wireless mobile communication network; A dual-path architecture is adopted to process the concave metric and additive metric of the resource scheduling process respectively, and obtain the concave metric q value and the additive metric q value, which specifically include: The concavity measurement index is processed using a reinforcement learning algorithm Q-Learning to obtain a q value of the concavity measurement index; The additive metric is processed using an improved reinforcement learning algorithm DGQN to obtain the additive metric q value; the improved reinforcement learning algorithm DGQN is a network based on GhostNet, which uses a Ghost bottleneck structure to replace the convolution kernel in the traditional reinforcement learning algorithm DQN; The q value of the comprehensive model is calculated according to the following formula: Among them, q(s,a * ) is the comprehensive model q value, s is the state, a * is the action taken, q w (s,a * ) is the concavity metric q value, w thr is the bandwidth requirement of the service flow, q d (s,a * ) is the additive metric q value, qMin d =min(q d (s,a * )), μ is the weight of the additive metric q value, and A is the action set.

2. The wireless mobile communication network resource scheduling method based on DGQN according to claim 1, characterized in that: Preprocessing the index data of each node in the wireless mobile communication network to obtain preprocessed index data, specifically including: For any missing values ​​in the indicator data of a node, fill them with the same indicator data of the previous time frame; All indicator data after filling in the missing values ​​are normalized to obtain preprocessed indicator data.

3. The wireless mobile communication network resource scheduling method based on DGQN according to claim 1, characterized in that: The concavity metric q-value can be calculated by the following formula: Among them, q w (s,a) is the concavity metric q value of taking action a in the current state s, q w (s -1 ,a * ) is the concavity metric q value of taking action a* in the state s at the previous moment, and A is the action set.

4. The wireless mobile communication network resource scheduling method based on DGQN according to claim 1, characterized in that: The remaining bandwidth on the routing path of the service flow can be expressed as follows: M W (IT ρ )=min(w ij ),ij∈E ρ 4 Among them, M W (E p ) is the remaining bandwidth on the routing path of service flow ρ, w ij is the bandwidth parameter of link ij, E ρ is the routing path of service flow ρ.

5. The wireless mobile communication network resource scheduling method based on DGQN according to claim 1, characterized in that: The transmission delay on the routing path of the service flow can be expressed as follows: Among them, M D (E ρ ) is the transmission delay on the routing path of the service flow ρ, d ij is the delay parameter of link ij, E ρ is the routing path of service flow ρ.

6. The wireless mobile communication network resource scheduling method based on DGQN according to claim 5, characterized in that: The delay parameter of link ij can be determined by the following formula: Among them, d ij is the delay parameter of link ij, α T-1 ,…,α is the forgetting factor, T is the preset time length, is the initial delay parameter of link ij at the current moment, is the delay parameter of link ij at the previous moment.

7. The wireless mobile communication network resource scheduling method based on DGQN according to claim 6, characterized in that: Forgetting Factor α T-1 ,…,α increases gradually to reduce the impact of delay parameters far away from the current moment.

8. A wireless mobile communication network resource scheduling system based on DGQN, characterized in that: include: A communication network parameter acquisition module is used to obtain the topology of the wireless mobile communication network and the index data of each node in the wireless mobile communication network; the index data includes the delay parameters and bandwidth parameters of each node passing through different links; An indicator data preprocessing module is used to preprocess the indicator data of each node in the wireless mobile communication network to obtain preprocessed indicator data; A decision scheduling process mapping module is used to map the resource scheduling process of the wireless mobile communication network into a Markov decision process in a reinforcement learning algorithm according to the topology of the wireless mobile communication network; A dual-path indicator processing module is used to process the concave metric and additive metric of the resource scheduling process using a dual-path architecture to obtain a concave metric q value and an additive metric q value; the concave metric is the remaining bandwidth on the routing path of the service flow, and the additive metric is the transmission delay on the routing path of the service flow; the additive metric is processed using an improved reinforcement learning algorithm DGQN to obtain the additive metric q value; the improved reinforcement learning algorithm DGQN is a network obtained by improving the traditional reinforcement learning algorithm DQN based on GhostNet; A comprehensive model q-value calculation module, configured to perform a weighted summation of the concavity metric q-value and the additive metric q-value to obtain a comprehensive model q-value; The q value of the comprehensive model is a q value representation that combines two types of indicators; A final routing path determination module, configured to determine a final strategy according to the q value of the comprehensive model, and determine a final routing path for any service flow based on the final strategy; A final routing path sending module, used to send the final routing path of the service flow to each node of the wireless mobile communication network to perform resource scheduling of the wireless mobile communication network; The dual-path indicator processing module is specifically used to: The concavity measurement index is processed using a reinforcement learning algorithm Q-Learning to obtain a q value of the concavity measurement index; The additive metric is processed using an improved reinforcement learning algorithm DGQN to obtain the additive metric q value; the improved reinforcement learning algorithm DGQN is a network based on GhostNet, which uses a Ghost bottleneck structure to replace the convolution kernel in the traditional reinforcement learning algorithm DQN; The q value of the comprehensive model is calculated according to the following formula: Among them, q(s,a * ) is the comprehensive model q value, s is the state, a * is the action taken, q w (s,a * ) is the concavity metric q value, w thr is the bandwidth requirement of the service flow, q d (s,a * ) is the additive metric q value, qMin d =min(q d (s,a * )), μ is the weight of the additive metric q value, and A is the action set.

Images