Bandwidth fragment adaptive arrangement method and device based on DT algorithm

Abstract

The present invention discloses a method and device for adaptively arranging bandwidth fragments based on a DT algorithm, which belongs to the field of power system communication technology. The method includes the following steps: obtaining basic data of a target optimization network and establishing a digital twin of the target optimization network; based on the digital twin, drawing on the logic of interaction between an intelligent agent and an environment in reinforcement learning, generating a data set; preprocessing the data set to obtain a training data set; building a fragment adaptive arrangement model of the target optimization network based on the DT algorithm; using the training data set to train the fragment adaptive arrangement model; and using the trained fragment adaptive arrangement model to arrange bandwidth fragments of the target optimization network. The present invention can not only quickly and accurately reflect the state of the target network, but also adaptively handle the fragmentation problem in the network, thereby effectively improving the utilization rate of network resources and reducing the degree of network congestion.

Description

Technical Field

[0001] The present invention relates to a bandwidth fragment adaptive arrangement method and device based on a DT algorithm (Decision Transformer deep learning network), belonging to the technical field of power system communication. Background Art

[0002] As distributed renewable energy sources equipped with power electronic devices such as photovoltaic power generation systems are increasingly integrated into the power grid, the operation mechanism and balance mode of the power grid have undergone profound changes. The randomness and volatility of distributed photovoltaic systems may cause intermittent power flow problems on the medium and low voltage busbars. The frequent changes in such services may lead to improper allocation of communication network resources and serious fragmentation of bandwidth resources, which not only affects network performance but also leads to uneconomical use of resources. In order to meet these challenges, it is necessary to apply network defragmentation technology in the communication network in a timely manner to reconfigure network resources, improve the performance and reliability of the power communication network, and thus support the development of the power grid in a smarter and more efficient direction.

[0003] Bandwidth defragmentation faces many difficulties: (1) Large network scale: The scale of the power communication network is huge and the structure is complex, involving a wide geographical area and a large number of devices. The huge state space makes it extremely difficult to find the optimal solution. (2) Adaptive defragmentation is difficult: The resource allocation of the power grid needs to be dynamically adjusted according to the actual network status, requiring bandwidth defragmentation to make appropriate decisions based on the real-time network status. It also involves dynamic adjustment of multiple services, making it difficult to find a comprehensive and effective adjustment plan.

[0004] Defragmentation can be viewed as a linear programming problem. There are heuristic algorithms such as the greedy defragmentation algorithm and the shortest path defragmentation algorithm for large-scale networks to maximize the utilization of network resources. Waya Fadini et al. proposed a subcarrier slot partitioning scheme for wavelength assignment in the paper "A subcarrier-slot partition scheme for wavelength assignment in elastic optical networks". By dividing the total subcarrier slot into multiple parts and assigning wavelengths to each part based on the link used for the connection, more continuous and aligned available subcarrier slots are generated, effectively reducing the fragmentation in the network and reducing the blocking probability. Although the algorithm can improve network performance to some extent, in a large-scale network environment, a large number of training iterations are required to optimize the defragmentation strategy, which not only increases the service interruption time, but also may cause the network performance to drop sharply in the face of complex and dynamically changing network states.

[0005] In recent years, with the development of artificial intelligence technology, defragmentation algorithms based on reinforcement learning have also achieved certain results. Ruoxing Li et al. proposed a cognitive hitless spectrum defragmentation scheme based on deep reinforcement learning in the paper "Learning-based cognitive hitless spectrum defragmentation for dynamic provisioning in elastic optical networks". Through online training, the knowledge learned from previous operations is used to adaptively optimize spectrum utilization, which can continuously adapt to traffic changes, reduce fragmentation, and improve network resource utilization efficiency. At present, reinforcement learning mainly uses Markov decision process for modeling, fits value function or policy function through neural network, and uses temporal difference learning method to approximate the optimal strategy. Defragmentation generally involves the adjustment of multiple services. Each adjustment will affect the state of the network, and then affect the next decision. The effectiveness of a decision may not be reflected until many steps later. It is a typical sequential decision problem. However, traditional reinforcement learning based on Markov decision process is difficult to effectively allocate credit when dealing with such problems, and it is difficult to strike a balance between improving immediate performance and optimizing long-term benefits. Summary of the invention

[0006] Aiming to overcome the shortcomings of the existing technology, especially its limitations in processing sequential decisions in a dynamic network environment, the present invention proposes a bandwidth fragment adaptive sorting method and device based on the DT algorithm, which can continuously improve the overall resource utilization efficiency, enhance the adaptability and flexibility of the network, and efficiently optimize the configuration and use of network resources.

[0007] The technical solution adopted by the present invention to solve the technical problem is:

[0008] In a first aspect, an embodiment of the present invention provides a method for adaptively sorting bandwidth fragments based on a DT algorithm, comprising the following steps:

[0009] Obtain basic data of the target optimized network and establish a digital twin of the target optimized network, wherein the digital twin can accurately and quickly reflect the carrier signal-to-noise ratio, transmission signal power, and noise power status of the target network;

[0010] Based on digital twins, we draw on the logic of interaction between agents and environments in reinforcement learning to generate data sets;

[0011] Preprocess the data set to obtain a training data set;

[0012] Building a fragment adaptive arrangement model for target optimization network based on DT algorithm;

[0013] Using the training data set to train the fragment adaptive defragmentation model;

[0014] The trained adaptive defragmentation model is used to defragment the bandwidth of the target optimized network.

[0015] As a possible implementation of this embodiment, the digital twin-based method generates a data set by drawing on the logic of interaction between an intelligent agent and an environment in reinforcement learning, including:

[0016] Based on the digital twin, the intelligent agent randomly reallocates network resources for the pending services of the target optimization network in turn. The environment generates rewards, next states, and end flags based on the actions of the intelligent agent, and so on, to generate a trajectory; and,

[0017] Generate several trajectories, and all trajectories constitute a dataset.

[0018] As a possible implementation of this embodiment, the state includes the signal-to-noise ratio of each link carrier through which the service to be adjusted currently passes and the service transmission rate, the action is to adjust the carrier sequence number of the service, and the reward is a reduction value of the network bandwidth fragmentation degree;

[0019] The calculation formula of the network bandwidth fragmentation is:

[0020] ,

[0021] In the formula, is the bandwidth fragmentation, k is the weight coefficient, is the utilization factor, is the consistency factor, is the capacity of link i and carrier j, For Business The carrier number occupied on link i, n is the number of carriers actually transmitting services on each link, m is the number of links actually transmitting services in the network, is the service transmission rate at link i and carrier j.

[0022] As a possible implementation of this embodiment, preprocessing the data set to obtain the training data set includes:

[0023] Cut or complete the trajectories in the data set to make them of uniform length, and add a mask to indicate the time step where the valid data is located;

[0024] Normalize the state vector and reward vector so that the data is limited to [0,1];

[0025] The immediate reward at each time step is the average of all future rewards from the current moment to the end of the sequence.

[0026] As a possible implementation manner of this embodiment, the state vector includes a network state and a service state.

[0027] As a possible implementation of this embodiment, the fragment adaptive sorting model is built based on the GPT-2 architecture, and uses a multi-layer Transformer architecture to extract deep time series information and capture the relationship and change rules between states, actions and rewards between time steps in the input sequence.

[0028] As a possible implementation of this embodiment, the method of using the training data set to perform model training on the adaptive fragment defragmentation model includes:

[0029] Model training process in the forward propagation phase;

[0030] Model training process during back propagation and optimization phases.

[0031] As a possible implementation of this embodiment, the model training process in the forward propagation stage includes:

[0032] The state, action, reward, and time step are converted into embedding vectors through their respective embedding layers. The time step embedding is added to the other three embedding vectors as position information. After that, the new three embedding vectors are stacked, rearranged, and normalized to form The sequence is used as the input of Decision Transformer. is the reward vector, is the state vector, is the action vector;

[0033] The output sequence of the input layer is sent to the encoder, and is processed by the Transformer encoder such as multi-head attention mechanism, normalization, feedforward, normalization, etc., and then passes through the linear decoder to output the predicted action.

[0034] As a possible implementation of this embodiment, the model training process in the back propagation and optimization phase includes:

[0035] Calculate the cross entropy loss between the forward propagation prediction value and the true value, calculate the gradient of the model parameters after clearing the gradient information, and then adjust the model weights to reduce the loss;

[0036] After each batch of data training is completed, the learning rate is adjusted through the learning rate scheduler to optimize the training process and speed up the model convergence;

[0037] Repeat the above back-propagation and optimization phase model training process until the model converges.

[0038] In a second aspect, an embodiment of the present invention provides a bandwidth fragment adaptive arrangement device based on a DT algorithm, comprising:

[0039] A data acquisition module is used to obtain basic data of the target optimization network and establish a digital twin of the target optimization network, which can accurately and quickly reflect the carrier signal-to-noise ratio, transmission signal power and noise power status of the target network;

[0040] The data generation module is used to generate data sets based on digital twins and the logic of interaction between agents and environments in reinforcement learning;

[0041] A data processing module is used to preprocess the data set to obtain a training data set;

[0042] Model building module, used to build a fragment adaptive arrangement model of the target optimization network based on the DT algorithm;

[0043] A model training module, used for training the fragment adaptive defragmentation model using a training data set;

[0044] The defragmentation module is used to perform bandwidth defragmentation on the target optimization network by using the trained fragmentation adaptive defragmentation model.

[0045] As a possible implementation of this embodiment, the data generation module includes:

[0046] The trajectory generation module is used to randomly reallocate network resources for the pending services of the target network based on the digital twin. The environment generates rewards, next states, and end flags based on the actions of the agents, and a trajectory is generated in this cycle.

[0047] The dataset composition module is used to compose all trajectories generated by the trajectory generation module into a dataset.

[0048] As a possible implementation of this embodiment, the state includes the signal-to-noise ratio of each link carrier through which the service to be adjusted currently passes and the service transmission rate, the action is to adjust the carrier sequence number of the service, and the reward is a reduction value of the network bandwidth fragmentation degree;

[0049] The calculation formula of the network bandwidth fragmentation is:

[0050] ,

[0051] In the formula, is the bandwidth fragmentation, k is the weight coefficient, is the utilization factor, is the consistency factor, is the capacity of link i and carrier j, For Business The carrier number occupied on link i, n is the number of carriers actually transmitting services on each link, m is the number of links actually transmitting services in the network, is the service transmission rate at link i and carrier j.

[0052] As a possible implementation of this embodiment, the data processing module includes:

[0053] The trajectory processing module is used to cut or complete the trajectories in the data set to make them of uniform length, and add a mask to indicate the time step of the valid data;

[0054] The normalization module is used to normalize the state vector and reward vector so that the data is limited to [0,1];

[0055] The immediate reward module is used to take the average of all future rewards from the current moment to the end of the sequence as the immediate reward for each time step.

[0056] As a possible implementation manner of this embodiment, the state vector includes a network state and a service state.

[0057] As a possible implementation of this embodiment, the fragment adaptive sorting model is built based on the GPT-2 architecture, and uses a multi-layer Transformer architecture to extract deep time series information and capture the relationship and change rules between states, actions and rewards between time steps in the input sequence.

[0058] As a possible implementation of this embodiment, the model training module includes:

[0059] Forward training module, used for model training process in the forward propagation stage;

[0060] The reverse training module is used for the model training process in the back-propagation and optimization stages.

[0061] As a possible implementation of this embodiment, the forward training module is specifically used to:

[0062] The state, action, reward, and time step are converted into embedding vectors through their respective embedding layers. The time step embedding is added to the other three embedding vectors as position information. After that, the new three embedding vectors are stacked, rearranged, and normalized to form The sequence is used as the input of Decision Transformer. is the reward vector, is the state vector, is the action vector;

[0063] The output sequence of the input layer is sent to the encoder, and is processed by the Transformer encoder such as multi-head attention mechanism, normalization, feedforward, normalization, etc., and then passes through the linear decoder to output the predicted action.

[0064] As a possible implementation of this embodiment, the reverse training module is specifically used to:

[0065] Calculate the cross entropy loss between the forward propagation prediction value and the true value, calculate the gradient of the model parameters after clearing the gradient information, and then adjust the model weights to reduce the loss;

[0066] After each batch of data training is completed, the learning rate is adjusted through the learning rate scheduler to optimize the training process and speed up the model convergence;

[0067] Repeat the above back-propagation and optimization phase model training process until the model converges.

[0068] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory through the bus, and the processor executes the machine-readable instructions to perform any step of the above-mentioned bandwidth fragmentation adaptive defragmentation method based on the DT algorithm.

[0069] In a fourth aspect, an embodiment of the present invention provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for adaptively defragmenting bandwidth fragments based on the DT algorithm are executed.

[0070] The beneficial effects of the technical solution of the embodiment of the present invention are as follows:

[0071] A method for adaptively arranging bandwidth fragments based on a DT algorithm according to the technical solution of an embodiment of the present invention comprises the following steps: obtaining basic data of a target optimized network, and establishing a digital twin of the target optimized network, wherein the digital twin can accurately and quickly reflect the carrier signal-to-noise ratio, transmission signal power, and noise power state of the target network; based on the digital twin, drawing on the logic of the interaction between the agent and the environment in reinforcement learning, a data set is generated; the data set is preprocessed to obtain a training data set; a fragment adaptive arrangement model of the target optimized network is built based on the DT algorithm; the fragment adaptive arrangement model is trained using the training data set; and the bandwidth fragments of the target optimized network are arranged using the trained fragment adaptive arrangement model. By combining the advantages of the DT algorithm and the digital twin, the present invention reallocates network resources for services with unreasonable resource allocation in the network in real time according to the network status, which can not only quickly and accurately reflect the state of the target network, but also adaptively handle the fragmentation problem in the network, thereby effectively improving the utilization rate of network resources and reducing the degree of network congestion.

[0072] Aiming to overcome the shortcomings of the existing technology, especially its limitations in processing sequential decisions in a dynamic network environment, the present invention introduces the DT algorithm and proposes a new bandwidth fragmentation adaptive defragmentation method, which can not only respond to changes in network status in real time and autonomously learn the optimal defragmentation strategy, but also find a long-term good sequential strategy, thereby continuously improving the overall resource utilization efficiency, enhancing the adaptability and flexibility of the network, and efficiently optimizing the configuration and use of network resources.

[0073] The device for adaptively arranging bandwidth fragments based on the DT algorithm according to the technical solution of the embodiment of the present invention has the same beneficial effects as the method for adaptively arranging bandwidth fragments based on the DT algorithm according to the technical solution of the embodiment of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 is a flow chart of a method for adaptively sorting bandwidth fragments based on a DT algorithm according to an exemplary embodiment;

[0075] Figure 2 The diagram is a schematic structural diagram of a device for adaptively sorting bandwidth fragments based on a DT algorithm according to an exemplary embodiment. DETAILED DESCRIPTION

[0076] In order to more clearly illustrate the technical features of the solution of the present invention, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0077] like Figure 1 As shown, an embodiment of the present invention provides a method for adaptively sorting bandwidth fragments based on a DT algorithm, comprising the following steps:

[0078] Acquire basic data of the target optimization network and establish a digital twin of the target optimization network, wherein the digital twin can accurately and quickly reflect the carrier signal-to-noise ratio (SNR), transmission signal power, and noise power status of the target network;

[0079] Based on digital twins, we draw on the logic of interaction between agents and environments in reinforcement learning to generate data sets;

[0080] Preprocess the data set to obtain a training data set;

[0081] Building a fragment adaptive arrangement model for target optimization network based on DT algorithm;

[0082] Using the training data set to train the fragment adaptive defragmentation model;

[0083] The trained adaptive defragmentation model is used to defragment the bandwidth of the target optimized network.

[0084] As a possible implementation of this embodiment, the digital twin-based method generates a data set by drawing on the logic of interaction between an intelligent agent and an environment in reinforcement learning, including:

[0085] Based on digital twins, the intelligent agents randomly optimize the network resources for the services to be sorted out in turn, and the environment adjusts according to the actions of the intelligent agents. , generate rewards , next state and end mark , and so on until True, get a trajectory;

[0086] Repeat the above steps to generate a certain number of trajectories, and all trajectories constitute a data set.

[0087] As a possible implementation of this embodiment, the state includes the signal-to-noise ratio of each link carrier through which the service to be adjusted currently passes and the service transmission rate, the action is to adjust the carrier sequence number of the service, and the reward is a reduction value of the network bandwidth fragmentation degree;

[0088] The setting of the state space needs to fully reflect the current network status, while ensuring that the state space is not too complicated to avoid problems such as difficulty in feature extraction. The network status should include the capacity and configuration of each carrier, as well as the transmission rate of the service and the influence of other external environmental factors. Since bandwidth fragments can be sorted out by reallocating carriers for services, and the adjustment of a single service only affects the status of other service carriers in the same link, the network status does not need to cover all links in the entire network, but only needs to include the carrier status of the link where the service to be adjusted is currently located. The carrier status includes transmission signal power, SNR, capacity, etc. Since the carrier selection of the service is affected by SNR, the transmission signal power is included in SNR, and the carrier capacity is determined by SNR, in order to simplify the representation, the network status only includes the SNR of all carriers of the link where the service to be adjusted is currently located. Since Transformer requires the same dimension of input data, and the number of links occupied by each service is uncertain, the maximum number of links occupied by the service M should be counted or estimated first, and the first part of the state space should be set to a fixed length M. When the number of links occupied by the service is less than M, the state value is set to 0. The service status includes information such as service path, transmission rate, start time, and duration. Combining the network status and service status into a state vector can provide basic network information for current decision-making.

[0089] The action space is defined as a representation of the adjustment scheme for reallocation of network resources. The dimension of the action space is determined by the number of types of network resources involved, and the value range of each dimension depends on the specific characteristics of various network resources. The present invention only considers adjusting the carrier of the service, so the action space is a one-dimensional vector:

[0090] ,

[0091] Wherein, N is the maximum carrier number.

[0092] The reward function provides a numerical evaluation of a certain adjustment action taken in the current state, reflecting the effect of bandwidth defragmentation. If the selected solution does not support the transmission of the current service to be defragmented, the reward value , to avoid unreasonable decisions and improve the compliance and effectiveness of decisions; if the selected solution supports the transmission of the current business, the bandwidth fragmentation of the network before and after the action is executed is calculated respectively and , reward value The network bandwidth fragmentation degree is calculated as follows:

[0093] ,

[0094] In the formula, is the bandwidth fragmentation, k is the weight coefficient, is the utilization factor, is the consistency factor, is the capacity of link i and carrier j, For Business The carrier number occupied on link i, n is the number of carriers actually transmitting services on each link, m is the number of links actually transmitting services in the network, is the service transmission rate at link i and carrier j.

[0095] The logic of the interaction function between the environment and the agent is as follows: First, determine the sequence of services to be sorted. Select the first service to be sorted, and the agent randomly selects a variable in the action space to check whether the selected solution meets the conditions for transmitting the current service. If not, the reward value is -100 and the end flag is set to True; otherwise, the corresponding network resources are allocated to the service, the network status is updated, and the corresponding reward value is calculated, and the end flag is set to False. If the service is the last service in the service sequence, the end flag is set to True; otherwise, the current service to be sorted is updated to the next service in the sequence, and the above steps are repeated. If the end flag is True, a trajectory is generated. Repeat the above steps until the expected number of trajectories are generated.

[0096] As a possible implementation of this embodiment, preprocessing the data set to obtain the training data set includes:

[0097] Cut or complete the trajectories in the data set to make them of uniform length, and add a mask to indicate the time step where the valid data is located;

[0098] Normalize the state vector and reward vector so that the data is limited to [0,1];

[0099] The immediate reward at each time step is the average of all future rewards from the current moment to the end of the sequence.

[0100] During the preprocessing process of each trajectory of the data set. Since the Decision Transformer requires the input data to be of the same size, it is necessary to cut or complete the trajectory to a uniform length, and add a mask to indicate the time step where the valid data is located; the state vector and reward vector are normalized so that the data is limited to a certain range, thereby eliminating the adverse effects caused by singular sample data; in order to evaluate the long-term effectiveness of the action, the average value of the future cumulative rewards of each time step is calculated, that is, the average value of all rewards from the current moment to the end of the sequence. At this point, the model can evaluate the long-term effectiveness of taking the action in the current state. At the same time, the cumulative reward value is independent of the sequence length. The model can handle decision problems of any sequence length, balance the reward values ​​of sequences of different lengths, and enhance the scalability of the model.

[0101] As a possible implementation of this embodiment, the fragment adaptive sorting model is built based on the GPT-2 architecture, and uses a multi-layer Transformer architecture to extract deep time series information and capture the relationship and change rules between states, actions and rewards between time steps in the input sequence.

[0102] The fragmentation adaptive defragmentation model built based on the Decision Transformer includes the input layer, codec, and output layer designs. The Decision Transformer model is based on the GPT-2 architecture and adopts an N-layer Transformer architecture. Each layer of the Transformer contains a multi-head attention mechanism, a feedforward neural network, and corresponding normalization functions and activation functions to capture the relationship and change rules between each time step in the input sequence. In order to adapt to sequential decision-making problems, this architecture removes position embedding and adds time step embedding compared to the traditional GPT-2 model, so that the model can more effectively extract information that is key to the decision-making process from the input data, thereby improving the performance and efficiency of the model in the process of defragmenting bandwidth fragments.

[0103] As a possible implementation of this embodiment, the method of using the training data set to perform model training on the adaptive fragment defragmentation model includes:

[0104] Model training process in the forward propagation phase;

[0105] Model training process during back propagation and optimization phases.

[0106] As a possible implementation of this embodiment, the model training process in the forward propagation stage includes:

[0107] The state, action, reward, and time step are converted into embedding vectors through their respective embedding layers. The time step embedding is added to the other three embedding vectors as position information. After that, the new three embedding vectors are stacked, rearranged, and normalized to form The sequence is used as the input of Decision Transformer. is the reward vector, is the state vector, is the action vector;

[0108] The output sequence of the input layer is sent to the encoder, and is processed by the Transformer encoder such as multi-head attention mechanism, normalization, feedforward, normalization, etc., and then passes through the linear decoder to output the predicted action.

[0109] In the forward propagation phase, the input layer mainly performs data embedding, converting the state, action, reward, and time step into embedding vectors through their respective embedding layers. The time step embedding is added to the other three embedding vectors as position information. After that, the new three embedding vectors are stacked, rearranged, and normalized to form The output sequence of the input layer is sent to the encoder, which is processed by the Transformer encoder such as multi-head attention mechanism, normalization, feedforward, normalization, etc., and then passes through the linear decoder to output the predicted action.

[0110] As a possible implementation of this embodiment, the model training process in the back propagation and optimization phase includes:

[0111] Calculate the cross entropy loss between the forward propagation prediction value and the true value, calculate the gradient of the model parameters after clearing the gradient information, and then adjust the model weights to reduce the loss;

[0112] After each batch of data training is completed, the learning rate is adjusted through the learning rate scheduler to optimize the training process and speed up the model convergence;

[0113] Repeat the above back-propagation and optimization phase model training process until the model converges.

[0114] In the back propagation and optimization phase, the model first calculates the cross entropy loss between the forward propagation predicted value and the true value, calculates the gradient of the model parameters after clearing the gradient information, and then adjusts the model weights to reduce the loss, thereby improving the future performance of the model. In addition, after each batch of data training is completed, the learning rate is adjusted through the learning rate scheduler to optimize the training process, speed up the model convergence, and improve the final performance of the model. The above process is repeated many times until the model converges.

[0115] like Figure 2 As shown, an embodiment of the present invention provides a bandwidth fragment adaptive arrangement device based on a DT algorithm, comprising:

[0116] A data acquisition module is used to obtain basic data of the target optimization network and establish a digital twin of the target optimization network, which can accurately and quickly reflect the carrier signal-to-noise ratio, transmission signal power and noise power status of the target network;

[0117] The data generation module is used to generate data sets based on digital twins and the logic of interaction between agents and environments in reinforcement learning;

[0118] A data processing module is used to preprocess the data set to obtain a training data set;

[0119] Model building module, used to build a fragment adaptive arrangement model of the target optimization network based on the DT algorithm;

[0120] A model training module, used for training the fragment adaptive defragmentation model using a training data set;

[0121] The defragmentation module is used to perform bandwidth defragmentation on the target optimization network by using the trained fragmentation adaptive defragmentation model.

[0122] As a possible implementation of this embodiment, the data generation module includes:

[0123] The trajectory generation module is used to randomly reallocate network resources for the pending services of the target network based on the digital twin. The environment generates rewards, next states, and end flags based on the actions of the agents, and a trajectory is generated in this cycle.

[0124] The dataset composition module is used to compose all trajectories generated by the trajectory generation module into a dataset.

[0125] As a possible implementation of this embodiment, the state includes the signal-to-noise ratio of each link carrier through which the service to be adjusted currently passes and the service transmission rate, the action is to adjust the carrier sequence number of the service, and the reward is a reduction value of the network bandwidth fragmentation degree;

[0126] The calculation formula of the network bandwidth fragmentation is:

[0127] ,

[0128] In the formula, is the bandwidth fragmentation, k is the weight coefficient, is the utilization factor, is the consistency factor, is the capacity of link i and carrier j, For Business The carrier number occupied on link i, n is the number of carriers actually transmitting services on each link, m is the number of links actually transmitting services in the network, is the service transmission rate at link i and carrier j.

[0129] As a possible implementation of this embodiment, the data processing module includes:

[0130] The trajectory processing module is used to cut or complete the trajectories in the data set to make them of uniform length, and add a mask to indicate the time step of the valid data;

[0131] The normalization module is used to normalize the state vector and reward vector so that the data is limited to [0,1];

[0132] The immediate reward module is used to take the average of all future rewards from the current moment to the end of the sequence as the immediate reward for each time step.

[0133] As a possible implementation manner of this embodiment, the state vector includes a network state and a service state.

[0134] As a possible implementation of this embodiment, the fragment adaptive sorting model is built based on the GPT-2 architecture, and uses a multi-layer Transformer architecture to extract deep time series information and capture the relationship and change rules between states, actions and rewards between time steps in the input sequence.

[0135] As a possible implementation of this embodiment, the model training module includes:

[0136] Forward training module, used for model training process in the forward propagation stage;

[0137] The reverse training module is used for the model training process in the back-propagation and optimization stages.

[0138] As a possible implementation of this embodiment, the forward training module is specifically used to:

[0139] The state, action, reward, and time step are converted into embedding vectors through their respective embedding layers. The time step embedding is added to the other three embedding vectors as position information. After that, the new three embedding vectors are stacked, rearranged, and normalized to form The sequence is used as the input of Decision Transformer. is the reward vector, is the state vector, is the action vector;

[0140] The output sequence of the input layer is sent to the encoder, and is processed by the Transformer encoder such as multi-head attention mechanism, normalization, feedforward, normalization, etc., and then passes through the linear decoder to output the predicted action.

[0141] As a possible implementation of this embodiment, the reverse training module is specifically used to:

[0142] Calculate the cross entropy loss between the forward propagation prediction value and the true value, calculate the gradient of the model parameters after clearing the gradient information, and then adjust the model weights to reduce the loss;

[0143] After each batch of data training is completed, the learning rate is adjusted through the learning rate scheduler to optimize the training process and speed up the model convergence;

[0144] Repeat the above back-propagation and optimization phase model training process until the model converges.

[0145] In the embodiment of the present invention, taking the arrangement of bandwidth fragments in a distributed photovoltaic network as an example, the specific process of adaptive arrangement of bandwidth fragments is as follows.

[0146] 1. Establish a digital twin of the target optimization network.

[0147] Initialize the network environment and services. The digital twin needs to be able to accurately simulate the state of the multi-band optical network and quickly and accurately update the state of each link in the network according to the service distribution. Use the random-fit RWA method to randomly allocate a number of services. Allocating or releasing services requires timely updating the carrier status of each link involved, and updating the bandwidth utilization of each carrier and each service.

[0148] 2. Based on the digital twin, we draw on the logic of interaction between the intelligent agent and the environment in reinforcement learning to generate a data set.

[0149] Select services with bandwidth utilization rates lower than 30% from the existing services in the network to form a sequence of services to be sorted, and take out one service from the sequence in turn as the current service to be sorted. The state space is the generalized signal-to-noise ratio of each carrier of the link involved in the service, as well as the status of the current service. The service status includes information such as service path, transmission rate, start time, duration, etc., among which the service path information is already included in the network status; to simplify the problem, this embodiment only involves immediate service adjustment, and does not consider the start time and duration. Therefore, the service status only needs to include the service transmission rate. Since Transformer requires the same dimension of input data and the number of links occupied by each service is uncertain, the maximum number of links M occupied by the service should be counted or estimated first, and the first part of the state space should be set to a fixed length of M. When the number of links occupied by the service is less than M, the state value is set to 0. According to statistics, the maximum number of links for each service in the network is 10, so the state space for:

[0150] ,

[0151] Among them, SNR (i,j) represents the SNR of the jth carrier on the i-th link. If the number of links m for the current service is less than 10, then SNR (m+1,1) To SNR (10,80)are all set to 0. Then, randomly select an action from the action space to determine whether the action satisfies the carrier allocation constraints such as carrier uniqueness constraint, carrier non-repetition constraint, carrier consistency constraint, and carrier capacity constraint. If so, adjust the current service to the carrier, update the network status, calculate the reward value corresponding to the action, set the end flag to False, and add (network status before adjustment, action, reward, end flag) to the current trajectory; otherwise, keep the carrier of the service unchanged, and the reward value is -100. Set the current service to the next service in the service sequence. If the current service is already the last service to be sorted, set the end flag to True. Repeat the above steps until the end flag is False, and a trajectory is generated. Repeat the above steps to generate a specified number of trajectories as the initial data set.

[0152] 3. Preprocess the data set.

[0153] First, each trajectory is uniformly cut or padded to 50 time steps, and the corresponding mask is generated to represent the valid time steps. Then the reward value is normalized. According to statistics, more than 99% of the reward values ​​for successful business allocation are between [-10, 20]. This reward value is first constrained to [-10, 20], and then mapped to [0, 1]. Finally, the immediate reward of each time step is changed to the average of all rewards from the current time step to the end of the sequence. Then the state is normalized. The link state is constrained to [0, 30] and then mapped to [0, 1]. Since this example only involves two modulation formats, QKSP and 16-QAM, which support business transmission rates of 400Gbps and 800Gbps respectively, the business state is set to 1 when the business transmission rate is greater than 400Gbps, otherwise it is set to 0. After all data sets are preprocessed as above, they are used as data sets for training bandwidth defragmentation models.

[0154] 4. Build a fragment adaptive sorting model for the target optimization network based on the DT algorithm.

[0155] A network bandwidth fragmentation adaptive management model is built based on the Decision Transformer. The Transformer part is based on the GPT-2 architecture, which extracts deep time series information and accurately captures the relationship and change rules between states, actions, and rewards between time steps in the input sequence.

[0156] 5. Use the training data set to train the fragment adaptive defragmentation model.

[0157] The model first performs data embedding, converting the state, action, reward, and time step into an embedding vector through the embedding layer. The size of the embedding vector is uniformly set to 500. The state vector, action vector, reward vector, and time step vector are all mapped to the embedding layer vector through their respective linear layers. Then the time step vector is added after the state, action, and reward embedding vectors. The new three vectors are stacked, rearranged, and normalized to form a shape like The sequence is used to adapt the input of the DecisionTransformer model. After that, the sequence is used as the input of the encoder, and the multi-head attention mechanism is used to extract features, and the two layers of feedforward fully connected layers are used to enhance the model's fitting ability. The normalization layer is used to normalize the values ​​of the features so that the values ​​of the features are within a reasonable range to prevent the increase in the number of network layers from causing the parameters to be too large or too small, which in turn causes the learning process to be abnormal and the model converges slowly. After residual processing, some neurons in the network are randomly stopped to prevent overfitting. Finally, the extracted features are processed by the linear decoder and the Softmax layer to obtain the score of each carrier selection, and the action with the highest score and the score of each action are returned. After receiving the score of each action, the trainer uses the cross entropy loss function to calculate the loss value of the score and the actual action, and updates the model parameters through back propagation.

[0158] 6. Use the trained fragmentation adaptive defragmentation model to defragment the bandwidth of the target optimization network.

[0159] The trained fragmentation adaptive defragmentation model is used to defragment the bandwidth of the target optimization network, and the effectiveness of the model is evaluated by reconfiguring network resources. The model is set to evaluation mode and the target cumulative reward value is set. The model creates an evaluation function based on the target cumulative reward value, selects the best carrier adjustment action for the current service to be defragmented according to the network status and the evaluation function, executes the action in the environment, calculates the actual reward, and adjusts the action generation strategy according to the target reward value and the actual reward value. Repeat the above steps until the carrier reconfiguration of all services to be defragmented is completed, and the total reward value of the model is output to evaluate the model. After that, the model is tested in the digital twin for reducing bandwidth fragmentation and blocking rate. Specifically, the bandwidth utilization of the network is monitored in real time. When the bandwidth utilization is lower than 45% or the new service is blocked, the bandwidth defragmentation algorithm is executed to observe the changes in the bandwidth utilization and blocking rate of the network. If the test is passed, this algorithm can be used in the optimization process of the actual network.

[0160] The present invention can not only respond to changes in network status in real time and autonomously learn the optimal defragmentation strategy, but also find a long-term good sequential strategy, thereby continuously improving the overall resource utilization efficiency, enhancing the adaptability and flexibility of the network, and efficiently optimizing the configuration and use of network resources.

[0161] Compared with the prior art, the present invention has the following characteristics:

[0162] 1. Use Decision Transformer to capture the strong correlation between network bandwidth fragmentation and resource allocation. By observing the input layer vector space, the model can quickly and effectively perceive changes in network status, and adaptively select actions to adjust services according to the network status to minimize network bandwidth fragmentation and improve network resource utilization efficiency. Compared with heuristic algorithms, the present invention can dynamically perceive the state change rules of large-scale networks and make better response strategies.

[0163] 2. The immediate reward of each step is changed to the average of the future cumulative rewards in the trajectory, so that the model is not limited by the length of the sequence, accurately learns the impact of a certain action on future rewards, and can more accurately evaluate and optimize each decision. Compared with the reinforcement learning algorithm based on the Markov decision process, the present invention can better deal with complex sequential decision problems that are highly dependent on historical decision sequences; compared with the traditional Decision Transformer, the present invention can eliminate the influence of sequence length, allowing the model to deal with sequential decision problems of different lengths, and improve the scalability of the model.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A bandwidth fragmentation adaptive arrangement method based on the Decision Transformer algorithm, characterized in that: include: Obtain the basic data of the target optimization network and establish a digital twin of the target optimization network; Based on the digital twin, the intelligent agent randomly reallocates network resources for the pending services of the target optimization network in turn. The environment generates rewards, next states, and end flags based on the actions of the intelligent agent, and this cycle generates a trajectory. as well as, Generate several trajectories, and all trajectories form a data set; Preprocess the data set to obtain a training data set; A fragment adaptive sorting model based on the Decision Transformer algorithm to build a target optimization network; Using the training data set to train the fragment adaptive defragmentation model; Utilize the trained adaptive defragmentation model to defragment the bandwidth of the target optimization network; The method of using the training data set to perform model training on the fragment adaptive defragmentation model includes: Model training process in the forward propagation phase; Model training process during back propagation and optimization phases; The model training process in the forward propagation stage includes: The state, action, reward, and time step are converted into embedding vectors through their respective embedding layers. The time step embedding is added to the other three embedding vectors as position information. After that, the new three embedding vectors are stacked, rearranged, and normalized to form The sequence is used as the input of Decision Transformer. is the reward vector, is the state vector, is the action vector; The output sequence of the input layer is sent to the encoder, which goes through the multi-head attention mechanism, normalization, feedforward, normalization, and then passes through the linear decoder to output the predicted action; The model training process in the back propagation and optimization phase includes: Calculate the cross entropy loss between the forward propagation prediction value and the true value, calculate the gradient of the model parameters after clearing the gradient information, and then adjust the model weights to reduce the loss; After each batch of data training is completed, the learning rate is adjusted through the learning rate scheduler to optimize the training process and speed up the model convergence; Repeat the above back-propagation and optimization phase model training process until the model converges.

2. The method for adaptively arranging bandwidth fragments based on the Decision Transformer algorithm according to claim 1, characterized in that: The state includes the signal-to-noise ratio of each link carrier through which the service to be adjusted currently passes and the service transmission rate, the action is to adjust the carrier sequence number of the service, and the reward is the reduction value of the network bandwidth fragmentation; The calculation formula of the network bandwidth fragmentation is: ， In the formula, is the bandwidth fragmentation, k is the weight coefficient, is the utilization factor, is the consistency factor, is the capacity of link i and carrier j, For Business The carrier number occupied on link i, n is the number of carriers actually transmitting services on each link, m is the number of links actually transmitting services in the network, is the service transmission rate at link i and carrier j.

3. The method for adaptively arranging bandwidth fragments based on the Decision Transformer algorithm according to claim 1, characterized in that: The preprocessing of the data set to obtain a training data set includes: Cut or complete the trajectories in the data set to make them of uniform length, and add a mask to indicate the time step where the valid data is located; Normalize the state vector and reward vector so that the data is limited to [0,1]; The immediate reward at each time step is the average of all future rewards from the current moment to the end of the sequence.

4. The method for adaptively arranging bandwidth fragments based on the Decision Transformer algorithm according to claim 1, characterized in that: The fragment adaptive defragmentation model is built based on the GPT-2 architecture and uses a multi-layer Transformer architecture to extract deep time series information and capture the relationship and change rules between states, actions and rewards between time steps in the input sequence.

5. A bandwidth fragmentation adaptive arrangement device based on the Decision Transformer algorithm, characterized in that: include: Data acquisition module, used to obtain basic data of the target optimization network and establish a digital twin of the target optimization network; The trajectory generation module is used to randomly reallocate network resources for the pending services of the target optimized network based on the digital twin. The environment generates rewards, next states, and end flags according to the actions of the agents to generate a trajectory. A data set composition module is used to compose all trajectories generated by the trajectory generation module into a data set; A data processing module is used to preprocess the data set to obtain a training data set; Model building module, used to build a fragment adaptive sorting model of the target optimization network based on the Decision Transformer algorithm; A model training module, used for training the fragment adaptive defragmentation model using a training data set; A defragmentation module, used for defragmenting bandwidth of a target optimization network using a trained fragmentation adaptive defragmentation model; The model training module includes: The forward training module is used for the model training process in the forward propagation stage; Backward training module, used for the model training process in the back-propagation and optimization stages; The forward training module is specifically used for: The state, action, reward, and time step are converted into embedding vectors through their respective embedding layers. The time step embedding is added to the other three embedding vectors as position information. After that, the new three embedding vectors are stacked, rearranged, and normalized to form The sequence is used as the input of Decision Transformer. is the reward vector, is the state vector, is the action vector; The output sequence of the input layer is sent to the encoder, which goes through the multi-head attention mechanism, normalization, feedforward, normalization, and then passes through the linear decoder to output the predicted action; The reverse training module is specifically used for: Calculate the cross entropy loss between the forward propagation prediction value and the true value, calculate the gradient of the model parameters after clearing the gradient information, and then adjust the model weights to reduce the loss; After each batch of data training is completed, the learning rate is adjusted through the learning rate scheduler to optimize the training process and speed up the model convergence; Repeat the above back-propagation and optimization phase model training process until the model converges.

6. The bandwidth fragmentation adaptive arrangement device based on the Decision Transformer algorithm according to claim 5, characterized in that: The state includes the signal-to-noise ratio of each link carrier through which the service to be adjusted currently passes and the service transmission rate, the action is to adjust the carrier sequence number of the service, and the reward is the reduction value of the network bandwidth fragmentation; The calculation formula of the network bandwidth fragmentation is: ， In the formula, is the bandwidth fragmentation, k is the weight coefficient, is the utilization factor, is the consistency factor, is the capacity of link i and carrier j, For Business The carrier number occupied on link i, n is the number of carriers actually transmitting services on each link, m is the number of links actually transmitting services in the network, is the service transmission rate at link i and carrier j.

7. The bandwidth fragmentation adaptive arrangement device based on the Decision Transformer algorithm according to claim 5, characterized in that: The data processing module comprises: The trajectory processing module is used to cut or complete the trajectories in the data set to make them of uniform length, and add a mask to indicate the time step of the valid data; The normalization module is used to normalize the state vector and reward vector so that the data is limited to [0,1]; The immediate reward module is used to take the average of all future rewards from the current moment to the end of the sequence as the immediate reward for each time step.

8. The bandwidth fragmentation adaptive arrangement device based on the Decision Transformer algorithm according to claim 5, characterized in that: The fragment adaptive defragmentation model is built based on the GPT-2 architecture and uses a multi-layer Transformer architecture to extract deep time series information and capture the relationship and change rules between states, actions and rewards between time steps in the input sequence.

Images