Double-agent-based resource allocation method in space division multiplexing quantum optical network

Abstract

The invention discloses a double-agent-based resource allocation method in a space division multiplexing quantum optical network. The method is suitable for realizing cooperative transmission of classical optical signals and quantum key distribution signals in a multi-core optical network. The method mainly solves the problem that high-performance intelligent cooperative transmission is difficult to realize due to noise influence and resource competition in the common-fiber transmission process of classical signals and quantum signals in a multi-core optical network. A state representation method combining service requirements and environment information is constructed, a unified reward function considering classic signal transmission performance and quantum signal transmission performance at the same time is introduced, and a multi-agent near-end strategy optimization algorithm is adopted for joint training, so that a resource allocation algorithm based on double agents is realized; dynamic collaborative resource allocation of classical signals and quantum signals is realized, so that the network performance can be optimized, the network blocking rate is reduced, the transmission performance of the two signals is improved, and theoretical research and practical application of a space division multiplexing quantum optical network are promoted.

Description

Technical Field

[0001] This invention relates to the field of quantum communication, and more particularly to a space-division multiplexing quantum optical network using quantum key distribution. The method primarily addresses resource allocation for the coordinated transmission of classical and quantum signals in multi-core fiber optic networks. It establishes a dual-agent reinforcement learning model to intelligently allocate resources such as fiber cores and spectrum, effectively reducing the impact of physical layer noise and service blocking rates, and improving overall network resource utilization and the coordinated transmission performance of classical and quantum signals. Background Technology

[0002] With the continuous growth of data traffic, optical networks are constantly evolving towards higher capacity and greater security. Multi-core optical fibers, as an important form of spatial multiplexing, are a key technology supporting high-capacity transmission; meanwhile, quantum key distribution (QKD) can provide information-theoretic security guarantees. Therefore, the integration of these two technologies is considered an important direction for building future high-performance optical communication networks. However, achieving the coordinated transmission of classical and quantum signals in multi-core optical fibers still faces many challenges.

[0003] Cooperative transmission is a transmission technology that collaboratively enhances the transmission performance of classical and quantum channels. It achieves this by establishing precise coupling theoretical models to evaluate system performance, designing suitable network architectures, and proposing efficient resource allocation schemes. Compared to studies that simply allow the two signals to coexist, cooperative transmission employs denser wavelength division multiplexing (WDM) and space division multiplexing (SDM) methods to better achieve collaboration between network communication and encryption capabilities. However, in the cooperative transmission of quantum and classical signals, the performance of both is difficult to guarantee simultaneously. On one hand, quantum signals have extremely low power and are highly susceptible to interference from classical signals and other factors, leading to a decline in transmission quality. On the other hand, classical signals themselves can experience performance fluctuations due to uneven resource allocation and noise interference. Furthermore, there is a contradiction in the utilization and allocation of resources for both signals: if the supply of quantum resources exceeds demand, node devices will be wasted; if the supply is insufficient, encryption may fail, reducing communication security. These issues collectively make it difficult to achieve a balance between overall network performance.

[0004] Existing research largely employs heuristic resource allocation methods, using fixed rules to reduce interference and improve utilization. However, resource allocation is inherently a dynamic, high-dimensional optimization problem involving various metrics such as blocking rate, key rate, and network throughput. Traditional methods lack adaptability and flexibility, making it difficult to achieve effective optimization in complex environments. In recent years, deep reinforcement learning (DRL) has demonstrated certain advantages in optical network resource allocation, but existing research mostly focuses on single scenarios. How to utilize reinforcement learning techniques to achieve collaborative optimization allocation of classical and quantum signals in multi-core fiber optic networks is a hot research topic. Summary of the Invention

[0005] This invention proposes a resource allocation method based on a dual-agent architecture for the coordinated transmission of classical and quantum signals in multi-core optical fibers. The method designs two agents, one responsible for the resource allocation of classical signals and the other for quantum signals, aiming to improve network coordination performance, reduce blocking rates, and collaboratively improve the transmission quality of both quantum and classical signals. It mainly includes three key technical points:

[0006] 1. A state representation method for multi-core optical fiber networks is proposed, which can integrate service information, network resource information and noise interference information to provide effective environmental feature input for intelligent agents;

[0007] 2. A unified reward function for classical and quantum cooperative transmission is proposed to achieve joint measurement of classical and quantum performance indicators, and a penalty mechanism is introduced when resource allocation fails, thereby guiding the agent to form a cooperative optimization strategy.

[0008] 3. A dual-agent resource allocation algorithm based on reinforcement learning is proposed. It adopts a collaborative decision-making approach between an agent responsible for classical signals and an agent responsible for quantum signals to dynamically allocate the core and spectrum resources of multi-core optical fibers. This algorithm can effectively suppress interference noise, improve resource utilization, and enhance network robustness in complex environments.

[0009] The first technical point is explained in detail below:

[0010] A state representation method for multi-core optical networks is proposed. This method provides effective input features for the decision-making process of dual agents by comprehensively describing service information, network resource status, and noise interference, enabling them to accurately perceive the environment and make reasonable resource allocation decisions.

[0011] The main idea of ​​this method is to abstract a multi-core fiber optic network as a graph structure G(V,E), and transform each arriving service request and its current network occupancy state into a state vector. This facilitates the agent's reading and understanding of global information. The spectrum is divided into two functional regions: a classical service region and a cooperative transmission region, corresponding to the different transmission ranges of classical and quantum signals. The overall information required for the state includes: service information, environmental information, and additional information. This information is transmitted through S... q S e S a In the graph structure G(V,E), an idle frequency slot can be located using data in three dimensions: link i, fiber core j, and frequency slot k.

[0012] S={S q ,{S e ,S a}| i,j,k} (1)

[0013] Business Information S q The description outlines the basic characteristics of the service to be processed, including source node, destination node, duration, and transmission power information. For ease of reading by the intelligent agent, the source and destination nodes are represented using one-hot encoding.

[0014] S q ={s,d,t,P} (2)

[0015] Where s represents the source node of the newly generated service, d represents the destination node of the newly generated service, t represents the duration of the service, and P represents the transmission power of the service.

[0016] Environmental information reflects the real-time status of network resources, and it is necessary to locate the available idle frequency bands in the network through link i, fiber core j, and frequency slot k.

[0017] S e ={i,j,k} (3)

[0018] Building on this, the state vector provides additional information for achieving the optimization objective, enabling the agent to obtain information related to the optimization objective more directly and efficiently by reading the state.

[0019] S a ={z1,z2,…} (4)

[0020] In our study, we use z1 to indicate whether the current slot is in the classical quantum co-transmission region, and z2 to indicate the degree to which the current slot is affected by noise, which is proportional to the noise received by the current slot.

[0021] The second technical point is explained in detail below:

[0022] A unified reward function for classical and quantum cooperative transmission is proposed. This method guides two agents to form a cooperative strategy in a dynamic environment by comprehensively measuring the performance of classical and quantum signals within the same optimization framework, thus optimizing both the transmission performance of the two signals and the network's cooperative performance. To guide the two agents to reach a cooperative allocation strategy, we employ a unified cooperative reward function in each decision step. This function adds the classical and quantum rewards and penalizes resource allocation failures. Specifically, in each time slot t, we define the reward function as follows:

[0023] R t =α·R C +β·R Q -γ·Penalty (5)

[0024] Where R CIt is a classical sub-reward, which uses the optical signal-to-noise ratio of the classical signal to reflect classical performance indicators; R Q α is the quantum reward, which uses the average key generation rate of the quantum signal to reflect the quantum performance index; Penalty is a penalty term, which equals 1 when the request is blocked, and 0 otherwise; α, β, γ>0 are weighting factors, used to adjust the weight of each part of the reward function to the overall contribution. They are hyperparameters in machine learning and need to be continuously adjusted during training.

[0025] By employing the aforementioned collaborative unified reward function, the behavior of the two agents will exhibit the following preferences:

[0026] Maximizing overall transmission performance: The reward function considers the contributions of both classical signal performance and quantum key generation rate, prompting the agent to prioritize improving overall transmission performance during policy learning.

[0027] Minimize blocking events: Since allocation failure will immediately bring negative penalties, the agent will tend to choose a strategy that can guarantee successful allocation, thereby effectively reducing the blocking rate;

[0028] Balanced resource sharing: Classical and quantum performance are both incorporated into the optimization objective. The two agents can gradually develop an understanding of performance trade-offs during training, achieving reasonable allocation and collaborative optimization of resources.

[0029] The third technical point is explained in detail below:

[0030] A reinforcement learning-based dual-agent resource allocation algorithm is proposed. This algorithm achieves dynamic optimization of resource allocation in multi-core fiber optic networks through the collaborative training of an agent responsible for classical signals and an agent responsible for quantum signals. Its core steps are as follows:

[0031] Step 1: Construct the network topology G(V,E), generate a set of route candidates based on the given topology, and set the service load parameter λ. Simultaneously initialize the parameters of both agents.

[0032] Step 2: In each time slot t, the environment generates a business request D, and constructs a state vector S by combining the candidate route R with the current resource occupancy. t :

[0033] Step 3: The agent responsible for classical signals and the agent responsible for quantum signals generate action A based on their respective policy functions. c Or A q :

[0034] Step 4: The environment returns to the new state S. t+1 And calculate the instant reward R based on the aforementioned unified reward function. t :

[0035] Step 5: Transfer the state transition tuple (S) t A,S t+1 ,R t The data is stored in the experience pool. When the update cycle arrives, the agent parameters are updated using the Multi-Agent Proximity Policy Optimization (MAPPO) algorithm.

[0036] Step 6: After completing all training iterations, output the converged agent models Agent C and Agent Q, and test and evaluate the performance based on the cumulative reward R and the total loss L. Attached Figure Description

[0037] Figure 1 The channel allocation strategy of the resource allocation method proposed in this patent divides the spectrum in each fiber core into a dedicated classical transmission region and a classical quantum co-transmission region.

[0038] Figure 2 This is the overall framework for the interaction between intelligent agents, network environment, and core information flow (state, action, reward) in the resource allocation method proposed in this patent.

[0039] Figure 3 The diagram shows the multi-core fiber optic network topology used in a specific embodiment of this patent. Detailed Implementation

[0040] To make the objectives, technical methods, and advantages of this invention clearer, the following description is provided in conjunction with the appendix. Figure 1 Appendix Figure 2 Appendix Figure 3 The present invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0041] In a specific embodiment of the resource allocation method based on dual agents in the space-division multiplexing quantum optical network proposed in this patent, a spectrum allocation strategy is adopted, in which the spectrum is divided into two regions: a classical service region and a cooperative transmission region, as shown in the appendix. Figure 1 As shown, the specific planning method is as follows:

[0042] When classical signals arrive, their spectrum selection range covers the entire spectrum resource pool, meaning they can be allocated in either the classical service area or the cooperative transmission area. The agent responsible for classical signals prioritizes link availability during the selection process to ensure a low blocking rate. Since the number of classical signals is far greater than that of quantum signals, the classical service area bears the majority of the traffic load, thus significantly improving the flexibility and efficiency of resource allocation.

[0043] When quantum signals arrive, their spectral selection is limited to high-frequency cooperative transmission regions. Quantum signals typically require far fewer channels than classical signals; therefore, concentrating them in specific regions effectively reduces the action space dimension of quantum agents and lowers training complexity. Simultaneously, quantum signal transmission is highly sensitive to noise levels in the region. Restricting the frequency bands used for quantum signals not only suppresses interference and maintains the stability of the quantum key generation rate (SKR), but also relatively improves signal transmission performance in classical service areas, thereby ensuring overall network performance optimization.

[0044] The flowchart illustrating the interaction between agents and network environment, as well as service and state elements, in the dual-agent-based resource allocation method for spatial multiplexing quantum optical networks proposed in this patent is attached. Figure 2 As shown, the specific planning method is as follows:

[0045] First, when a service request arrives, the network environment generates corresponding state information S based on the service requirements, current fiber core and spectrum occupancy, and interference / noise levels. t This signal is then transmitted to the agent responsible for classical signals or the agent responsible for quantum signals, as shown in the appendix. Figure 2 As shown in (a). Next, the two quantum agents analyze the received state information based on their respective policy models and output corresponding resource allocation actions. The environment executes resource allocation operations based on the actions of the two agents and updates the network state, as shown in the appendix. Figure 2 As shown in (b).

[0046] Subsequently, after completing the allocation, the unified reward function module returns a unified reward to both agents based on the transmission performance of the classical service, the key generation rate of the quantum service, and the existence of blocking events, as shown in the appendix. Figure 2 As shown in (c). After a certain number of training iterations, the two agents will update their internal decision-making policy models based on the rewards and experience cache accumulated during the training period, as shown in the appendix. Figure 2 As shown in (d), through multiple rounds of iterative training, the two agents gradually learn a cooperative allocation strategy in complex network environments.

[0047] In a specific embodiment of the resource allocation method based on dual agents in the space-division multiplexing quantum optical network proposed in this patent, an appended... Figure 3 The network topology is shown, and the simulation parameters are shown in Table 1. In this embodiment, the scenario is set as service transmission between data centers within a metropolitan area, with a network service load of 500 Erlang. The network environment includes mixed transmission requirements of classical and quantum signals. Two agents make independent decisions in the same environment and achieve joint optimization through a unified reward function. The entire process includes a training phase and a testing phase.

[0048] Table 1 Simulation Parameters

[0049]

[0050] During the training phase, the system first initializes the multi-core fiber optic network environment and the agent policy network, generating a set of candidate routes based on the given topology. Subsequently, the environment randomly generates new service requests based on the Poisson distribution of traffic volume and constructs a state vector based on the current link occupancy and service requirements. This state information is then passed to the agent responsible for classical signals and the agent responsible for quantum signals, respectively. The two agents analyze their states based on their respective policy models and output resource allocation actions. The environment executes the corresponding resource allocation based on the agents' decisions and provides feedback on the updated network state. After allocation, the environment calculates a unified reward value. In each interaction, the agents store their state, actions, rewards, and new states in an experience pool and call the MAPPO algorithm to train and update their policy models within a set update cycle. After multiple iterations, the two agents gradually converge, forming a stable resource allocation policy.

[0051] During the testing phase, the same environmental parameters as in the training phase were maintained. The two trained agents were directly applied to this environment to perform real-time resource allocation for continuously arriving dynamic service requests. During testing, the system recorded the allocation results for various services, the performance indicators of the quantum signal, and the transmission performance of the classical signal. The results show that under this service load, the method of this invention can stabilize the quantum key generation rate at approximately 68.4 kbps while maintaining the optical signal-to-noise ratio of the classical signal at approximately 37 dB. The generated quantum key can meet the requirements of classical signal transmission in high-security encryption scenarios.

[0052] The results demonstrate that the resource allocation method based on dual agents described in this invention can simultaneously guarantee the performance of quantum signals and classical signals in multi-core optical fiber networks, significantly improving the overall transmission efficiency and collaborative transmission capability of the network.

Claims

1. A resource allocation method based on dual agents in a space-division multiplexing quantum optical network, mainly addressing the dynamic collaborative resource allocation problem in the co-fiber transmission of classical and quantum signals in multi-core optical fiber networks. By introducing two agents to manage channel allocation for classical and quantum signals respectively, and combining state representation methods, a unified reward function, and a resource allocation method based on dual-agent reinforcement learning, dynamic allocation of channel resources such as fiber cores and spectrum is achieved. This enables collaborative transmission that simultaneously improves the transmission performance of both classical and quantum signals, thereby increasing the overall resource utilization of the network and reducing network congestion rate. The method is characterized by… Comprise: A, a state representation method: business information, network resource occupation information and noise interference information are uniformly encoded, and a state vector is formed and input to an intelligent agent performing resource allocation tasks; B, a unified reward function: simultaneously measure the performance of classical signal transmission and quantum signal transmission, and introduce a penalty when allocation fails, so that the intelligent agent tends to improve the transmission performance of classical signals and quantum signals at the same time, and can improve the resource utilization of the network and reduce the network blocking rate; C, a method for training and decision-making of double-agent reinforcement learning: using reinforcement learning algorithm to train and make decisions for two intelligent agents respectively responsible for classical signal and quantum signal channel allocation, the method can perform dynamic coordinated resource allocation of classical signals and quantum signals in a multi-core fiber network environment, and achieve the cooperative transmission goal of simultaneously improving the transmission performance of classical signals and quantum signals.

2. The method of claim 1, proposes a state representation method in multi-core optical network, which is used to map the service demand and network environment into the input of agent, by providing additional information to the agent, the agent can understand the noise influence degree of candidate idle frequency band without reading the whole network spectrum state, abstract the high-dimensional network state into low-dimensional vector, reduce the dimension of agent state input, and thus reduce the difficulty of training, characterized in that, Comprise the following steps: S1, abstract the multi-core fiber network into a graph structure G(V, E), where V is the node set and E is the link set, and the frequency spectrum is divided into two functional areas: classical service area and cooperative transmission area; S2, generating a new service request in each time slot, extracting its source and destination node, duration, transmission power information, forming service information S q ; S3, in combination with the occupation of the link, the core and the frequency slot, build environment information S e , for describing available resources; S4, by reading the position relationship and occupation of each frequency slot and the surrounding frequency slot, the noise influence on each idle frequency band is obtained, and the additional information S is formed by the discrete numerical value of the position relationship a . Wherein the element z1 represents whether the current frequency slot is in the classical quantum co-transmission area, and the element z2 represents the number of signals around the current frequency slot which have noise influence on the current frequency slot, and can reflect the degree of noise influence on the current frequency slot; S5, S q , S e , S a are combined into a state vector S as input to the agent.

3. The method of claim 1, proposing a unified reward function method for classical and quantum signal cooperative transmission, by simultaneously considering the transmission performance of classical signals and quantum signals, for guiding the joint optimization of two agents, realizing dynamic cooperative resource allocation, characterized in that, Comprise the following steps: S1, transition from state and extract the required transmission performance indicators from the environment. The optical signal-to-noise ratio of the classical signal as the classical performance reward R C , the average key generation rate of the quantum signal as the quantum performance reward R Q ; S2, to reduce the network congestion caused by the failure of business allocation resources, a penalty term Penalty is introduced, which is equal to 1 when the business is blocked, and 0 otherwise; S3, combined with the goal of cooperative transmission, calculate the influence of this allocation action on the performance of classical signals and quantum signals, and calculate the unified reward function: R t = a · R C + β · R Q - γ · Penalty (1) Where, α, β, γ>0 are weight coefficients used to adjust the weight of each part of the reward function to the overall contribution, which belongs to the hyperparameters in machine learning and needs to be adjusted constantly during the training process.

4. The method of claim 1, proposes a double-agent reinforcement learning training and decision-making method, adopts a multi-agent proximal policy optimization algorithm for training, and the two agents obtained can realize dynamic allocation of channel resources such as fiber cores and frequency spectra, perform collaborative transmission to simultaneously improve the transmission performance of classical signals and the transmission performance of quantum signals, and improve the overall resource utilization rate of the network and reduce the network blocking rate, characterized in that, Comprise the following steps: S1, initialize the environment, set the business load and network topology, generate classical business or quantum business requests and calculate the candidate route set R; S2, at each time slot t, the agent in charge of the classical signal or the agent in charge of the quantum signal respectively receives the state vector S t , outputs an action A C or A Q to the environment; S3, the environment performs resource allocation according to the actions of the two agents and returns a new state S t+1 S4, from state S t to S t+1 the transition action and the corresponding change in the transmission performance indicator of the environment are passed to the reward function, which gives a reward R t to the agent performing the action; S4, store the tuple (S t ,A,S t+1 ,R t ) into the experience pool and update the policy parameters of the two agents periodically; S5, after multiple iterations, output the converged two intelligent agents, which can be used for coordinated resource allocation of classical signals and quantum signals in dynamic multi-core fiber networks.

Images