A deep reinforcement learning allocation method based on multi-link state and related devices

Abstract

The present invention discloses a deep reinforcement learning allocation method based on multi-link status and related devices; it belongs to the technical field of optical network resource allocation; it constructs a network status matrix of high-frequency used links, and based on this, calculates a multi-link fragmentation that can represent the resource status of multiple links in the network. In the process of constructing the state matrix, a breadth-first numbering algorithm is used to sort the links, so that adjacent links in the network state matrix are also connected to each other in the actual topology, thereby ensuring the rationality of the multi-link fragmentation calculation results. Then, based on the information of the used links, a resource occupancy is calculated, which represents the occupancy of resources in the link. Finally, the reward function of deep reinforcement learning is optimized in combination with the above-mentioned evaluation parameters, which provides an accurate evaluation for the decision-making of the intelligent agent and improves the algorithm performance, thereby reducing network congestion and improving network transmission performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical network resource allocation, and relates to a deep reinforcement learning allocation method based on multi-link status and related devices. Background Art

[0002] With the continuous improvement of network infrastructure, the advancement of the Internet of Things (IoT), and the continuous development of new network consumption models, people are increasingly demanding high-quality and diverse networks in a variety of areas, including communications, healthcare, transportation, and daily life. Faced with complex and ever-changing business needs, how to efficiently utilize network resources and flexibly provide customized services to various users has become a key focus of network research. Deep reinforcement learning combines the advantages of reinforcement learning and deep learning. Its decision-making process can handle highly complex and dynamic environments. In optical network environments, its driving strategies rely on frequent interactions between the agent and the environment. Compared with algorithms based on fixed rules, these algorithms are more flexible and efficient in terms of environmental adaptability and feature capture. In deep reinforcement learning, rewards guide the training of the agent. They not only determine how the agent evaluates the quality of its actions but also directly influence the agent's learning efficiency and ultimate performance.

[0003] Existing planning algorithms for resource allocation in elastic optical networks are mostly based on expert systems with fixed strategies, making them difficult to adapt to the complex dynamic network allocation requirements. However, deep reinforcement learning techniques, with their perception and decision-making capabilities, can effectively address this bottleneck. Currently, mainstream research only uses fixed rewards based on the success or failure of allocation requests, which significantly restricts algorithm performance. A small number of studies have attempted to optimize reward design based on the degree of spectrum fragmentation, but these only consider the current state of the link to be allocated and lack analysis of the overall network state, making it impossible to provide accurate evaluation of agent decisions. Therefore, a new resource allocation method is needed that can more accurately evaluate agent decisions, improve algorithm learning efficiency and performance, and thus reduce network congestion. Summary of the Invention

[0004] The purpose of the present invention is to provide a deep reinforcement learning allocation method and related devices based on multi-link status to solve the technical problems in the prior art that the optical network resource allocation method is difficult to adapt to complex and changeable network environments and large-scale requests, and has poor optimization effect.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a deep reinforcement learning allocation method based on multi-link state, comprising the following steps:

[0007] Obtain network topology information, initialize the spectrum resource status of each link, and use the breadth-first numbering algorithm to sort and number each link;

[0008] Count the K shortest paths between all pairs of nodes in the network and mark the high-frequency links;

[0009] Obtain new arrival request information and determine candidate routes;

[0010] Determine the input state information s based on the request information and the resource information of the links in the candidate path t , input status information s t Input the intelligent agent to allocate spectrum resources;

[0011] Calculate resource occupancy O based on the path information used for spectrum resource allocation; concatenate links in the order of link numbers to form a high-frequency link state matrix, and calculate the multi-link fragmentation X;

[0012] Construct a reward function r, calculate the reward based on the reward function r, resource occupancy O, and multi-link fragmentation X, feed the reward result back to the agent, and update the network status.

[0013] Furthermore, the steps of initializing the spectrum resource status of each link and sorting and numbering each link using a breadth-first numbering algorithm specifically include:

[0014] Set a node as the source node and number all the links connected to the source node. The numbering is based on the number of unnumbered links connected to the other node of the link.

[0015] After all the links connected to the source node are numbered, the other end node of the link with the largest number is selected as the new source node;

[0016] The numbering process continues with the new source node until all links are numbered.

[0017] Furthermore, it is characterized in that the step of counting K shortest paths between all pairs of nodes in the network and marking high-frequency links specifically includes:

[0018] The KSP algorithm is used to calculate and record the K shortest paths between all pairs of nodes in the network, count the frequency of each link in all paths, and mark the links with frequencies higher than the average as high-frequency links.

[0019] Furthermore, the request information includes duration τ, required bandwidth b i , source node s and destination node d.

[0020] Furthermore, the input state information s is determined based on the request information and the resource information of the links in the candidate path. t, input status information s t The steps of allocating spectrum resources in the input agent include:

[0021] Determine the modulation format and then the modulation level based on the length of the candidate path; calculate the required number of frequency slots FS in each candidate path k The specific calculation formula is:

[0022]

[0023] Where b i is the required bandwidth; M is the modulation level;

[0024] According to the request information, the frequency slot number FS k and other resource information of the links in the candidate path to determine the input state information s t ; The input status information s t The expression is:

[0025]

[0026] in, is the starting index of the first available spectrum block on the kth alternative path, For its size; B k is the number of available spectrum blocks, for its average size;

[0027] Will input status information s t Input the spectrum resource allocation in the intelligent agent.

[0028] Furthermore, the calculation formula of the resource occupancy O is:

[0029]

[0030] Where M is the modulation level and H is the number of hops in the currently used path;

[0031] The calculation formula of the multi-link fragmentation degree X is:

[0032]

[0033] Where S o Indicates the number of occupied frequency slots in the state matrix, l o It indicates the number of unoccupied frequency slots “adjacent” to the occupied frequency slots; F indicates the degree of dispersion of all currently occupied frequency slots.

[0034] Furthermore, the reward function r is expressed as:

[0035]

[0036] If the request is blocked, the reward will be -1. If the request is successfully accepted, the reward will be returned.

[0037] In a second aspect, the present invention provides a deep reinforcement learning allocation system based on multi-link states, comprising:

[0038] The link numbering module is used to obtain network topology information, initialize the spectrum resource status of each link, and use the breadth-first numbering algorithm to sort and number each link;

[0039] The high-frequency link marking module is used to count the K shortest paths between all node pairs in the network and mark the high-frequency links;

[0040] Request acquisition module, used to obtain new arrival request information and determine candidate paths;

[0041] The resource allocation module is used to determine the input state information s according to the request information and the resource information of the links in the candidate path. t , input status information s t Input the intelligent agent to allocate spectrum resources;

[0042] An indicator calculation module is used to calculate the resource occupancy O based on the path information used for spectrum resource allocation; link splicing is performed according to the link number sequence to form a high-frequency link state matrix, and the multi-link fragmentation degree X is calculated;

[0043] Reward calculation module: used to construct the reward function r, calculate the reward based on the reward function r, resource occupancy O, and multi-link fragmentation X, feed the reward result back to the agent, and update the network status.

[0044] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0045] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and is characterized in that when the computer program is executed by a processor, the steps of the above method are implemented.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention discloses a deep reinforcement learning allocation method and related devices based on multi-link status, constructs a network status matrix of high-frequency used links, and based on this, calculates a multi-link fragmentation that can represent the resource status of multiple links in the network. Then, based on the information of the used links, a resource occupancy rate is calculated, which represents the occupancy of resources in the link. Finally, the two are combined to optimize the reward function of deep reinforcement learning. Compared with the existing technology, the present invention enriches the reward content with more network status information, rather than focusing only on the status of specific used links, providing an accurate evaluation for the decision-making of the intelligent agent, which helps the intelligent agent gradually learn how to formulate an allocation strategy during training to facilitate the network to accept subsequent requests. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0050] Figure 2 This is a diagram of the reward function design in an embodiment of the present invention;

[0051] Figure 3 Flowchart of a breadth-first numbering algorithm according to an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of the network structure of an embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of a network status matrix according to an embodiment of the present invention;

[0054] Figure 6 It is a schematic diagram of the computer device structure of the present invention. DETAILED DESCRIPTION

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0056] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0057] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0058] In the description of the embodiments of the present invention, it should be noted that if the terms "upper," "lower," "horizontal," "inner," etc. appear, the orientation or positional relationship indicated is based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the inventive product is typically placed when in use. These terms are merely for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or component referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on the present invention. In addition, the terms "first," "second," etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0059] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0060] In the description of the embodiments of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "disposed," "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0061] The present invention is described in further detail below with reference to the accompanying drawings:

[0062] See also Figure 1 , an embodiment of the present invention discloses a deep reinforcement learning allocation method based on multi-link state, comprising the following steps:

[0063] 1) Obtain network topology information, initialize the spectrum resource status of each link, and use the breadth-first numbering algorithm to sort and number the links.

[0064] In this step, the breadth-first numbering algorithm process is as follows:

[0065] Step 1.1: Set a node as the source node and number all the links connected to the node. The numbering is based on the number of unnumbered links connected to the other node of the link.

[0066] Step 1.2: After all the links connected to the current node are numbered, the other end node of the link with the largest number is selected as the new source node;

[0067] Step 1.3: Repeat step 1.1 until all links are numbered.

[0068] 2) Using the KSP algorithm, calculate and record the K shortest paths between all pairs of nodes in the network, count the frequency of each link in all paths, and mark the links with a frequency higher than the average as high-frequency links;

[0069] 3) Get the information of the newly arrived request, which includes the duration τ and the required bandwidth b i , source node s, destination node d;

[0070] 4) Determine the modulation format based on the candidate path length, and then determine the modulation level, and calculate the required number of frequency slots FS in each candidate path. k ;

[0071]

[0072] Where M is the modulation level.

[0073] 5) Construct the agent's input state information s based on the request and the resource information of the links in the candidate path t , and input it into the agent. Then, the agent selects an action and specifies a strategy to select a path for spectrum resource allocation;

[0074] The status information s t for:

[0075]

[0076] in, is the starting index of the first available spectrum block on the kth alternative path, For its size; B k is the number of available spectrum blocks, Its average size.

[0077] 6) Calculate the reward based on the reward function r; perform link splicing according to the link number sequence, construct a high-frequency link state matrix, calculate the multi-link fragmentation X based on the state matrix, and calculate the resource occupancy O based on the path information used; feed the reward back to the agent and update the network state;

[0078] The multi-link fragmentation degree X is:

[0079]

[0080] Among them, S o Indicates the number of occupied frequency slots in the state matrix, l o Indicates the number of unoccupied frequency slots that are "adjacent" to occupied frequency slots (adjacent frequency slots of the same link and co-frequency slots of adjacent links are both considered "adjacent"). max =4, F min =0.

[0081] The resource occupancy O is:

[0082]

[0083] Where M is the modulation level, and H is the number of hops of the currently used path.

[0084] The reward function r is:

[0085]

[0086] Example:

[0087] This embodiment discloses a deep reinforcement learning allocation algorithm based on multi-link state. The overall process is as follows: Figure 1 As shown, the specific steps include:

[0088] 1) Obtain network topology information, initialize the spectrum resource status of each link, and use the breadth-first numbering algorithm to sort and number the links.

[0089] Step 1) The specific operations are:

[0090] As Figure 4 Take the network model shown in the figure as an example. The network contains 6 nodes and 8 links. The links are numbered using the breadth-first numbering algorithm. Figure 3 In the process shown, with node 1 as the source node, links 1-2 and 1-6 are numbered. Because node 6 has more unnumbered connected links than node 2, link 1-2 is numbered Link 1 and link 1-6 is numbered Link 2. At this point, all connected links to node 1 are numbered, and the source node is updated. Since link 1-6 is numbered Link 2, node 6 is designated as the new source node, and numbering continues until all eight links are numbered.

[0091] 2) Using the KSP algorithm, calculate and record the K shortest paths between all pairs of nodes in the network, count the frequency of each link in all paths, and mark the links with a frequency higher than the average as high-frequency links;

[0092] 3) Get the information of the newly arrived request, which includes the duration τ and the required bandwidth b i , source node s, destination node d;

[0093] Step 3) The specific operations are:

[0094] Assume that the information of the request currently arriving at the network is: duration τ is 10, required bandwidth b i The source node s is node 1 and the destination node d is node 5;

[0095] 4) Determine the modulation format based on the candidate path length and calculate the required number of frequency slots FS in each candidate path k ;

[0096] Step 4) The specific operations are:

[0097] Assume K = 2. The candidate paths between nodes 1 and 5 are 1-2-3-5 and 1-6-5. The physical distance of path 1-2-3-5 is 800 km, and the physical distance of path 1-6-5 is 300 km. Assuming that the network uses G.652 fiber, according to Table 1, the applicable modulation formats for the candidate paths are 8-QAM and 16-QAM, respectively, and the modulation levels M are 3 and 4, respectively. Calculate the required number of frequency slots FS in the candidate path using equation (1): k , which are 3 and 2 respectively.

[0098] Table 1 Correspondence between transmission distance and modulation format in G.652 optical fiber

[0099] M Modulation format Maximum transmission distance (km) 1 BPSK 4800 2 QPSK 2400 3 8-QAM 1200 4 16-QAM 600

[0100] 5) Construct the agent's input state information s based on the request and the resource information of the links in the candidate path t , and input it into the agent. Then, the agent selects an action and chooses a path for spectrum resource allocation;

[0101] Step 5) The specific operations are:

[0102] Calculate the starting index of the first available spectrum block on the two candidate paths and its size and the number of available spectrum blocks B k and its average size Then, state information is constructed according to Equation (2) and input to the agent. Based on the input state information, the agent selects a path from the candidate paths in the action space and allocates frequency slot resources. If the request is accepted, it proceeds to step 6. If the request is rejected, a reward of -1 is directly fed back. The agent receives the reward and updates the state information.

[0103] 6) Calculate the multi-link fragmentation X based on the resource status of the high-frequency links and the resource occupancy O based on the used path information; then, feed the reward back to the agent and update the network status;

[0104] Step 6) The specific operations are:

[0105] The design of reward function is as follows Figure 2 Links are spliced ​​in the order of link numbers to form a high-frequency link state matrix. The state matrix is ​​as follows: Figure 5 As shown. It can be seen that ∑l o =80, S o =74, F max =4, F min =0. Then, according to formula (3), we can get X = 0.831. This time, the agent uses the candidate path 1-6-5, so we know that M = 4, H = 2, and M max =4,H min = 1. Then, according to formula (4), we can get O = 0.782.

[0106] Finally, r = 0.807 is calculated according to formula (5), which is fed back to the agent and the network status is updated.

[0107] An embodiment of the present invention discloses a deep reinforcement learning allocation system based on multi-link status, including a link numbering module, a high-frequency link labeling module, a request acquisition module, a resource allocation module, an indicator calculation module and a reward module.

[0108] The link numbering module is used to obtain network topology information, initialize the spectrum resource status of each link, and sort and number each link using breadth-first numbering; the high-frequency link marking module is used to calculate and record the K shortest paths between all node pairs in the network, count and mark high-frequency links; the request acquisition module is used to obtain newly arrived request information and determine candidate paths; the resource allocation module is used to determine the input status information s based on the request information and the resource information of the links in the candidate paths. t , input status information s t Spectrum resource allocation is performed in the input agent; the indicator calculation module is used to calculate the resource occupancy O based on the path information used for spectrum resource allocation; links are spliced ​​in the order of link numbers to form a high-frequency link state matrix, and the multi-link fragmentation X is calculated; the reward module is used to construct a reward function r, calculate the reward based on the reward function r, the resource occupancy O, and the multi-link fragmentation X, feed the reward result back to the agent, and update the network status.

[0109] In one embodiment of the present invention, a computer device is provided. Figure 6, the computer device includes a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the deep reinforcement learning allocation method based on multi-link status.

[0110] The present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the deep reinforcement learning allocation method based on multi-link state in the above embodiment.

[0111] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0112] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0113] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A deep reinforcement learning allocation method based on multi-link state, characterized in that: The following steps are involved: Obtain network topology information, initialize the spectrum resource status of each link, and use the breadth-first numbering algorithm to sort and number each link. Specifically, it includes: Set a node as the source node and number all the links connected to the source node. The numbering is based on the number of unnumbered links connected to the other node of the link. After all the links connected to the source node are numbered, the other end node of the link with the largest number is selected as the new source node; Continue numbering with the new source node until all links are numbered; Statistics of the relationships between all pairs of nodes in the network The shortest paths are marked with high-frequency links; Obtain new arrival request information and determine candidate routes; Determine input status information based on request information and resource information of links in candidate paths , the input status information Input the intelligent agent to allocate spectrum resources; Calculate resource occupancy based on the path information used for spectrum resource allocation ; Link splicing is performed according to the link number sequence to form a high-frequency link state matrix and calculate the multi-link fragmentation degree ; The resource occupancy The calculation formula is: Where, is the modulation level, is the number of hops of the currently used path; is the maximum value of the modulation level; is the minimum value of the path hop count; The multi-link fragmentation The calculation formula is: Where, represents the number of occupied frequency slots in the state matrix, Indicates the number of unoccupied frequency slots "adjacent" to occupied frequency slots; Indicates the degree of dispersion of all currently occupied frequency slots; Indicates the maximum value of the discreteness of the occupied frequency slot distribution; Indicates the minimum value of the discreteness of the occupied frequency slot distribution; Building a reward function , combined with the reward function and resource usage and multi-link fragmentation Calculate the reward, feed the reward result back to the agent, and state.

2. A multi-link state-based deep reinforcement learning allocation method according to claim 1, characterized in that: The statistical network has a relationship between all pairs of nodes. The shortest paths are marked with the steps of the high-frequency links, including: Through the KSP algorithm, the relationships between all pairs of nodes in the network are calculated and recorded. The shortest paths are selected, the frequency of occurrence of each link in all paths is counted, and the links with frequencies higher than the average are marked as high-frequency links.

3. The multi-link state-based deep reinforcement learning allocation method according to claim 1, characterized in that: The request information includes duration , Required bandwidth , source node and destination node .

4. The multi-link state-based deep reinforcement learning allocation method according to claim 1, characterized in that: The input state information is determined based on the request information and the resource information of the links in the candidate path. , the input status information The steps of allocating spectrum resources in the input agent include: Determine the modulation format based on the length of the candidate path, and then determine the modulation level; calculate the required number of frequency slots requested in each candidate path The specific calculation formula is: Where, is the required bandwidth; is the modulation level; According to the request information, the number of frequency slots and other resource information of links in the candidate path to determine the input state information ; The input status information The expression is: in, For the The starting index of the first available spectrum block on the alternative path, For its size; is the number of available spectrum blocks, for its average size; Will enter status information Input the spectrum resource allocation in the intelligent agent.

5. The multi-link state-based deep reinforcement learning allocation method according to claim 1, characterized in that: The reward function The expression is: If the request is blocked, the reward will be -1. If the request is successfully accepted, the reward will be returned. .

6. A deep reinforcement learning distribution system based on multi-link state, characterized in that: A multi-link state-based deep reinforcement learning allocation method according to any one of claims 1 to 5, comprising: The link numbering module is used to obtain network topology information, initialize the spectrum resource status of each link, and sort and number each link using the breadth-first numbering algorithm. Specifically, it includes: Set a node as the source node and number all the links connected to the source node. The numbering is based on the number of unnumbered links connected to the other node of the link. After all the links connected to the source node are numbered, the other end node of the link with the largest number is selected as the new source node; Continue numbering with the new source node until all links are numbered; High-frequency link annotation module, used to count the links between all node pairs in the network The shortest paths are marked with high-frequency links; Request acquisition module, used to obtain new arrival request information and determine candidate paths; Resource allocation module, used to determine input status information based on request information and resource information of links in candidate paths , the input status information Input the intelligent agent to allocate spectrum resources; The indicator calculation module is used to calculate the resource occupancy based on the path information used for spectrum resource allocation ; Link splicing is performed according to the link number sequence to form a high-frequency link state matrix and calculate the multi-link fragmentation degree ; The resource occupancy The calculation formula is: Where, is the modulation level, is the number of hops of the currently used path; is the maximum value of the modulation level; is the minimum value of the path hop count; The multi-link fragmentation The calculation formula is: Where, represents the number of occupied frequency slots in the state matrix, Indicates the number of unoccupied frequency slots "adjacent" to occupied frequency slots; Indicates the degree of dispersion of all currently occupied frequency slots; Indicates the maximum value of the discreteness of the occupied frequency slot distribution; Indicates the minimum value of the discreteness of the occupied frequency slot distribution; Reward calculation module: used to build reward function , combined with the reward function and resource usage and multi-link fragmentation Calculate the reward, feed the reward result back to the agent, and state.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images