Deep reinforcement learning distribution method based on multi-link state and related device

Abstract

The invention discloses a deep reinforcement learning distribution method based on a multi-link state and a related device. Belongs to the technical field of optical network resource allocation. A network state matrix of a high-frequency use link is constructed, and a multi-link fragmentation degree capable of representing the resource state of multiple links in the network is calculated based on the network state matrix. In a state matrix construction process, a breadth-first numbering algorithm is used for carrying out link sorting, so that adjacent links in a network state matrix are mutually connected in an actual topology, and the reasonability of a multi-link fragmentation degree calculation result is ensured. Then, a resource occupancy rate is calculated based on the information of using the link, and the resource occupancy rate represents the occupancy condition of the resource in the link. Finally, the reward function of deep reinforcement learning is optimized in combination with the evaluation parameters, accurate evaluation is provided for decision making of the intelligent agent, the algorithm performance is improved, network congestion is reduced, and the network transmission performance is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical network resource allocation, and relates to a deep reinforcement learning allocation method based on multi-link states and related devices. Background Art

[0002] With the continuous improvement of network infrastructure, the continuous advancement of the construction of the Internet of Things, and the continuous development of new network consumption patterns, people have increasingly higher demands for high-quality and diversified networks in multiple fields such as communications, medical care, transportation, and life. Faced with complex and changing business needs, how to efficiently use network resources and flexibly provide exclusive customized services for various users has become a focus of network research. Deep reinforcement learning combines the advantages of reinforcement learning and deep learning. It can handle highly complex and dynamic environments in the decision-making process. In the optical network environment, its driving strategy depends on the frequent interaction between the agent and the environment. Compared with algorithms based on fixed rules, they are more flexible and efficient in environmental adaptability and feature capture. In deep reinforcement learning, rewards play a guiding role in the training of agents. It not only determines how the agent evaluates the quality of its actions, but also directly affects the learning efficiency and final performance of the agent.

[0003] For the resource allocation problem in elastic optical networks, the existing planning algorithms are mostly based on expert systems with fixed strategies, which are difficult to adapt to the complex dynamic network allocation needs. However, deep reinforcement learning technology with perception and decision-making capabilities can better solve this bottleneck problem. At present, the mainstream related research only provides fixed value rewards based on whether the request allocation is successful or not, which greatly restricts the performance of the algorithm; a small number of studies have attempted to optimize the reward design from the perspective of the degree of spectrum fragmentation, but they only consider the status of the current link to be allocated, lack analysis of the overall network status, and cannot provide accurate evaluation for the decision-making of the intelligent agent. Therefore, it is necessary to design a new resource allocation method that makes its evaluation of the intelligent agent's decision more accurate, so as to improve the learning efficiency and performance of the algorithm, thereby reducing network congestion. Summary of the invention

[0004] The purpose of the present invention is to provide a deep reinforcement learning allocation method and related devices based on multi-link states to solve the technical problem that the optical network resource allocation method in the prior art is difficult to adapt to the complex and changeable network environment and large-scale requests, and has poor optimization effect.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a deep reinforcement learning allocation method based on multi-link states, comprising the following steps:

[0007] Obtain network topology information, initialize the spectrum resource status of each link, and use the breadth-first numbering algorithm to sort and number each link;

[0008] Count the K shortest paths between all pairs of nodes in the network and mark the high-frequency links;

[0009] Get new arrival request information and determine candidate routes;

[0010] Determine the input state information s based on the request information and the resource information of the links in the candidate path t , the input status information s t Input the intelligent agent to allocate spectrum resources;

[0011] Calculate resource occupancy O according to the path information used for spectrum resource allocation; perform link splicing according to the link numbering sequence to form a high-frequency link state matrix, and calculate the multi-link fragmentation X;

[0012] Construct a reward function r, calculate the reward based on the reward function r, resource occupancy O, and multi-link fragmentation X, feed the reward result back to the agent, and update the network status.

[0013] Furthermore, the steps of initializing the spectrum resource status of each link and using a breadth-first numbering algorithm to sort and number each link specifically include:

[0014] Set a node as the source node, and number all the connected links of the source node, and the numbering basis is the number of unnumbered connected links at the other node of the link;

[0015] After all the links connected to the source node are numbered, the other end node of the link with the largest number is selected as the new source node;

[0016] The numbering process continues with the new source node until all links are numbered.

[0017] Furthermore, it is characterized in that the step of counting K shortest paths between all pairs of nodes in the network and marking high-frequency links specifically includes:

[0018] Through the KSP algorithm, the K shortest paths between all node pairs in the network are calculated and recorded, the frequency of occurrence of each link in all paths is counted, and the links with frequencies higher than the average are marked as high-frequency links.

[0019] Furthermore, the request information includes duration τ, required bandwidth b i , source node s and destination node d.

[0020] Furthermore, the input state information s is determined according to the request information and the resource information of the link in the candidate path. t, the input status information s t The steps of inputting spectrum resource allocation into the intelligent agent specifically include:

[0021] Determine the modulation format based on the length of the candidate path, and then determine the modulation level; calculate the required number of frequency slots FS requested in each candidate path k ; The specific calculation formula is:

[0022]

[0023] Where b i is the required bandwidth; M is the modulation level;

[0024] According to the request information, the frequency slot number FS k and other resource information of the links in the candidate path to determine the input state information s t ; The input status information s t The expression is:

[0025]

[0026] in, is the starting index of the first available spectrum block on the kth candidate path, For its size; B k is the number of available spectrum blocks, for its average size;

[0027] Will input status information s t Input the spectrum resource allocation in the intelligent agent.

[0028] Furthermore, the calculation formula of the resource occupancy O is:

[0029]

[0030] Where M is the modulation level and H is the number of hops of the currently used path;

[0031] The calculation formula of the multi-link fragmentation degree X is:

[0032]

[0033] In the formula, S o represents the number of occupied frequency slots in the state matrix, l o It indicates the number of unoccupied frequency slots “adjacent” to the occupied frequency slots; F indicates the degree of dispersion of all currently occupied frequency slots.

[0034] Furthermore, the expression of the reward function r is:

[0035]

[0036] If the request is blocked, the reward will be -1, if the request is successfully accepted, the reward will be returned

[0037] In a second aspect, the present invention provides a deep reinforcement learning allocation system based on multi-link states, comprising:

[0038] The link numbering module is used to obtain network topology information, initialize the spectrum resource status of each link, and use the breadth-first numbering algorithm to sort and number each link;

[0039] The high-frequency link marking module is used to count the K shortest paths between all node pairs in the network and mark the high-frequency links;

[0040] The request acquisition module is used to obtain the new arrival request information and determine the candidate path;

[0041] The resource allocation module is used to determine the input state information s according to the request information and the resource information of the links in the candidate path. t , the input status information s t Input the intelligent agent to allocate spectrum resources;

[0042] The indicator calculation module is used to calculate the resource occupancy O according to the path information used for spectrum resource allocation; link splicing is performed according to the link numbering sequence to form a high-frequency link state matrix, and the multi-link fragmentation degree X is calculated;

[0043] Reward calculation module: used to construct the reward function r, calculate the reward by combining the reward function r with the resource occupancy O and the multi-link fragmentation X, feed back the reward result to the agent, and update the network status.

[0044] In a third aspect, the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0045] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and wherein the computer program implements the steps of the above method when executed by a processor.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention discloses a deep reinforcement learning allocation method and related devices based on multi-link status, constructs a network status matrix of high-frequency used links, and based on this, calculates a multi-link fragmentation degree that can represent the resource status of multiple links in the network. Then, based on the information of the used links, a resource occupancy rate is calculated, which represents the occupancy of resources in the link. Finally, the two are combined to optimize the reward function of deep reinforcement learning. Compared with the prior art, the present invention enriches the reward content with more network status information, rather than focusing only on the status of specific used links, providing an accurate evaluation for the decision-making of the intelligent agent, and helping the intelligent agent to gradually learn how to formulate allocation strategies during training to facilitate the network to accept subsequent requests. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0050] Figure 2 A reward function design diagram in an embodiment of the present invention;

[0051] Figure 3 Flow chart of breadth-first numbering algorithm in an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of the network structure of an embodiment of the present invention;

[0053] Figure 5 A schematic diagram of a network status matrix according to an embodiment of the present invention;

[0054] Figure 6 It is a schematic diagram of the computer device structure of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0056] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0058] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0059] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0060] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0061] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0062] See also Figure 1 The embodiment of the present invention discloses a deep reinforcement learning allocation method based on multi-link state, comprising the following steps:

[0063] 1) Obtain network topology information, initialize the spectrum resource status of each link, and use the breadth-first numbering algorithm to sort and number the links.

[0064] In this step, the breadth-first numbering algorithm process is:

[0065] Step 1.1, set a node as the source node, and number all the connected links of the node, and the numbering basis is the number of unnumbered connected links at the other node of the link;

[0066] Step 1.2: After all the links connected to the current node are numbered, select the other end node of the link with the largest number as the new source node;

[0067] Step 1.3: Repeat step 1.1 to continue numbering until all links are numbered.

[0068] 2) Using the KSP algorithm, calculate and record the K shortest paths between all pairs of nodes in the network, count the frequency of each link in all paths, and mark the links with a frequency higher than the average as high-frequency links;

[0069] 3) Get the information of the newly arrived request, which includes the duration τ and the required bandwidth b i , source node s, destination node d;

[0070] 4) Determine the modulation format based on the candidate path length, and then determine the modulation level, and calculate the required number of frequency slots FS in each candidate path. k ;

[0071]

[0072] Where M is the modulation level.

[0073] 5) Construct the agent's input state information s based on the request and the resource information of the links in the candidate path t , and input it into the agent. Then, the agent selects actions and specifies the strategy to select the path for spectrum resource allocation;

[0074] The status information t for:

[0075]

[0076] in, is the starting index of the first available spectrum block on the kth candidate path, For its size; B k is the number of available spectrum blocks, Its average size.

[0077] 6) Calculate the reward in combination with the reward function r; perform link splicing according to the link numbering order, construct a high-frequency link state matrix, calculate the multi-link fragmentation X based on the state matrix, and calculate the resource occupancy O based on the path information used; feed back the reward to the agent and update the network status;

[0078] The multi-link fragmentation degree X is:

[0079]

[0080] Among them, S o represents the number of occupied frequency slots in the state matrix, l o Indicates the number of unoccupied frequency slots that are "adjacent" to occupied frequency slots (adjacent frequency slots of the same link and co-frequency slots of adjacent links are considered "adjacent"). max =4, F min =0.

[0081] The resource occupancy O is:

[0082]

[0083] Wherein, M is the modulation level, and H is the number of hops of the currently used path.

[0084] The reward function r is:

[0085]

[0086] Example:

[0087] This embodiment discloses a deep reinforcement learning allocation algorithm based on multi-link state. Figure 1 As shown, the specific steps include:

[0088] 1) Obtain network topology information, initialize the spectrum resource status of each link, and use the breadth-first numbering algorithm to sort and number the links.

[0089] Step 1) The specific operations are:

[0090] As Figure 4 Take the network model shown in the figure as an example. The network contains 6 nodes and 8 links. The links are numbered using the breadth-first numbering algorithm. Figure 3 In the process shown, node 1 is used as the source node, and links 1-2 and 1-6 are numbered. Since the number of unnumbered connected links at node 6 is greater than that at node 2, link 1-2 is numbered as Link1 and link 1-6 is numbered as Link2. At this point, all the links connected to node 1 are numbered, and the source node is updated. Since link 1-6 is numbered as Link2, node 6 is used as the new source node, and the numbering operation continues until all 8 links are numbered.

[0091] 2) Using the KSP algorithm, calculate and record the K shortest paths between all pairs of nodes in the network, count the frequency of each link in all paths, and mark the links with a frequency higher than the average as high-frequency links;

[0092] 3) Get the information of the newly arrived request, which includes the duration τ and the required bandwidth b i , source node s, destination node d;

[0093] Step 3) The specific operations are:

[0094] Assume that the information of the request currently arriving at the network is: duration τ is 10, required bandwidth b i is 50GHz. The source node s is node 1 and the destination node d is node 5;

[0095] 4) Determine the modulation format based on the candidate path length and calculate the required number of frequency slots FS in each candidate path k ;

[0096] Step 4) The specific operations are:

[0097] Assume K = 2, the candidate paths between nodes 1 and 5 are 1-2-3-5 and 1-6-5. The physical distance of path 1-2-3-5 is 800 km, and the physical distance of path 1-6-5 is 300 km. Assuming that the network uses G.652 optical fiber, according to Table 1, the applicable modulation formats of the candidate paths are 8-QAM and 16-QAM, and the modulation levels M are 3 and 4 respectively. According to formula (1), the required number of frequency slots FS requested in the candidate path is calculated k , which are 3 and 2 respectively.

[0098] Table 1 Correspondence between transmission distance and modulation format in G.652 optical fiber

[0099] M Modulation format Maximum transmission distance (km) 1 BPSK 4800 2 QPSK 2400 3 8-QAM 1200 4 16-QAM 600

[0100] 5) Construct the agent's input state information s based on the request and the resource information of the links in the candidate path t , and input it into the agent. Then, the agent selects an action and chooses a path to allocate spectrum resources;

[0101] Step 5) The specific operations are:

[0102] Calculate the starting index of the first available spectrum block on the two candidate paths and its size And the number of available spectrum blocks B k and its average size Then, the state information is constructed according to formula (2) and input into the agent. Based on the input state information, the agent selects a path from the candidate paths in the action space and allocates frequency slot resources. If the request is accepted, proceed to step 6. If the request is rejected, directly feedback the reward -1. The agent obtains the reward and updates the state information.

[0103] 6) Calculate the multi-link fragmentation X based on the resource status of the high-frequency links, and calculate the resource occupancy O based on the path information used; then, feed back the reward to the agent and update the network status;

[0104] Step 6) The specific operations are:

[0105] The design of the reward function is as follows Figure 2 Links are spliced ​​in the order of link numbers to form a high-frequency link state matrix. The state matrix is ​​as follows: Figure 5 As shown. It can be seen that ∑l o =80, S o =74, F max =4, F min =0. Then, according to formula (3), we can get X = 0.831. This time, the agent uses the candidate path 1-6-5, so we know that M = 4, H = 2, and M max =4,H min = 1. Then, according to formula (4), we can get O = 0.782.

[0106] Finally, r = 0.807 is calculated according to formula (5), which is fed back to the agent and the network status is updated.

[0107] An embodiment of the present invention discloses a deep reinforcement learning allocation system based on multi-link states, including a link numbering module, a high-frequency link labeling module, a request acquisition module, a resource allocation module, an indicator calculation module and a reward module.

[0108] The link numbering module is used to obtain network topology information, initialize the spectrum resource status of each link, and use breadth-first numbering to sort and number each link; the high-frequency link marking module is used to calculate and record the K shortest paths between all node pairs in the network, count and mark high-frequency links; the request acquisition module is used to obtain new arrival request information and determine candidate paths; the resource allocation module is used to determine the input status information s according to the request information and the resource information of the links in the candidate paths t , the input status information s t Spectrum resource allocation is performed in the input agent; the indicator calculation module is used to calculate the resource occupancy O according to the path information used for spectrum resource allocation; the links are spliced ​​in the order of link numbers to form a high-frequency link state matrix, and the multi-link fragmentation X is calculated; the reward module is used to construct a reward function r, calculate the reward by combining the reward function r, the resource occupancy O and the multi-link fragmentation X, feed back the reward result to the agent, and update the network status.

[0109] In one embodiment of the present invention, a computer device is provided. Figure 6, the computer device includes a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions in a computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of a deep reinforcement learning allocation method based on multi-link states.

[0110] The present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here may include a built-in storage medium in a computer device, and of course may also include an extended storage medium supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions may be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk memory. The processor may load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the deep reinforcement learning allocation method based on multi-link states in the above-mentioned embodiment.

[0111] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0112] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0113] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A deep reinforcement learning allocation method based on multi-link states, characterized in that: The following steps are involved: Obtain network topology information, initialize the spectrum resource status of each link, and use the breadth-first numbering algorithm to sort and number each link; Count the K shortest paths between all pairs of nodes in the network and mark the high-frequency links; Get new arrival request information and determine candidate routes; Determine the input state information s based on the request information and the resource information of the links in the candidate path t , the input status information s t Input the intelligent agent to allocate spectrum resources; Calculate resource occupancy O according to the path information used for spectrum resource allocation; perform link splicing according to the link numbering sequence to form a high-frequency link state matrix, and calculate the multi-link fragmentation X; Construct a reward function r, calculate the reward based on the reward function r, resource occupancy O, and multi-link fragmentation X, feed the reward result back to the agent, and update the network status.

2. According to a multi-link state-based deep reinforcement learning allocation method according to claim 1, it is characterized in that: The steps of initializing the spectrum resource status of each link and using a breadth-first numbering algorithm to sort and number each link specifically include: Set a node as the source node, and number all the connected links of the source node, and the numbering basis is the number of unnumbered connected links at the other node of the link; After all the links connected to the source node are numbered, the other end node of the link with the largest number is selected as the new source node; The numbering process continues with the new source node until all links are numbered.

3. The multi-link state-based deep reinforcement learning allocation method according to claim 1, characterized in that: The step of counting K shortest paths between all pairs of nodes in the network and marking high-frequency links specifically includes: Through the KSP algorithm, the K shortest paths between all node pairs in the network are calculated and recorded, the frequency of occurrence of each link in all paths is counted, and the links with frequencies higher than the average are marked as high-frequency links.

4. The multi-link state-based deep reinforcement learning allocation method according to claim 1, characterized in that: The request information includes duration τ, required bandwidth b i , source node s and destination node d.

5. The multi-link state-based deep reinforcement learning allocation method according to claim 1, characterized in that: The input state information s is determined according to the request information and the resource information of the links in the candidate path. t , the input status information s t The steps of inputting spectrum resource allocation into the intelligent agent specifically include: Determine the modulation format based on the length of the candidate path, and then determine the modulation level; calculate the required number of frequency slots FS requested in each candidate path k ; The specific calculation formula is: Where b i is the required bandwidth; M is the modulation level; According to the request information, the frequency slot number FS k and other resource information of the links in the candidate path to determine the input state information s t ; The input status information s t The expression is: in, is the starting index of the first available spectrum block on the kth candidate path, For its size; B k is the number of available spectrum blocks, for its average size; Will input status information s t Input the spectrum resource allocation in the intelligent agent.

6. The multi-link state-based deep reinforcement learning allocation method according to claim 1, characterized in that: The calculation formula of the resource occupancy O is: Where M is the modulation level and H is the number of hops of the currently used path; The calculation formula of the multi-link fragmentation degree X is: In the formula, S o represents the number of occupied frequency slots in the state matrix, l o represents the number of unoccupied frequency slots "adjacent" to the occupied frequency slots; F represents the degree of dispersion of all currently occupied frequency slots.

7. The multi-link state-based deep reinforcement learning allocation method according to claim 1, characterized in that: The expression of the reward function r is: If the request is blocked, the reward will be -1, if the request is successfully accepted, the reward will be returned 8. A multi-link state-based deep reinforcement learning allocation system, characterized in that: include: The link numbering module is used to obtain network topology information, initialize the spectrum resource status of each link, and use the breadth-first numbering algorithm to sort and number each link; The high-frequency link marking module is used to count the K shortest paths between all node pairs in the network and mark the high-frequency links; The request acquisition module is used to obtain the new arrival request information and determine the candidate path; The resource allocation module is used to determine the input state information s according to the request information and the resource information of the links in the candidate path. t , the input status information s t Input the intelligent agent to allocate spectrum resources; The indicator calculation module is used to calculate the resource occupancy O according to the path information used for spectrum resource allocation; link splicing is performed according to the link numbering sequence to form a high-frequency link state matrix, and the multi-link fragmentation degree X is calculated; Reward calculation module: used to construct the reward function r, calculate the reward by combining the reward function r with the resource occupancy O and the multi-link fragmentation X, feed back the reward result to the agent, and update the network status.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images