Intelligent optical fiber wiring scheduling method and device based on reinforcement learning decision

Abstract

The application provides an intelligent optical fiber distribution scheduling method and device based on reinforcement learning decision, belongs to the technical field of optical fiber communication, and abstracts fiber cores of different intelligent optical fiber distribution robots respectively accessed in a cluster into a logical resource pool; priority of the logical resource pool and intelligent optical fiber distribution robots contained in the logical resource pool is determined by using dynamic weight value partition, physical limitations of traditional resource allocation are broken, and the intelligent optical fiber distribution robots are no longer limited to the range of fiber cores managed by a single intelligent optical fiber distribution robot; through the reinforcement learning technology, service demand and resource state are matched in real time, and an optimal fiber core path is dynamically generated; for application scenarios with high requirements on service opening and recovery speed, the fiber core path meeting the service requirements can be quickly found, service response efficiency is improved, and service waiting time is reduced.

Description

Technical Field

[0001] This application relates to the field of optical fiber communication technology, and more specifically, to an intelligent optical fiber distribution scheduling method and apparatus based on reinforcement learning decision-making. Background Technology

[0002] With the surge in demand from 5G, data centers, and other services, the scale of optical fiber networks continues to expand, with both the number of network nodes and the length of optical cables growing exponentially, creating an urgent need for intelligent operation and maintenance (O&M). Currently, optical fiber network O&M still relies heavily on manual operations, with tasks such as patching and troubleshooting dependent on on-site manual intervention, resulting in low efficiency and susceptibility to human error. As the scale of optical fiber networks continues to expand, intelligent optical fiber distribution robots are gradually replacing traditional manual methods. These robots use robotic arms to automate functions such as patching and monitoring the quality of optical fiber cores, enabling dynamic resource allocation and intelligent management of optical fiber networks. These robots can remotely perform core insertion and removal, path switching, and performance testing, significantly improving O&M efficiency.

[0003] However, in application scenarios with high requirements for service activation and recovery speed, the fiber core resources in traditional solutions are managed independently by a single robot, lacking a global resource pooling mechanism. Cross-robot scheduling requires manual intervention or fixed routing strategies, i.e., relying on preset static routing tables or single-machine local search, which leads to the problem that critical services cannot be scheduled in a timely manner. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide an intelligent fiber optic distribution scheduling method and device based on reinforcement learning decision-making, which can match business needs and resource status in real time through reinforcement learning technology, dynamically generate the optimal fiber core path, and improve business response efficiency.

[0005] This application provides an intelligent fiber optic distribution scheduling method based on reinforcement learning decision-making, applied to a cluster of multiple intelligent fiber optic distribution robots. The method includes the following steps:

[0006] The fiber cores of different intelligent fiber optic distribution robots connected to the cluster are abstracted into logical resource pools.

[0007] Determine the priority of the logical resource pool and the intelligent fiber optic distribution robots it contains;

[0008] The algorithm uses reinforcement learning to determine the scheduling path that meets the business requirements. First, a search algorithm is used to filter the set of feasible paths that meet the business requirements from the logical resource pool determined based on the priority. Then, the current state of the fiber optic network is obtained, and a path is selected from the set of feasible paths as an action to be executed. The immediate reward and the expected value of the cumulative reward are calculated based on the execution result. Finally, the path is iteratively optimized based on the calculated expected value of the cumulative reward to determine the scheduling path that meets the business requirements.

[0009] In some embodiments, abstracting the fiber cores of different intelligent fiber optic distribution robots connected to the cluster into a logical resource pool includes the following steps:

[0010] Configure logical identifiers for the fiber core range managed by each intelligent fiber distribution robot;

[0011] The dynamic weight of each logical resource pool is calculated based on its load and the average health of the fiber core.

[0012] In some embodiments, determining the priority of the logical resource pool and its included intelligent fiber optic distribution robots includes the following steps:

[0013] The priority of the logical resource pool is determined by sorting the data according to the magnitude of the dynamic weights.

[0014] The priority of each intelligent fiber optic distribution robot in the logical resource pool is determined based on the health status and the number of idle fiber cores of the intelligent fiber optic distribution robot.

[0015] In some embodiments, the step of using a search algorithm to filter a set of feasible paths that meet business requirements from a logical resource pool determined based on the priority includes the following steps:

[0016] A set number of logical resource pools and their contained intelligent fiber optic distribution robots are selected according to priority order to obtain the managed fiber core set;

[0017] Based on network topology information, the A* algorithm is used to calculate the set of feasible paths from the source node to the destination node from the fiber core set;

[0018] Determine whether there is a path in the set of feasible paths that meets the business requirements. If there is, use the set of feasible paths as the action space for reinforcement learning. If not, expand the selected logical resource pool and the number of intelligent fiber optic distribution robots it contains according to priority order, and recalculate the set of feasible paths until a path that meets the business requirements is found.

[0019] In some embodiments, the state of the optical fiber network includes network topology information, dynamic weights of logical resource pools, and the health of intelligent optical fiber distribution robots; the network topology information includes inter-node connectivity, link length, and optical power attenuation.

[0020] In some embodiments, selecting a path from the set of feasible paths as an action to be executed, and calculating the immediate reward and the expected value of the cumulative reward based on the execution result, includes the following steps:

[0021] A greedy strategy is used to select a path from the set of feasible paths as the action to be executed; wherein, if the generated random number is less than the set exploration rate, a path is randomly selected from the set of feasible paths as the action to be executed; if the generated random number is greater than the set exploration rate, the path with the largest expected cumulative reward value is selected from the set of feasible paths as the action to be executed.

[0022] The immediate reward after an action is executed is calculated based on the set reward strategy, and the expected cumulative reward after an action is executed is calculated based on the immediate reward, the set learning rate and discount factor, the expected cumulative reward when the action is not executed, and the expected cumulative reward among all feasible paths.

[0023] In some embodiments, the reward strategy is set based on one or more metrics among connection quality, fiber core health, business objectives, and path efficiency.

[0024] In some embodiments, a reinforcement learning-based intelligent fiber optic cabling scheduling device is also provided, applied to a cluster of multiple intelligent fiber optic cabling robots, the device comprising:

[0025] The module is used to abstract the fiber cores of different intelligent fiber optic distribution robots connected to the cluster into a logical resource pool;

[0026] A determination module is used to determine the priority of the logical resource pool and the intelligent fiber optic distribution robots it contains;

[0027] The path matching module is used to determine the scheduling path that meets the business requirements based on the reinforcement learning algorithm. First, a search algorithm is used to filter a set of feasible paths that meet the business requirements from the logical resource pool determined based on the priority. Then, the current state of the fiber optic network is obtained, and a path is selected from the set of feasible paths as an action to be executed. The immediate reward and the expected cumulative reward are calculated based on the execution result. Finally, the path is iteratively optimized based on the calculated expected cumulative reward to determine the scheduling path that meets the business requirements.

[0028] In some embodiments, an electronic device is also provided, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the intelligent fiber optic distribution scheduling method based on reinforcement learning decision-making described above are performed.

[0029] In some embodiments, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, performs the steps of the intelligent fiber optic distribution scheduling method based on reinforcement learning decision-making as described above.

[0030] This application describes an intelligent fiber optic distribution scheduling method and apparatus based on reinforcement learning decision-making, applied to a cluster of multiple intelligent fiber optic distribution robots. The method abstracts fiber cores connected to different intelligent fiber optic distribution robots within the cluster into logical resource pools. It determines the priorities of these logical resource pools and the intelligent fiber optic distribution robots they contain. A reinforcement learning algorithm is used to determine scheduling paths that meet business requirements. First, a search algorithm is used to filter a set of feasible paths that meet business requirements from the logical resource pool determined by the priorities. Then, the current state of the fiber optic network is obtained, and a path is selected from the set of feasible paths as an action to be executed. The immediate reward and the expected cumulative reward are calculated based on the execution result. Finally, the path is iteratively optimized based on the calculated expected cumulative reward to determine the scheduling path that meets business requirements. Thus, by combining reinforcement learning with dynamic weighted scheduling, business needs and resource states are matched in real time, and the optimal fiber core path is dynamically generated. For application scenarios with high requirements for service activation and recovery speed, it can quickly find fiber core paths that meet business requirements, improve service response efficiency, and reduce service waiting time. Attached Figure Description

[0031] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 A flowchart of the intelligent fiber optic distribution scheduling method based on reinforcement learning decision-making described in an embodiment of this application is shown;

[0033] Figure 2 A flowchart illustrating the process of determining the priority of the logical resource pool and its contained intelligent fiber optic distribution robots, as described in an embodiment of this application, is shown.

[0034] Figure 3 The flowchart illustrates an embodiment of this application, showing how to select a path from the set of feasible paths as an action to be executed, and how to calculate the immediate reward and the expected value of the cumulative reward based on the execution result.

[0035] Figure 4 This paper shows a schematic diagram of the structure of the intelligent fiber optic distribution scheduling device based on reinforcement learning decision-making according to an embodiment of this application;

[0036] Figure 5 A schematic diagram of the structure of the electronic device described in an embodiment of this application is shown. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0038] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0039] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0040] In view of the technical problems raised in the background, this application provides an intelligent optical fiber distribution scheduling method and device based on reinforcement learning decision-making, which can match service needs and resource status in real time through reinforcement learning technology, dynamically generate the optimal fiber core path, and improve service response efficiency.

[0041] See the instruction manual appendix Figure 1 This application provides an intelligent fiber optic distribution scheduling method based on reinforcement learning decision-making, applied to a cluster of multiple intelligent fiber optic distribution robots, comprising the following steps:

[0042] S1. Abstract the fiber cores of different intelligent fiber optic distribution robots connected to the cluster into a logical resource pool.

[0043] S2. Determine the priority of the logical resource pool and the intelligent fiber optic distribution robot it contains;

[0044] S3. Determine the scheduling path that meets the business requirements based on the reinforcement learning algorithm; wherein, firstly, a search algorithm is used to filter the set of feasible paths that meet the business requirements from the logical resource pool determined based on the priority; then, the current state of the optical fiber network is obtained, and a path is selected from the set of feasible paths as an action to be executed, and the immediate reward and the expected value of the cumulative reward are calculated based on the execution result; finally, the path is iteratively optimized based on the calculated expected value of the cumulative reward to determine the scheduling path that meets the business requirements.

[0045] In step S1, the main task is to construct logical resource pools. Specifically, the fiber optic cores connected to different intelligent fiber optic distribution robots are abstracted as logical resource pools. A unique logical identifier is assigned to the range of fiber cores managed by each intelligent fiber optic distribution robot. For example, logical resource pool 1 includes two intelligent fiber optic distribution robots, A and B. Intelligent fiber optic distribution robot A manages fiber cores 1-288; intelligent fiber optic distribution robot B manages fiber cores 289-480. The logical resource pool identifier corresponding to intelligent fiber optic distribution robots A and B is Pool_1. Logical resource pool 2 includes one intelligent fiber optic distribution robot, C. Intelligent fiber optic distribution robot C manages fiber cores 481-768, and the corresponding logical resource pool identifier is Pool_2. Then, the logical identifiers of the fiber cores managed by all intelligent fiber optic distribution robots are integrated to form a global logical resource pool.

[0046] When a business request requires selecting different fiber core connections for two intelligent fiber distribution robots, its dynamic weight W is calculated based on the real-time load of each logical resource pool and the average health of the fiber core. i This is to balance the load of intelligent fiber optic distribution robots and prioritize tasks to intelligent fiber optic distribution robots with high health status that manage fiber core areas.

[0047]

[0048] Among them, W i ω1 represents the dynamic weight of the i-th logical resource pool; a higher weight indicates that the logical resource pool is selected with higher priority during task allocation. ω1 and ω2 are weighting coefficients used to adjust the relative importance of load conditions and fiber core average health, ω1 + ω2 = 1. IdleCores i TotalCore represents the number of currently idle fiber cores in the i-th logical resource pool, reflecting the load status of that logical resource pool. More idle fiber cores indicate a lower load. i Hpool represents the total number of fiber cores in the i-th logical resource pool. i The average health score of the fiber core in the i-th logical resource pool is obtained by averaging the health scores of all fiber cores in the logical resource pool. It reflects the health status of the fiber core; the higher the score, the better the fiber core health.

[0049]

[0050] Where n is the number of intelligent fiber optic distribution robots in the logical resource pool; RH i RH represents the health score of the i-th intelligent fiber optic distribution robot, with a value ranging from 0 to 1. i =Hscore i / 100; IC i This represents the number of idle fiber cores accessed by the i-th intelligent fiber optic distribution robot. This parameter reflects the remaining resources available for task allocation by the intelligent fiber optic distribution robot. The more idle fiber cores a robot has, the more likely it should be considered for task allocation in the next round.

[0051] Among them, Hscore i The health prediction value of the i-th intelligent fiber optic distribution robot is expressed as a percentage. In practical applications, there are many models to evaluate the health of various types of intelligent fiber optic robot equipment and operating conditions. For example, a trained LSTM model can be used to analyze the relevant operating data of the currently collected intelligent fiber optic distribution robot. This is not a key technical point of this patent and will not be elaborated here.

[0052] In step S2, see the appendix to the instruction manual. Figure 2 Determining the priority of the logical resource pool and its contained intelligent fiber optic distribution robots includes the following steps:

[0053] S201. Determine the priority of the logical resource pool by sorting according to the size of the dynamic weight;

[0054] S202. Determine the priority of each intelligent fiber optic distribution robot in the logical resource pool based on the health status and number of idle fiber cores of the intelligent fiber optic distribution robot.

[0055] Specifically, in steps S201 and S202, based on the dynamic weight W... i The sorting results are used to generate a priority queue for the logical resource pool. If the logical resource pool contains multiple intelligent fiber optic distribution robots, the priority queue is further determined based on the health rating (RH) of the intelligent fiber optic distribution robots. i and idle fiber core count IC iThis process generates a priority queue for intelligent fiber optic distribution robots within the logical resource pool. During subsequent route calculations, the logical resource pool with the highest weight and the fiber cores managed by the intelligent fiber optic distribution robots are prioritized for connection. Step S2 identifies one or more high-priority logical resource pools and relatively better intelligent fiber optic distribution robots within those pools. This defines the resource selection range for subsequent reinforcement learning path matching, allowing reinforcement learning to perform fiber core path selection based on the logical resource pools, rather than on a large and unordered set of physical fiber cores, thus significantly reducing the search space.

[0056] In step S3, a search algorithm is used to filter a set of feasible paths that meet the business requirements from the logical resource pools determined based on the priorities. To determine the path search range, firstly, several logical resource pools with the highest weights are selected to ensure that the source and destination nodes belong to these pools. Then, for each selected logical resource pool, the intelligent fiber optic distribution robots participating in the path search are determined based on their internal intelligent fiber optic distribution robot priority queues. The set of fiber cores managed by the selected intelligent fiber optic distribution robots is used as the initial range for the path search. Thus, from the large physical fiber core set, based on the priorities of the logical resource pools and intelligent fiber optic distribution robots, a relatively small logical resource pool-intelligent fiber optic distribution robot-fiber core set is selected, ensuring fiber core quality and a balanced business fiber core load, providing a reasonable starting space for subsequent path searches. Then, based on network topology information, a search algorithm is used to calculate a set of feasible paths from the source node to the destination node from this fiber core set, serving as the action space for reinforcement learning.

[0057] It should be noted that, due to the selection of a relatively small logical resource pool – intelligent fiber optic distribution robot – fiber core set, there is a possibility that a feasible path cannot be found from this fiber core set. When this occurs, the logical resource pool is expanded downwards according to its priority queue, from high to low. Furthermore, based on the priority of the intelligent fiber optic distribution robot queue, the logical resource pool – intelligent fiber optic distribution robot – fiber core subset is redefined until a feasible path is found or all logical resource pools are traversed.

[0058] In one embodiment, the A* algorithm is used to compute the set of alternative paths A. f When a service request is received and the start and end nodes of the Availability Zone (AZ) are determined, the A* algorithm is used to calculate the possible fiber core paths from the start to the end point based on the network topology information. The A* algorithm, based on breadth-first search, uses heuristic search to estimate the cost from node n to the target node. At each step, when selecting the next node to expand, the A* algorithm prioritizes the node with the smallest f(n) value.

[0059] f(n)=φ1×g(n)+φ2×h(n)+φ3×(1-RH n )

[0060] Where g(n) is the actual cost from the starting node to node n, characterized by a normalized value of the sum of the physical lengths of all fiber optic lines traversed from the starting node to node n; h(n) is the estimated cost from node n to the target node, characterized by a normalized value of the shortest hop count traversed from node n to the target node; 1-RH n The health loss cost (RH) of the intelligent fiber optic distribution robot corresponding to node n. n The robot's health score is normalized and ranges from 0 to 1; φ1, φ2, and φ3 are weights, and their sum is 1. This way, when calculating the link, the impact of cable length on optical signal attenuation is considered, as well as the connection relationships of subsequent nodes and the current health of the selected intelligent fiber optic distribution robot, minimizing the impact of the intelligent fiber optic distribution robot's patching on optical power attenuation.

[0061] See the instruction manual appendix Figure 3 In reinforcement learning path matching, a path is selected from the set of feasible paths as an action to be executed, and the immediate reward and the expected value of the cumulative reward are calculated based on the execution result, including the following steps:

[0062] S301. A greedy strategy is used to select a path from the set of feasible paths as the action to be executed; wherein, if the generated random number is less than the set exploration rate, a path is randomly selected from the set of feasible paths as the action to be executed; if the generated random number is greater than the set exploration rate, the path with the largest expected cumulative reward value is selected from the set of feasible paths as the action to be executed.

[0063] S302. Calculate the immediate reward after the action is executed based on the set reward strategy, and calculate the expected cumulative reward after the action is executed based on the immediate reward, the set learning rate and discount factor, the expected cumulative reward when the action is not executed, and the expected cumulative reward among all feasible paths.

[0064] Specifically, for each path in the set of feasible paths, its value is evaluated based on the current Q-value. This involves learning to maintain a Q-value table to record the long-term cumulative reward expectation of performing an action in a given state. In the fiber optic cabling scenario, each row of the Q-value table corresponds to the real-time state s at a given time t, and each column corresponds to a possible action (i.e., the fiber core connection path, calculated by the A* algorithm). The evaluation value Q′(s) after performing this action is... t ,a t Its update formula is:

[0065] Q′(s t ,at )=Q(s t ,a t )+α[r t +γmax a′ Q(s t+1 ,a′)-Q(s t ,a t )]

[0066] Where Q(s) t ,a t ) is in state s t The fiber core connection path action a was not executed. t The Q value stored at time, Q′(s) t ,a t ) to execute the fiber core connection path action a t The Q value after that indicates the state s. t The expected long-term cumulative reward for choosing this path; α represents the learning rate, which takes values ​​between [0,1]. It controls the degree to which new information is learned each time the Q-value is updated. The closer α is to 1, the greater the impact of newly acquired reward information on the Q-value update; the closer α is to 0, the more the Q-value update depends on previous experience; r t To perform action a t The immediate reward obtained afterward is used to measure the direct effect of the action; the γ discount factor, ranging from [0,1], is used to measure the importance of future rewards. The closer γ is to 1, the more importance is placed on future rewards, considering the long-term impact of the current action on subsequent states and rewards; the closer γ is to 0, the more attention is paid to immediate rewards; s t+1 Indicates the execution of action a t The new state to which the user transitions reflects the change in state caused by the execution of an action; max a′ Q(s t+1 ,a′) represents the new state s t+1 The largest Q-value among all feasible fiber core connection paths a′ represents the maximum expected long-term cumulative reward that can be obtained by choosing the optimal path in the new state. Initially, for each path in the set of feasible paths, the Q-value is usually initialized to a default value, such as 0. This is because sufficient experience has not yet been accumulated, and the value of each path is not clearly determined.

[0067] Wherein, state s t It can be represented by a multidimensional vector, s t =[T,W,H score[M], T represents network topology information used to determine network physical connectivity, including inter-node connections, link length, and optical power attenuation, and is composed of three N×N matrices representing connectivity, link length, and optical power attenuation, respectively; W represents the dynamic weights of each logical resource pool, used to guide path search, and is a one-dimensional array. H score The health prediction value of each intelligent fiber optic distribution robot is represented by a one-dimensional array. M represents the fiber core occupancy status. Fiber cores are numbered and marked with their occupancy status according to the "resource pool-robot-fiber core" sequence, which allows for quick location of available fiber cores in the resource pool. This is a two-dimensional array.

[0068] In the Q-value calculation formula for each path, action 'a' represents the selection of a feasible fiber connection path from the source node to the destination node, i.e., selecting a specific combination of fiber cores to establish an optical path connection. For example, selecting fiber core 1 from node A, passing through fiber core 5 at intermediate node B, and finally connecting to fiber core 8 at the destination node C, constitutes one action. In a certain state, an action (i.e., selecting a path) is chosen from the set of feasible paths, following a greedy strategy: a path is randomly selected from the set of feasible paths with probability ε; the path with the highest Q-value is selected with probability 1-ε.

[0069]

[0070] Among them, a t Let A be the action chosen at time t; ε is the exploration rate, ranging from [0,1]. In the early stages of learning, ε is set to a larger value (e.g., 0.3) to allow more opportunities to explore different actions; as learning progresses, ε gradually decreases, favoring the action currently considered optimal. f It is a set of feasible paths that meet the optical power requirements, calculated by the A* algorithm based on network topology and service requests (originating and ending nodes, optical power requirements, etc.). In complex fiber optic networks, the A* algorithm uses heuristic functions to estimate the cost from the current node to the target node, thereby quickly selecting feasible paths; Q′(s t a) is in state s t The Q value of the action 'a' is the value in state 's'. t The expected long-term cumulative reward of choosing action a.

[0071] Assume the current state is s t Path a was selected from the set of feasible paths. t After executing this path selection action, an immediate reward r is obtained based on information such as whether the connection was successfully established and whether the optical power meets the requirements. tThen, based on the Q-value update formula, the new Q-value is calculated. For example, if the connection is successfully established and the optical power meets the requirements, r t The value is positive (e.g., +1); if the connection fails or the optical power is insufficient, r t It is negative (e.g., -1). α is the learning rate, which controls the influence of new reward information on Q-value updates; γ is the discount factor, which measures the importance of future rewards. a′ Q(s t+1 ,a′) represents the new state s t+1 The maximum Q value among all possible actions (i.e., all feasible fiber core connection paths) is continuously updated iteratively, and the Q value gradually reflects the value of each path under different states. With accumulated experience, the master node evaluates the value of paths based on the updated Q value and prioritizes paths with higher Q values ​​to achieve better decision-making results.

[0072] The immediate reward after an action is performed is calculated based on a set reward strategy. This reward strategy can be set according to one or more indicators such as connection quality, fiber core health, service objectives, and path efficiency. For example, based on the selected action, a fiber jumper command is issued to a designated intelligent fiber optic distribution robot. The intelligent fiber optic distribution robot performs a fiber core connection operation. If the connection is successfully established and meets the optical power requirements, an immediate reward of +1 is given; if the connection fails, an immediate reward of -1 is given; if the connection is successfully established but exceeds the optical power threshold, a small negative reward (such as -0.5) is given. The specific settings are determined according to the actual application, and this application does not limit or fix this.

[0073] The aforementioned processes of state information acquisition, action selection, action execution, reward calculation, and policy update are iterated repeatedly. With each iteration, the policy is continuously learned and optimized, gradually finding a better core path selection method. Thus, by combining the A* algorithm with Q-value reinforcement learning, network topology information can be utilized more effectively during the action selection phase to find better core paths for service requests, overcoming the limitations of traditional path selection methods and avoiding blind searching or reliance on a single factor for decision-making. Simultaneously, the learning capabilities of reinforcement learning are used to continuously optimize the policy, improving SLA (Service-Level Agreement) satisfaction and resource utilization to adapt to dynamic changes in network conditions.

[0074] As can be seen, the proposed intelligent fiber optic distribution scheduling method based on reinforcement learning abstracts physical fiber cores into logical resource pools and employs dynamic weighted partitioning scheduling to achieve optimized resource allocation. This breaks through the physical limitations of traditional resource allocation and is no longer restricted to the range of fiber cores managed by a single intelligent fiber optic distribution robot. By calculating dynamic weights for each logical resource pool based on real-time load and fiber core health, service requests can be accurately allocated to the optimal robot management area, achieving robot load balancing and overload avoidance. Furthermore, by using reinforcement learning technology to match service requirements with resource status in real time and dynamically generate optimal fiber core paths, it can quickly find fiber core paths that meet service requirements in application scenarios with high requirements for service activation and recovery speed, improving service response efficiency and reducing service waiting time.

[0075] Based on the same inventive concept, this application also provides an intelligent fiber optic distribution scheduling device based on reinforcement learning decision-making. Since the principle of the device in this application is similar to the intelligent fiber optic distribution scheduling method based on reinforcement learning decision-making described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0076] As per the instruction manual Figure 4 As shown in the illustration, this application also provides an intelligent fiber optic distribution scheduling device based on reinforcement learning decision-making, applied to a cluster composed of multiple intelligent fiber optic distribution robots. The device includes:

[0077] Module 401 is used to abstract the fiber cores of different intelligent fiber optic distribution robots connected to the cluster into a logical resource pool.

[0078] The determining module 402 is used to determine the priority of the logical resource pool and the intelligent fiber optic distribution robot it contains;

[0079] The path matching module 403 is used to determine the scheduling path that meets the business requirements based on the reinforcement learning algorithm. First, a search algorithm is used to filter a set of feasible paths that meet the business requirements from the logical resource pool determined based on the priority. Then, the current state of the optical fiber network is obtained, and a path is selected from the set of feasible paths as an action to be executed. The immediate reward and the expected cumulative reward are calculated based on the execution result. Finally, the path is iteratively optimized based on the calculated expected cumulative reward to determine the scheduling path that meets the business requirements.

[0080] In one embodiment, the construction module 401 abstracts the fiber cores of different intelligent fiber optic distribution robots in the cluster into logical resource pools, including: configuring logical identifiers for the fiber core range managed by each intelligent fiber optic distribution robot; and calculating dynamic weights based on the load status and average health of each logical resource pool.

[0081] In one embodiment, the determining module 402 determines the priority of the logical resource pool and the intelligent fiber optic distribution robots it contains, including: determining the priority of the logical resource pool by sorting according to the size of the dynamic weight; and determining the priority of each intelligent fiber optic distribution robot in the logical resource pool according to the health status and the number of idle fiber cores of the intelligent fiber optic distribution robots.

[0082] In one embodiment, the path matching module 403 employs a search algorithm to filter a set of feasible paths that meet business requirements from a logical resource pool determined based on the priority. This includes: selecting a predetermined number of logical resource pools and their contained intelligent fiber optic distribution robots according to priority order to obtain a managed fiber core set; calculating a set of feasible paths from the source node to the destination node from the fiber core set using the A* algorithm based on network topology information; determining whether a path that meets business requirements exists in the set of feasible paths; if so, using the set of feasible paths as the action space for reinforcement learning; if not, expanding the number of selected logical resource pools and their contained intelligent fiber optic distribution robots according to priority order, and recalculating the set of feasible paths until a path that meets business requirements is found.

[0083] In one embodiment, the path matching module 403 selects a path from the set of feasible paths as an action to be executed, and calculates an immediate reward and an expected cumulative reward based on the execution result. This includes: selecting a path from the set of feasible paths as an action to be executed using a greedy strategy; wherein, if the generated random number is less than a set exploration rate, a path is randomly selected from the set of feasible paths as an action to be executed; if the generated random number is greater than the set exploration rate, the path with the largest expected cumulative reward is selected from the set of feasible paths as an action to be executed; calculating the immediate reward after the action is executed based on a set reward strategy; and calculating the expected cumulative reward after the action is executed based on the immediate reward, a set learning rate and discount factor, the expected cumulative reward when the action is not executed, and the largest expected cumulative reward among all feasible paths. The reward strategy is set based on one or more indicators selected from connection quality, fiber core health, business objectives, and path efficiency.

[0084] The intelligent fiber optic distribution scheduling device based on reinforcement learning decision-making described in this application is applied to a cluster composed of multiple intelligent fiber optic distribution robots. A construction module abstracts the fiber cores connected to different intelligent fiber optic distribution robots in the cluster into a logical resource pool. A determination module determines the priority of the logical resource pool and the intelligent fiber optic distribution robots it contains. A path matching module determines the scheduling path that meets the business requirements based on a reinforcement learning algorithm. Specifically, a search algorithm is first used to filter a set of feasible paths that meet the business requirements from the logical resource pool determined based on the priorities. Then, the current state of the fiber optic network is obtained, and a path is selected from the set of feasible paths as an action to be executed. The immediate reward and the expected cumulative reward are calculated based on the execution result. Finally, the path is iteratively optimized based on the calculated expected cumulative reward to determine the scheduling path that meets the business requirements. Thus, by combining reinforcement learning and dynamic weighted scheduling, business needs and resource states are matched in real time, and the optimal fiber core path is dynamically generated. In application scenarios with high requirements for service activation and recovery speed, it can quickly find fiber core paths that meet business requirements, improve service response efficiency, and reduce service waiting time.

[0085] Based on the same concept of the present invention, as shown in the appendix to the specification. Figure 5 As shown in the figure, an embodiment of this application provides the structure of an electronic device 500, which includes: at least one processor 501, at least one network interface 504 or other user interface 503, a memory 505, and at least one communication bus 502. The communication bus 502 is used to realize the connection and communication between these components. The electronic device 500 may optionally include a user interface 503, including a display (e.g., touch screen, LCD, CRT, holographic imaging, or projector, etc.), a keyboard, or a clicking device (e.g., mouse, trackball, touchpad, or touch screen, etc.).

[0086] Memory 505 may include read-only memory and random access memory, and provides instructions and data to processor 501. A portion of memory 505 may also include non-volatile random access memory (NVRAM).

[0087] In some implementations, memory 505 stores executable modules or data structures, or subsets thereof, or extended sets thereof:

[0088] The 5051 operating system contains various system programs used to implement various basic business functions and handle hardware-based tasks.

[0089] Application module 5052 contains various applications, such as desktop launcher, media player, and browser, to implement various application functions.

[0090] In this embodiment, the processor 501 executes steps such as those of an intelligent fiber optic distribution scheduling method based on reinforcement learning decision-making by calling programs or instructions stored in the memory 505.

[0091] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs steps such as those in an intelligent fiber optic distribution scheduling method based on reinforcement learning decision-making.

[0092] Specifically, the storage medium can be a general-purpose storage medium, such as a portable disk or hard drive. When the computer program on the storage medium is run, it can use reinforcement learning technology to match business needs and resource status in real time, dynamically generate the optimal fiber core path, and improve business response efficiency.

[0093] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interface, and the indirect coupling or communication connection of the apparatus or units may be electrical, mechanical, or other forms.

[0094] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0095] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0096] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] Finally, it should be noted that the above embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for intelligent fiber optic distribution scheduling based on reinforcement learning decision-making, characterized in that, Applied to a cluster of multiple intelligent fiber optic distribution robots, the method includes the following steps: The fiber cores of different intelligent fiber optic distribution robots connected to the cluster are abstracted into logical resource pools, including the following steps: configuring logical identifiers for the fiber core range managed by each intelligent fiber optic distribution robot; calculating the dynamic weight of each logical resource pool based on its load and average fiber core health. Determining the priority of the logical resource pool and its contained intelligent fiber optic distribution robots includes the following steps: determining the priority of the logical resource pool by sorting according to the size of the dynamic weight; determining the priority of each intelligent fiber optic distribution robot in the logical resource pool according to the health status and number of idle fiber cores of the intelligent fiber optic distribution robots. The algorithm uses reinforcement learning to determine the scheduling path that meets the business requirements. First, a search algorithm is used to filter the set of feasible paths that meet the business requirements from the logical resource pool determined based on the priority. Then, the current state of the fiber optic network is obtained, and a path is selected from the set of feasible paths as an action to be executed. The immediate reward and the expected value of the cumulative reward are calculated based on the execution result. Finally, the path is iteratively optimized based on the calculated expected value of the cumulative reward to determine the scheduling path that meets the business requirements.

2. The intelligent fiber optic distribution scheduling method based on reinforcement learning decision-making according to claim 1, characterized in that, The step of using a search algorithm to filter a set of feasible paths that meet business requirements from the logical resource pool determined based on the priority includes the following steps: A set number of logical resource pools and their contained intelligent fiber optic distribution robots are selected according to priority order to obtain the managed fiber core set; Based on network topology information, the A* algorithm is used to calculate the set of feasible paths from the source node to the destination node from the fiber core set; Determine whether there is a path in the set of feasible paths that meets the business requirements. If there is, use the set of feasible paths as the action space for reinforcement learning. If not, expand the selected logical resource pool and the number of intelligent fiber optic distribution robots it contains according to priority order, and recalculate the set of feasible paths until a path that meets the business requirements is found.

3. The intelligent fiber optic distribution scheduling method based on reinforcement learning decision-making according to claim 2, characterized in that, in, The status of the fiber optic network includes network topology information, dynamic weights of logical resource pools, and the health of intelligent fiber optic distribution robots; the network topology information includes inter-node connectivity, link length, and optical power attenuation.

4. The intelligent fiber optic distribution scheduling method based on reinforcement learning decision-making according to claim 3, characterized in that, The step of selecting a path from the set of feasible paths as an action and calculating the immediate reward and the expected value of the cumulative reward based on the execution result includes the following steps: A greedy strategy is used to select a path from the set of feasible paths as the action to be executed; wherein, if the generated random number is less than the set exploration rate, a path is randomly selected from the set of feasible paths as the action to be executed; if the generated random number is greater than the set exploration rate, the path with the largest expected cumulative reward value is selected from the set of feasible paths as the action to be executed. The immediate reward after an action is executed is calculated based on the set reward strategy, and the expected cumulative reward after an action is executed is calculated based on the immediate reward, the set learning rate and discount factor, the expected cumulative reward when the action is not executed, and the expected cumulative reward among all feasible paths.

5. The intelligent fiber optic distribution scheduling method based on reinforcement learning decision-making according to claim 4, characterized in that, in, The reward strategy is set based on one or more of the following indicators: connection quality, fiber core health, business objectives, and path efficiency.

6. An intelligent fiber optic distribution scheduling device based on reinforcement learning decision-making, characterized in that, For use in a cluster of multiple intelligent fiber optic distribution robots, the device includes: The module is used to abstract the fiber cores of different intelligent fiber optic distribution robots connected to the cluster into logical resource pools. This includes: configuring logical identifiers for the fiber core range managed by each intelligent fiber optic distribution robot; and calculating the dynamic weights of each logical resource pool based on its load and the average health of the fiber cores. A determining module is used to determine the priority of the logical resource pool and the intelligent fiber optic distribution robots it contains; including: determining the priority of the logical resource pool by sorting according to the size of the dynamic weight; and determining the priority of each intelligent fiber optic distribution robot in the logical resource pool according to the health status and the number of idle fiber cores of the intelligent fiber optic distribution robots. The path matching module is used to determine the scheduling path that meets the business requirements based on the reinforcement learning algorithm. First, a search algorithm is used to filter a set of feasible paths that meet the business requirements from the logical resource pool determined based on the priority. Then, the current state of the fiber optic network is obtained, and a path is selected from the set of feasible paths as an action to be executed. The immediate reward and the expected cumulative reward are calculated based on the execution result. Finally, the path is iteratively optimized based on the calculated expected cumulative reward to determine the scheduling path that meets the business requirements.

7. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of an intelligent fiber optic distribution scheduling method based on reinforcement learning decision-making as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of an intelligent fiber optic distribution scheduling method based on reinforcement learning decision-making as described in any one of claims 1 to 5.

Images