Optical fiber optimal distribution method and device based on fiber distribution rule constraint

Abstract

The invention provides an optical fiber optimal distribution method and device based on optical fiber distribution rule constraint, and the method comprises the steps: determining the current optical fiber distribution state information according to the maximum group number of optical fibers in the current period and remaining orders to be distributed; inputting the current fiber distribution state information into a decision model to obtain current fiber distribution decision action information; performing optical fiber optimal distribution according to an optical fiber length combination distribution strategy in the current optical fiber distribution decision action information to obtain an optical fiber distribution result of the current period; wherein the decision-making model is obtained by training a deep reinforcement learning model according to sample data circularly obtained from an experience pool; the environmental model is constructed based on a first target condition for constraining attribute information of the optical fiber and / or a second target condition for constraining a quantitative score of the optical fiber. According to the invention, intelligent and reasonable optical fiber optimal distribution is realized, and the production and manufacturing performance of the optical cable is improved.

Description

Technical Field

[0001] The present invention relates to the field of optimized access technology, and in particular to an optical fiber optimized distribution method and device based on fiber distribution rule constraints. Background Art

[0002] Optical fiber is the main raw material of optical cables. Different optical cables have different numbers of sheaths and the number of optical fibers in each sheath. Therefore, how to achieve convenient and reasonable optical fiber distribution is an important issue that needs to be urgently solved in the optical cable manufacturing process.

[0003] In the related art, during the production and manufacturing of optical cables, optical fiber distribution is performed by relying on manual experience. However, when distributing optical fibers, it is necessary not only to strictly match the combination of optical fibers and casings according to customer needs, but also to consider multiple goals such as the color matching of the optical fibers, the remaining inventory quantity, the length of the optical fibers, and the time of entry into the warehouse, which leads to the high complexity of optical fiber distribution. Therefore, if optical fiber distribution is performed one by one based on manual experience alone, it is difficult to fully meet the production and manufacturing requirements of each optical cable, which makes it difficult to ensure the rationalization of optical fiber distribution.

[0004] Therefore, there is an urgent need for an optical fiber optimization distribution method and device based on fiber distribution rule constraints to solve the above technical problems. Summary of the invention

[0005] The present invention provides a method and device for optimizing optical fiber distribution based on fiber distribution rule constraints, which is used to solve the defect that the prior art relies on manual experience to distribute optical fibers one by one, which is difficult to fully meet the production and manufacturing requirements of various optical cables, resulting in difficulty in ensuring the rationalization of optical fiber distribution. The method realizes automatic optimization of fiber distribution strategy, so as to optimize optical fiber distribution in an intelligent and rational manner, thereby improving the production and manufacturing performance of optical cables.

[0006] The present invention provides an optical fiber optimization distribution method based on fiber distribution rule constraints, comprising: According to the fiber allocation results of the previous cycle, determine the maximum number of fiber groups and the remaining orders to be allocated in the current cycle; Determine current fiber distribution status information according to the maximum number of optical fiber groups and the remaining orders to be distributed; Inputting the current fiber distribution state information into the decision model to obtain the current fiber distribution decision action information; According to the optical fiber length combination allocation strategy in the current optical fiber allocation decision action information, optical fiber optimization allocation is performed to obtain the optical fiber allocation result of the current cycle; Among them, the decision model is obtained by training the deep reinforcement learning model according to the sample data cyclically obtained from the experience pool; the sample data is formed by interacting with the environmental model, and the sample data includes sample fiber matching state information, sample fiber matching decision action information corresponding to the sample fiber matching state information, the reward value corresponding to the sample fiber matching decision action information, and the next fiber matching state information corresponding to the sample fiber matching decision action information; the environmental model is constructed based on the fiber matching rule constraint conditions; the fiber matching rule constraint conditions include a first target condition for constraining the attribute information of the optical fiber and / or a second target condition for constraining the quantitative score of the optical fiber.

[0007] According to a method for optimizing optical fiber distribution based on fiber distribution rule constraints provided by the present invention, the first target condition is constructed based on the following steps: Constructing a first constraint condition according to a constraint relationship between the length of each optical fiber to be allocated and the longest length of the optical fiber in stock; Constructing a second constraint condition according to the constraint relationship between the length of each optical fiber to be distributed and the major diameter, minor diameter and outer diameter of each optical cable casing to be produced; The first target condition is determined according to the first constraint condition and the second constraint condition.

[0008] According to a method for optimizing optical fiber distribution based on fiber distribution rule constraints provided by the present invention, the second target condition is constructed based on the following steps: Constructing a third constraint condition according to the constraint relationship between the length quantification score of each optical fiber to be allocated and each preset length; Constructing a fourth constraint condition according to the constraint relationship between the color quantitative score of each of the optical fibers to be allocated and each preset color; Constructing a fifth constraint condition according to the constraint relationship between the return-to-stock quantitative score of each of the optical fibers to be allocated and each preset return-to-stock category; Constructing a sixth constraint condition according to the constraint relationship between the time quantification score of each optical fiber to be allocated and the storage time and usage time of each optical fiber to be allocated; Constructing a seventh constraint condition according to the constraint relationship between the inventory score of all the optical fibers to be allocated and the length quantitative score, the color quantitative score, the return to inventory quantitative score and the time quantitative score; The second target condition is determined based on the third constraint condition, the fourth constraint condition, the fifth constraint condition, the sixth constraint condition and the seventh constraint condition.

[0009] According to a fiber optic optimization allocation method based on fiber distribution rule constraints provided by the present invention, the reward value corresponding to the sample fiber distribution decision action information is determined based on the following steps: Combine the number of broken fibers, the number of fiber matching times, the first inventory score, the second inventory score, and the action state value corresponding to the sample fiber matching decision action information to obtain the reward value corresponding to the sample fiber matching decision action information; The first inventory score is the inventory score before optical fiber allocation is performed using the sample fiber distribution decision action information, and the second inventory score is the inventory score after optical fiber allocation is performed using the sample fiber distribution decision action information.

[0010] According to a fiber distribution optimization method based on fiber distribution rule constraints provided by the present invention, the deep reinforcement learning model includes an evaluation network and a target network; The decision model is trained based on the following steps: For the current training, if it is determined that the number of sample data in the experience pool does not reach the preset number, sample data is generated, and the generated sample data is stored in the experience pool in a loop until it is determined that the number of sample data in the experience pool reaches the preset number, then a plurality of sample data are randomly extracted from the experience pool to construct a training data set for the current training; According to the training data set, the parameters of the evaluation network after the last training are optimized to obtain the evaluation network after the current training; When the number of iterative training reaches an integer multiple of the first preset step length, soft-update the parameters of the target network after the previous training based on the parameters of the evaluation network after the current training to obtain the target network after the current training; When the number of iterative training reaches an integer multiple of the second preset step length, verifying the performance of the evaluation network after the current training; According to the verification result, whether the evaluation network after the current training has converged is determined, and if it is determined that the evaluation network after the current training has not converged, the next training is performed iteratively until the evaluation network after the training converges or the maximum number of iterative training times is reached; Among the evaluation networks trained multiple times, the evaluation network trained with the best performance is determined to construct the decision model.

[0011] According to a method for optimizing optical fiber distribution based on fiber distribution rule constraints provided by the present invention, the parameters of the evaluation network after the last training are optimized according to the training data set to obtain the evaluation network after the current training, including: Inputting each of the sample fiber distribution state information in the training data set into the evaluation network after the last training to obtain the value function value of the sample fiber distribution decision action information corresponding to each of the sample fiber distribution state information; Input each of the next fiber distribution state information in the training data set into the evaluation network after the last training, and obtain the optimized fiber distribution decision action information corresponding to the next fiber distribution state information, wherein the value function value of the optimized fiber distribution decision action information is optimal; Input each of the next fiber distribution state information in the training data set into the target network after the last training, and obtain the first value function value of the optimized fiber distribution decision action information output by the first branch network of the target network after the last training, and the second value function value of the optimized fiber distribution decision action information output by the second branch network of the target network after the last training; Determine the minimum value function value between the first value function value and the second value function value as the target value function value of the optimized fiber distribution decision action information; According to the value function value of the sample fiber matching decision action information, the target value function value of the optimized fiber matching decision action information, and the reward value corresponding to the sample fiber matching decision action information, the parameters of the evaluation network after the last training are optimized to obtain the evaluation network after the current training.

[0012] According to a method for optimizing optical fiber distribution based on fiber distribution rule constraints provided by the present invention, the parameters of the evaluation network after the last training are optimized according to the value function value of the sample fiber distribution decision action information, the target value function value of the optimized fiber distribution decision action information, and the reward value corresponding to the sample fiber distribution decision action information to obtain the evaluation network after the current training, including: Obtaining a first objective function value according to the objective value function value of the optimized fiber distribution decision action information and the reward value corresponding to the sample fiber distribution decision action information; Obtaining a second objective function value according to the value function value of the sample fiber distribution decision action information; Obtaining a loss function value according to a deviation between the first objective function value and the second objective function value; Obtaining a gradient coefficient of the loss function value according to the gradient value of the loss function value and a preset coefficient; According to the gradient value and the gradient coefficient, the parameters of the evaluation network after the previous training are optimized to obtain the evaluation network after the current training.

[0013] According to a method for optimizing optical fiber distribution based on fiber distribution rule constraints provided by the present invention, if it is determined that the number of sample data in the experience pool does not reach a preset number, sample data is generated, and the generated sample data is cyclically stored in the experience pool until it is determined that the number of sample data in the experience pool reaches the preset number, including: If the current training is the initial training, then encoding and generating the sample fiber distribution decision action information corresponding to the sample fiber distribution state information according to the sample fiber distribution state information, the fiber distribution quantity constraint condition, the fiber distribution length constraint condition and the fiber distribution strategy constraint condition; If the current training is other training than the initial training, obtaining the sample fiber matching decision action information corresponding to the sample fiber matching state information according to the evaluation network after the last training; According to the sample fiber distribution state information and the sample fiber distribution decision action information, interact with the environmental model to obtain a reward value corresponding to the sample fiber distribution decision action information and next fiber distribution state information corresponding to the sample fiber distribution decision action information; Generate sample data according to the sample fiber distribution state information, the sample fiber distribution decision action information, the reward value corresponding to the sample fiber distribution decision action information, and the next fiber distribution state information corresponding to the sample fiber distribution decision action information, and store the generated sample data into the experience pool; The next fiber distribution state information is used as new sample fiber distribution state information, sample data is iteratively generated, and the generated sample data is cyclically stored in the experience pool until it is determined that the number of sample data in the experience pool reaches the preset number.

[0014] The present invention also provides an optical fiber optimization distribution device based on fiber distribution rule constraints, comprising: The first processing unit is used to determine the maximum number of optical fiber groups and the remaining orders to be allocated in the current cycle according to the optical fiber allocation result of the previous cycle; A second processing unit is used to determine the current fiber distribution status information according to the maximum number of optical fiber groups and the remaining orders to be distributed; An optimization unit, used for inputting the current fiber distribution state information into a decision model to obtain current fiber distribution decision action information; An allocation unit, configured to perform optical fiber optimization allocation according to the optical fiber length combination allocation strategy in the current optical fiber allocation decision action information, and obtain the optical fiber allocation result of the current cycle; Among them, the decision model is obtained by training the deep reinforcement learning model according to the sample data cyclically obtained from the experience pool; the sample data is formed by interacting with the environmental model, and the sample data includes sample fiber matching state information, sample fiber matching decision action information corresponding to the sample fiber matching state information, the reward value corresponding to the sample fiber matching decision action information, and the next fiber matching state information corresponding to the sample fiber matching decision action information; the environmental model is constructed based on the fiber matching rule constraint conditions; the fiber matching rule constraint conditions include a first target condition for constraining the attribute information of the optical fiber and / or a second target condition for constraining the quantitative score of the optical fiber.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the optical fiber optimization distribution method based on fiber distribution rule constraints as described above is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the optical fiber optimization distribution method based on fiber distribution rule constraints as described above is implemented.

[0017] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned optical fiber optimization distribution methods based on fiber distribution rule constraints.

[0018] The optical fiber optimization allocation method and device based on fiber distribution rule constraints provided by the present invention construct an environmental model that can meet various fiber distribution needs based on various fiber distribution rule constraints, and fill the sample data formed by interacting with the environmental model into the experience pool, so as to optimize the evaluation network and the target network in the deep reinforcement model according to the sample fiber distribution state information, sample fiber distribution decision action information, the reward value corresponding to the sample fiber distribution decision action information, and the next fiber distribution state information cyclically obtained from the experience pool, and obtain the decision model, so that the obtained decision model can establish a complete set of algorithm processes for automatic optical fiber distribution optimization, so that the optical fiber distribution comprehensively considers multiple fiber distribution rule constraints such as the maximum number of optical fiber groups, the remaining orders to be distributed, the optical fiber attribute information and the quantitative scoring, realizes the automation and intelligence of the optical fiber distribution process, significantly improves the accuracy, efficiency and rationality of the optical fiber distribution, ensures the comprehensive satisfaction of the optical cable production and manufacturing requirements, effectively reduces the overall inventory optical fiber storage time, improves the optical fiber utilization rate and the fiber distribution efficiency as well as the optical cable production efficiency, thereby effectively solving the problems of high complexity of optical fiber distribution and difficulty in ensuring rationality. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flow chart of the optical fiber optimization distribution method based on fiber distribution rule constraints provided by the present invention.

[0021] Figure 2 It is a structural schematic diagram of the optical cable provided by the present invention.

[0022] Figure 3 It is a schematic diagram of the training process of the traditional D3QN model provided by the present invention.

[0023] Figure 4 It is a schematic diagram of the training process of the improved D3QN model provided by the present invention.

[0024] Figure 5 This is one of the simulation schematic diagrams of the optical fiber optimization distribution method provided by the present invention.

[0025] Figure 6 This is the second simulation schematic diagram of the optical fiber optimization distribution method provided by the present invention.

[0026] Figure 7 This is the third simulation schematic diagram of the optical fiber optimization distribution method provided by the present invention.

[0027] Figure 8 This is the fourth simulation schematic diagram of the optical fiber optimization distribution method provided by the present invention.

[0028] Fig. 9 This is the fifth simulation schematic diagram of the optical fiber optimization distribution method provided by the present invention.

[0029] Fig.10 This is the sixth simulation schematic diagram of the optical fiber optimization distribution method provided by the present invention.

[0030] Fig.11 It is a structural schematic diagram of an optical fiber optimization distribution device based on fiber distribution rule constraints provided by the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] In optical cable manufacturing companies, there are usually three ways to produce optical cables: the first is to manufacture according to customer orders. Orders are accumulated over a long period of time, and the delivery date requirements of different customers are different; the second is to produce according to the company's development plan, and then produce according to customer needs when there are new orders; the third is to combine the above two methods, that is, to produce according to orders while maintaining an appropriate amount of inventory. When manufacturing according to customer orders, the biggest challenge is the rationalization of optical fiber distribution. Optical fiber is the main raw material of optical cable. The number of sleeves and the number of optical fibers in each sleeve of different optical cables are different, and they need to be paired one by one. Short optical fibers generated during the pairing process are usually difficult to reuse, and optical fibers that have been stored in inventory for a long time need to be given priority. Therefore, when matching fibers, not only should the combination of optical fiber and sleeve be strictly matched according to customer needs, but also multiple objectives such as optical fiber color matching, remaining inventory quantity, optical fiber length, and storage time should be considered. Due to the complexity of multi-objective calculations, the traditional method of relying on manual fiber matching is difficult to fully meet the production and manufacturing requirements of various optical cables, resulting in difficulty in ensuring the rationalization of optical fiber distribution. The manufacturing cost of optical fiber accounts for more than 30% of the total optical cable cost. The accuracy and efficiency of fiber distribution directly affect product cost and production efficiency, and have a great impact on corporate competitiveness. Therefore, an optical fiber optimization distribution method and device based on fiber distribution rule constraints are urgently needed to solve the above technical problems.

[0033] In view of the above problems, the present invention proposes a method for optimizing optical fiber allocation based on fiber distribution rule constraints, which interacts with an environmental model constructed based on fiber distribution rule constraints to form sample data, and fills the sample data into an experience pool, so as to optimize the evaluation network and the target network in the deep enhancement model according to the sample fiber distribution state information, sample fiber distribution decision action information, the reward value corresponding to the sample fiber distribution decision action information, and the next fiber distribution state information cyclically obtained from the experience pool, and obtain a decision model, so that the obtained decision model can establish a complete set of algorithm processes for automatic optical fiber allocation optimization, so as to automatically formulate high-quality optical fiber allocation strategies, and then intelligently and rationally perform optical fiber optimization allocation, thereby reducing the overall inventory optical fiber storage time, improving optical fiber utilization and fiber distribution efficiency as well as optical cable production efficiency.

[0034] Figure 1 is a flow chart of an optical fiber optimization allocation method based on fiber distribution rule constraints provided by the present invention; Figure 1 As shown, the method includes: step 110, step 120, step 130 and step 140. The method can be applied to various scenarios where optical fiber optimization allocation is required, such as optical cable production and manufacturing scenarios. The optical fiber optimization allocation method provided in this embodiment is described below using the optical cable production and manufacturing scenario as an example.

[0035] Figure 2 A schematic diagram of the structure of the optical cable provided by the present invention; Figure 2 As shown in the figure, in order to more intuitively introduce the structure of the optical cable, a cross-sectional view of a certain type of optical cable product is used as an example for explanation. The optical cable includes optical fiber, fiber paste (a gel-like substance filled around the optical fiber bundle to protect and fix the optical fiber), casing, aluminum-plastic composite tape, water-blocking cable paste, steel wire reinforcement, and steel wire sheath. Different small circles represent casings, and the number of small circles is defined as the number of casings. The small dots in each casing represent optical fibers, and the number of small dots is defined as the number of cores. That is, the optical cable consists of 8 casings, and each casing consists of 6 optical fibers.

[0036] Step 110, according to the optical fiber allocation result of the previous cycle, determine the maximum number of optical fiber groups and the remaining orders to be allocated in the current cycle.

[0037] Optionally, when the current cycle needs to optimize the fiber allocation, the fiber allocation results of the previous cycle can be obtained first. Since after the corresponding length of fiber is selected according to the previous fiber allocation decision action information, the maximum number of groups and the remaining orders to be allocated in the current cycle will change with the selection action, the maximum number of groups and the remaining orders to be allocated in the current cycle can be dynamically determined based on the fiber allocation results of the previous cycle.

[0038] Here, the steps for obtaining the fiber optic allocation results of the previous cycle include: inputting the previous fiber optic allocation status information formed by the maximum number of fiber groupings and the remaining orders to be allocated in the previous cycle into the decision model, and the decision model outputs the previous fiber optic allocation decision action information, and optimizes the fiber optic allocation for the remaining orders to be allocated in the previous cycle based on the fiber length combination allocation strategy in the previous fiber optic allocation decision action information, thereby obtaining the fiber optic allocation results of the previous cycle.

[0039] Here, the remaining orders to be matched in the current cycle are the orders to be matched except the orders that have been matched in the previous cycle.

[0040] Here, the steps for obtaining the maximum number of fiber groups in the current cycle can be: before the formal fiber distribution, the inventory fiber information needs to be counted according to the input order information. First, the number of sleeves and cores required for the order is clarified, and optical fibers of different colors are selected according to the number of cores, and the maximum number of groups (group_number) is calculated; the size of the number of groups and the number of sleeves is judged to determine whether it is necessary to break the fiber to supplement the number of groups. The length is considered in fiber selection (because the excess part will be wasted, mainly to reduce the waste rate), and the score is considered in fiber breaking (because the remaining part will be put back into the inventory, so the length factor is secondary, mainly to improve the inventory score). The fiber selection logic is judged by the size of the number of groups and the number of sleeves. When the number of groups is greater than the number of sleeves, there is no need to dye and distribute the fibers; when the number of groups is less than the number of sleeves, the fiber selection logic is carried out in the following order until the number of groups is greater than or equal to the number of sleeves: 1. First select the corresponding color fiber in this interval, 2. Dye the colorless length of this interval, 3. Break the same color long fiber, 4. Dye the colorless long fiber first and then break the fiber. As can be seen from the above logic, there is a mapping relationship between the state of the inventory fiber information and the maximum number of groups. Therefore, the maximum number of fiber groups in the current cycle can be obtained by association based on the state of the inventory fiber information after the fiber allocation in the previous cycle through the above logic.

[0041] Step 120: Determine current fiber distribution status information according to the maximum number of optical fiber groups and the remaining orders to be distributed.

[0042] Optionally, after obtaining the maximum number of optical fiber groups and the remaining orders to be matched in the current cycle, the current fiber matching status information can be jointly constructed based on the maximum number of optical fiber groups and the remaining orders to be matched in the current cycle, so that the current fiber matching status information can complete the order status information and optical fiber information, so as to optimize and obtain the fiber matching decision action information of the current cycle efficiently and accurately based on this fiber matching status information. Here, the maximum number of optical fiber groups and the remaining orders to be matched in the current cycle can be combined to form the current fiber matching status information. At this time, the dimension of the current fiber matching status information is the sum of the dimension of the maximum number of optical fiber groups and the dimension of the remaining orders to be matched, that is, the current fiber matching status information St, which can be expressed as: St = {maximum number of groups, remaining orders to be matched}.

[0043] Step 130, input the current fiber distribution state information into the decision model to obtain the current fiber distribution decision action information. Wherein, the decision model is obtained by training the deep reinforcement learning model according to each sample data obtained cyclically from the experience pool; the sample data is formed by interacting with the environmental model, and the sample data includes sample fiber distribution state information, sample fiber distribution decision action information corresponding to the sample fiber distribution state information, reward value corresponding to the sample fiber distribution decision action information, and next fiber distribution state information corresponding to the sample fiber distribution decision action information; the environmental model is constructed based on the fiber distribution rule constraint condition; the fiber distribution rule constraint condition includes a first target condition for constraining the attribute information of the optical fiber and / or a second target condition for constraining the quantitative score of the optical fiber.

[0044] Optionally, after obtaining the current fiber distribution state information, the current fiber distribution state information can be used as the input of the decision model, and the decision model selects the candidate fiber distribution state information with the best value function value from multiple candidate fiber distribution state information based on the current fiber distribution state information as the current fiber distribution decision action information. Among them, the current fiber distribution decision action information includes all length combination allocation strategies that meet the order length limit, that is, the current fiber distribution state information At, which can be expressed as: At={length combination allocation strategy that meets the order length limit}.

[0045] The decision model here can be trained based on the following steps: First, an environmental model is constructed based on a first target condition for constraining the attribute information of the optical fiber and / or a second target condition for constraining the quantitative score of the optical fiber. The attribute information is information on the optical fiber attributes, including but not limited to the number of optical fibers and the length of the optical fiber; the quantitative score is formed based on various fiber selection principles, including but not limited to one or more of the length quantitative score, color quantitative score, return to stock quantitative score, time quantitative score and inventory score.

[0046] In some embodiments, the first target condition is constructed based on the following steps: constructing a first constraint condition based on the constraint relationship between the length of each optical fiber to be distributed and the longest length of the optical fiber in stock; constructing a second constraint condition based on the constraint relationship between the length of each optical fiber to be distributed and the major diameter, minor diameter and outer diameter of each optical cable casing to be produced; determining the first target condition based on the first constraint condition and the second constraint condition.

[0047] In some embodiments, the second target condition is constructed based on the following steps: constructing a third constraint condition based on the constraint relationship between the length quantitative score of each optical fiber to be allocated and each preset length; constructing a fourth constraint condition based on the constraint relationship between the color quantitative score of each optical fiber to be allocated and each preset color; constructing a fifth constraint condition based on the constraint relationship between the return to warehouse quantitative score of each optical fiber to be allocated and each preset return to warehouse category; constructing a sixth constraint condition based on the constraint relationship between the time quantitative score of each optical fiber to be allocated and the storage time and usage time of each optical fiber to be allocated; constructing a seventh constraint condition based on the constraint relationship between the inventory score of all the optical fibers to be allocated and the length quantitative score, the color quantitative score, the return to warehouse quantitative score and the time quantitative score; determining the second target condition based on the third constraint condition, the fourth constraint condition, the fifth constraint condition, the sixth constraint condition and the seventh constraint condition.

[0048] Optionally, the steps of constructing the environment model specifically include: The fiber distribution problem in optical cable production can be described as follows: In the production scheduling task, the bundle number set N, the length set L, the sleeve number K, the tube core number P are known, Li is the length of product i, Ni is the bundle number of product i, and each length l corresponds to a bundle number n, that is, the bundle number set N and the length set L have a one-to-one correspondence. Under the premise of meeting the production scheduling task requirements, the manufacturer allocates the bundle number n based on factors such as the inventory fiber quantity and length and the total length of the production scheduling task ∑Li Ni, the number of sleeves K, and the number of cores P provide a reasonable optical fiber length combination plan, namely the fiber matching plan, which increases the proportion of high-quality optical fibers in inventory and reduces the number of broken fibers during fiber matching.

[0049] In addition, the fiber selection principles during fiber distribution include but are not limited to the following: (1) The maximum length of a single optical fiber is a preset length, such as 66 kilometers; (2) The length of the production section does not exceed the limited length; (3) The number of broken fibers during optical cable production is reduced; (4) The number of fiber distribution times during optical cable generation is reduced; (5) Colored fibers are given priority to reduce the proportion of colored fibers in inventory; (6) Returned fibers are given priority to reduce the proportion of returned fibers in inventory; (7) Fibers that have been stored for a long time are given priority to reduce the proportion of fibers that have been stored for a long time in inventory.

[0050] Therefore, in order to quantitatively describe the complex fiber matching scenarios with multiple constraints, the following mathematical model of fiber matching rule constraints was established based on the above fiber selection principles: ; ; ; ; ; ; ; Where li is the length of optical fiber i to be distributed, is the longest length of the stock optical fiber, C1 is the first constraint, which is used to constrain the length of a single optical fiber to be allocated to be less than or equal to the longest length of the stock optical fiber; Rb, Rs, , and are the major diameter, minor diameter, width, pi, and outer diameter of the casing, respectively; C2 is the second constraint condition, which is used to constrain the length of the optical fiber to be allocated, that is, the relationship between the length limit of the produced casing and the major diameter, minor diameter, and outer diameter of the casing; Li is the length quantitative score of the optical fiber i to be allocated, and I is the optical fiber set to be allocated, which includes multiple optical fibers to be allocated; C3 is the third constraint condition, which is used to establish a constraint relationship between the length quantitative score of each optical fiber to be allocated and each preset length, and the preset length here can be 8km to ensure that short fibers of 8km and below are used preferentially; Ci is the color quantitative score of the optical fiber i to be allocated, ci is the color of the optical fiber i to be allocated, G and M are the color sets {B, OR, G, BR, GR, R, BL, Y, V, P, AQ}, {N}, where N, B, OR, G, BR, GR, R, BL, Y, V, P, AQ are colorless, blue, orange, green, brown, white, red, black, yellow, blue-purple, purple, and light green, respectively. C4 is the fourth constraint condition, which is used to establish a constraint relationship between the color quantitative score of each optical fiber to be allocated and each preset color, so as to ensure the priority use of colored fiber; Ui is the return to warehouse quantitative score of the optical fiber i to be allocated, ki is the return to warehouse status of the optical fiber i to be allocated, which is used to characterize whether the optical fiber i to be allocated is a return to warehouse optical fiber; and are respectively a set of fiber returning to the library and a set of fiber not returning to the library; C5 is the fifth constraint condition, which is used to establish a constraint relationship between the quantitative score of the fiber returning to the library to be allocated and each preset return category to ensure that the fiber returning to the library is used preferentially; Di is the time quantitative score of the fiber i to be allocated, and are the usage time and storage time respectively; C6 is the sixth constraint condition, which is used to establish the constraint relationship between the time quantification score of each optical fiber to be allocated and the storage time and usage time of each optical fiber to be allocated, so as to ensure that the optical fiber with longer storage time is used first; Ψ is the inventory score, , , , is the weight coefficient, is the total number of optical fibers contained in the optical fiber set to be allocated; C7 is the seventh constraint condition, which is used to establish the constraint relationship between the inventory score and the length quantitative score, color quantitative score, return to inventory quantitative score and time quantitative score, so as to ensure that the four fiber selection principles are taken into account at the same time when selecting fibers.

[0051] Subsequently, based on the mathematical model constrained by the above fiber distribution rule, the state space, executable action space, state transition rules and action timely reward rules that can be identified by the evaluation network in the deep reinforcement learning model are constructed, thereby obtaining the environment model. The environment model is the place where the network learns the fiber distribution strategy, and also feedbacks the network's fiber distribution performance through rewards and state transitions. It is a core concept in reinforcement learning. It is an external system that the evaluation network interacts with to obtain experience and feedback. It defines the impact of the agent's actions on its state, as well as the reward mechanism.

[0052] After obtaining the environment model, we can collect trajectory fragments (also called trajectory) formed by multiple interactions with the environment module. For example, the trajectory generated by the interaction between the evaluation network and the environment model is (s0, a0, r0, s1, s1,a1, r1, s2, s2, a2 ​​r2, s3, s3, a3, r3, s4,..., sn, an, rn, sn+1), and split the trajectory into (si, ai, ri, si+1) in the form of , and then put it into the experience pool to provide sample data for the subsequent deep reinforcement learning model. If the storage quantity of sample data in the experience pool is not full, trajectory segmentation will continue to be generated according to the above method until the experience pool is full. Subsequently, sample data including sample fiber matching state information, sample fiber matching decision action information corresponding to the sample fiber matching state information, reward value corresponding to the sample fiber matching decision action information, and next fiber matching state information corresponding to the sample fiber matching decision action information are obtained cyclically from the experience pool, so as to iteratively train the deep reinforcement learning model based on each sample data, and construct a decision model based on the trained deep reinforcement learning model, which can efficiently and accurately optimize and make decisions on real-time fiber matching decision action information according to the real-time fiber matching state information.

[0053] Here, the experience pool (or replay buffer) is used to store the data generated by the interaction between the agent and the environment, and to sample from it during training to update the agent's strategy. The replay buffer has two main functions: 1. Break data correlation; 2. Make more effective use of past experience.

[0054] The sample fiber matching status information here may be status information generated by the fiber matching during the sample cycle. The specific acquisition steps may be referred to step 110 and will not be described again here.

[0055] The sample fiber matching decision action information here can be the sample fiber matching decision action information encoded and generated according to the sample fiber matching state information, fiber matching quantity constraints, fiber matching length constraints and fiber matching strategy constraints; or, the sample fiber matching decision action information output by the evaluation network decision in the deep reinforcement learning model can be adaptively selected according to different training stages.

[0056] The reward value corresponding to the sample fiber matching decision action information and the next fiber matching state information corresponding to the sample fiber matching decision action information are the reward and state change estimation value generated by executing the sample fiber matching decision action information obtained by interacting with the environment model.

[0057] Step 140: Perform optical fiber optimization allocation according to the optical fiber length combination allocation strategy in the current optical fiber allocation decision action information to obtain the optical fiber allocation result of the current cycle.

[0058] Optionally, after obtaining the current fiber distribution decision action information, all fiber length combination allocation strategies that meet the order requirements in the current cycle can be obtained, so as to allocate fiber to the remaining orders to be distributed in the current cycle according to the fiber length combination allocation strategy, so as to obtain the fiber distribution result of the current cycle. And continue to obtain the maximum number of fiber groups and the remaining orders to be distributed in the next cycle based on the fiber distribution result of the current cycle, so as to obtain the next fiber distribution state information, and optimize the fiber distribution decision action information of the next fiber distribution state information based on the decision model, and repeat this process until all orders to be distributed are configured, that is, the remaining orders to be distributed are empty.

[0059] The method provided in this embodiment constructs an environmental model that can meet various fiber distribution needs based on various fiber distribution rule constraints, and fills the sample data formed by interacting with the environmental model into the experience pool, so as to optimize the evaluation network and the target network in the deep reinforcement model according to the sample fiber distribution state information, sample fiber distribution decision action information, the reward value corresponding to the sample fiber distribution decision action information, and the next fiber distribution state information cyclically obtained from the experience pool, and obtain a decision model, so that the obtained decision model can establish a complete set of algorithm processes for automatic fiber distribution optimization, so that the fiber distribution comprehensively considers multiple fiber distribution rule constraints such as fiber attribute information and quantitative scoring, realizes the automation and intelligence of the fiber distribution process, significantly improves the accuracy, efficiency and rationality of fiber distribution, ensures the comprehensive satisfaction of optical cable production and manufacturing requirements, effectively reduces the overall inventory fiber storage time, improves fiber utilization and fiber distribution efficiency as well as optical cable production efficiency, thereby effectively solving the problems of high complexity of fiber distribution and difficulty in ensuring rationality.

[0060] In some embodiments, the reward value corresponding to the sample fiber distribution decision action information is determined based on the following steps: combining the number of broken fibers, the number of fiber distribution times, the first inventory score, the second inventory score and the action status value corresponding to the sample fiber distribution decision action information to obtain the reward value corresponding to the sample fiber distribution decision action information; wherein the first inventory score is the inventory score before the sample fiber distribution decision action information is used for fiber distribution, and the second inventory score is the inventory score after the sample fiber distribution decision action information is used for fiber distribution.

[0061] Optionally, in order to quantitatively evaluate the quality of the selected action, the reward value Rt corresponding to the sample fiber matching decision action information can be determined based on the weighted sum of the number of broken fibers, the number of fiber matching times and the inventory score, as well as the action state value. The specific calculation formula can be expressed as follows: ; in, The number of broken fibers corresponding to the sample fiber matching decision action information, The stock score after fiber distribution (also known as the second stock score) is Indicates the inventory score before fiber distribution (also known as the first inventory score). Indicates the number of fiber distribution, that is, the number of fiber distribution; , , Represents the weight coefficient for calculating the reward function score; Indicates the action status value corresponding to the sample fiber matching decision action information. The action status value is used to indicate whether the action is legal. =1, indicating that the action is a legal action, =0, it indicates that the action is illegal. and The specific calculation formula of can refer to the calculation formula of the seventh constraint condition C7, which will not be repeated here.

[0062] The method provided in this embodiment accurately calculates and determines the reward value of the sample fiber distribution decision action information by comprehensively considering the number of broken fibers, the number of fiber distribution times, and the changes in inventory scores before and after distribution during the fiber distribution process, thereby achieving an objective quantitative evaluation of the effect of the fiber distribution decision, and further enhancing the training effect and decision-making accuracy of the deep reinforcement learning model in fiber distribution optimization.

[0063] In some embodiments, the deep reinforcement learning model includes an evaluation network and a target network; the decision model is trained based on the following steps: for the current training, if it is determined that the number of sample data in the experience pool does not reach a preset number, sample data is generated, and the generated sample data is stored in the experience pool in a loop until it is determined that the number of sample data in the experience pool reaches a preset number, then multiple sample data are randomly extracted from the experience pool to construct a training data set for the current training; based on the training data set, the parameters of the evaluation network after the previous training are optimized to obtain the evaluation network after the current training; when the number of iterative training reaches the first preset step length When the number of iterative training reaches an integer multiple of the second preset step length, the performance of the evaluation network after the current training is verified; according to the verification result, it is determined whether the evaluation network after the current training has converged. When it is determined that the evaluation network after the current training has not converged, the next training is iterated until the evaluation network after the training converges or the maximum number of iterative training is reached; among the evaluation networks after multiple trainings, the evaluation network after the training with the best performance is determined to construct a decision model.

[0064] Figure 3 A schematic diagram of the training process of the traditional D3QN model provided by the present invention; Figure 4 A schematic diagram of the training process of the improved D3QN model provided by the present invention; Figure 3 As shown, in the traditional D3QN (Dueling Double Deep Q-Network) algorithm, it combines the advantages of Double Q-Learning, Dueling Network Architectures and Deep Q-Network (DQN). By using two Q networks to reduce the Q value overestimation bias, and using a dual network architecture to separate the state value function and the advantage function, D3QN can more accurately evaluate the action value and improve the efficiency of strategy learning. However, this method still has the problem of overestimation of action value; in order to further alleviate the problem of overestimation of action value, the embodiment of the present application is improved on the basis of the traditional D3QN to form an improved D3QN (that is, a deep reinforcement learning model), so that the improved D3QN can use the idea of ​​​​competing between the dual action-advantage function and the state-value function to further alleviate the problem of overestimation of action value and improve the stability of the algorithm during the learning process.

[0065] like Figure 4As shown in the figure, the improved D3QN includes an evaluation network and a target network. The input of the evaluation network is the fiber distribution state information, and the output is the Q value of all fiber distribution decision action information under the fiber distribution state information, that is, the value function value. These Q values ​​represent the expected return of taking a specific action in a given state, and are used to continuously update and train the parameters of the evaluation network.

[0066] The target network is introduced mainly to solve an important problem in reinforcement learning: the target Q value during training is constantly changing, which may lead to unstable training. The role of the target network is to provide a relatively stable target Q value so that the evaluation network can learn from a relatively stable target. In general, the frequency of parameter updates of the target network is much lower than that of the evaluation network. In the embodiment of the present application, a soft update method is used to update the target network parameters, and the update frequency can be updated once when the first preset step length is reached. If the first preset step length is 1, the update frequency of the evaluation network and the target network is the same. That is, each time the evaluation network updates the parameters, the target network parameters will be soft-updated with the new evaluation network parameters, so as to synchronize and update the target network parameters to the latest state of the evaluation network in real time, thereby realizing the close coupling of the evaluation network and the target network parameters, thereby effectively reducing the instability during the training process and improving the smoothness of learning, and promoting the improvement of the model convergence speed.

[0067] Among them, the evaluation network and the target network have the same structure, and both include two branch networks, namely the first branch network and the second branch network, wherein the two branch networks share an input layer and a hidden layer, but different output layers. The number of hidden layers here can be set according to actual needs, such as 2 layers, etc., and this embodiment does not specifically limit this. The input layer, hidden layer and output layer here can all be formed based on a network with the same structure, or they can be formed based on a network with different structures, such as all based on a fully connected network. The activation function of each layer can be adaptively selected according to actual needs, such as selecting a ReLU activation function, etc., and this embodiment does not specifically limit this.

[0068] Optionally, since the improved D3QN is an offline reinforcement learning algorithm, that is, training, evaluation, and exploration are performed alternately within a set step size.

[0069] For the training steps: for the current training, if it is judged that the number of sample data in the experience pool does not reach the preset number, it iteratively interacts with the environment module to generate sample data, and the generated sample data is cyclically stored in the experience pool until it is judged that the number of sample data in the experience pool reaches the preset number, then multiple sample data are randomly selected from the experience pool, that is, a batch of fragments (si, ai, ri, si+1) are randomly selected from the experience pool to construct a training data set for the current training.

[0070] Using the training data set of the current training, the parameters of the evaluation network after the previous training are optimized to obtain the evaluation network after the current training; And determine whether the number of iterative trainings accumulated up to the current training has reached an integer multiple of the first preset step size. If so, the parameters of the target network after the previous training are soft-updated based on the parameters of the evaluation network after the current training to obtain the target network after the current training; if not, the parameters of the target network after the previous training are used as the parameters of the target network after the current training.

[0071] The soft update here can be a weighted fusion of the parameters of the evaluation network after the current training and the parameters of the target network after the previous training, wherein the weight coefficient of the parameters of the evaluation network after the current training is greater than the weight coefficient of the parameters of the target network after the previous training, so as to slowly fuse the parameters of the evaluation network into the target network to maintain a certain stability of the target network while gradually learning new information.

[0072] In addition, it is determined whether the number of iterative training times reaches an integer multiple of the second preset step length. If so, the performance of the evaluation network after training to be evaluated is verified to determine whether the evaluation network after the current training has converged. If the evaluation network after the current training has not converged, the next batch of new fragments are randomly drawn from the experience pool to iterate the next training until the evaluation network after the training converges or the maximum number of iterative training times is reached; if converged, the model training process is terminated.

[0073] For the evaluation phase, it only evaluates the trained evaluation network without back-propagation to update the network parameters. The specific implementation steps may include: First, given an initial fiber distribution state information, named s0, it is input into the trained evaluation network to obtain the Q(s0,a0) value, and the max Q(s0,a0) is found. Then, the fiber distribution decision action information a0 corresponding to the initial fiber distribution state information is input into the environmental model to obtain the reward value r0 corresponding to the fiber distribution decision action information a0 and the next fiber distribution state information s1. Then, the next fiber distribution state information s1 is input into the trained evaluation network to obtain a1, r1, s2, and so on until the interaction ends, and a trajectory is obtained: (s0, a0, r0, s1, s1, a1, r1, s2, s2, a2 ​​r2, s3,s3, a3, r3, s4..., sn, an, rn, sn+1). The reward value r of a trajectory is weightedly added to obtain the performance score. , The larger the value, the better the performance of the evaluation network. It should be noted that if the state transition of the environmental model is uncertain, the average of multiple performance scores needs to be taken as the final performance score result; otherwise, only one performance score is needed as the final performance score result. The performance of the trained evaluation network can be verified based on the final performance score result. Preferably, the state transition of the environmental model for this fiber distribution is a deterministic transition, so only one evaluation is needed to verify the performance of the trained evaluation network.

[0074] Finally, based on the performance scores of the evaluation networks after multiple trainings, the trained evaluation network with the highest performance score is selected to obtain the trained evaluation network with the best performance, so as to build a decision model based on this, which can automatically decide the optimized fiber distribution decision action information according to the real-time fiber distribution status information.

[0075] The method provided in this embodiment effectively realizes an efficient and stable training process by dynamically constructing a training data set, optimizing evaluation network parameters, and implementing soft updates of the target network and evaluation network performance verification at a specific iteration step size, ensuring that before reaching the convergence condition or the maximum number of iterations, the training can be automatically iterated until the evaluation network performance is optimal, thereby building a decision model with excellent performance.

[0076] In some embodiments, according to the training data set, the parameters of the evaluation network after the last training are optimized to obtain the evaluation network after the current training, including: inputting each sample fiber distribution state information in the training data set into the evaluation network after the last training to obtain the value function value of the sample fiber distribution decision action information corresponding to each sample fiber distribution state information; inputting each next fiber distribution state information in the training data set into the evaluation network after the last training to obtain the optimized fiber distribution decision action information corresponding to the next fiber distribution state information, and the value function value of the optimized fiber distribution decision action information is optimal; inputting each next fiber distribution state information in the training data set into the target network after the last training , obtain the first value function value of the optimized fiber distribution decision action information output by the first branch network of the target network after the last training, and the second value function value of the optimized fiber distribution decision action information output by the second branch network of the target network after the last training; determine the minimum value function value between the first value function value and the second value function value as the target value function value of the optimized fiber distribution decision action information; according to the value function value of the sample fiber distribution decision action information, the target value function value of the optimized fiber distribution decision action information, and the reward value corresponding to the sample fiber distribution decision action information, optimize the parameters of the evaluation network after the last training to obtain the evaluation network after the current training.

[0077] Optionally, in the training phase, for the current iteration training, the sample fiber distribution state information formed by the tth decision can be The fiber distribution status information of the samples output by the evaluation network is input into the evaluation network after the last training. Select the sample fiber distribution state information from the value function values ​​of the corresponding multiple candidate fiber distribution decision action information Corresponding sample fiber matching decision action information The corresponding value function value.

[0078] In addition, the next fiber distribution status information Input into the evaluation network after the last training to obtain the next fiber distribution state information output by the first branch network of the evaluation network Multiple candidate fiber matching decision action information The value function value of the fiber distribution decision action information with the largest value function value is selected as the optimized fiber distribution decision action information corresponding to the next fiber distribution state information. .

[0079] In addition, the next fiber distribution status information Input to the target network after the last training to obtain the next fiber distribution state information output by the first branch network of the evaluation network The value function values ​​of the corresponding multiple candidate fiber distribution decision action information are used to select the optimal fiber distribution decision action information. The corresponding first value function value; and obtain the next fiber distribution state information output by the second branch network of the evaluation network The value function values ​​of the corresponding multiple candidate fiber distribution decision action information, and the optimal fiber distribution decision action information is selected from them The corresponding second value function value determines the minimum value function value between the first value function value and the second value function value as the target value function value for optimizing the fiber distribution decision action information.

[0080] Subsequently, the value function value of the sample fiber matching decision action information, the target value function value of the optimized fiber matching decision action information, and the reward value corresponding to the sample fiber matching decision action information are combined to optimize the parameters of the evaluation network after the last training to obtain the evaluation network after the current training.

[0081] The method provided in this embodiment inputs the sample fiber matching state information and the next fiber matching state information in the training data set into the evaluation network and the target network after the last training respectively, uses the minimum value function value output by the dual-branch target network as the optimization benchmark, and combines the value evaluation and reward feedback of the sample fiber matching decision to accurately optimize the evaluation network parameters, thereby effectively improving the prediction accuracy and optimization ability of the evaluation network for future fiber matching decision actions, and obtaining a more accurate evaluation network after the current training.

[0082] In some embodiments, the step of obtaining the evaluation network after the current training further includes: obtaining the first objective function value according to the target value function value of the optimized fiber matching decision action information and the reward value corresponding to the sample fiber matching decision action information; obtaining the second objective function value according to the value function value of the sample fiber matching decision action information; obtaining the loss function value according to the deviation between the first objective function value and the second objective function value; obtaining the gradient coefficient of the loss function value according to the gradient value and the preset coefficient of the loss function value; optimizing the parameters of the evaluation network after the last training according to the gradient value and the gradient coefficient to obtain the evaluation network after the current training.

[0083] Optionally, the step of obtaining the evaluation network after the current training further includes: According to the target value function value of the optimized fiber distribution decision action information and the reward value corresponding to the sample fiber distribution decision action information, the first target function value is obtained by weighted addition: ; in, is the first objective function value; Provide fiber decision action information for samples The corresponding reward value; is the weight coefficient; is the calculation function of the target network; Provide fiber status information for samples The next fiber distribution status information; Optimized fiber distribution decision action information for the next fiber distribution state information; The target network The model parameters of the branch network.

[0084] In addition, according to the value function value of the sample fiber matching decision action information, the second objective function value is obtained, that is, ; Then, the loss function value is obtained according to the deviation between the first objective function value and the second objective function value. , the specific calculation formula is: ; Then, the loss gradient of the loss function value is calculated , the specific calculation formula is: ; Then, using the loss gradient value and the preset coefficient, the gradient coefficient of the loss function value is calculated , the specific formula is: , , =3; Then, using the gradient coefficient Update the loss gradient value to obtain the updated loss gradient value. The specific calculation formula is as follows: = ; in, Hyperparameters set for gradient clipping; To evaluate the network The model parameters of the branch network.

[0085] Subsequently, the updated loss gradient value is used to iteratively back-propagate to update the parameters of the evaluation network after the last training to obtain the evaluation network after the current training.

[0086] In some embodiments, the step of filling the experience pool further includes: if the current training is the initial training, then encoding and generating sample fiber matching decision action information corresponding to the sample fiber matching state information according to the sample fiber matching state information, the fiber matching quantity constraint conditions, the fiber matching length constraint conditions and the fiber matching strategy constraint conditions; if the current training is other training except the initial training, then obtaining the sample fiber matching decision action information corresponding to the sample fiber matching state information according to the evaluation network after the last training; according to the sample fiber matching state information and the sample fiber matching decision action information, interacting with the environmental model to obtain the reward value corresponding to the sample fiber matching decision action information and the next fiber matching state information corresponding to the sample fiber matching decision action information; generating sample data according to the sample fiber matching state information, the sample fiber matching decision action information, the reward value corresponding to the sample fiber matching decision action information and the next fiber matching state information corresponding to the sample fiber matching decision action information, and storing the generated sample data in the experience pool; using the next fiber matching state information as the new sample fiber matching state information, iteratively generating sample data, and cyclically storing the generated sample data in the experience pool until it is determined that the number of sample data in the experience pool reaches a preset number.

[0087] Optionally, when the current training is the initial training, due to factors such as the number of optical fibers, the length of optical fiber segments, the fiber selection principle, and the random initialization of network parameters, it is difficult for the network to output successful fiber matching results in the early stage. In order to help the network learn successful fiber matching results in a short time, and on this basis, self-learn better fiber matching strategies, the method of randomly initializing feasible solutions can be used to replace the traditional process of filling the experience pool by exploring the environment in the early stage of the intelligent agent, thereby increasing the proportion of successful samples in the initial stage, thereby significantly improving the decision-making ability of the model. The specific implementation steps include: According to the sample fiber distribution status information, fiber quantity, fiber segment length, fiber selection principles and other limiting factors, a permutation coding is used to generate a fiber distribution strategy that meets the order requirements. The coding result is then converted into an action set. Based on this, the complete Markov decision chain (s0, a0, r0, s1, s1, a1, r1, s2, s2, a2 ​​r2,s3, s3, a3, r3, s4..., sn, an, rn, sn+1) is obtained by interacting with the environmental model. Finally, the Markov decision chain is divided into (si, ai, ri, si+1) forms and put into the experience pool to provide training data for subsequent network training.

[0088] If the current training is other than the initial training, the fiber distribution state information s0 of each sample in the experience pool can be collected and input into the evaluation network after the last training to obtain the value function value of the fiber distribution decision action information corresponding to the fiber distribution state information of each sample. Strategy, find the decision action corresponding to max Q(s0, a0) or randomly select a decision action as the sample decision action a0 according to the exploration rate.

[0089] Here, the formula for exploring sample decision actions is as follows: ; in, , is the set of action spaces; is the value function value of the fiber distribution decision action information output by the evaluation network after the last training, A random number generated randomly; is the exploration rate.

[0090] Subsequently, the sample decision action a0 is input into the environmental model, and the reward value r0 corresponding to the sample decision action and the next processing state information s1 corresponding to the sample decision action s0 are obtained through interaction with the environmental model. Then s1 is input into the evaluation network to obtain a1, r1, s2, and so on, to obtain a trajectory: (s0, a0, r0, s1, s1, a1, r1, s2,s2, a2 ​​r2, s3, s3, a3, r3, s4..., sn, an, rn, sn+1), and the trajectory is divided into (si, ai, ri, si+1) and put into the experience pool. This cycle is repeated until the end condition is met.

[0091] The method provided in this embodiment replaces the traditional process of the intelligent agent filling the experience pool by exploring the environment in the early stage by adopting a method of randomly initializing feasible solutions, thereby increasing the proportion of successful samples in the initial stage, thereby significantly improving the decision-making ability of the model.

[0092] The optical fiber optimization allocation process provided in this embodiment is described below with reference to a specific example.

[0093] Training parameter settings: exploration rate 0.25, maximum step size 3e5, discount factor 0.98, learning rate 1e-4, batch size 64, sample pool capacity 5e4, model evaluation every few steps 1e3, number of evaluations 1, soft update 5e-3, number of feasible solutions 50, gradient clipping 3.

[0094] First, 50 feasible solutions are randomly initialized and interacted with the environment to convert them into 50 trajectories; secondly, they are divided into segments (si, ai, ri, si+1) and put into the experience pool; then, training is started, 64 segments are extracted from the experience pool, the loss value is calculated by the above method, the evaluation network parameters are updated by back propagation, and then the target network parameters are updated by soft update based on the evaluation network parameters. When the total number of samples extracted for training is equal to the number of samples in the experience pool, training is stopped and the evaluation phase is entered, and the evaluation indicators are output. After the evaluation is completed, the exploration phase begins, and the action selection strategy is explored. Strategy, explore the generated samples and fill them into the experience pool for the next network training. Determine whether the experience pool is full. If the experience pool is full, delete the earliest sample until the new sample is filled into the experience pool. If it is not full, fill it in directly. Determine whether the maximum step length is reached. If it is reached, output the evaluation network and target network parameters and other values ​​saved during training; if not, repeat the above operation. After the training is completed, give the trained evaluation network an initial state s0, and through continuous interaction with the environment, finally obtain a high-quality fiber matching strategy.

[0095] In order to verify the effectiveness of the method (D3QNTF) provided in this embodiment, in the following embodiment, the present invention is used to analyze the most common order combinations of three common orders in optical cable manufacturers: 20 (3km long optical cable, 20 rolls produced), 3 20+5 20,3 20+5 20+7 20 for intelligent fiber matching, and compared it with the strategies formulated by greedy algorithm (Greedy), heuristic algorithms such as genetic algorithm (GA), deep Q network algorithm (DQN), double deep Q network algorithm (Double DQN), and traditional D3QN to verify the effectiveness of this method.

[0096] In order to more intuitively show the changes of this method during training, Figure 5 and Figure 6 As shown, this method will combine the common length 3 20+5 20. In the experiment, the number of sleeves and cores was set to 4 and 12 respectively, the outer diameter of the tube was set to 2.55, the major diameter was set to 100, the minor diameter was set to 50, the width was set to 63, the fiber selection interval was set to 1.02, the redundant length was set to 0.15, and the length factor was set to 1.025. These parameter selections are intended to simulate the actual application environment in order to comprehensively evaluate the performance and stability of this method. In the upper part of the figure, it can be seen that in the initial stage of the training process, due to the random initialization of the network parameters, the returns and rewards obtained by the interaction between the agent and the environment fluctuate greatly. As the training progresses, the fluctuation gradually decreases and finally converges. In the lower part of the figure, as the number of training steps increases, the estimated value of Q gradually stabilizes, and the loss also tends to be stable and basically converges to zero. This shows that as the training progresses, the prediction accuracy and stability of the model are improved. The small oscillations that occur during convergence are mainly caused by random action selection with a small probability.

[0097] In order to intuitively compare the fiber matching effects of different algorithms, Figure 7 , 8 , 9, and 10, the learning curves of each algorithm are smoothed and the results are plotted on Figure 7-10 The results show that DQN has the lowest return value and average reward value. Although the return curve tends to be stable, there are still oscillations and even declines. This is mainly due to the fact that DQN failed to fully converge within a limited step size and its Q value estimation error is large. In contrast, D3QN and Double DQN showed significant improvements. Their loss functions basically converged to zero, the estimation error of the Q value was reduced, and higher rewards and returns were obtained. This method performs better than the previous two, thanks to its sample initialization operation, which further enhances the decision-making ability of the model. Figure 7-10 As shown in the figure, although the convergence speed of this method is reduced due to the need to fit more network parameters, its return and average reward value curves during the learning process are more stable. This is because this method uses a dual action-advantage function and a state-value function, which further alleviates the problem of overestimation of action value and improves the stability of algorithm learning. In contrast, the D3QN curve has obvious oscillations between 110,000 and 200,000 steps. The Q value estimation curves of different algorithms are shown in Figure 1. Fig. 9 shown.

[0098] Table 1 Simulation comparison table

[0099] To further verify the superiority of this method, as shown in Table 1, this method is compared with two traditional optimization algorithms (i.e., greedy algorithm and heuristic algorithm) and three deep reinforcement learning algorithms suitable for discrete space (DQN, DoubleDQN and D3QN). In this experiment, considering the limitation of computer performance, three most common order combinations are selected: 3 20,3 20+5 20,3 20+5 20+7 20. For different order combinations, the state vector St dimensions of DQN, DoubleDQN, D3QN and this method are set to 14, 64, 165 respectively; the action vector At dimensions are set to 13, 62, 162 respectively. The experimental results show that on different test sets, this method is surprisingly better than DQN, Double DQN, D3QN and conventional optimization algorithms. As shown in the figure, in 3 In the 20th example, all algorithms found the optimal combination of fiber distribution, and the final reward reached -9.9958, which shows that the search rules of the greedy algorithm and the mathematical model of the genetic algorithm are both suitable for this scenario. 20+5 When the number of fiber matching is 20, the fiber matching results of the greedy algorithm are significantly lower than those of other algorithms, indicating that the greedy algorithm performs poorly in large-scale solution space problems. Compared with the greedy algorithm, the genetic algorithm searches for high-quality feasible solutions that meet the requirements in the integer programming model for building fiber matching requirements based on the evaluation index function, and the number of broken fibers and the number of fiber matching times are reduced by 100% and 16.67%, respectively. However, the solution obtained by the genetic algorithm is still not as good as that of Double DQN, D3QN and this method. When the case is 3 20+5 20+7 20, this method shows a superior optimization ability to other algorithms. By comparing the result evaluation indicators of different reinforcement learning algorithms, it can be determined that this method will comprehensively consider the number of fiber matching, the number of broken fibers and the inventory score when making decisions. For example, increasing the number of fiber matching can reduce the number of broken fibers, thereby improving the final score. In addition, this method adopts a dual action-advantage and state-value function architecture and uses randomly initialized feasible solutions to fill the experience pool, which makes it show a learning ability that is superior to traditional D3QN. This architecture not only improves the stability of the algorithm, but also significantly enhances its optimization performance in complex environments. Overall, this method demonstrates its excellent ability in dealing with large-scale and high-complexity fiber matching problems.

[0100] The optical fiber optimization distribution device based on fiber distribution rule constraints provided by the present invention is described below. The optical fiber optimization distribution device based on fiber distribution rule constraints described below and the optical fiber optimization distribution method based on fiber distribution rule constraints described above can correspond to each other.

[0101] like Fig.11 As shown, the device includes: a first processing unit 1110 is used to determine the maximum number of optical fiber groups and the remaining orders to be matched in the current cycle according to the optical fiber distribution result of the previous cycle; a second processing unit 1120 is used to determine the current fiber distribution state information according to the maximum number of optical fiber groups and the remaining orders to be matched; an optimization unit 1130 is used to input the current fiber distribution state information into a decision model to obtain the current fiber distribution decision action information; an allocation unit 1140 is used to optimize the optical fiber distribution according to the optical fiber length combination allocation strategy in the current fiber distribution decision action information, and obtain the optical fiber distribution result of the current cycle; wherein the decision model, It is obtained by training the deep reinforcement learning model according to the sample data cyclically obtained from the experience pool; the sample data is formed by interacting with the environmental model, and the sample data includes sample fiber matching state information, sample fiber matching decision action information corresponding to the sample fiber matching state information, the reward value corresponding to the sample fiber matching decision action information, and the next fiber matching state information corresponding to the sample fiber matching decision action information; the environmental model is constructed based on the fiber matching rule constraint conditions; the fiber matching rule constraint conditions include a first target condition for constraining the attribute information of the optical fiber and / or a second target condition for constraining the quantitative score of the optical fiber.

[0102] The device provided in this embodiment realizes the automation and intelligence of the fiber distribution process by comprehensively considering multiple fiber distribution rule constraints such as the maximum number of fiber groups, remaining orders to be distributed, fiber attribute information and quantitative scores, significantly improving the accuracy, efficiency and rationality of fiber distribution, ensuring that the requirements of optical cable production and manufacturing are fully met, so as to effectively reduce the overall inventory fiber storage time, improve fiber utilization and fiber distribution efficiency and optical cable production efficiency, thereby effectively solving the problems of high complexity of fiber distribution and difficulty in ensuring rationality.

[0103] The device provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0104] This embodiment also provides an electronic device, which may include: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus. The processor may call the logic instructions in the memory to execute the optical fiber optimization distribution method provided in the above embodiments.

[0105] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the optical fiber optimization distribution method based on fiber distribution rule constraints provided by the above methods.

[0106] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the optical fiber optimization distribution method based on fiber distribution rule constraints provided by the above methods.

[0107] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0108] Through the description of the above implementation modes, those skilled in the art can clearly understand that each implementation mode can be implemented by means of software plus a necessary general hardware platform, or of course by hardware. Based on such an understanding, the above technical solution can essentially or in other words be embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for optimizing optical fiber distribution based on fiber distribution rule constraints, characterized in that: include: According to the fiber allocation results of the previous cycle, determine the maximum number of fiber groups and the remaining orders to be allocated in the current cycle; Determine current fiber distribution status information according to the maximum number of optical fiber groups and the remaining orders to be distributed; Inputting the current fiber distribution state information into the decision model to obtain the current fiber distribution decision action information; According to the optical fiber length combination allocation strategy in the current optical fiber allocation decision action information, optical fiber optimization allocation is performed to obtain the optical fiber allocation result of the current cycle; Among them, the decision model is obtained by training the deep reinforcement learning model according to the sample data cyclically obtained from the experience pool; the sample data is formed by interacting with the environmental model, and the sample data includes sample fiber matching state information, sample fiber matching decision action information corresponding to the sample fiber matching state information, the reward value corresponding to the sample fiber matching decision action information, and the next fiber matching state information corresponding to the sample fiber matching decision action information; the environmental model is constructed based on the fiber matching rule constraint conditions; the fiber matching rule constraint conditions include a first target condition for constraining the attribute information of the optical fiber and / or a second target condition for constraining the quantitative score of the optical fiber.

2. The optical fiber optimization allocation method based on fiber distribution rule constraints according to claim 1 is characterized in that: The first target condition is constructed based on the following steps: Constructing a first constraint condition according to a constraint relationship between the length of each optical fiber to be allocated and the longest length of the optical fiber in stock; Constructing a second constraint condition according to the constraint relationship between the length of each optical fiber to be distributed and the major diameter, minor diameter and outer diameter of each optical cable casing to be produced; The first target condition is determined according to the first constraint condition and the second constraint condition.

3. The optical fiber optimization allocation method based on fiber distribution rule constraints according to claim 1 is characterized in that: The second target condition is constructed based on the following steps: Constructing a third constraint condition according to the constraint relationship between the length quantification score of each optical fiber to be allocated and each preset length; Constructing a fourth constraint condition according to the constraint relationship between the color quantitative score of each of the optical fibers to be allocated and each preset color; Constructing a fifth constraint condition according to the constraint relationship between the return-to-stock quantitative score of each of the optical fibers to be allocated and each preset return-to-stock category; Constructing a sixth constraint condition according to the constraint relationship between the time quantification score of each optical fiber to be allocated and the storage time and usage time of each optical fiber to be allocated; Constructing a seventh constraint condition according to the constraint relationship between the inventory score of all the optical fibers to be allocated and the length quantitative score, the color quantitative score, the return to inventory quantitative score and the time quantitative score; The second target condition is determined based on the third constraint condition, the fourth constraint condition, the fifth constraint condition, the sixth constraint condition and the seventh constraint condition.

4. The optical fiber optimization allocation method based on fiber distribution rule constraints according to any one of claims 1 to 3, characterized in that: The reward value corresponding to the sample fiber matching decision action information is determined based on the following steps: Combine the number of broken fibers, the number of fiber matching times, the first inventory score, the second inventory score, and the action state value corresponding to the sample fiber matching decision action information to obtain the reward value corresponding to the sample fiber matching decision action information; The first inventory score is the inventory score before optical fiber allocation is performed using the sample fiber distribution decision action information, and the second inventory score is the inventory score after optical fiber allocation is performed using the sample fiber distribution decision action information.

5. The optical fiber optimization distribution method based on fiber distribution rule constraints according to any one of claims 1 to 3, characterized in that: The deep reinforcement learning model includes an evaluation network and a target network; The decision model is trained based on the following steps: For the current training, if it is determined that the number of sample data in the experience pool does not reach the preset number, sample data is generated, and the generated sample data is stored in the experience pool in a loop until it is determined that the number of sample data in the experience pool reaches the preset number, then a plurality of sample data are randomly extracted from the experience pool to construct a training data set for the current training; According to the training data set, the parameters of the evaluation network after the last training are optimized to obtain the evaluation network after the current training; When the number of iterative training reaches an integer multiple of the first preset step length, soft-update the parameters of the target network after the previous training based on the parameters of the evaluation network after the current training to obtain the target network after the current training; When the number of iterative training reaches an integer multiple of the second preset step length, verifying the performance of the evaluation network after the current training; According to the verification result, whether the evaluation network after the current training has converged is determined, and if it is determined that the evaluation network after the current training has not converged, the next training is performed iteratively until the evaluation network after the training converges or the maximum number of iterative training times is reached; Among the evaluation networks trained multiple times, the evaluation network trained with the best performance is determined to construct the decision model.

6. The optical fiber optimization allocation method based on fiber distribution rule constraints according to claim 5 is characterized in that: The step of optimizing the parameters of the evaluation network after the last training according to the training data set to obtain the evaluation network after the current training includes: Inputting each of the sample fiber distribution state information in the training data set into the evaluation network after the last training to obtain the value function value of the sample fiber distribution decision action information corresponding to each of the sample fiber distribution state information; Input each of the next fiber distribution state information in the training data set into the evaluation network after the last training, and obtain the optimized fiber distribution decision action information corresponding to the next fiber distribution state information, wherein the value function value of the optimized fiber distribution decision action information is optimal; Input each of the next fiber distribution state information in the training data set into the target network after the last training, and obtain the first value function value of the optimized fiber distribution decision action information output by the first branch network of the target network after the last training, and the second value function value of the optimized fiber distribution decision action information output by the second branch network of the target network after the last training; Determine the minimum value function value between the first value function value and the second value function value as the target value function value of the optimized fiber distribution decision action information; According to the value function value of the sample fiber matching decision action information, the target value function value of the optimized fiber matching decision action information, and the reward value corresponding to the sample fiber matching decision action information, the parameters of the evaluation network after the last training are optimized to obtain the evaluation network after the current training.

7. The optical fiber optimization allocation method based on fiber distribution rule constraints according to claim 6 is characterized in that: The method optimizes the parameters of the evaluation network after the previous training according to the value function value of the sample fiber distribution decision action information, the target value function value of the optimized fiber distribution decision action information, and the reward value corresponding to the sample fiber distribution decision action information to obtain the evaluation network after the current training, including: Obtaining a first objective function value according to the objective value function value of the optimized fiber distribution decision action information and the reward value corresponding to the sample fiber distribution decision action information; Obtaining a second objective function value according to the value function value of the sample fiber distribution decision action information; Obtaining a loss function value according to a deviation between the first objective function value and the second objective function value; Obtaining a gradient coefficient of the loss function value according to the gradient value of the loss function value and a preset coefficient; According to the gradient value and the gradient coefficient, the parameters of the evaluation network after the previous training are optimized to obtain the evaluation network after the current training.

8. The optical fiber optimization allocation method based on fiber distribution rule constraints according to claim 5 is characterized in that: If it is determined that the amount of sample data in the experience pool does not reach the preset amount, sample data is generated, and the generated sample data is cyclically stored in the experience pool until it is determined that the amount of sample data in the experience pool reaches the preset amount, including: If the current training is the initial training, then the sample fiber distribution decision action information corresponding to the sample fiber distribution state information is encoded and generated according to the sample fiber distribution state information, the fiber distribution quantity constraint condition, the fiber distribution length constraint condition and the fiber distribution strategy constraint condition; If the current training is other training than the initial training, obtaining the sample fiber matching decision action information corresponding to the sample fiber matching state information according to the evaluation network after the last training; According to the sample fiber distribution state information and the sample fiber distribution decision action information, interact with the environmental model to obtain a reward value corresponding to the sample fiber distribution decision action information and next fiber distribution state information corresponding to the sample fiber distribution decision action information; Generate sample data according to the sample fiber distribution state information, the sample fiber distribution decision action information, the reward value corresponding to the sample fiber distribution decision action information, and the next fiber distribution state information corresponding to the sample fiber distribution decision action information, and store the generated sample data into the experience pool; The next fiber distribution state information is used as new sample fiber distribution state information, sample data is iteratively generated, and the generated sample data is cyclically stored in the experience pool until it is determined that the number of sample data in the experience pool reaches the preset number.

9. An optical fiber optimization distribution device based on fiber distribution rule constraints, characterized in that: include: The first processing unit is used to determine the maximum number of optical fiber groups and the remaining orders to be allocated in the current cycle according to the optical fiber allocation result of the previous cycle; A second processing unit is used to determine the current fiber distribution status information according to the maximum number of optical fiber groups and the remaining orders to be distributed; An optimization unit, used for inputting the current fiber distribution state information into a decision model to obtain current fiber distribution decision action information; An allocation unit, configured to perform optical fiber optimization allocation according to the optical fiber length combination allocation strategy in the current optical fiber allocation decision action information, and obtain the optical fiber allocation result of the current cycle; Among them, the decision model is obtained by training the deep reinforcement learning model according to the sample data cyclically obtained from the experience pool; the sample data is formed by interacting with the environmental model, and the sample data includes sample fiber matching state information, sample fiber matching decision action information corresponding to the sample fiber matching state information, the reward value corresponding to the sample fiber matching decision action information, and the next fiber matching state information corresponding to the sample fiber matching decision action information; the environmental model is constructed based on the fiber matching rule constraint conditions; the fiber matching rule constraint conditions include a first target condition for constraining the attribute information of the optical fiber and / or a second target condition for constraining the quantitative score of the optical fiber.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the optical fiber optimization allocation method based on fiber distribution rule constraints as described in any one of claims 1 to 8 is implemented.

Images