D3QN-based wind power blade girder plate production intelligent scheduling method
Through the deep reinforcement learning algorithm based on D3QN, the complexity and constraints in the production schedule of wind power blade beam slabs are solved, efficient equipment utilization and order fulfillment are achieved, and production efficiency is improved.
Patent Information
- Application Number
- CN202410945111.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-12
AI Technical Summary
The production scheduling problem of wind power blade beam slabs is complicated, and the existing metaheuristic algorithms are difficult to find an ideal solution within a limited time, and special constraints such as product sequence and mold replacement time are not fully considered.
Using a deep reinforcement learning algorithm based on D3QN, a multi-objective collaborative optimization model is constructed, transformed into a Markov decision-making process, designed an experience replay pool and improved ε-greedy search strategy to optimize device utilization and order fulfillment rate.
The equipment utilization rate and order fulfillment rate of wind power blade beam slab production have been improved, production time has been shortened, and scheduling efficiency and production efficiency have been improved.
Smart Images

Figure CN118941002A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of parallel machine scheduling and artificial intelligence algorithm, and in particular relates to a D3QN-based intelligent scheduling method for wind turbine blade beam production. Background Art
[0002] Wind turbine blades are important components of wind turbines, which directly affect the conversion efficiency of wind power. Their bearing capacity depends largely on the strength and stiffness of the main beam. The wind turbine blade beam plate is a chord-wise spliced structural component of the wind blade main beam. Its main manufacturing process includes four steps: fiber supply, glue dipping, mold heating and curing molding, traction and winding packaging.
[0003] The beam plate of wind turbine blades is a component product of multiple coils. It adopts a sequential and complete continuous production method, and the product model strictly corresponds to the wind blade model. Depending on the model of the wind blade, a single set of beam plates consists of 6-10 coils of the same model and process but different lengths. The production process and molds of different types of coils are different. When changing the product model, the equipment needs to be shut down and the mold needs to be switched. When starting the equipment, it is necessary to preheat the equipment and prepare the raw materials before entering stable production, which takes a long time and will produce trial machine waste. Most companies adopt a regional production line organization form. Based on the requirements for product performance consistency and rapid assembly, the coil components of a single set of products need to be produced in the same area. And because the coils are large in volume and weight and difficult to move, in order to meet customer needs, they need to be stacked in the order of coil layer numbers when packaging and shipped in sets. Therefore, the production scheduling problem of wind turbine blade beam plates is essentially a distributed parallel machine scheduling problem that considers complex features such as sequential and complete constraints and mold change time.
[0004] In recent years, domestic and foreign scholars have mostly used meta-heuristic algorithms to solve the parallel machine scheduling problem. The existing technology has the following shortcomings: 1. The production scheduling problem is complex and the solution space is huge. It is difficult for meta-heuristic algorithms to find an ideal solution within a limited time; 2. The parallel machine scheduling problem currently studied is not sufficient to consider special constraints such as product sequence and mold change time. With the advent of the era of artificial intelligence, deep reinforcement learning has achieved better performance in solving optimization problems such as site selection and path planning. Therefore, applying deep reinforcement learning to solving production scheduling problems is a feasible research direction. At present, no research has been found on using deep reinforcement learning to solve the scheduling problem of wind turbine blade beams. Summary of the invention
[0005] In order to overcome the above shortcomings, the present invention provides a D3QN-based intelligent scheduling method for wind turbine blade beam plate production, which uses the D3QN algorithm to solve the wind turbine blade pultruded plate scheduling problem to obtain high-quality scheduling results, thereby optimizing the current operating equipment utilization rate and order fulfillment rate.
[0006] In order to achieve the above object, the technical solution adopted by the present invention is: A D3QN-based intelligent scheduling method for wind turbine blade beam production, the specific steps are as follows: S1. Construct a multi-objective collaborative optimization model; S2. Establish a Markov decision process and design two sets of experience replay pools. The establishment of the Markov decision process includes establishing the state space, action space and reward function. The specific method is as follows: S21, state space: select 5 key features to describe the state of the problem, state feature 1 represents the average utilization rate of the running machines, state feature 2 represents the standard deviation of the average utilization rate of the running machines, state feature 3 represents the average processing completion rate of all orders, state feature 4 represents the standard deviation of the average processing completion rate of all orders, and state feature 5 represents the total completion rate of all orders; S22, action space: when selecting scheduling actions, the action selection includes the combination relationship between product-production area, coil-mold, and mold-production equipment, where 5 and 2 scheduling rules are set for coils and equipment respectively. The above two rule sets are combined in pairs to obtain 10 combination rules, which are used as the actions of the deep reinforcement learning algorithm to solve the production scheduling problem of wind turbine blade beams; S23. Reward function: The reward function is defined as follows: r1(t)=f1(t+1) r2(t)=f2(t+1) Among them, r1(t) represents the relationship between the utilization rate of the current operating equipment and the reward, r2(t) represents the relationship between the order fulfillment rate and the reward, and the immediate reward obtained at the decision time t is as follows: r t =1000r1(t)+1000r2(t) The final reward function is expressed as R, which contains the sum of the rewards obtained at U decision moments, as shown in the following formula: Among them, U represents the total number of moments that the agent makes decisions; S3, determine whether the samples in the experience replay pool are full, if not, repeat step S2, if full, execute step S4; S4. Based on the D3QN algorithm, build a valuation network Q E(s,a;θ) and the target value network Q T (s,a;θ′); S5, randomly extract multiple samples in proportion from the two sets of experience replay pools, and train and update the target value network parameters; S6, judging whether the current estimation network has converged, if not, returning to step S5, if converged, executing step S7; S7. The intelligent agent uses the trained improved D3QN algorithm to select the best production schedule and executes it repeatedly to complete the intelligent production scheduling of wind turbine blade beams.
[0007] Further optimization, the model in step S1 is established by describing the production scheduling problem of wind turbine blade beams, with the goal of maximizing the utilization rate of the current operating equipment and minimizing the delivery date deviation.
[0008] Further optimization, the establishment of the Markov decision process in step S2 takes the current state of the production system as input and the scheduling rules as output, and transforms the solution to the production scheduling problem of wind turbine blade beams by designing elements such as state S, action A and reward function R.
[0009] Further optimization, the method for establishing the two groups of experience replay pools in step S2 is: samples are divided into two categories based on Q value and reward level, and are stored in two groups of experience replay pools, each with a capacity of C. When updating the network, samples from the two groups of experience replay pools are drawn in proportion.
[0010] Further optimization, in step S4, the valuation network is used for action selection, and its parameter θ is continuously updated; the target value network is used to calculate the time difference target value Y, and its parameter θ′ is initially fixed and replaced by the latest valuation network parameter θ after a certain time interval.
[0011] Further optimization, in step S6, the ε-greedy strategy is used as the agent action selection strategy, where ε is the greediness, and the value is between 0 and 1. According to the ratio of legal and illegal actions in the current environment, different greediness parameters ε1 and ε2 are set to explore on the legal action set and the full action set respectively.
[0012] Further optimization, the step Step7 is specifically: if the valuation network has converged in step Step6, then use the obtained trained neural network to input the current state of the production system to obtain a decision result; the decision result includes selecting the coil to be processed at the current time and selecting the machine to process the coil; by executing the optimal value scheduling action, refreshing the real-time state and the coil to be processed, and inputting the trained neural network again for optimal production scheduling, and repeatedly executing until the intelligent production scheduling of the wind turbine blade beam plate is completed.
[0013] The beneficial effects of the present invention are: 1. The present invention uses an intelligent algorithm based on deep reinforcement learning to replace manual experience in the production scheduling of wind turbine blade beams; for the production scheduling of wind turbine blade beams with complex characteristics such as sequential set constraints and mold change time, a multi-objective collaborative optimization model is constructed with the goals of maximizing the utilization rate of the current operating equipment and maximizing the order fulfillment rate; 2. The process of solving the wind turbine blade beam production scheduling problem is transformed into a Markov decision process, and the D3QN algorithm is improved and designed for solving it. The ε-greedy search strategy is improved by designing and improving the experience pool block, which can avoid the abandonment of excellent samples on the one hand, and accelerate the convergence speed and improve the speed of learning strategy on the other hand. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 This is a structural diagram of a single set of wind turbine blade beam plate products of the present invention; Figure 2 It is the production scheduling logic model diagram of the present invention; Figure 3 The Markov decision model for the production scheduling problem of the wind turbine blade beam plate of the present invention; Figure 4 This is a flow chart of the improved D3QN algorithm of the present invention; Figure 5 Gantt chart of production scheduling results. DETAILED DESCRIPTION
[0015] In order to more clearly understand the above-mentioned purposes, features and advantages of the present invention, the present invention is described in detail below in conjunction with specific embodiments. The following embodiments are implemented based on the technical solutions of the present invention, and detailed implementation methods and specific operating procedures are given. However, the present invention can also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the following embodiments.
[0016] The D3QN-based intelligent scheduling method for wind turbine blade beam production includes the following steps: S1. Describe the production scheduling problem of wind turbine blade beams and build a multi-objective collaborative optimization model with the goal of maximizing the utilization rate of the current equipment and minimizing the delivery date deviation; S2. Taking the current state of the production system as input and the scheduling rules as output, the solution of the wind turbine blade beam production scheduling problem is transformed into a Markov decision process by designing elements such as state S, action A and reward function R. The sample data is divided into two categories based on the Q value and the high and low reward, and stored in two groups of experience replay pools with a capacity of C respectively; S3, determine whether the samples in the experience replay pool are full, if not, repeat step S2, if full, execute step S4; S4. Based on the D3QN algorithm, two neural networks with the same structure are constructed, namely the valuation network Q E (s,a;θ) and the target value network Q T (s,a;θ′); S5. Randomly extract a number of samples from the two experience replay pools in proportion, adopt an improved ε-greedy search strategy, and train and update the target value network parameters through continuous interaction between the agent and the environment in the deep reinforcement learning network; S6, judging whether the current estimation network has converged, if not, returning to step S5, if converged, executing step S7; S7. The intelligent agent uses the trained improved D3QN algorithm to select the best production schedule and executes it repeatedly to complete the intelligent production scheduling of wind turbine blade beams.
[0017] The problem description in step S1 and the process of establishing the problem model for the production scheduling of wind turbine blade beam plates are as follows: S11. The production scheduling problem of wind turbine blade beams with complex features such as sequential matching constraints and mold changeover time is described as follows: Within a period of time, the company receives G orders for n product models from different customers. Each order contains only one product model and has its own order quantity and delivery date. A single set of products consists of several coils, all of which are produced continuously. There are M production equipments available for scheduling for coils of multiple models, which are distributed in K areas. Each area has L parallel machines, and it is necessary to produce several models and multiple sets of products in parallel. The coils of each set of products can be scheduled to any area, but according to the process requirements, the coils of the same set must be processed on the parallel machines in the same area. The coils of the same set of products must be completed in the order specified by the BOM. When the parallel machine changes the product model, it is necessary to prepare for the changeover. The goal of scheduling is to reasonably arrange the processing equipment and start and completion time of each coil.
[0018] S12. Model input and assumptions: (1) Model input: 1) Order requirements for a certain period of time, including product model, order quantity and delivery date; 2) Production area, number of processing equipment, equipment start-up and shutdown, and mold matching; 3) Component BOM of each model and processing time on the equipment; 4) The adjustment time required for product changeover; (2) Model assumptions: 1) Only consider existing orders and exclude order disturbances; 2) Each processing equipment is an identical parallel machine; 3) Each coil needs to be processed only once and on at most one parallel machine; 4) Each device can only process one coil at a time; 5) Once the processing of each coil begins, the processing cannot be interrupted; 6) There is time for changeover and adjustment when different types of coils are processed successively; 7) Both the parallel machine and the coil can be processed at time t=0, and no adjustment is required; 8) Eliminate abnormal equipment and quality disturbances during processing; 9) The sequential fit constraint only exists between coils of the same set of products.
[0019] S13. The mathematical symbol definitions of the mathematical model of the wind turbine blade beam plate production scheduling problem are established based on the above problem model: O g : Order number, g=1,2,...,G; R g : Order quantity of order g; D g : Delivery date of order g; i: product model, i=1,2,...,n; k: area number, k = 1, 2, ..., K; l: parallel machine number in the region, l = 1, 2, ..., L; M kl : The lth parallel machine in region k; t B : The current planned start time; t C : Current planned completion time; M T : The number of equipment in operation during the current period; q: product number of a certain model, q=1,2,...,Q; j: coil number in a single set of products, j = 1, 2, ..., J; T iqjkl : The jth coil of the qth set of product of model i is in parallel machine M kl Processing time on S ii′kl Parallel Machine M kl From Model Mod i Product Switch to Model Mod i′ The changeover adjustment time; B iqjkl : Continuous variable, the jth coil of the qth set of product of model i is on the parallel machine M kl The start time of work; C iqjkl: Continuous variable, the jth coil of the qth set of product of model i is on the parallel machine M kl Completion time on C g : continuous variable, completion time of order g; v iq :0-1 variable, if the qth set of model i is completed before the delivery date, v iq =1, otherwise v iq =0; x klii′ :0-1 variable, if Mod i ≠Mod i′ , parallel machine M kl From Model Mod i Product Switch to Model Mod i′ If the model needs to be changed and adjusted, then x klii′ =1, otherwise x klii′ =0; y iqjkl :0-1 variable, if the j-th coil of the q-th set of product of model i is in parallel machine M kl On processing, then y iqjkl =1, otherwise y iqjkl =0; wkl iqji′q′j′: 0-1 variable, if the j′ coil of the q′th set of product of model i′ is immediately followed by the coil of the qth set of product of model i in parallel machine M kl If the processing is done on the upper part, then wkl iqji′q′j′=1, otherwise wkl iqji′q′j′=0; z qjkl :0-1 variable, if the j-th coil of a set of products is in parallel machine M kl On processing, then z qjkl =1, otherwise z qjkl =1; S14. Based on the above definition, the following mathematical model can be established for the production scheduling problem of wind turbine blade beams: Objective function: F=(maxf1,maxf2) (1) Constraints: C iqjkl ≥C iq(j+1)kl (5) C iqjkl =B iqjkl +T iqjkl (7) Bi'q'j'kl =C iqjkl +S ii'kl w′x klii ' (8) B iqjkl ≥0,C iqjkl ≥0,C g ≥0 (9) v iq ∈(0,1),x klii' ∈(0,1),y iqjkl ∈(0,1),w′∈(0,1),z qjkl ∈(0,1)(10) Where, i,i′=1,2,...,n; q,q′=1,2,...,Q; j,j′=1,2,...,J; k=1,2,...,K; l=1,2,...,L. Formula (1) indicates that the overall optimization goal of the model is to maximize the utilization rate of the current operating equipment and maximize the order fulfillment rate; Formula (2) indicates the calculation formula of the utilization rate of the current operating equipment; Formula (3) indicates the calculation formula of the order fulfillment rate; Formula (4) indicates the regional production constraint, which ensures that the coils in the same set of products in any order are produced on parallel machines in the same area; Formula (5) indicates the sequence set constraint, which ensures that the larger the coil number j in the same set of products in any order, the greater the completion priority and the earlier the completion time; Formula (6) indicates the coil exclusive constraint Formula (7) represents the continuous production constraint, which ensures that any coil can only be processed on one parallel machine and only once; Formula (7) represents the continuous production constraint, which ensures that the processing process of any coil is continuous and cannot be interrupted once it starts; Formula (8) represents the changeover adjustment constraint. When two coils are processed successively on any parallel machine, there is only adjustment time when the product is changed. The start time of the next coil is determined by the completion time of the previous coil and the changeover adjustment time; Formula (9) ensures that all parameter variables are positive numbers; Formula (10) represents the value range of the decision variable.
[0020] The Markov decision process in step S2 is established and the dual experience replay pool is designed as follows: S21. Deep reinforcement learning is a technology for dealing with sequential decision-making problems. By setting training goals, the intelligent agent can continuously explore and learn in the interaction with the environment to obtain the global optimal solution. In order to build a deep reinforcement learning model based on D3QN, the production scheduling problem of wind turbine blade beams is transformed into a Markov decision process.
[0021] (1) State space The state space of the wind turbine blade beam production scheduling problem represents the relationship between subsequent coils and machine selections. It needs to reflect the main characteristics of the production environment and be closely related to the scheduling goals. Considering the complexity and variability of state characteristics, this paper selects 5 key characteristics to describe the state of the problem. State characteristics 1 and 2 describe the characteristics of the machine: state characteristic 1 represents the average utilization rate of the started machine; state characteristic 2 represents the standard deviation of the average utilization rate of the started machine. State characteristics 3-5 describe the characteristics of the order: state characteristic 3 represents the average processing completion rate of all orders; state characteristic 4 represents the standard deviation of the average processing completion rate of all orders; state characteristic 5 represents the total completion rate of all orders.
[0022] Status feature 1: Status feature 2: Status feature 3: Among them, z g Indicates that at this decision point, order O g The number of coils processed; Status feature 4: Status feature 5: Therefore, the numerical range of the selected input parameters is controlled in the interval [0,1], which can effectively avoid the problem of unsatisfactory final performance of the algorithm due to excessive variation range of related variables such as the number of coils and machines in the actual production process.
[0023] (2) Action Space When selecting a scheduling action, the action selection includes the combination relationship between product-production area, coil-mold, and mold-production equipment.
[0024] For coil selection, set 5 scheduling rules: JR1: Select the coil with the shortest processing time JR2: Select the coil with the longest processing time JR3: Select the coil with the earliest delivery deadline JR4: Select the coil with the lowest critical ratio (critical ratio = (delivery time - current time) / coil processing time) JR5: Select the coil with the shortest relaxation time (relaxation time = coil delivery time - coil processing time) For device selection, set up 2 scheduling rules: MR1: Select the equipment with the lowest utilization MR2: Select the equipment that completes the current task first Combining the above two rule sets together in pairs, a total of 10 combination rules are obtained, which are used as actions for the deep reinforcement learning algorithm to solve the production scheduling problem of wind turbine blade beams.
[0025] (3) Reward Function Rewards are divided into instant rewards and cumulative rewards; For the production scheduling problem mentioned in this article, the reward function needs to take into account the changes in the utilization rate of the current equipment and the order fulfillment rate, in addition to the corresponding production constraints. Therefore, the reward function is defined as follows: r1(t)=f1(t+1) (16) r2(t)=f2(t+1) (17) Among them, r1(t) represents the relationship between the utilization rate of the current operating equipment and the reward, and r2(t) represents the relationship between the order fulfillment rate and the reward. The immediate reward obtained at the decision time t is shown in formula (18): r t =1000r1(t)+1000r2(t) (18) The final reward function is expressed as R, which contains the sum of the rewards obtained at U decision moments, as shown in formula (19) Where U represents the total number of moments the agent makes decisions.
[0026] S22. Double experience replay pool design The samples are divided into two categories based on the Q value and the reward, and are stored in two groups of experience replay pools, each with a capacity of C. When updating the network, samples from the two groups of experience replay pools are drawn in proportion to ensure that the experience pool contains relatively more excellent samples for limited learning of the intelligent agent, avoiding the abandonment of excellent samples.
[0027] The step S4 of constructing two neural networks with the same structure is specifically as follows: Construct two neural networks with the same structure, one as the valuation network Q E (s, a; θ), used for action selection, its parameter θ will be continuously updated; the other is used as the target value network Q T (s, a; θ′) is used to calculate the time difference target value Y. Its parameter θ′ is initially fixed and replaced by the latest estimated network parameter θ after a certain time interval. The calculation formula of the target value network Y is as follows: Y=r+γQ T (s′,argmax a′ Q E (s′,a′; θ); θ′) (20) Keeping the target network parameter θ′ unchanged for a period of time helps to estimate the network Q E Stable convergence. When Q E and Q T When there is disagreement in choosing an action, the valuation network Q E Evaluate action value, target network Q T Calculating the target value can avoid choosing overestimated suboptimal actions, thereby improving algorithm performance and stability.
[0028] Improve the neural network structure design and divide its output into state value function V(s; θ,θ V ) and advantage function A(s,a;θ,θ A ) The former is used to evaluate the quality of each state, and the latter is used to identify the quality of actions in a specific state. Q(s,a;θ,θ A ,θ V )=V(s;θ,θ V )+A(s,a;θ,θ A ) (twenty one) Among them, θ A represents the weight of the advantage function, θ V represents the weight of the state value function; Finally, the target value of D3QN is: Y=r+γQ T (s′,argmax a′ Q E (s′,a′;θ,θ A ,θ V );θ′,θ A ′,θ V ') (twenty two) The network parameters are updated using mean square error as the loss function as follows: L(θ,θ A ,θ V )=E[(YQ E (s,a;θ,θ A ,θ V )) 2 ] (twenty three) The improved ε-greedy search strategy and target value network parameter update in step S6 are specifically: This paper proposes an improved exploration strategy, using the ε-greedy strategy as the agent action selection strategy, where ε is the greediness, ranging from 0 to 1. According to the ratio of legal and illegal actions in the current environment, different greediness parameters ε1 and ε2 are set to explore on the legal action set and the full action set respectively, so as to improve the efficiency of the learning strategy.
[0029] Therefore, the Q network is updated as follows: In the early stages of exploration, the agent has less experience, so a higher exploration rate should be set to prevent it from falling into a local optimum. As the number of training times increases, the rate of decay is set at ε = (1-1 / decay) × ε, where ε0 = 1, decay = 12000, and the minimum value of ε is set to 0.05, so that the agent can choose actions with the greatest behavioral value more often and make better use of environmental experience.
[0030] The step S7 specifically comprises: If the valuation network has converged in step S6, the trained neural network is used to input the current state of the production system to obtain a decision result; the decision result includes selecting the coil to be processed at the current time and selecting the machine to process the coil; by executing the optimal value scheduling action, refreshing the real-time state and the coil to be processed, and inputting the trained neural network again for optimal production scheduling, and repeatedly executing until the intelligent production scheduling of the wind turbine blade beam plate is completed, thereby improving the utilization rate of the current operating equipment and the order fulfillment rate, and improving the production efficiency and customer satisfaction of the workshop.
[0031] The above shows and describes the main features, methods of use, basic principles and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements according to actual conditions, and these changes and improvements fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.
Claims
1. The intelligent scheduling method for wind turbine blade beam production based on D3QN is characterized by: The specific steps are as follows: S1. Construct a multi-objective collaborative optimization model; S2. Establish a Markov decision process and design two sets of experience replay pools. The establishment of the Markov decision process includes establishing the state space, action space and reward function. The specific method is as follows: S21, state space: select 5 key features to describe the state of the problem, state feature 1 represents the average utilization rate of the running machines, state feature 2 represents the standard deviation of the average utilization rate of the running machines, state feature 3 represents the average processing completion rate of all orders, state feature 4 represents the standard deviation of the average processing completion rate of all orders, and state feature 5 represents the total completion rate of all orders; S22, action space: when selecting scheduling actions, the action selection includes the combination relationship between product-production area, coil-mold, and mold-production equipment, where 5 and 2 scheduling rules are set for coils and equipment respectively. The above two rule sets are combined in pairs to obtain 10 combination rules, which are used as the actions of the deep reinforcement learning algorithm to solve the production scheduling problem of wind turbine blade beams; S23. Reward function: The reward function is defined as follows: r1(t)=f1(t+1) r2(t)=f2(t+1) Among them, r1(t) represents the relationship between the utilization rate of the current operating equipment and the reward, r2(t) represents the relationship between the order fulfillment rate and the reward, and the immediate reward obtained at the decision time t is as follows: r t =1000r1(t)+1000r2(t) The final reward function is expressed as R, which contains the sum of the rewards obtained at U decision moments, as shown in the following formula: Among them, U represents the total number of moments that the agent makes decisions; S3, determine whether the samples in the experience replay pool are full, if not, repeat step S2, if full, execute step S4; S4. Based on the D3QN algorithm, build a valuation network Q E (s,a;θ) and the target value network Q T (s,a;θ′); S5. Extract multiple samples randomly in proportion from the two sets of experience replay pools, and train and update the target value network parameters; S6, judging whether the current estimation network has converged, if not, returning to step S5, if converged, executing step S7; S7. The intelligent agent uses the trained improved D3QN algorithm to select the best production schedule and executes it repeatedly to complete the intelligent production scheduling of wind turbine blade beams.
2. The D3QN-based intelligent scheduling method for wind turbine blade beam production according to claim 1, characterized in that: The model in step S1 is established by describing the production scheduling problem of wind turbine blade beams, with the goal of maximizing the utilization rate of the currently operating equipment and minimizing the delivery date deviation.
3. The D3QN-based intelligent scheduling method for wind turbine blade beam production according to claim 1, characterized in that: The establishment of the Markov decision process in step S2 takes the current state of the production system as input and the scheduling rules as output. By designing elements such as state S, action A and reward function R, the solution to the production scheduling problem of wind turbine blade beams is transformed.
4. The D3QN-based intelligent scheduling method for wind turbine blade beam production according to claim 1, characterized in that: The method for establishing the two groups of experience replay pools in step S2 is as follows: samples are divided into two categories based on Q value and reward level, and are stored in two groups of experience replay pools, each with a capacity of C. When updating the network, samples from the two groups of experience replay pools are drawn in proportion.
5. The D3QN-based intelligent scheduling method for wind turbine blade beam production according to claim 1, characterized in that: In step S4, the valuation network is used for action selection, and its parameter θ is continuously updated. The target value network is used to calculate the time difference target value Y, and its parameter θ′ is initially fixed and replaced by the latest valuation network parameter θ after a certain time interval.
6. The D3QN-based intelligent scheduling method for wind turbine blade beam production according to claim 1, characterized in that: In step S6, the ε-greedy strategy is used as the agent action selection strategy, where ε is the greediness, and its value is between 0 and 1. According to the ratio of legal and illegal actions in the current environment, different greediness parameters ε1 and ε2 are set to explore on the legal action set and the full action set respectively.
7. The D3QN-based intelligent scheduling method for wind turbine blade beam production according to claim 1, characterized in that: The step S7 is specifically as follows: if the valuation network has converged in step Step 6, the trained neural network is used to input the current state of the production system to obtain a decision result; the decision result includes selecting the coil to be processed at the current time and selecting the machine to process the coil; by executing the optimal value scheduling action, the real-time state and the coil to be processed are refreshed, and the trained neural network is input again for optimal production scheduling, which is repeated until the intelligent production scheduling of the wind turbine blade beam plate is completed.
