Electric power material verification and production scheduling method based on reinforcement learning
By applying reinforcement learning methods in the field of power material verification, the shortcomings of the existing technology in dealing with high-dimensional state space and real-time response to environmental changes are solved, and the accuracy and efficiency of power material verification and production scheduling are achieved.
Patent Information
- Application Number
- CN202411373362.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-29
- Publication Date
- 2025-02-07
AI Technical Summary
The existing power material verification and production optimization methods are insufficient in dealing with high-dimensional state space and responding to environmental changes in real time, and lack practical applications in the field of power material verification.
Using reinforcement learning methods, by obtaining the status information of the power materials to be checked, building the status space and action space, designing reward functions, creating a deep Q network algorithm model, conducting reinforcement learning training, and obtaining the optimal strategy for power materials verification and ranking Produce.
It has achieved comprehensive consideration of multiple factors in power material verification, and the decision-making quality of the agent is evaluated in multiple dimensions to ensure the accuracy and efficiency of production scheduling decisions.
Smart Images
Figure CN119398378A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of power material management, and in particular to a power material verification and production scheduling method based on reinforcement learning. Background Art
[0002] In the management and testing of power materials, efficient and accurate testing and reasonable scheduling are important links to ensure the stable operation of the power system. However, there are some problems in the existing power material testing and scheduling optimization methods.
[0003] With the continuous development of optimization algorithms and computing technologies, production scheduling methods based on optimization models have gradually been applied to the field of inspection and testing of power materials. Existing optimization methods introduce mathematical models and optimization algorithms such as particle swarm optimization (PSO), ant colony optimization (ACO), taboo search algorithm (TS) and immune algorithm (IA) into the automated inspection system. The PSO algorithm simulates the foraging behavior of bird flocks to find the optimal solution, and can quickly converge to a better solution in a complex search space. The ACO algorithm simulates the behavior of ants in the process of finding food, guides the search direction through the transmission of pheromones, and is suitable for solving combinatorial optimization problems. The TS algorithm avoids repeated searches by introducing a taboo table, and can effectively jump out of the local optimal solution. The IA algorithm draws on the mechanism of the biological immune system and has strong robustness and adaptability. However, most of the existing intelligent production scheduling methods are still in the theoretical research or experimental stage, lacking practical applications in the field of power material inspection, and these methods still have deficiencies in processing high-dimensional state space and responding to environmental changes in real time.
[0004] In view of this, a method for power material inspection and scheduling based on reinforcement learning is needed. Summary of the invention
[0005] In view of the shortcomings of the existing technology in processing high-dimensional state space and responding to environmental changes in real time, the present invention provides a method for power material verification and production scheduling based on reinforcement learning, which can comprehensively consider multiple factors in power material verification, evaluate the decision quality of intelligent agents in multiple dimensions, and schedule power material verification with the optimal strategy to ensure the accuracy and efficiency of production scheduling decisions. The specific technical solution is as follows:
[0006] A method for power material verification and production scheduling based on reinforcement learning, comprising:
[0007] Obtaining the status information of the power material to be inspected, and obtaining the status space according to the status information;
[0008] Set the action space based on the allocation and verification decision of power materials and verification equipment;
[0009] According to the influencing factors of power materials in the inspection and scheduling process, a reward function for evaluating the effectiveness of the agent's actions is constructed;
[0010] The state space is used as the input state of the intelligent agent, the action space is used as the execution range of the intelligent agent, and the reward function is combined to create a deep Q network algorithm model as the intelligent agent;
[0011] By training the intelligent agent through reinforcement learning in the constructed environment, an intelligent agent that can perform power material inspection and scheduling according to the optimal strategy is obtained;
[0012] The trained intelligent agent is deployed into the power production system, and the intelligent agent is used to process the real-time power materials to be inspected, output and automatically adjust the inspection scheduling strategy.
[0013] Preferably, the step of obtaining the status information of the electric power material to be tested and obtaining the status space according to the status information comprises:
[0014] Obtain the quantity, type, urgency and status information of various types of power materials to be tested, and construct a state space based on the state information; let the state space be S, expressed as:
[0015] S=(n1,n2,…,n k ,e1,e2,…,e m )
[0016] Where n k represents the quantity of the kth type of electric power materials, where k is the number of types of electric power materials; e m Indicates the status of the mth calibration device, where m is the number of calibration devices.
[0017] Preferably, the setting of the action space according to the allocation and verification decision of the electric power materials and the verification equipment includes:
[0018] According to the allocation and verification decision of power materials and verification equipment, the action space is set. Let the action space be A, which represents the range of actions taken by the agent. A is expressed as:
[0019] A=(a1,a2,…,a l )
[0020] In the formula, a l It represents the action of allocating electric power material a to the testing equipment l for testing.
[0021] Preferably, the reward function for evaluating the action effect of the intelligent agent is constructed according to the influencing factors of the electric power materials in the verification and scheduling process, including:
[0022] According to the factors affecting the completion time of the calibration of power materials, equipment utilization rate and material urgency in the calibration and scheduling process, a reward function for evaluating the effect of the agent's actions is constructed; let the reward function be R, which can be expressed as:
[0023] R=w1×T -1 +w2×U+w3×E
[0024] Where T represents the time for completion of verification; U represents the equipment utilization rate; E represents the weighted sum of the urgency of materials; w1, w2, and w3 are the weight coefficients of different influencing factors.
[0025] Preferably, the verification completion time T is obtained by accumulating the completion time of each verification task, and the calculation formula is:
[0026]
[0027] Where N is the total number of tasks to be tested, t i represents the completion time of the i-th task;
[0028] The equipment utilization rate U is calculated by measuring the ratio of the equipment working time to the total time, and the calculation formula is:
[0029]
[0030] In the formula, m is the total number of calibration equipment, t work,j represents the working time of the jth calibration equipment, t total,j represents the total time for the jth machine to be tested and prepared;
[0031] The weighted sum E of material urgency is calculated according to the urgency level of power materials, and the calculation formula is:
[0032]
[0033] In the formula, e i represents the urgency level of the i-th material, w e,i is the weight coefficient of urgency.
[0034] Preferably, the intelligent agent that is trained by performing reinforcement learning training on the intelligent agent in the constructed environment and performs power material inspection and production scheduling according to the learned optimal strategy includes:
[0035] The agent selects an action a based on the current state s;
[0036] The environment executes the action selected by the agent and returns the reward r and the next state s′;
[0037] The agent stores the experience data (s, a, r, s′) in the experience replay pool;
[0038] Randomly extract a batch of experience data from the experience replay pool, and use the extracted experience data to train the neural network and update the parameter θ;
[0039] Repeat the above steps until the preset number of training times or convergence conditions are reached. After the training is completed, the intelligent agent performs power material inspection and production scheduling according to the learned optimal strategy.
[0040] Compared with the prior art, the present invention has the following beneficial effects:
[0041] The method for power material inspection and production scheduling based on reinforcement learning of the present invention obtains the state information of the power material to be inspected, and obtains the state space according to the state information; sets the action space according to the allocation inspection decision of the power material and the inspection equipment; and constructs a reward function for evaluating the action effect of the intelligent agent according to the influencing factors of the power material in the inspection and production scheduling process;
[0042] The state space is used as the input state of the intelligent agent, the action space is used as the execution range of the intelligent agent, and the reward function is combined to create a deep Q network algorithm model as the intelligent agent; by performing reinforcement learning training on the intelligent agent in the constructed environment, an intelligent agent that performs power material inspection and scheduling according to the optimal strategy is obtained; the trained intelligent agent is deployed to the power production system, and the real-time power materials to be inspected are processed by the intelligent agent, and the inspection and scheduling strategy is output and automatically adjusted. The present invention uses a reinforcement learning algorithm to realize an intelligent scheduling strategy, comprehensively considers multiple factors in the inspection of power materials, evaluates the decision-making quality of the intelligent agent through multiple dimensions, and performs power material inspection and scheduling according to the optimal strategy to ensure the accuracy and efficiency of the scheduling decision. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.
[0044] Figure 1 This is a flow chart of a method for power material inspection and production scheduling based on reinforcement learning of the present invention.
[0045] Figure 2 This is a flow chart of another embodiment of a method for power material inspection and scheduling based on reinforcement learning of the present invention. DETAILED DESCRIPTION
[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0047] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0048] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0049] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0050] See the following examples Figure 1 to Figure 2 .
[0051] The embodiment of the present application provides a method for power material verification and production scheduling based on reinforcement learning, including:
[0052] Step S01, obtaining status information of the power material to be tested, and obtaining a status space according to the status information;
[0053] In this embodiment, the state space S is a multidimensional vector, which includes the quantity, type, urgency, and status information of each testing device of the power materials to be tested.
[0054] Specifically, the step of obtaining the status information of the power material to be tested and obtaining the status space according to the status information includes:
[0055] Obtain the quantity, type, urgency and status information of various types of power materials to be tested, and construct a state space based on the state information; let the state space be, expressed as:
[0056] S=(n1,n2,…,n k ,e1,e2,…,e m )
[0057] Where n k represents the quantity of the kth type of electric power materials, where k is the number of types of electric power materials; e m Indicates the status of the mth calibration device. The calibration device status includes idle, busy, faulty, etc., and m is the number of calibration devices.
[0058] Step S02: setting the action space according to the allocation and verification decision of the power materials and the verification equipment;
[0059] In this embodiment, the action space A represents the actions that the agent can take, that is, the allocation decision of electric power materials.
[0060] Specifically, the setting of the action space according to the allocation and verification decision of the electric power materials and the verification equipment includes:
[0061] According to the allocation and verification decision of power materials and verification equipment, the action space is set. Let the action space be A, which represents the range of actions taken by the agent. A is expressed as:
[0062] A=(a1,a2,…,a l )
[0063] In the formula, a l It represents the action of allocating electric power material a to the testing equipment l for testing.
[0064] Step S03: construct a reward function for evaluating the action effect of the intelligent agent according to the influencing factors of the electric power materials in the verification and scheduling process;
[0065] In this embodiment, the reward function R is used to evaluate the quality of the actions taken by the agent. The design of the reward function should take into account multiple factors, including the completion time of the inspection, the utilization rate of equipment, the urgency of materials, etc.
[0066] Specifically, the reward function for evaluating the action effect of the intelligent agent is constructed according to the influencing factors of the electric power materials in the verification and scheduling process, including:
[0067] According to the factors affecting the completion time of the calibration of power materials, equipment utilization rate and material urgency in the calibration and scheduling process, a reward function for evaluating the effect of the agent's actions is constructed; let the reward function be R, which can be expressed as:
[0068] R=w1×T -1 +w2×U+w3×E
[0069] Where T represents the time for completion of verification; U represents the equipment utilization rate; E represents the weighted sum of the urgency of materials; w1, w2, and w3 are the weight coefficients of different influencing factors.
[0070] Specifically, the verification completion time T is obtained by accumulating the completion time of each verification task, and the calculation formula is:
[0071]
[0072] Where N is the total number of tasks to be tested, t i represents the completion time of the i-th task;
[0073] The equipment utilization rate U is calculated by measuring the ratio of the equipment working time to the total time, and the calculation formula is:
[0074]
[0075] In the formula, m is the total number of calibration equipment, t work,j represents the working time of the jth calibration equipment, t total,j represents the total time for the jth machine to be tested and prepared;
[0076] The weighted sum E of material urgency is calculated according to the urgency level of power materials, and the calculation formula is:
[0077]
[0078] In the formula, e i represents the urgency level of the i-th material, w e,i is the weight coefficient of urgency.
[0079] Step S04: Use the state space as the input state of the agent, the action space as the execution range of the agent, and combine the reward function to create a deep Q network algorithm model as the agent;
[0080] In this embodiment, the deep Q network algorithm model serves as the intelligent agent, and the power simulation system serves as the environment of the intelligent agent.
[0081] Step S05: by performing reinforcement learning training on the intelligent agent in the constructed environment, an intelligent agent that performs power material inspection and production scheduling according to the optimal strategy is obtained;
[0082] In this embodiment, the Deep Q-Network (DQN) algorithm is used for reinforcement learning. The DQN algorithm consists of a neural network for estimating the state-action value function Q(s,a;θ), where s represents the state, a represents the action, and θ represents the parameters of the neural network. During the training process, the agent collects experience data (s,a,r,s′) by interacting with the environment, where r represents the reward and s′ represents the next state. Then, the neural network is trained using this experience data and the parameter θ is updated so that Q(s,a;θ) approaches the true state-action value function.
[0083] Specifically, the intelligent agent that is trained by performing reinforcement learning training on the intelligent agent in the constructed environment and performs power material inspection and production scheduling according to the learned optimal strategy includes:
[0084] Step S51, the agent selects an action a according to the current state s;
[0085] The current state of the agent in the process of interacting with the environment is:
[0086] s=(n1,n2,…,n k ,e1,e2,…,e m )
[0087] Step S52: The environment executes the action selected by the agent and returns the reward r and the next state s′;
[0088] According to the current state s, an action a is selected, and the environment performs corresponding operations according to the action of the agent, and returns the reward r and the next state s′, that is:
[0089] s′=(n1′,n′2,…,n′ k ,e1',e'2,…,e' m )
[0090] Step S53, the agent stores the experience data (s, a, r, s′) in the experience replay pool;
[0091] The agent stores experience data (i.e., a combination of state, action, reward, and next state) into an experience replay pool.
[0092] Step S54, randomly extracting a batch of experience data from the experience replay pool, and using the extracted experience data to train the neural network, and updating the parameter θ;
[0093] A batch of experience data is randomly extracted from the experience replay pool to train the neural network. The training goal is to minimize the loss function so that the state action value function Q(s,a;θ) estimated by the neural network is close to the true state action value function. The loss function is usually defined as the mean square error between the predicted value and the target value, that is:
[0094]
[0095] in:
[0096]
[0097] Where γ is a discount factor used to weigh the importance of future rewards; θ′ is a parameter of the target network; and the reward r is associated with the previous reward function R design, namely:
[0098] r=w1×T -1 +w2×U+w3×E
[0099] Among them, T is the verification completion time, U is the equipment utilization rate, and E is the weighted sum of the material urgency.
[0100] In the specific training, in order to improve the stability of training, the DQN algorithm uses the target network technology. The structure of the target network is the same as that of the evaluation network, but the parameter update frequency is lower. During the training process, the parameters of the evaluation network are copied to the target network at a certain time step to maintain the stability of the target network. Let the state action value function of the target network be Q′(s,a;θ′). When updating the parameters of the target network, θ′←θ is used, and this operation is performed at a certain time step.
[0101] Step S55, repeat the above steps S51 to S54 until the preset number of training times or convergence condition is reached; at this time, when the loss function L(θ) is expressed as:
[0102] L(θ)<ò convergence
[0103] Among them, convergence is a preset positive number or a preset number of training times N iterations .
[0104] After the training is completed, the agent performs power material inspection and production scheduling according to the learned optimal strategy, that is, selects actions:
[0105]
[0106] In reinforcement learning, the agent needs to strike a balance between exploring unknown actions and exploiting known optimal actions. To achieve this balance, Strategy. Each time you choose an action, you Randomly select an action, with probability 1-ò selecting the action with the highest current value estimate. As training progresses, the probability gradually decreases. to increase the utilization of known optimal actions. The probability of selecting action a is: Randomly select an action a∈A; with probability 1-ò, select action a=A max .
[0108] Step S06: deploy the trained intelligent agent to the power production system, and use the intelligent agent to process the real-time power materials to be inspected, output and automatically adjust the inspection scheduling strategy.
[0109] The trained agent is deployed to the power production system to guide the calibration and production scheduling of power materials. The agent will automatically adjust the calibration and production scheduling strategy based on real-time production data and demand forecasts to achieve the optimal scheduling decision.
[0110] In the embodiments of the present application, the construction of a unique mathematical model is closely combined with the reinforcement learning algorithm: the precise definition of the state space covers multi-dimensional information such as the quantity, type and equipment status of electric power materials, and is closely related to the state representation in reinforcement learning, providing a basis for the intelligent agent to accurately perceive the production scheduling environment. The clear definition of the action space is combined with the action selection in the reinforcement learning algorithm to ensure that the intelligent agent can make reasonable allocation decisions for electric power materials. The reward function comprehensively considers the weighted sum of the verification completion time, equipment utilization rate and material urgency, and is deeply integrated with the reward mechanism in reinforcement learning to guide the intelligent agent to learn the optimal production scheduling strategy.
[0111] The method for power material inspection and production scheduling based on reinforcement learning of the present invention obtains the state information of the power material to be inspected, and obtains the state space according to the state information; sets the action space according to the allocation inspection decision of the power material and the inspection equipment; and constructs a reward function for evaluating the action effect of the intelligent agent according to the influencing factors of the power material in the inspection and production scheduling process;
[0112] The state space is used as the input state of the intelligent agent, the action space is used as the execution range of the intelligent agent, and the reward function is combined to create a deep Q network algorithm model as the intelligent agent; by performing reinforcement learning training on the intelligent agent in the constructed environment, an intelligent agent that performs power material inspection and scheduling according to the optimal strategy is obtained; the trained intelligent agent is deployed to the power production system, and the intelligent agent will automatically adjust the inspection and scheduling strategy according to the real-time power materials to be inspected. The present invention uses a reinforcement learning algorithm to implement an intelligent scheduling strategy, comprehensively considers multiple factors in the inspection of power materials, evaluates the decision-making quality of the intelligent agent in multiple dimensions, and performs power material inspection and scheduling according to the optimal strategy to ensure the accuracy and efficiency of scheduling decisions.
[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.
Claims
1. A method for power material verification and production scheduling based on reinforcement learning, characterized in that: include: Obtaining the status information of the power material to be inspected, and obtaining the status space according to the status information; Set the action space based on the allocation and verification decision of power materials and verification equipment; According to the influencing factors of power materials in the inspection and scheduling process, a reward function for evaluating the effectiveness of the agent's actions is constructed; The state space is used as the input state of the intelligent agent, the action space is used as the execution range of the intelligent agent, and the reward function is combined to create a deep Q network algorithm model as the intelligent agent; By training the intelligent agent through reinforcement learning in the constructed environment, an intelligent agent that can perform power material inspection and scheduling according to the optimal strategy is obtained; The trained intelligent agent is deployed into the power production system, and the intelligent agent is used to process the real-time power materials to be inspected, output and automatically adjust the inspection scheduling strategy.
2. According to the method of claim 1, the method is characterized in that: The step of obtaining the status information of the power material to be tested and obtaining the status space according to the status information includes: Obtain the quantity, type, urgency and status information of various types of power materials to be tested, and construct a state space based on the state information; let the state space be S, expressed as: S=(n1,n2,…,n k ,e1,e2,…,e m ) Where n k represents the quantity of the kth type of electric power materials, where k is the number of types of electric power materials; e m Indicates the status of the mth calibration device, where m is the number of calibration devices.
3. The method for power material verification and production scheduling based on reinforcement learning according to claim 2 is characterized in that: The setting of the action space according to the allocation and verification decision of the electric power materials and the verification equipment includes: According to the allocation and verification decision of power materials and verification equipment, the action space is set, and the action space is set as A, which can be expressed as: <h2 style=";text-align:left;direction:ltr">A = (a1, a2,…, a)<h2 style=";text-align:left;direction:ltr"> l <h2 style=";text-align:left;direction:ltr"> ) In the formula, a l It represents the action of allocating electric power material a to the testing equipment l for testing.
4. The method for power material verification and production scheduling based on reinforcement learning according to claim 3 is characterized in that: The reward function for evaluating the action effect of the intelligent agent based on the influencing factors of the electric power materials in the verification and scheduling process includes: According to the factors affecting the completion time of the calibration of power materials, equipment utilization rate and material urgency in the calibration and scheduling process, a reward function for evaluating the action effect of the intelligent agent is constructed; let the reward function be R, which is expressed as: R=w1×T -1 +w2×U+w3×E Where T represents the time for completion of verification; U represents the equipment utilization rate; E represents the weighted sum of the urgency of materials; w1, w2, and w3 are the weight coefficients of different influencing factors.
5. The method for power material verification and production scheduling based on reinforcement learning according to claim 4 is characterized in that: The verification completion time T is obtained by accumulating the completion time of each verification task, and the calculation formula is: Where N is the total number of tasks to be tested, t i represents the completion time of the i-th task; The equipment utilization rate U is calculated by measuring the ratio of the equipment working time to the total time, and the calculation formula is: In the formula, m is the total number of calibration equipment, t work,j represents the working time of the jth calibration equipment, t total,j represents the total time for the jth machine to be tested and prepared; The weighted sum E of material urgency is calculated according to the urgency level of power materials, and the calculation formula is: In the formula, e i represents the urgency level of the i-th material, w e,i is the weight coefficient of urgency.
6. The method for power material verification and production scheduling based on reinforcement learning according to claim 5 is characterized in that: The intelligent agent that is trained by performing reinforcement learning training on the intelligent agent in the constructed environment and performs power material inspection and production scheduling according to the learned optimal strategy includes: The agent selects an action a based on the current state s; The environment executes the action selected by the agent and returns the reward r and the next state s′; The agent stores the experience data (s, a, r, s′) in the experience replay pool; Randomly extract a batch of experience data from the experience replay pool, and use the extracted experience data to train the neural network and update the parameter θ; Repeat the above steps until the preset number of training times or convergence conditions are reached. After the training is completed, the intelligent agent performs power material inspection and production scheduling according to the learned optimal strategy.