Power transmission network power failure maintenance scheduling optimization method based on Q learning

Abstract

The invention provides a power transmission network power failure maintenance scheduling optimization method based on Q learning, and the method comprises the steps: generating a Gantt chart through obtaining an original topological file of a power grid, and determining a state judgment rule and a target function of reinforcement learning; integrating the maintenance plans of the subordinate units to a Gantt chart to form an initial maintenance plan; calculating an initial power flow set by using a Matlab built-in power flow solver, and evaluating the operation state of the power grid; in the agent training process, an epsilone-greedy method is adopted to select a maintenance action, update a Gantt chart and a topological file, and update a Q value through a Bellman equation to optimize a maintenance plan; and after training is completed, outputting a Gantt chart of a final power failure maintenance plan. According to the method, the power grid dispatching problem is abstracted into the Q learning problem, power failure dispatching is optimized by utilizing the Q learning algorithm through the reasonable constraint and the objective function, the working intensity of dispatching personnel is relieved, the talent training cost is reduced, the dispatching efficiency and reliability are improved, and the key technical problem of power grid dispatching is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of power grid planned power outage scheduling, and in particular to a transmission network power outage maintenance scheduling optimization method based on Q learning. Background Art

[0002] In order to ensure the long-term stable operation of the transmission grid, the reasonable arrangement of the power outage and maintenance plan of the transmission grid in the dispatching area is not only related to the normal production and life of power users, but also reflects the service quality and operational efficiency of the grid dispatching department.

[0003] At present, the main means of dispatching power outages in regional transmission networks is manual arrangement. With the development of social economy, the topological complexity of regional transmission networks continues to increase, and the types of power users in the same region continue to increase, making it more difficult to arrange dispatching manually. At the same time, due to the high time cost of learning dispatching, dispatching cannot quickly train talents to meet the development of regional power grids.

[0004] In addition, when faced with special period dispatching, since most dispatchers propose plans based on experience, the proposed power outage dispatching plans are not necessarily the most appropriate. These factors together lead to the fact that the current manually proposed power outage plans for the transmission network may have the potential risk of being detrimental to the long-term safe and reliable operation of the transmission network, affecting the continuity and reliability of power supply, reducing user power supply satisfaction, aggravating user complaints, and increasing the workload of power grid staff.

[0005] Therefore, it is urgent to propose a Q-learning-based method for optimizing the scheduling of power outage maintenance plans, and to use the Q-learning concept in reinforcement learning in artificial intelligence to achieve reasonable scheduling of power outage scheduling, reduce the workload of dispatchers, and reduce the training cost of dispatchers. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention aims to provide a Q-learning-based method for optimizing power outage maintenance scheduling in a transmission network, and solve the problems raised in the above background technology.

[0007] In order to achieve the above object, the present invention is implemented through the following technical solution: a Q-learning-based transmission network outage maintenance scheduling optimization method, comprising the steps of:

[0008] Generate the corresponding Gantt chart by obtaining the original topology file of the power grid, and establish the state judgment rules and objective function of the current intelligent agent reinforcement learning;

[0009] Integrate the maintenance plans of subordinate units and map them to the Gantt chart to form an initial maintenance plan;

[0010] Use the built-in power flow solver of Matlab to calculate the initial power flow collection and evaluate the initial operation status of the power grid;

[0011] In the double-loop nested process of agent training, the epsilon-greedy method is used to select maintenance actions and update the Gantt chart and topology file, and the Bellman equation is used to update the Q value to optimize the maintenance plan and improve the safety and efficiency of power grid operation.

[0012] After the training of the intelligent agent is completed, the Gantt chart corresponding to the final power outage maintenance plan is output to ensure the high efficiency and reliability of power grid dispatching.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] The present invention abstracts the power grid scheduling problem into a Q-learning problem, establishes reasonable constraints and objective functions, and uses the Q-learning algorithm in reinforcement learning to optimize power outage scheduling. This can reduce the workload of dispatchers, reduce the cost of training dispatchers, and improve the efficiency and reliability of power outage scheduling. At the same time, through iterative training and optimization, the method can provide the power grid with the optimal power outage maintenance plan, ensure the long-term stable operation of the power grid, improve the continuity and reliability of power supply, and reduce user complaints, effectively solving the key technical problems in power grid scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The disclosure of the present invention is described with reference to the accompanying drawings. It should be understood that the drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. In the drawings, the same reference numerals are used to refer to the same components. Among them:

[0016] Figure 1 It is a schematic flow chart of a method for optimizing and arranging a power outage maintenance plan based on Q learning proposed in one embodiment of the present invention;

[0017] Figure 2 A schematic diagram of a planned power outage topology of an IEEE 39 node system using an embodiment proposed in an embodiment of the present invention;

[0018] Figure 3 A Gantt chart G0 corresponding to the initial power outage plan proposed in an embodiment of the present invention;

[0019] Figure 4 FIG. 1 is a power outage plan diagram G after being optimized by an intelligent agent in an embodiment of the present invention. * ;

[0020] Figure 5 This is a sum-reward diagram of the training effect proposed in one embodiment of the present invention. DETAILED DESCRIPTION

[0021] It is easy to understand that according to the technical solution of the present invention, without changing the essential spirit of the present invention, a person skilled in the art can propose a variety of interchangeable structural modes and implementation modes. Therefore, the following specific implementation modes and drawings are only exemplary descriptions of the technical solution of the present invention, and should not be regarded as the whole of the present invention or as a limitation or restriction to the technical solution of the present invention.

[0022] The present invention is further described in detail below with reference to the accompanying drawings, but is not intended to limit the present invention.

[0023] As an understanding of the technical concept and implementation principle of the present invention, the present invention proposes a Q-learning-based transmission network power outage maintenance scheduling optimization method. The main purpose is to solve the problem in the prior art of how to abstract the scheduling problem into a problem that fits Q learning, and establish reasonable constraints and objective function dimensions for the Q learning agent, so that it can give the optimal power outage scheduling of the current regional transmission network within a reasonable training time.

[0024] When implementing it, Figure 1 As shown, as an embodiment of the present invention, a transmission network outage maintenance scheduling optimization method based on Q learning is proposed, comprising the steps of:

[0025] S1, by obtaining the original topology file of the power grid to generate the corresponding Gantt chart, and establish the state judgment rules and objective function of the current agent reinforcement learning, to provide a clear initial state and optimization direction for power grid maintenance.

[0026] S2, integrate the maintenance plans of subordinate units and map them to the Gantt chart to form an initial maintenance plan to ensure the systematicness and coordination of the maintenance plan;

[0027] S3, using the built-in power flow solver of Matlab to calculate the initial power flow collection representing the operating state of the power grid under the initial power outage maintenance plan, and evaluate the operating state of the power grid under the initial maintenance plan;

[0028] S4, in the double-loop nested process of agent training, selects maintenance actions and updates the Gantt chart and topology file through the epsilon-greedy method, calculates the reward value reflecting the power grid operation status and the quality of the maintenance plan, and uses the Bellman equation to update the Q value to optimize the maintenance plan and improve the safety and efficiency of power grid operation;

[0029] S5, after the training of the intelligent agent is completed, the Gantt chart corresponding to the final power outage maintenance plan and the training effect test are output to verify the effectiveness of the optimization scheduling method and ensure the high efficiency and reliability of power grid dispatching.

[0030] In one embodiment of the present invention, it should be noted that in step S1, the specific implementation method of obtaining the original topology file of the power grid to generate the corresponding Gantt chart is: obtaining the original topology file of the power grid, and generating the corresponding two-dimensional Gantt chart G according to n，row，col (n=1,2,…, is the number of Gantt charts generated under this patent method), row=30 is the number of days in the month, col is the total number of repairable equipment in the current regional power grid (including the number of transmission lines and the number of motors in the current topology), G in the Gantt chart i,j =1 means that the jth device works normally on the i-th day, G i,j =0 means that the jth device will be shut down for maintenance on the i-th day. The initial Gantt chart should not have any equipment shut down for maintenance. G can be used when the Gantt chart does not involve the modification of the power outage plan. n Direct expression.

[0031] When establishing the state judgment rules of the current agent reinforcement learning, we first need to determine the number of transmission lines branch and the number of motors gen through the original topology file of the power grid, and calculate the total number of repairable equipment col in the power grid. After providing the data basis for the state judgment rule, the state judgment rule C is composed of a single rule c1, C = {c1}, where the single rule c1 ensures that on any given date, the number of devices k that are shut down for maintenance month No more than the total number of devices that can be repaired on the power grid minus the maximum number of devices in operation on that day row is the number of days, is the number of devices running on that day. That is, the state judgment rule of the current agent reinforcement learning is:

[0032] C = {c1}(2)

[0033]

[0034] In one embodiment of the present invention, k month The number of devices that have power outages on a day with power outages for maintenance in the month. It refers to the number of devices in operation on a day in the month when there is a power outage for maintenance.

[0035] Example: Figure 2 As shown, according to the current topology diagram, it can be determined that branch=46, gen=10, col=branch+gen=56.

[0036] In step S1, it is necessary to establish the objective function F based on the state judgment rule C of the intelligent agent reinforcement learning by considering the minimization of voltage deviation of the grid nodes, the minimum active network loss of the entire network transmission line, and the convergence of the power flow. The specific implementation method is as follows:

[0037] According to the obtained Gantt chart Gn Modify the original topology file and use the built-in power flow solver in MATLAB to calculate the power flow collection PF of the power outage maintenance plan for the month in units of days. n The optimization result goal is set as follows: after optimization, the number of over-limit voltages of nodes in the entire network is the least, the active network loss of the entire network transmission line is the lowest, and the power flow converges after optimization. The above objective function is expressed as:

[0038]

[0039] PF n is the power flow collection of the outage plan after optimization by the intelligent agent, PF·V is the per-unit value of the node voltage in the current power flow collection (with the reference node voltage of the current network topology as 1), PF·P loss is the active network loss of the transmission line in the current power flow collection, and PF·success is the power flow convergence mark of each day in the power flow collection (a value of 1 represents convergence, and a value of 0 represents non-convergence).

[0040] Example: Assume that the maximum number of training rounds for the reinforcement learning agent in this example is episode = 1200, and the maximum number of steps that can be taken in each training round is step = 50.

[0041] In one embodiment of the present invention, it should be noted that in step S2, the specific implementation method of forming the initial maintenance plan is: the maintenance days of the corresponding equipment submitted by the subordinate unit should be an array E equp = [Day1, Day2, …, Day n ](Day n , n=1,2,……is the equipment maintenance date of the month, and the subscript equp is the current equipment in the Gantt chart G n In the example of the present invention, Figure 2 As shown in the figure, the power outage plan equipment submitted in the current month has been marked in the topology diagram. The maintenance plans submitted by all subordinate units will appear in the form of a collection E = {E equp1 ,E equp2 ,…E equpn}, all submitted maintenance plans are mapped to the Gantt chart to form the initial plan G0. Figure 2 It can be seen that the number of equipment under maintenance in this instance is 5, and the corresponding equipment numbers in the topology file are 2, 10, 28, 31, and 44. Then the collection of equipment under maintenance in this month and the array corresponding to a single maintenance equipment are:

[0042]

[0043] It can be seen that all equipment are under maintenance at the same time, and there are serious conflicts that need to be replanned. From E, it can be determined that there are 55 possible actions for a device in the current month, so the action array a of a single device is:

[0044] a2=[1,2,…,27,0,-27,…,-2,-1](6)

[0045] A positive number n means that the overall plan of the current equipment is adjusted down by n days, a negative number -n means that the overall plan of the current equipment is adjusted up by n days, and 0 means that the current plan will not be adjusted;

[0046] The action set A is: A = {a2, a 10 ,a 28 ,a 31 ,a 32}(7).

[0047] Based on the above technical concept, it should be noted that after forming the initial maintenance plan and before calculating the initial power flow collection, a Q table needs to be initialized to store the expected return value of each power grid operation state and maintenance action pair to provide a starting point for the intelligent agent so that it can be continuously updated and optimized through learning in the subsequent training process. The process is: first, according to the number of power outage maintenance equipment in the month and combined with the Nk principle in the power system, the state space dimension of the Q table is determined. This dimension reflects all possible state combinations of the power grid when considering that k devices may be maintained at the same time; secondly, the action space is defined, that is, the maintenance actions that the intelligent agent can take in each state; then, a two-dimensional array is created as the Q table, so that the number of rows corresponds to the dimension of the state space, and the number of columns corresponds to the dimension of the action space; finally, all entries in the Q table are initialized to zero or small values ​​close to zero, which means that at the beginning of training, the intelligent agent lacks understanding of the expected return value of any state-action pair and needs to be gradually learned and adjusted through subsequent reinforcement learning processes.

[0048] In one embodiment of the present invention, it should be noted that in step S3, the specific implementation method of evaluating the operating status of the power grid under the initial maintenance plan is: modifying the corresponding topology file according to the Gantt chart G0, and using the built-in power flow solver in Matlab to calculate the monthly power flow set PF0 of the initial power outage plan = {pf1, pf2, ..., pf row}(If there are multiple equipment maintenance in one day, it is also considered as a power outage plan for the day); where pf nThe power grid flow of the power outage on the nth maintenance day of the month is determined according to the judgment rule C. The corresponding state S1 of the current Gantt chart G0 is determined. S1=C{G0}=5(8). The dimension of the Q table used to store Q learning is determined according to the number of power outage maintenance equipment in the month and the Nk principle in the power system. The current Q table should be a table with 5 rows and 275 columns, where the rows represent the state S, corresponding to the maximum number of power outage equipment k in a single day of the month in the Nk principle, and the columns represent the action index of all equipment. All values ​​in the Q table are initialized to -1000.

[0049] As an embodiment of the present invention, the initial operation state of the power grid is the operation state S1 of the power grid when implementing the initial power outage maintenance plan, and the operation state S1 is determined by the state judgment rule C, S1 = C{G0}, where G0 is the current Gantt chart. In S4, the maintenance action is selected by the epsilon-greedy method and the Gantt chart and topology file are updated, the reward value reflecting the operation state of the power grid and the quality of the maintenance plan is calculated, and the Q value is updated using the Bellman equation to optimize the maintenance plan and improve the safety and efficiency of the power grid operation.

[0050] In the specific implementation, the double-loop nested process of agent training is entered. The operating state S1 is required to be the starting point of the agent training process. After selecting the maintenance action through the epsilon-greedy method and updating the Gantt chart and topology file, the state judgment rule C is also used to determine the optimized operating state S2 of the power grid to reflect the impact of the maintenance action on the operating state of the power grid, so as to achieve a better voltage level, lower network loss and higher power flow convergence. Specifically: In the current step, only one meaningful action a is selected from the action set A according to the epsilon-greedy method equpn,k , modify E m {……,E equpn}, modify the Gantt chart to G according to the selected action m , according to G m Modify the topology file for calculation, and still use the built-in power flow solver in MATLAB to calculate the power outage power flow plan collection PF after the action m ={pf1,pf2,…,pf row}, determine the current state S2 according to the judgment rule C. Set the action reward value mapping function f(PF);

[0051]

[0052] PF m is the collection of power outage plan trends after the agent’s actions, PF m V is the per-unit value of the node voltage in the current power flow collection (the reference node voltage of the current network topology is 1), PF m ·P lossis the total active power loss of the transmission line in the current power flow, PF m ·success is the daily trend convergence mark in the trend collection (a value of 1 represents convergence, and a value of 0 represents non-convergence);

[0053] According to the trend collection PF0, PF m And the action-value mapping rule calculates the immediate reward value obtained after the current action;

[0054] reward m =f(PF m )-f(PF0)(10)

[0055] Cover S2 to S1 and calculate the Q value using the Bellman equation;

[0056] Q(S1,a equpn,k )=Q(S1,a equpn,k )+α[reward m +γmax a， Q(S2,a′)-Q(S1,a equpn,k )](11)

[0057] Where: Q(S1,a equpn,k ) is the update to Q table S1 row a after executing the current action equpn,k The column corresponding to the action index, α is the learning rate, γ is the discount factor, max a′ Q(S2,a′) is the maximum Q value among the set of meaningful actions a′ in row S2 of the Q-table.

[0058] Based on the above technical concept, in the double loop nesting process of agent training, it is also necessary to check the results of its training process, where the completion of the training process depends on whether the preset conditions are met. In specific implementation, the agent will continue to execute the training loop until any of the following conditions are met: the maximum number of training rounds is reached or the maximum number of steps is reached in any training round, where:

[0059] If in any training round, the agent executes the maximum number of steps but fails to meet the optimization goal or achieve the expected performance index, the current training process is considered incomplete and needs to be continued; otherwise,

[0060] If in any training round, the agent completes all the preset steps and reaches the optimization goal or performance index, the current training process is considered completed, and after the training is completed, the agent outputs the two-dimensional Gantt chart G corresponding to the final power outage maintenance plan * And the sum of the reward values ​​of all steps in each training round is used as the basis for subsequent evaluation of the training effect, that is, the output of the final power outage plan corresponds to the Gantt chart G *And the sum of rewards of all steps in each episode is used as a test of training effect.

[0061]

[0062] It can be understood that the above-mentioned results ensure the effectiveness of the optimization scheduling method of the power outage maintenance plan to meet the safety and efficiency requirements of power grid operation.

[0063] As an embodiment of the present invention, in order to better introduce the present invention, the following is a detailed description through a specific simulation example:

[0064] like Figure 2 As shown, in the IEEE39 node topology, the current state judgment rules (2), (3) and the objective function (4) are set. The equipment that needs to be overhauled this month is: 2, 10, 28, 31, 44, and the maintenance plan time period submitted this month is (5). According to the above conditions, the Gantt chart G0 can be determined as follows Figure 3 As shown, the action and action set are determined as (6) and (7). According to (3) and G0, the state S1=5 is determined, and the Q table is initialized according to the above conditions as shown in Table 1. After the steps (4) and (5) are iterated, the final Q table is shown in Table 2. The dispatcher can further learn or adjust according to the iterated Q table. The final power outage plan Gantt chart G after the agent optimization * like Figure 4 As shown; the final running effect sum-reward is as follows Figure 5 As shown, it can be seen that the training trend finally converges and moves upward.

[0065] Table 1 Initialization Q table

[0066]

[0067]

[0068] Table 2 Q table after iteration

[0069]

[0070] The technical scope of the present invention is not limited to the contents in the above description. Those skilled in the art can make various deformations and modifications to the above embodiments without departing from the technical idea of ​​the present invention, and these deformations and modifications should all fall within the protection scope of the present invention.

Claims

1. A Q-learning-based power transmission network outage maintenance scheduling optimization method, characterized by: Includes steps: Generate the corresponding Gantt chart by obtaining the original topology file of the power grid, and establish the state judgment rules and objective function of the current intelligent agent reinforcement learning; Integrate the maintenance plans of subordinate units and map them to the Gantt chart to form an initial maintenance plan; Use the built-in power flow solver of Matlab to calculate the initial power flow collection and evaluate the initial operation status of the power grid; In the double-loop nested process of agent training, the epsilon-greedy method is used to select maintenance actions and update the Gantt chart and topology file, and the Bellman equation is used to update the Q value to optimize the maintenance plan and improve the safety and efficiency of power grid operation. After the training of the intelligent agent is completed, the Gantt chart corresponding to the final power outage maintenance plan is output to ensure the high efficiency and reliability of power grid dispatching.

2. According to the Q-learning-based power transmission network outage maintenance scheduling optimization method of claim 1, it is characterized by: After forming the initial maintenance plan and before calculating the initial power flow set, a Q table needs to be initialized to store the expected return value of each grid operation state and maintenance action pair to provide a starting point for the agent so that it can be continuously updated and optimized through learning in the subsequent training process. The process is as follows: First, the state space dimension of the Q-table is determined according to the number of equipment that was outage for maintenance that month and combined with the Nk principle in the power system. Secondly, the action space is defined, and a two-dimensional array is created as the Q-table so that the number of rows corresponds to the dimension of the state space and the number of columns corresponds to the dimension of the action space. Finally, all entries in the Q-table are initialized to zero or close to zero decimal values, and gradually learned and adjusted through the subsequent reinforcement learning process.

3. The method for optimizing power outage maintenance scheduling in a power transmission network based on Q learning according to claim 1 is characterized in that: In the double loop nesting process of agent training, the agent will continue to execute the training loop until any of the following conditions are met: the maximum number of training rounds is reached or the maximum number of steps is reached in any training round, where: If in any training round, the agent executes the maximum number of steps but fails to meet the optimization goal or achieve the expected performance index, the current training process is considered incomplete and needs to be continued; otherwise, If in any training round, the agent completes all preset steps and reaches the optimization goal or performance indicator, the current training process is considered completed. After the training is completed, the agent outputs a two-dimensional Gantt chart corresponding to the final power outage maintenance plan and the sum of the reward values ​​of all steps in each training round as a basis for subsequent evaluation of the training effect.

4. The method for optimizing power outage maintenance scheduling in a power transmission network based on Q learning according to claim 1 is characterized in that: When establishing the state judgment rule of the current agent reinforcement learning, it is first necessary to determine the number of transmission lines branch and the number of motors gen through the original topology file of the power grid, calculate the total number of devices that can be repaired in the power grid col, and provide a data basis for the state judgment rule. Then, a single rule c1 is used to form a state judgment rule C, C = {c1}, where the single rule c1 ensures that on any given date, the number of devices k that are out of power for maintenance in the power grid is month No more than the total number of devices that can be repaired on the power grid minus the maximum number of devices in operation on that day row is the number of days, The number of devices running on that day.

5. A Q-learning-based power transmission network outage maintenance scheduling optimization method according to claim 1 or 4, characterized in that: On the basis of establishing the state judgment rule C of the agent reinforcement learning, the objective function F is established by considering the minimization of voltage deviation of the grid nodes, the minimum active network loss of the entire network transmission line, and the convergence of the power flow: In the formula, PF n is the power flow collection of the outage plan after optimization by the intelligent agent, PF·V is the per-unit value of the node voltage in the current power flow collection, PF·P loss is the active network loss of the transmission line in the current power flow collection. PF·success is the power flow convergence mark of each day in the power flow collection. The value of 1 represents convergence, and the value of 0 represents non-convergence.

6. A Q-learning-based power transmission network outage maintenance scheduling optimization method according to claim 1 or 4, characterized in that: The initial operation state of the power grid is the operation state S1 of the power grid when the initial power outage maintenance plan is implemented. The operation state S1 is determined by the state judgment rule C, S1 = C{G0}, where G0 is the current Gantt chart; and When entering the double-loop nested process of agent training, the operating state S1 is required to be the starting point of the agent training process. After selecting the maintenance action through the epsilon-greedy method and updating the Gantt chart and topology file, the state judgment rule C is also used to determine the optimized operating state S2 of the power grid to reflect the impact of the maintenance action on the operating state of the power grid, so as to achieve a better voltage level, lower network loss and higher power flow convergence.

7. The method for optimizing power outage maintenance scheduling in a power transmission network based on Q learning according to claim 6 is characterized in that: The specific process of determining the running state S2 is as follows: first, a meaningful action a is selected from the action set A according to the epsilon-greedy method. equpn,k , modify the subordinate unit maintenance plan total E m {……,E equpn }; Secondly, modify the Gantt chart to G according to the selected action m , according to the Gantt chart G m Modify the topology file for calculation; again, use the built-in power flow solver in MATLAB to calculate the power outage plan power flow collection PF after the action m ={pf1,pf2,…,pf row }; Finally, the state S2 is determined according to the judgment rule C, wherein the action set A is determined according to the maintenance plan submitted by the subordinate unit and the operating status of the power grid.

8. The method for optimizing power outage maintenance scheduling in a power transmission network based on Q learning according to claim 1 or 3, characterized in that: In the double loop nesting process of agent training, by setting the action reward value mapping function f(PF), according to the initial power flow collection PF0, the power outage plan power flow collection PF m And the action-value mapping rule is used to calculate the reward value reward, cover the running state S2 to the running state S1, and then use the Bellman equation to calculate the Q value.

9. The method for optimizing power outage maintenance scheduling in a power transmission network based on Q learning according to claim 8 is characterized in that: The action reward value mapping function f(PF) formula is: In the formula, PF m is the collection of power outage plan trends after the agent’s actions, PF m V is the per unit value of the node voltage in the current power flow collection, PF m ·P loss is the total active power loss of the transmission line in the current power flow, PF m ·success is the daily trend convergence sign in the trend collection. A value of 1 indicates convergence and a value of 0 indicates non-convergence.

10. The method for optimizing power outage maintenance scheduling of a power transmission network based on Q learning according to claim 8, characterized in that: The reward value is calculated as follows: m =f(PF m )-f(PF0); The formula for calculating the Q value using the Bellman equation is: Q(S1,a equpn,k )=Q(S1,a equpn,k )+α[reward m +γmax a′ Q(S2,a′)-Q(S1,a equpn,k )] In the formula, Q(S1,a equpn,k ) is the update to Q table S1 row a after executing the current action equpn,k The column corresponding to the action index, α is the learning rate, γ is the discount factor, max a′ Q(S2,a′) is the maximum Q value among the set of meaningful actions a′ in row S2 of the Q-table.

Images