A Reinforcement Learning-Based Task Planning Method and System for Converter Sampling Robots
By dividing the tasks of the converter variable sampling robot into independent behaviors and generating a conditional behavior tree, the problems of complexity in manual adjustment and inconvenience in applying reinforcement learning results in the prior art are solved, and the visualization and simplification of state conditions and behavior selection are realized.
Patent Information
- Application Number
- CN202611610871.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-19
- Publication Date
- 2026-11-17
AI Technical Summary
The existing commutator sampling robot's task flow requires manual adjustment, which can easily lead to repeated conditions, missing states, or branch conflicts. Furthermore, the training results of reinforcement learning are not easily applied directly to the robot's online control.
The tasks of the converter sampling robot are divided into five independently encapsulated behaviors. The behavior mapping is trained in a simulation environment through reinforcement learning, Boolean predicates are generated and a conditional behavior tree is constructed. The underlying controller signals are converted into Boolean predicate combination states to generate the final behavior tree.
It enables visualization of state conditions and behavior selection, reduces the complexity of manual adjustments, avoids infinite state growth, simplifies the behavior tree generation process, and facilitates inspection and maintenance by engineers.
Smart Images

Figure CN123103702A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot automatic control technology, and in particular to a task planning method and system for a commutator variable sampling robot based on reinforcement learning. Background Technology
[0002] The operational workflow of existing converter sampling robots is typically pre-programmed by developers. Developers need to manually determine the sequence of five actions: navigation to the oil sampling location, robotic arm docking with the oil sampling port, oil circuit flushing, oil sampling, and return to the designated location. They also need to define the activation conditions for each action and the switching rules for abnormal states before writing this information into the sequence control program or behavior tree. When the oil sampling point location, docking determination method, sampling operation status, or available actions change, the conditions need to be re-evaluated and the tree structure modified. However, as the number of states and actions increases, manual adjustments are prone to problems such as repeated conditions, omitted states, or branch conflicts.
[0003] Reinforcement learning can derive behavioral selection strategies for different states through repeated trials, but the training results are typically stored in a Q-table. While the Q-table indicates the appropriate behavior for a given state, it cannot be directly used as an online robot control program, nor is it convenient for personnel to view the task sequence and branches. Therefore, a technical method is needed to automatically convert the state-behavior mapping obtained from reinforcement learning into an executable, viewable, and modifiable behavior tree without rewriting the underlying controllers of the navigation, robotic arm, and sampling devices. Summary of the Invention
[0004] The purpose of this invention is to provide a task planning method and system for commutation variable sampling robots based on reinforcement learning, which aims to solve the problems that can easily occur when manually adjusting conditions, omissions of states, or branch conflicts as the number of states and behaviors increases.
[0005] In a first aspect, the present invention provides a task planning method for a commutator variable sampling robot based on reinforcement learning, the method comprising: The operation tasks of the converter sampling robot are divided into five independently encapsulated behaviors. In each control cycle, the operation signals of the robot's underlying controller for each behavior are collected and converted into Boolean predicates. The Boolean predicates include five task state predicates and five behavior success predicates. Compare the Boolean predicates of the current control cycle with those of the previous control cycle, and determine the new state only when at least one predicate changes; In the simulation environment, reinforcement learning is used to repeatedly execute various behaviors. Each time a new state is generated, a new row is created for that state in the preset Q table and the state behavior values of each column in that row are initialized to zero. Rewards are generated based on the behavior execution results and the state behavior values of the corresponding state and the called behavior in the preset Q table are updated. After training is completed, the behavior corresponding to the maximum state behavior value in each row of the preset Q table is taken as the optimal behavior in that state, and the mapping relationship between the state and the optimal behavior is obtained. Based on the mapping relationship, states that map to any behavior are selected from all the existing states and constitute the minterm set of that behavior. States that did not appear during training are aggregated into irrelevant term set. Logical simplification is performed using the minterm set as the states that must be covered and the irrelevant terms as the states that can be freely processed, to obtain the simplified execution conditions for each behavior. Using the selection node as the root node, five sequential nodes are created under the root node. Each sequential node contains the corresponding simplified execution conditions and behaviors, forming a conditional behavior tree. The conditional behavior tree is then subjected to inverse condition processing, and the final behavior tree is obtained based on the inverse condition processing result. The final behavior tree is deployed to the robot task planning computer. When the robot is running, the signals of the underlying controller are collected according to the control cycle and converted into real-time Boolean predicates. This allows the root node to check the execution conditions of each node in turn according to the real-time Boolean predicates. When the conditions are met, the corresponding behavior is called. If the behavior is running, the behavior continues to be executed. After the behavior returns successfully, the Boolean predicate is updated. The updated Boolean predicate is then entered into the behavior tree for judgment again until the five behaviors are executed in turn.
[0006] Secondly, this invention provides a task planning system for a commutator variable sampling robot based on reinforcement learning, the system comprising: The behavior segmentation module is used to divide the operation tasks of the converter sampling robot into five independently encapsulated behaviors. In each control cycle, it collects the operation signals of the robot's underlying controller regarding each behavior and converts them into Boolean predicates. The Boolean predicates include five task state predicates and five behavior success predicates. The predicate combination module is used to compare the Boolean predicates of the current control cycle with those of the previous control cycle. Only when at least one predicate changes, the changed predicate combination is determined as a new state. The training module is used to repeatedly execute various behaviors using reinforcement learning in a simulation environment. Each time a new state is generated, a new row is created for that state in the preset Q table, and the state behavior values of each column in that row are initialized to zero. Rewards are generated based on the behavior execution results, and the state behavior values of the corresponding state and the called behavior in the preset Q table are updated. After training is completed, the behavior corresponding to the maximum state behavior value in each row of the preset Q table is taken as the optimal behavior in that state, and the mapping relationship between the state and the optimal behavior is obtained. The simplification module is used to filter out the states mapped to any behavior from all the existing states according to the mapping relationship and form the minterm set of the behavior. The states that did not appear in the training are summarized into the irrelevant term set. Logical simplification is performed with the minterm set as the states that must be covered and the irrelevant terms as the states that can be freely processed, to obtain the simplified execution conditions of each behavior. The behavior tree construction module is used to build five sequential nodes under the root node with the selection node as the root node. Each sequential node contains the corresponding simplified execution conditions and behaviors to form a conditional behavior tree. The conditional behavior tree is then subjected to inverse condition processing, and the final behavior tree is obtained based on the inverse condition processing result. The behavior execution module is used to deploy the final behavior tree to the robot task planning computer. When the robot is running, it collects the signals of the underlying controller according to the control cycle and converts them into real-time Boolean predicates. This allows the root node to check the execution conditions of each node in turn according to the real-time Boolean predicates. When the conditions are met, the corresponding behavior is called. The behavior continues to be executed while it is running. After the behavior returns successfully, the Boolean predicate is updated. The updated Boolean predicate is then entered into the behavior tree for judgment again until all five behaviors are executed in turn.
[0007] Thirdly, the present invention provides a storage medium that stores one or more programs, which, when executed by a processor, implement the above-described reinforcement learning-based commutation variable sampling robot task planning method.
[0008] Fourthly, the present invention provides an electronic device, the electronic device comprising a memory and a processor, wherein: The memory is used to store computer programs; When the processor executes the computer program stored in the memory, it implements the above-mentioned reinforcement learning-based commutator sampling robot task planning method.
[0009] Compared with the prior art, the present invention has the following advantages: This invention encapsulates the five operational behaviors of the converter sampling robot independently. Reinforcement learning only selects and combines complete task behaviors, eliminating the need to relearn the continuous control quantities of the chassis, robotic arm, and sampling device, and allowing direct use of existing control modules. By converting the signals of the underlying controller into finite Boolean predicates and combining them into states, each state can serve as a definite row number in the Q-table, avoiding the infinite growth of states caused by directly inputting continuous values into the table. After training, the states mapped to each behavior are selected from the Q-table to form a minterm set. After logical simplification, simplified execution conditions are obtained, and a conditional behavior tree is generated. Then, the conditional behavior tree is re-processed through inverse condition processing. The group forms the final behavior tree, enabling the state conditions, behavior selections, and sequential relationships in the strategy to be displayed in a tree structure, facilitating inspection and subsequent maintenance by engineers. The entire behavior tree generation process does not require searching for all permutations and tree structures of the five behaviors. Instead, it first generates a fixed-structure conditional behavior tree and then reorganizes the conditions and nodes, reducing the process of repeatedly adding, deleting, and exchanging nodes. The inverse condition processing only changes the expression structure of the behavior tree and does not change the state-behavior correspondences already learned in the Q-table. Therefore, the final behavior tree can still call the five behaviors of autonomous navigation, robotic arm docking, oil flushing, oil retrieval, and return according to the training strategy to complete the commutation and sampling operation. Attached Figure Description
[0010] Figure 1 This is a flowchart of a task planning method for a commutator variable sampling robot based on reinforcement learning, as proposed in an embodiment of the present invention. Figure 2 This is a schematic diagram of a task planning system for a commutator variable sampling robot based on reinforcement learning, as proposed in an embodiment of the present invention.
[0011] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation
[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art. The terms "comprising" and similar expressions used herein mean that the element or object preceding the word covers the element or object listed after the word and its equivalents, but does not exclude other elements or objects.
[0013] like Figure 1As shown, an embodiment of the present invention proposes a task planning method for a commutator variable sampling robot based on reinforcement learning. The method includes steps S101 to S106, wherein: Step S101: Divide the operation task of the converter sampling robot into five independently encapsulated behaviors. In each control cycle, collect the operation signals of the robot's underlying controller for each behavior and convert them into Boolean predicates. The Boolean predicates include five task status predicates and five behavior success predicates. It's important to note that the converter sampling operation itself is continuous: the chassis is moving, the robotic arm is extending, and oil is flowing. However, if reinforcement learning directly processes these continuous quantities, the state space would become infinitely large, making it impossible to construct a Q-table. Therefore, the entire operation is first broken down into five independently encapsulated behaviors: navigation, docking, flushing, oil sampling, and return. Each behavior is only responsible for its own segment. The task planning program doesn't care how the underlying controller controls the motors, joints, valves, and pumps; it only reads the signals returned by the controller.
[0014] Specifically, a converter transformer sampling operation scenario is established in the simulation environment. The scenario is built based on the actual robot operating environment on-site. The scenario includes a robot body model, a converter transformer oil sampling port model, the robot's initial position, oil sampling position, and final return position, all within a unified scenario coordinate system. The robot body model is connected to three controllers: a mobile chassis controller, a robotic arm controller, and an oil sampling device controller. These three controllers control the robot's actions to complete the entire oil sampling operation. At the start of each training session, the robot needs to return to its initial position, the robotic arm retracts, the sampling device resets, and all signals are cleared.
[0015] This solution divides the robot sampling task into five actions: Behavior Read the current location and oil collection point coordinates, and call the mobile chassis controller to navigate to the oil collection location; behavior Read the oil inlet pose and the robotic arm pose, and call the robotic arm controller to extend the robotic arm to dock with the oil inlet; behavior The oil extraction device controller is invoked to flush the oil circuit; action The oil sampling device controller is invoked to sample oil from the transformer; Action The mobile chassis controller is invoked to return to the specified location.
[0016] Based on the above five actions, a set of behaviors is derived. : Each action, upon receiving a run signal, will only respond with one of three results: Running, Success, or Failure. Running indicates that the current action is being executed; Success indicates that the current action's goal has been achieved; and Failure indicates that the current action cannot be executed or has failed. The run signals for the five actions are generated by signals from three controllers. The task decision-making and planning system only reads the controller signals to determine whether the action is completed or not, without modifying the underlying control logic. The five actions have been independently encapsulated, and when invoked, the corresponding action is directly executed to complete the oil retrieval task.
[0017] In addition, in some embodiments, the signal of the underlying controller is read in each control cycle and converted into a Boolean predicate of 0 or 1, where a value of 1 indicates that the action is true and a value of 0 indicates that the condition is false. Define task state predicates corresponding to the five behaviors respectively. to , Used to read whether the robot has reached the oil collection point. Used to read the signal indicating that the robotic arm has completed docking with the oil inlet. Used to read the signal indicating completion of oil circuit flushing. Used to read the oil extraction completion signal. Used to read the return arrival signal; Define the behavioral success predicates corresponding to the five behaviors. to When any action returns success, the success predicate corresponding to that action is set to 1, and the success predicates for all other actions are set to 0. to Used for reading behavior respectively to The execution result.
[0018] In total, ten Boolean predicates were obtained. These predicates are combined in a fixed order to form the robot state at the current time t. : In summary, by breaking down the converter sampling operation into five clearly defined behaviors with well-defined controller affiliations—navigation, docking, flushing, oil sampling, and return—the task planning program only needs to read three return results—running, success, or failure—to determine whether the behavior is completed and whether a switch is needed. It doesn't need to concern itself with how the underlying controller drives motors, joints, or valves / pumps, thus completely separating high-level task planning from low-level motion control. Furthermore, by defining task state predicates to describe the completion status of each operation stage and defining behavior success predicates to record the most recent successful behavior, the robot's continuously changing operation process is transformed into a finite set of ten Boolean predicates. Each state can serve as a definite row number in the Q-table, preventing continuous values from being directly entered into the table and causing infinite state growth. This provides a unified and implementable state foundation for subsequent reinforcement learning training and Q-table establishment.
[0019] Step S102: Compare the Boolean predicates of the current control cycle with those of the previous control cycle, and determine the changed predicate combination as a new state only when at least one predicate changes; It's important to note that a new state isn't generated in every control cycle; it's only generated when the predicate changes. This is because a robot might execute the same behavior for dozens of consecutive control cycles with the predicate remaining unchanged. If a new state were recorded for each cycle, the Q-table would be overwhelmed by a large number of repetitive states.
[0020] Therefore, the program compares the ten predicates in two adjacent control cycles. If any one of them changes, the changed combination is defined as a new state, and a state transition occurs. If the predicates remain unchanged, the current behavior continues to be executed, without repeatedly writing to the Q-table. In this way, each row of the Q-table corresponds to a truly distinct state, rather than a point in time.
[0021] Step S103: In the simulation environment, reinforcement learning is used to repeatedly execute various behaviors. Each time a new state is generated, a new row is created for the state in the preset Q table and the state behavior values of each column in the row are initialized to zero. Rewards are generated according to the behavior execution results and the state behavior values of the corresponding state and the called behavior in the preset Q table are updated. After training is completed, the behavior corresponding to the maximum state behavior value in each row of the preset Q table is taken as the optimal behavior in that state, and the mapping relationship between the state and the optimal behavior is obtained. It should be noted that the preset Q-table is initially empty. During training, a new row is created for each new state, with five columns corresponding to five behaviors, all initially set to zero. The robot then repeatedly tries different behaviors in the simulation environment: selecting a behavior to perform in the current state, receiving a reward, and then moving to the next state. The Q-value update formula is used to write the experience back into the preset Q-table.
[0022] The reward rules can be as follows: a positive reward for successful oil retrieval, a larger positive reward for successful return, a negative reward for task failure, and a small negative reward for other actions. In this way, intermediate actions such as navigation, docking, and flushing do not directly provide positive rewards; they only earn higher Q-values through future rewards if they facilitate subsequent oil retrieval and return. Training is repeated until the action with the highest Q-value in each state no longer changes. The action corresponding to the highest Q-value in each row is then taken as the optimal action for that state, resulting in a mapping table of "state → optimal action".
[0023] Specifically, each row of the Q table corresponds to a Boolean state that has occurred, and the five columns correspond to the rows in order. to In the table This indicates that the robot is in state s. t Choose behavior B i The long-term benefits that can be obtained. At the start of training, all existing Q values are set to 0; when a new Boolean combination appears during training, a new row is created for that combination, and the five Q values of the new row are set to 0.
[0024] An ε-greedy strategy is used to select behaviors: During the initial training phase, the five behaviors are tested in turn or randomly. When differences in Q-values occur, the behavior with the larger current Q-value is prioritized, while trials of other behaviors are retained, ensuring that each column is updated based on the actual results. The execution flow for a single training cycle is as follows: Read the current state ; from to Choose an action And send behavior through a unified interface The running signal; The behavior continues to be invoked when it returns to running, until a predicate changes or the behavior returns to failure; Read the changed state, calculate the current reward, and update the state behavior value according to the following formula: in, , These represent the states of the robot at time t. The state at time t+1 Choose the i-th action The long-term benefits that can be obtained For learning rate, The instantaneous reward at time t. Discount factor; Continue selecting actions from the changed state. The training ends when the action that returns to the specified position successfully completes the return.
[0025] After training, find the maximum value among the five values in each row of the Q-table. The behavior corresponding to the column containing this maximum value is the optimal behavior for the current state. When there are multiple maximum Q-values in the same row, proceed according to... to We select one of the fixed numbering sequences so that the same Q-table always yields the same result. The final policy π(s) represents the optimal behavior in state s.
[0026] Step S104: Based on the mapping relationship, select the states that are mapped to any behavior from all the states that have appeared and form the minterm set of the behavior. Summarize the states that have not appeared in the training into the irrelevant term set. Use the minterm set as the states that must be covered and the irrelevant terms as the states that can be freely processed to perform logical simplification and obtain the simplified execution conditions of each behavior. It should be noted that, for behavior B i Find all pairs of data in the mapping table that satisfy π(s) = B i The states are placed into behavior B. i minterms(B) i A minterm is a logical expression corresponding to a complete Boolean state. According to... The fixed order of bitwise conversion is as follows: when the value is 1, it is written to the variable itself; when the value is 0, it is written to the negation of the variable; then the ten variables are connected by a logical AND.
[0027] Ten Boolean variables can form 2 10 There are several theoretical states. States that have appeared during training are retained in the mapping table, while Boolean combinations that have not appeared are placed in the dontcares set. Dontcares represent combinations for which no single behavior has been determined during current training, and can be incorporated into any adjacent condition during logical simplification. This allows variables that do not affect the actual behavior selection to be removed, avoiding the retention of a large number of condition nodes for states that will not appear.
[0028] Then use logical simplification. For behavior... ,Will As a state that must be covered, As a state that can be freely processed, the output behavior Simplified execution conditions .
[0029] During simplification, multiple minterms are compared, states with only one variable differing in value are merged, and variables whose changes in value do not affect the behavior selection are deleted. The final result is... to Five conditions for implementation: In the formula, It is behavior The simplified execution conditions, LogicMin is the logical simplification operation, minterms(B i Select B from the Q table. i All states that have appeared, and dontcares comes from states that did not appear during training.
[0030] Step S105: Using the selected node as the root node, establish five sequential nodes under the root node. Each sequential node contains the corresponding simplified execution conditions and behaviors to form a conditional behavior tree. Perform inverse condition processing on the conditional behavior tree and obtain the final behavior tree based on the inverse condition processing result. In this step, a selection node is created as the root node of the behavior tree, and five sequence nodes are created under the root node. Each sequence node contains only two child nodes: the left child node is the obtained execution condition t. i The right child node corresponds to behavior B. i Connect t1 and B1 to form the first sequential node, connect t2 and B2 to form the second sequential node, and connect t3 and B3, t4 and B4, and t5 and B5 in the same way. Then place the five sequential nodes under the root selection node in the order of their numbers from B1 to B5. The resulting conditional behavior tree is as follows: In the formula: This indicates that the sequential node only has condition t. i B will only be executed if the conditions are met. i , This indicates that the selected nodes are executed sequentially in order, and CBT is the generated conditional behavior tree.
[0031] When the conditional behavior tree is running, the root selection node sends run signals to the five sequential nodes from left to right. A sequential node first checks its own execution conditions; if a condition is false, it immediately returns "Failure," and the root selection node continues checking the sequential nodes to its right; if a condition is true, the corresponding behavior is invoked. After the behavior returns "Running" or "Success," the root selection node stops checking the nodes to its right, ensuring that only one robot behavior is activated within a control cycle. When all conditions are false, the root selection node returns "Failure" and records the currently uncovered state.
[0032] The five sequential nodes use the exact same structure, so there is no need to search the tree depth, node types, and connection methods during training, nor is it necessary to compare all permutations of the five behaviors. The priority of a behavior under the root node is automatically excluded by the execution conditions, ensuring that each existing state still executes the optimal behavior specified in the Q-table.
[0033] Furthermore, in some embodiments, two sequential nodes are randomly selected from the conditional behavior tree, and their execution conditions are extracted respectively. Complementary variables with opposite values in the two execution conditions are searched. When the complementary variable is a behavior success predicate, the two sequential nodes are combined into a memory sequential node. This node continues to execute the previous behavior when the previous behavior returns to running, and switches to the next behavior after the previous behavior returns to success. When the complementary variable is not a behavior success predicate, the two sequential nodes are combined into a conditional branch node. At runtime, the current actual value of the complementary variable is read. If the value is 1, the behavior in one of the sequential nodes is executed; if the value is 0, the behavior in the other sequential node is executed. After each combination, the minterms corresponding to the two sequential nodes are merged into the minterms of the new node. The search and combination are repeated until there are no more sequential nodes that can be combined. The uncombined sequential nodes are treated as independent nodes. All nodes are arranged in ascending order of execution condition length and then connected to the root selection node to obtain the final behavior tree. When multiple complementary variables exist between two execution conditions, the behavior success predicate is preferentially selected as the complementary variable because it directly reflects the sequential dependency between behaviors. The memory sequence node obtained by combining it can correctly express the sequential logic of turning to the next behavior after the previous behavior succeeds. If no behavior success predicate is available as a complementary variable, one of the remaining complementary variables is selected. The generated condition branch node only changes the organization of the condition judgment and does not change the optimal behavior corresponding to any existing state. If there are no complementary variables between two execution conditions, the two sequence nodes are not combined and are kept as independent nodes. The minterm set after merging is used for subsequent logical simplification. That is, after each combination, logical simplification is performed again on the new node obtained by merging, and predicate variables whose values do not affect the behavior selection after being changed are deleted. In other words, if it is found after merging that the truth value of a certain predicate does not change whether the behavior is selected, it indicates that the predicate does not play a decisive role in the selection result of the current behavior. The predicate is then deleted from the condition expression, thereby further simplifying the execution conditions of the new node. When multiple combination methods exist, node pairs with shorter execution condition lengths are combined first, followed by node pairs with longer execution condition lengths, resulting in a more compact final behavior tree structure. Since combination only changes the tree's organization and does not alter the correspondence between states and behaviors, the final behavior trees obtained from different combination orders still execute the same state-behavior mapping during robot operation. When two nodes have the same execution condition length, they are arranged in ascending order of behavior number, ensuring that the same conditional behavior tree yields the same final behavior tree each time it undergoes inverse condition processing, guaranteeing reproducible results.
[0034] Step S106: Deploy the final behavior tree to the robot task planning computer. When the robot is running, it collects the signals of the underlying controller according to the control cycle and converts them into real-time Boolean predicates so that the root node checks the execution conditions of each node in turn according to the real-time Boolean predicates. When the conditions are met, the corresponding behavior is called. When the behavior is running, the behavior continues to be executed. After the behavior returns successfully, the Boolean predicate is updated. The updated Boolean predicate enters the behavior tree for judgment again until the five behaviors are executed in turn.
[0035] It's important to note that this step truly implements the final behavior tree obtained through previous training, extraction, simplification, tree building, and inverse condition processing onto the robot. This transforms the behavior tree from a static policy table into a closed loop with the real-time state: Each control cycle reads the underlying controller signal, converts it into a Boolean predicate, and the root node determines which behavior to invoke based on the execution conditions. If a behavior returns "in progress," it indicates the current action is not yet complete, so the same behavior continues without switching. If a behavior returns "successful," it signifies the completion of this stage, and the task status predicate and behavior success predicate are updated, allowing the new predicate combination to re-enter the behavior tree for judgment, determining the next behavior to execute. This process continues round by round, allowing the robot to sequentially complete five tasks: navigation, docking, flushing, oil retrieval, and return. The entire execution sequence is generated by the Q-table, execution conditions, and tree structure, eliminating the need for manually writing the workflow before the behavior tree is generated.
[0036] In summary, this invention encapsulates the five operational behaviors of the converter sampling robot independently. Reinforcement learning only selects and combines complete task behaviors, eliminating the need to relearn the continuous control quantities of the chassis, robotic arm, and sampling device, and allowing direct use of existing control modules. By converting the signals of the underlying controller into finite Boolean predicates and combining them into states, each state can serve as a definite row number in the Q-table, avoiding the infinite growth of states caused by directly inputting continuous values into the table. After training, the states mapped to each behavior are selected from the Q-table to form a minterm set. After logical simplification, simplified execution conditions are obtained, and a conditional behavior tree is generated. Then, the conditional behavior tree is processed through inverse condition processing. The final behavior tree is reorganized, allowing the state conditions, behavior choices, and sequence relationships in the strategy to be displayed in a tree structure, facilitating inspection and subsequent maintenance by engineers. The entire behavior tree generation process does not require searching for all permutations and tree structures of the five behaviors. Instead, it first generates a fixed-structure conditional behavior tree and then reorganizes the conditions and nodes, reducing the process of repeatedly adding, deleting, and exchanging nodes. The inverse condition processing only changes the expression structure of the behavior tree and does not change the state-behavior correspondences already learned in the Q-table. Therefore, the final behavior tree can still call the five behaviors of autonomous navigation, robotic arm docking, oil flushing, oil retrieval, and return according to the training strategy to complete the converter sampling operation.
[0037] like Figure 2As shown, one embodiment of the present invention proposes a task planning system for a commutator variable sampling robot based on reinforcement learning, the system comprising: The behavior segmentation module 10 is used to divide the work tasks of the converter sampling robot into five independently encapsulated behaviors. In each control cycle, it collects the operation signals of the robot's underlying controller about each behavior and converts them into Boolean predicates. The Boolean predicates include five task status predicates and five behavior success predicates. The predicate combination module 20 is used to compare the Boolean predicates of the current control cycle with those of the previous control cycle, and to determine the changed predicate combination as a new state only when at least one predicate changes; Training module 30 is used to repeatedly execute various behaviors in a simulation environment using reinforcement learning. Each time a new state is generated, a new row is created for that state in the preset Q table and the state behavior values of each column in that row are initialized to zero. Rewards are generated based on the behavior execution results and the state behavior values of the corresponding state and the called behavior in the preset Q table are updated. After training is completed, the behavior corresponding to the maximum state behavior value in each row of the preset Q table is taken as the optimal behavior in that state, and the mapping relationship between the state and the optimal behavior is obtained. The simplification module 40 is used to filter out the states mapped to any behavior from all the existing states according to the mapping relationship and form the minterm set of the behavior. The states that did not appear in the training are summarized into the irrelevant term set. Logical simplification is performed with the minterm set as the states that must be covered and the irrelevant terms as the states that can be freely processed, to obtain the simplified execution conditions of each behavior. The behavior tree construction module 50 is used to build five sequential nodes under the root node with the selection node as the root node. Each sequential node contains the corresponding simplified execution conditions and behaviors to form a conditional behavior tree. The conditional behavior tree is subjected to inverse condition processing, and the final behavior tree is obtained based on the inverse condition processing result. The behavior execution module 60 is used to deploy the final behavior tree to the robot task planning computer. When the robot is running, it collects the signals of the underlying controller according to the control cycle and converts them into real-time Boolean predicates so that the root node checks the execution conditions of each node in turn according to the real-time Boolean predicates. When the conditions are met, the corresponding behavior is called. When the behavior is running, the behavior continues to be executed. After the behavior returns successfully, the Boolean predicate is updated. The updated Boolean predicate is re-entered into the behavior tree for judgment until the five behaviors are executed in turn.
[0038] In another aspect, the present invention also proposes a storage medium on which one or more programs are stored, which, when executed by a processor, implement the above-described reinforcement learning-based commutation variable sampling robot task planning method.
[0039] In another aspect, the present invention also proposes an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to realize the above-mentioned reinforcement learning-based commutator sampling robot task planning method.
[0040] Those skilled in the art will understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain stored, communicated, propagated, or transmitted programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.
[0041] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.
[0042] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0043] While embodiments of the present invention have been described in detail above, it will be apparent to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations fall within the scope and spirit of the invention as set forth in the claims. Furthermore, the invention described herein may have other embodiments and can be implemented or carried out in various ways.
Claims
1. A reinforcement learning based converter transformer sampling robot task planning method, characterized in that, The method includes: The operation tasks of the converter sampling robot are divided into five independently encapsulated behaviors. In each control cycle, the operation signals of the robot's underlying controller for each behavior are collected and converted into Boolean predicates. The Boolean predicates include five task state predicates and five behavior success predicates. Compare the Boolean predicates of the current control cycle with those of the previous control cycle, and determine the new state only when at least one predicate changes; In the simulation environment, reinforcement learning is used to repeatedly execute various behaviors. Each time a new state is generated, a new row is created for that state in the preset Q table and the state behavior values of each column in that row are initialized to zero. Rewards are generated based on the behavior execution results and the state behavior values of the corresponding state and the called behavior in the preset Q table are updated. After training is completed, the behavior corresponding to the maximum state behavior value in each row of the preset Q table is taken as the optimal behavior in that state, and the mapping relationship between the state and the optimal behavior is obtained. Based on the mapping relationship, states that map to any behavior are selected from all the existing states and constitute the minterm set of that behavior. States that did not appear during training are aggregated into irrelevant term set. Logical simplification is performed using the minterm set as the states that must be covered and the irrelevant terms as the states that can be freely processed, to obtain the simplified execution conditions for each behavior. Using the selection node as the root node, five sequential nodes are created under the root node. Each sequential node contains the corresponding simplified execution conditions and behaviors, forming a conditional behavior tree. The conditional behavior tree is then subjected to inverse condition processing, and the final behavior tree is obtained based on the inverse condition processing result. The final behavior tree is deployed to the robot task planning computer. When the robot is running, the signals of the underlying controller are collected according to the control cycle and converted into real-time Boolean predicates. This allows the root node to check the execution conditions of each node in turn according to the real-time Boolean predicates. When the conditions are met, the corresponding behavior is called. If the behavior is running, the behavior continues to be executed. After the behavior returns successfully, the Boolean predicate is updated. The updated Boolean predicate is then entered into the behavior tree for judgment again until the five behaviors are executed in turn.
2. The reinforcement learning based commutation variable sampling robot task planning method according to claim 1, characterized in that, The steps of dividing the tasks of the converter sampling robot into five independently packaged behaviors include: Divide the job task into: behavior , read the current position and the oil pickup point coordinates, call the mobile chassis controller to navigate to the oil pickup position; behavior , read the oil pickup port pose and the robot pose, call the robot controller to extend the robot to dock with the oil pickup port; behavior , call the oil pickup device controller to flush the oil circuit; behavior , call the oil pickup device controller to take oil from the transformer; behavior , call the mobile chassis controller to return to the specified position; Each action returns one of three results after receiving a run signal: running, success, or failure. Running means that the current action is being executed, success means that the goal of the current action has been achieved, and failure means that the current action cannot be executed or has failed to execute.
3. The reinforcement learning-based task planning method for commutator variable sampling robots according to claim 2, characterized in that, The step of acquiring the robot's underlying controller's operation signals for each behavior in each control cycle and converting them into Boolean predicates, wherein the Boolean predicates include five task state predicates and five behavior success predicates, includes: In each control cycle, the signal from the underlying controller is read and converted into a Boolean predicate of 0 or 1. A value of 1 indicates that the action is true and a value of 0 indicates that the condition is false. Define task state predicates corresponding to the five behaviors respectively. to , Used to read whether the robot has reached the oil collection point. Used to read the signal indicating that the robotic arm has completed docking with the oil inlet. Used to read the signal indicating that the oil circuit flushing is complete. Used to read the oil extraction completion signal. Used to read the return arrival signal; Define behavioral success predicates corresponding to the five behaviors. to When any action returns success, the success predicate corresponding to that action is set to 1, and the success predicates for all other actions are set to 0. to Used for reading behavior respectively to The execution result.
4. The reinforcement learning-based task planning method for commutator variable sampling robots according to claim 3, characterized in that, The steps of generating rewards based on the behavior execution results and updating the state-behavior values of the corresponding states and invoked behaviors in the preset Q table, and after training, taking the behavior corresponding to the maximum state-behavior value in each row of the preset Q table as the optimal behavior in that state to obtain the mapping relationship between states and optimal behaviors include: Read the current state ; from to Choose an action And send behavior through a unified interface The running signal; The behavior continues to be invoked when it returns to running, until a predicate changes or the behavior returns to failure; Read the changed state, calculate the current reward, and update the state behavior value according to the following formula: ; in, , These represent the states of the robot at time t. The state at time t+1 Choose the i-th action The long-term benefits that can be obtained For learning rate, The instantaneous reward at time t. Discount factor; Continue selecting actions from the changed state. The training ends when the action that returns to the specified position successfully completes the return.
5. The reinforcement learning-based task planning method for commutator variable sampling robots according to claim 4, characterized in that, The steps of filtering states mapped to any behavior from all existing states according to the mapping relationship and forming a minterm set for that behavior, summarizing states not appearing during training into an irrelevant term set, and logically simplifying the minterm set (representing states that must be covered) and the irrelevant terms (representing states that can be freely processed) to obtain the simplified execution conditions for each behavior include: Filter out from the preset Q table The target state is determined, and all target states are aggregated into a set of minterms: ; in, The optimal behavior in state s, For behavior The set of minterms; Each state is composed of ten Boolean predicates in a fixed order. Each state is converted into a logical expression bit by bit in the fixed order. The conversion rule is: when the predicate value is 1, it is written into the predicate itself; when the predicate value is 0, it is written into the negation form of the predicate. The bits are connected by a logical AND to obtain the minterm corresponding to the state. For each action, the set of minterms for that action is taken as the state that must be covered, and the set of irrelevant terms is taken as the state that can be freely processed. Multiple minterms in the set of minterms are compared, and minterms that differ by only one predicate value are merged. If after merging, it is found that the truth value of a certain predicate does not change whether the action is selected, it indicates that the predicate does not play a decisive role in the selection result of the current action, and the predicate is deleted from the conditional expression. Repeat merging and deleting until simplification can no longer be achieved, and output the simplification execution conditions for this behavior.
6. The reinforcement learning-based task planning method for commutator variable sampling robots according to claim 5, characterized in that, The steps of forming a condition-behavior tree by taking the selection node as the root node and creating five sequential nodes under the root node, each sequential node containing corresponding simplified execution conditions and behaviors, include: During runtime, the root node sends a run signal to the five sequential nodes in sequence, so that each sequential node can determine whether its execution condition is met based on its own run signal. If the condition is not met, it immediately returns failure and the root node continues to check the next sequential node. If the condition is met, the corresponding behavior is invoked. Once the invoked action returns to running or successful, the root node stops checking the next sequential node; When the execution conditions of all five sequential nodes are not met, the root node returns failure and records the current uncovered state.
7. The reinforcement learning-based task planning method for commutator variable sampling robots according to claim 6, characterized in that, The step of performing inverse condition processing on the conditional behavior tree and obtaining the final behavior tree based on the inverse condition processing result includes: Take any two sequential nodes from the conditional behavior tree, extract the execution conditions of the two sequential nodes respectively, and find complementary variables that have opposite values in the two execution conditions; When the complementary variable is a behavior success predicate, the two sequential nodes are combined into a memory sequential node. The memory sequential node continues to execute the previous behavior when the previous behavior returns to running, and switches to the next behavior after the previous behavior returns to success. When the complementary variable is not a successful predicate, the two sequential nodes are combined into a conditional branch node. When the conditional branch node is running, it reads the current actual value of the complementary variable. If the value of the complementary variable is 1, the behavior in one of the sequential nodes is executed. If the value of the complementary variable is 0, the behavior in the other sequential node is executed. Each time a combination is completed, the minterm sets corresponding to the two sequential nodes are merged into the minterm set of the new node. This search and combination is repeated until there are no more combinable sequential nodes in the conditional behavior tree. Treating ungrouped sequential nodes as independent nodes, all nodes are arranged in ascending order of execution condition length and then connected to the root selection node to obtain the final behavior tree.
8. A task planning system for a commutator variable sampling robot based on reinforcement learning, characterized in that, The system includes: The behavior segmentation module is used to divide the operation tasks of the converter sampling robot into five independently encapsulated behaviors. In each control cycle, it collects the operation signals of the robot's underlying controller regarding each behavior and converts them into Boolean predicates. The Boolean predicates include five task state predicates and five behavior success predicates. The predicate combination module is used to compare the Boolean predicates of the current control cycle with those of the previous control cycle. Only when at least one predicate changes, the changed predicate combination is determined as a new state. The training module is used to repeatedly execute various behaviors using reinforcement learning in a simulation environment. Each time a new state is generated, a new row is created for that state in the preset Q table, and the state behavior values of each column in that row are initialized to zero. Rewards are generated based on the behavior execution results, and the state behavior values of the corresponding state and the called behavior in the preset Q table are updated. After training is completed, the behavior corresponding to the maximum state behavior value in each row of the preset Q table is taken as the optimal behavior in that state, and the mapping relationship between the state and the optimal behavior is obtained. The simplification module is used to filter out the states mapped to any behavior from all the existing states according to the mapping relationship and form the minterm set of the behavior. The states that did not appear in the training are summarized into the irrelevant term set. Logical simplification is performed with the minterm set as the states that must be covered and the irrelevant terms as the states that can be freely processed, to obtain the simplified execution conditions of each behavior. The behavior tree construction module is used to build five sequential nodes under the root node with the selection node as the root node. Each sequential node contains the corresponding simplified execution conditions and behaviors to form a conditional behavior tree. The conditional behavior tree is then subjected to inverse condition processing, and the final behavior tree is obtained based on the inverse condition processing result. The behavior execution module is used to deploy the final behavior tree to the robot task planning computer. When the robot is running, it collects the signals of the underlying controller according to the control cycle and converts them into real-time Boolean predicates. This allows the root node to check the execution conditions of each node in turn according to the real-time Boolean predicates. When the conditions are met, the corresponding behavior is called. The behavior continues to be executed while it is running. After the behavior returns successfully, the Boolean predicate is updated. The updated Boolean predicate is then entered into the behavior tree for judgment again until all five behaviors are executed in turn.
9. A storage medium, characterized in that, The storage medium stores one or more programs that, when executed by a processor, implement the reinforcement learning-based task planning method for commutator variable sampling robots as described in any one of claims 1-7.
10. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein: The memory is used to store computer programs; When the processor executes the computer program stored in the memory, it implements the reinforcement learning-based commutator sampling robot task planning method as described in any one of claims 1-7.
