Autonomous Decision-making Method and Device Based on Dual-target Guidance

Abstract

The present invention provides a method and device for autonomous decision-making based on dual-objective guidance, which relates to the field of computer technology. The method includes: obtaining the current state of a target object; inputting the current state into a dual-objective guidance decision-making model to obtain a target decision corresponding to the current state output by the dual-objective guidance decision-making model; the dual-objective guidance decision-making model is trained based on a plurality of sample quadruples, and the sample quadruple includes a current sample state, a sample action, a sample reward, and a next sample state; the dual-objective guidance decision-making model is used to determine the target decision corresponding to the current state. Since the dual-objective guidance decision-making model is trained based on a plurality of sample quadruples, the dual-objective guidance decision-making model has better robustness and high decision-making accuracy of the model, improving the accuracy of the target decision.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular, to an autonomous decision-making method and device based on dual-goal guidance. Background Art

[0002] In offline reinforcement learning, complex state-action transitions and diverse trajectories lead to low learning efficiency. The goal-conditioned method provides an innovative perspective to drive the policy to make decisions based on the next state, thus simplifying the complex optimization problem into a goal-conditioned sub-problem.

[0003] Existing policy guidance methods integrate goal conditions into offline reinforcement learning by simultaneously learning a guidance policy to provide a goal state for the goal-conditioned execution policy. However, the policy guidance method uses a single policy to determine actions, and its performance depends to a large extent on the policy guidance network, resulting in a decrease in the robustness of the policy guidance network and thus low precision of the policy. Summary of the Invention

[0004] The present invention provides an autonomous decision-making method and device based on dual-goal guidance to solve the problem of low precision of the policy caused by the decrease in the robustness of the policy guidance network in the prior art.

[0005] The present invention provides an autonomous decision-making method based on dual-goal guidance, including:

[0006] Obtain the current state of the target object;

[0007] Input the current state into a dual-goal guidance decision model to obtain a target decision corresponding to the current state output by the dual-goal guidance decision model; the dual-goal guidance decision model is trained based on multiple sample quadruples, and the sample quadruple includes a current sample state, a sample action, a sample reward, and a next sample state; the dual-goal guidance decision model is used to determine the target decision corresponding to the current state.

[0008] According to the autonomous decision-making method based on dual-goal guidance provided by the present invention, the dual-goal guidance decision model includes a value network, a dual-guidance policy network, an integration network, and an execution policy network;

[0009] The step of inputting the current state into the dual-goal guidance decision model to obtain a target decision corresponding to the current state output by the dual-goal guidance decision model includes:

[0010] Input the current state into the value network to obtain a target value corresponding to the current state output by the value network;

[0011] Input the current state into the dual-guidance policy network to obtain multiple first states output by the dual-guidance policy network;

[0012] Input each of the first states into the value network to obtain first values corresponding to each of the first states output by the value network;

[0013] Input each of the first values into the integration network to obtain the target state output by the integration network;

[0014] Input the target state and the current state into the execution policy network to obtain the target decision output by the execution policy network.

[0015] According to an autonomous decision-making method based on dual-objective guidance provided by the present invention, the dual-guidance policy network and the execution policy network are trained based on the following steps:

[0016] Obtain a training data set, where the training data set includes the multiple sample quadruples;

[0017] Based on each of the sample quadruples, train the initial dual-guidance policy network and the initial execution policy network respectively to obtain the dual-guidance policy network and the execution policy network.

[0018] According to an autonomous decision-making method based on dual-objective guidance provided by the present invention, training the initial dual-guidance policy network and the initial execution policy network respectively based on each of the sample quadruples to obtain the dual-guidance policy network and the execution policy network includes:

[0019] Input the current sample state and the next sample state in each of the sample quadruples into the value network to obtain a first sample value corresponding to the current sample state and a second sample value corresponding to the next sample state output by the value network;

[0020] Input the current sample state into the initial dual-guidance policy network to obtain multiple first sample states corresponding to the current sample state output by the initial dual-guidance policy network;

[0021] Input the current sample state and the next sample state into the initial execution policy network to obtain a predicted action output by the initial execution policy network;

[0022] Based on the first sample value, the second sample value, each of the first sample states, the sample reward, the next sample state, and the sample action, determine the dual-guidance policy network and the execution policy network.

[0023] A self - decision - making method based on dual - objective guidance provided by the present invention, determining the dual - guidance policy network and the execution policy network based on the first sample value, the second sample value, each first sample state, the sample reward, and the sample action, includes:

[0024] Based on the first sample value, the second sample value, each first sample state, the sample reward, the next sample state, and the sample action, update the parameters of the initial dual - guidance policy network and the initial execution policy network to obtain an updated dual - guidance policy network and an updated execution policy network;

[0025] Based on the updated dual - guidance policy network and the updated execution policy network, determine the dual - guidance policy network and the execution policy network.

[0026] A self - decision - making method based on dual - objective guidance provided by the present invention, updating the parameters of the initial dual - guidance policy network and the initial execution policy network based on the first sample value, the second sample value, each first sample state, the sample reward, the next sample state, and the sample action to obtain an updated dual - guidance policy network and an updated execution policy network, includes:

[0027] Based on the first sample value, the second sample value, each first sample state, the sample reward, and the next sample state, determine a first optimization target value of the initial dual - guidance policy network; the first optimization target value is used to update the parameters of the initial dual - guidance policy network;

[0028] Based on the first sample value, the second sample value, each first sample state, the sample reward, and the sample action, determine a second optimization target value of the initial execution policy network; the second optimization target value is used to update the parameters of the initial execution policy network;

[0029] Based on the first optimization target value and the second optimization target value, determine the updated dual - guidance policy network and the updated execution policy network.

[0030] The present invention also provides a self - decision - making device based on dual - objective guidance, including:

[0031] A first acquisition module, configured to acquire the current state of the target object;

[0032] A decision-making module, configured to input the current state into a dual-objective guided decision-making model to obtain a target decision corresponding to the current state output by the dual-objective guided decision-making model; the dual-objective guided decision-making model is trained based on a plurality of sample quadruples, and the sample quadruple includes a current sample state, a sample action, a sample reward, and a next sample state; the dual-objective guided decision-making model is used to determine the target decision corresponding to the current state.

[0033] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method for decision-making based on dual-objective guidance as described in any one of the above is implemented.

[0034] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for autonomous decision-making based on dual-objective guidance as described in any one of the above is implemented.

[0035] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the method for autonomous decision-making based on dual-objective guidance as described in any one of the above is implemented.

[0036] The method and device for autonomous decision-making based on dual-objective guidance provided by the present invention obtain the current state of a target object; input the current state into a dual-objective guided decision-making model to obtain a target decision corresponding to the current state output by the dual-objective guided decision-making model; the dual-objective guided decision-making model is trained based on a plurality of sample quadruples, and the sample quadruple includes a current sample state, a sample action, a sample reward, and a next sample state; the dual-objective guided decision-making model is used to determine the target decision corresponding to the current state. Since the dual-objective guided decision-making model is trained based on a plurality of sample quadruples, the dual-objective guided decision-making model has good robustness and high decision accuracy of the model, improving the accuracy of the target decision. Description of the Drawings

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0038] Figure 1 is one of the flowcharts of the method for autonomous decision-making based on dual-objective guidance provided by the present invention;

[0039] Figure 2It is the second schematic flowchart of the autonomous decision-making method based on dual-objective guidance provided by the present invention;

[0040] Figure 3 It is the schematic structural diagram of the autonomous decision-making device based on dual-objective guidance provided by the present invention;

[0041] Figure 4 It is the schematic structural diagram of the electronic device provided by the present invention. Detailed implementation manners

[0042] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without creative efforts shall fall within the protection scope of the present invention.

[0043] To facilitate a clearer understanding of the embodiments of the present application, relevant background knowledge will be introduced first.

[0044] The current offline reinforcement learning technologies mainly include four categories: (1) Policy constraint: The training objective is to make the function distribution constraint of the generated policy close to the function distribution of the behavior policy. (2) Regularization: Set objectives such as controlling randomness and reducing overestimated weights to eliminate out-of-distribution action problems. (3) Single-step method: Reduce the out-of-distribution actions caused by the mutual interference between policy evaluation and policy improvement in the actor-critic structure. (4) Avoid policy evaluation to avoid interference in the actor-critic structure, thereby avoiding out-of-distribution actions.

[0045] However, policy constraint and regularization do not recognize some actions containing specific objectives, and their manifestations are similar to out-of-distribution actions. These actions come from the data set, and the algorithm does not consider the objectives from the data set; the single-step method splits policy evaluation and policy improvement; imitation learning does not make good use of rewards for policy evaluation.

[0046] The following combines Figure 1 - Figure 2 to describe the autonomous decision-making method based on dual-objective guidance of the present invention.

[0047] Figure 1 It is the first schematic flowchart of the autonomous decision-making method based on dual-objective guidance provided by the present invention. As Figure 1 shown, the method includes step 101-step 102; wherein,

[0048] Step 101, obtain the current state of the target object.

[0049] It should be noted that the autonomous decision-making method based on dual-goal guidance provided by the present invention can be applied to scenarios of intelligent agent autonomous decision-making. For example, in the autonomous decision-making task of a robot, the execution entity of this method can be an autonomous decision-making device based on dual-goal guidance, such as an electronic device, or a control module in the autonomous decision-making device based on dual-goal guidance for executing the autonomous decision-making method based on dual-goal guidance.

[0050] Specifically, the target object is an intelligent agent, for example, a robot. Obtain the current state s of the target object, that is, the state of the target object in the current environment.

[0051] Step 102: Input the current state into the dual-goal guidance decision-making model to obtain the target decision corresponding to the current state output by the dual-goal guidance decision-making model; the dual-goal guidance decision-making model is trained based on multiple sample quadruples, and the sample quadruple includes the current sample state, sample action, sample reward, and next sample state; the dual-goal guidance decision-making model is used to determine the target decision corresponding to the current state.

[0052] Specifically, the dual-goal guidance decision-making model is a trained decision-making model. The dual-goal guidance decision-making model is trained based on multiple sample quadruples, and the sample quadruple includes the current sample state, sample action, sample reward, and next sample state; the dual-goal guidance decision-making model is used to determine the target decision corresponding to the current state, and the target decision is the action to be executed.

[0053] After the dual-goal guidance decision-making model is trained, input the current state into the dual-goal guidance decision-making model, and the target decision corresponding to the current state output by the dual-goal guidance decision-making model can be obtained.

[0054] The autonomous decision-making method based on dual-goal guidance provided by the present invention obtains the current state of the target object; inputs the current state into the dual-goal guidance decision-making model to obtain the target decision corresponding to the current state output by the dual-goal guidance decision-making model; the dual-goal guidance decision-making model is trained based on multiple sample quadruples, and the sample quadruple includes the current sample state, sample action, sample reward, and next sample state; the dual-goal guidance decision-making model is used to determine the target decision corresponding to the current state. Since the dual-goal guidance decision-making model is trained based on multiple sample quadruples, the dual-goal guidance decision-making model has good robustness and high decision-making accuracy of the model, improving the accuracy of the target decision.

[0055] Optionally, the dual-goal guidance decision-making model includes a value network, a dual-guidance policy network, an integration network, and an execution policy network; the specific implementation manner of the above step 102 includes:

[0056] (1) Input the current state into the dual-guidance policy network to obtain multiple first states output by the dual-guidance policy network.

[0057] Specifically, input the current state s into the dual-guidance policy network, and multiple first states output by the dual-guidance policy network can be obtained; among them, the first state is s′ A or s′ B .

[0058]

[0059] Among them, g represents the dual-guidance policy network, and w i represents the parameters of the dual-guidance policy network.

[0060] (2) Input each of the first states into the value network to obtain the first values corresponding to each of the first states output by the value network.

[0061] Specifically, input each first state into the value network V, and the first values V(s′ i ) corresponding to each of the first states output by the value network can be obtained, where s′ i ∈{s′ A , s′ B}.

[0062] (3) Input each of the first values into the integration network to obtain the target state output by the integration network.

[0063] Specifically, input each first value into the integration network, and the target state s′ ref output by the integration network can be obtained.

[0064] The integration network obtains a high-precision target state by integrating these two first states and by assigning weights. The first values estimated according to the value function are used as the weights for the two first states. The first state with a lower value is assigned a lower weight η, while the weight (1 - η) is assigned to the other first state, where η ∈ [0, 1] represents the weight index. The integration network can be expressed as:

[0065]

[0066] (4) Input the target state and the current state into the execution policy network to obtain the target decision output by the execution policy network.

[0067] Specifically, input the target state s′ ref and the current state s into the execution policy network, and the target decision a output by the execution policy network can be obtained.

[0068] a = πθ (s, s' ref ) (3)

[0069] where π θ represents the execution policy network, and θ represents the parameters of the execution policy network.

[0070] In the process of generating the target decision of the present invention, by providing the current state of the target object, an accurate estimation of multiple first states is achieved, and finally a more robust target decision is obtained from a statistical perspective through the execution policy network, thereby improving the accuracy of the target decision.

[0071] Optionally, the dual guidance policy network and the execution policy network are trained based on the following steps:

[0072] (a) Obtain a training data set, where the training data set includes the multiple sample quadruples.

[0073] Specifically, the training data set is an offline data set, and the training data set includes multiple sample quadruples. The sample quadruple includes the current sample state, the sample action, the sample reward, and the next sample state.

[0074] (b) Based on each of the sample quadruples, train the initial dual guidance policy network and the initial execution policy network respectively to obtain the dual guidance policy network and the execution policy network.

[0075] Specifically, the initial dual guidance policy network and the initial execution policy network are the network models before the start of training. According to each of the sample quadruples in the training data set, train the initial dual guidance policy network and the initial execution policy network respectively, and the dual guidance policy network and the execution policy network after training can be obtained.

[0076] Optionally, the training of the initial dual guidance policy network and the initial execution policy network respectively based on each of the sample quadruples to obtain the dual guidance policy network and the execution policy network includes:

[0077] (b-1) Input the current sample state and the next sample state in each of the sample quadruples into the value network to obtain the first sample value corresponding to the current sample state output by the value network and the second sample value corresponding to the next sample state.

[0078] Specifically, input the current sample state s and the next sample state s' in each sample quadruple into the value network V, and the first sample value V φ (s) corresponding to the current sample state output by the value network and the second sample value V φ (s') corresponding to the next sample state can be obtained.

[0079] (b - 2) Input the current sample state into the initial dual - guidance policy network to obtain multiple first sample states corresponding to the current sample state output by the initial dual - guidance policy network.

[0080] Specifically, inputting the current sample state into the initial dual - guidance policy network can obtain multiple first sample states corresponding to the current sample state output by the initial dual - guidance policy network. The first sample state is s′ A or s′ B .

[0081] It should be noted that the role of the guidance policy network is to evaluate the target (the next state, that is, multiple first sample states) according to the current situation (the current sample state). When the next state is the target, it is correspondingly mapped to g ω : s → s′. Since (s, s′) is already included in the training dataset, the guidance policy network is trained through imitation learning.

[0082] (b - 3) Input the current sample state and the next sample state into the initial execution policy network to obtain the predicted action output by the initial execution policy network.

[0083] Specifically, inputting the current sample state s and the next sample state s′ into the initial execution policy network can obtain the predicted action a′ output by the initial execution policy network.

[0084] It should be noted that the role of the execution policy network is to generate an action according to the current sample state and the next sample state s′, and the corresponding mapping is π θ : (s, s′) → a. The current sample state, the next sample state, and the sample action in the training dataset are used to train the execution policy network. The execution policy network is also optimized using imitation learning, and the Bellman residual of the value function can also be used as the weight of the learning target, which is the same as the training process of the domain guidance policy network.

[0085] (b - 4) Based on the first sample value, the second sample value, each of the first sample states, the sample reward, the next sample state, and the sample action, determine the dual - guidance policy network and the execution policy network.

[0086] Specifically, according to the first sample value V φ (s), the second sample value V φ (s′), each of the first sample states (s′ A or s′ B ), the sample reward r, the next sample state s′, and the sample action a, the dual - guidance policy network and the execution policy network can be further determined.

[0087] Optionally, determining the dual-guided policy network and the execution policy network based on the first sample value, the second sample value, each of the first sample states, the sample reward, and the sample action includes:

[0088] 1) Updating the parameters of the initial dual-guided policy network and the initial execution policy network based on the first sample value, the second sample value, each of the first sample states, the sample reward, the next sample state, and the sample action to obtain an updated dual-guided policy network and an updated execution policy network.

[0089] Specifically, according to the first sample value V φ (s), the second sample value V φ (s′), each of the first sample states (s′ A or s′ B ), the sample reward r, the next sample state s′, and the sample action a, the parameters of the initial dual-guided policy network ω i and the initial execution policy network θ can be further updated to obtain an updated dual-guided policy network and an updated execution policy network.

[0090] 2) Determining the dual-guided policy network and the execution policy network based on the updated dual-guided policy network and the updated execution policy network.

[0091] Specifically, based on the updated dual-guided policy network and the updated execution policy network, repeat the step of updating the parameters of the initial dual-guided policy network and the initial execution policy network until the training stop condition is reached, and determine the finally updated dual-guided policy network and execution policy network as the dual-guided policy network and the execution policy network.

[0092] Optionally, updating the parameters of the initial dual-guided policy network and the initial execution policy network based on the first sample value, the second sample value, each of the first sample states, the sample reward, the next sample state, and the sample action to obtain an updated dual-guided policy network and an updated execution policy network includes:

[0093] a) Determining a first optimization target value of the initial dual-guided policy network based on the first sample value, the second sample value, each of the first sample states, the sample reward, and the next sample state; the first optimization target value is used to update the parameters of the initial dual-guided policy network.

[0094] Specifically, based on the first sample value V φ (s), the second sample value V φ(s′), each first sample state (s′ A or s′ B ), sample reward r, and the next sample state s′, use formula (4) to determine the first optimization objective value J(ω i ) of the initial dual guidance policy network; formula (4) is expressed as:

[0095]

[0096] Among them, γ represents the reward discount factor, represents the dual guidance policy network, that is i ∈ {A, B} represents two guidance policies (first sample state) output by the dual guidance policy network, ω i represents the dual guidance policy network parameter, that is, the learnable parameter of the two first sample states. In this application, the two estimation targets of the guidance policy are represented as s′ A and s′ B . [[ID=२४]]

[0097] By calculating the Bellman residual of the value function of the current state and the next state, and using the Bellman residual as the advantage weight of the guidance policy, a better dual guidance policy network can be obtained.

[0098] b) Based on the first sample value, the second sample value, each of the first sample states, the sample reward, and the sample action, determine the second optimization objective value of the initial execution policy network; the second optimization objective value is used to update the parameters of the initial execution policy network.

[0099] Specifically, based on the first sample value V φ (s), the second sample value V φ (s′), each first sample state (s′ A or s′ B ), sample reward r, and sample action a, use formula (5) to determine the second optimization objective value J(θ) of the initial execution policy network; formula (5) is expressed as:

[0100]

[0101] Among them, θ represents the parameter of the execution policy network, π θ (s, s′) represents the execution policy network, and E represents the expectation.

[0102] The execution policy network is also optimized using imitation learning, and the Bellman residual of the value function of the current state and the next state can also be used as the weight of the learning target, just like the training process of the dual guidance policy network.

[0103] c) Determine the updated dual-guidance policy network and the updated execution policy network based on the first optimization target value and the second optimization target value.

[0104] Specifically, according to the determined first optimization target value and second optimization target value, update the initial dual-guidance policy network and the initial execution policy network to obtain the updated dual-guidance policy network and the updated execution policy network. Furthermore, based on the updated dual-guidance policy network and the updated execution policy network, repeat the above steps of updating the parameters of the initial dual-guidance policy network and the initial execution policy network until the training stop condition is reached, and determine the finally updated dual-guidance policy network and execution policy network as the dual-guidance policy network and the execution policy network.

[0105] Figure 2 It is the second flow diagram of the autonomous decision-making method based on dual-objective guidance provided by the present invention. As Figure 2 shown, the method includes:

[0106] Step 1, obtain a training data set, where the training data set includes the multiple sample quadruples, and the sample quadruple includes the current sample state (current state), sample action, sample reward, and next sample state. The current sample state is the sample state in the current environment.

[0107] Step 2, input the current sample state and the next sample state in each sample quadruple into the value network to obtain the first sample value corresponding to the current sample state and the second sample value corresponding to the next sample state output by the value network.

[0108] Step 3, input the current sample state into the initial dual-guidance policy network to obtain multiple first sample states (target A and target B during training) corresponding to the current sample state output by the initial dual-guidance policy network.

[0109] Step 4, determine the first optimization target value of the initial dual-guidance policy network based on the first sample value, the second sample value, each first sample state, the sample reward, and the next sample state; the first optimization target value is used to update the parameters of the initial dual-guidance policy network.

[0110] Step 5, input the current sample state and the next sample state into the execution policy network to obtain the predicted action output by the execution policy network. During training, the next sample state is used as the optimization target.

[0111] Step 6, determine the second optimization target value of the initial execution policy network based on the first sample value, the second sample value, each first sample state, the sample reward, and the sample action; the second optimization target value is used to update the parameters of the initial execution policy network.

[0112] Step 7: Based on the first optimization objective value and the second optimization objective value, update the initial execution policy network and the parameters of the initial execution policy network. Based on the updated dual guidance policy network and the updated execution policy network, repeat the above steps of updating the parameters of the initial dual guidance policy network and the initial execution policy network until the training stop condition is reached.

[0113] Step 8: Determine the finally updated dual guidance policy network and execution policy network as the dual guidance policy network and the execution policy network.

[0114] Step 9: Obtain the current state of the target object (the current state in actual application).

[0115] Step 10: Input the current state into the dual guidance policy network to obtain multiple first states (target A and target B in actual application) output by the dual guidance policy network.

[0116] Step 11: Input each first state into the value network to obtain the first value corresponding to each first state output by the value network.

[0117] Step 12: Input each first value into the integration network to obtain the target state (the optimization objective in actual application) output by the integration network.

[0118] Step 13: Input the target state and the current state into the execution policy network to obtain the target decision (action) output by the execution policy network.

[0119] The autonomous decision-making method based on dual-objective guidance provided by the present invention is a high-robustness intelligent agent target-driven control technology based on existing trajectories and reward data, mainly a Policy-Guided Offline Reinforcement Learning (POR) strategy based on the guidance policy algorithm. In the offline reinforcement learning strategy based on the POR algorithm, it has been experimentally proven that good target estimation can generate more accurate policy actions. Based on the importance of target estimation, the present invention innovatively introduces a dual guidance policy network and an integration network, strengthening the model's estimation of the target, and the model shows stronger robustness in simulation tasks.

[0120] Generally speaking, for the path search problem in a complex environment, the present invention first uses a dual guidance policy network to estimate two target points that are expected to be passed through, estimates the target point with a higher value through a value function, and finally generates a final execution policy according to the current state of the current environment and the estimated target point, which can effectively improve the target-driven ability in the intelligent agent control of existing trajectories and reward data.

[0121] The following describes the autonomous decision-making device based on dual-objective guidance provided by the present invention. The autonomous decision-making device based on dual-objective guidance described below can be referred to in correspondence with the autonomous decision-making method based on dual-objective guidance described above.

[0122] Figure 3 It is a schematic structural diagram of the autonomous decision-making device based on dual-objective guidance provided by the present invention. As Figure 3 shown, the autonomous decision-making device 300 based on dual-objective guidance includes: a first acquisition module 301 and a decision-making module 302; wherein,

[0123] The first acquisition module 301 is configured to acquire the current state of the target object;

[0124] The decision-making module 302 is configured to input the current state into the dual-objective guidance decision-making model to obtain the target decision corresponding to the current state output by the dual-objective guidance decision-making model; the dual-objective guidance decision-making model is trained based on a plurality of sample quadruples, and the sample quadruple includes a current sample state, a sample action, a sample reward, and a next sample state; the dual-objective guidance decision-making model is used to determine the target decision corresponding to the current state.

[0125] For the autonomous decision-making device based on dual-objective guidance provided by the present invention, by acquiring the current state of the target object; inputting the current state into the dual-objective guidance decision-making model to obtain the target decision corresponding to the current state output by the dual-objective guidance decision-making model; the dual-objective guidance decision-making model is trained based on a plurality of sample quadruples, and the sample quadruple includes a current sample state, a sample action, a sample reward, and a next sample state; the dual-objective guidance decision-making model is used to determine the target decision corresponding to the current state. Since the dual-objective guidance decision-making model is trained based on a plurality of sample quadruples, the dual-objective guidance decision-making model has better robustness and higher decision-making accuracy of the model, improving the accuracy of the target decision.

[0126] Optionally, the dual-objective guidance decision-making model includes a value network, a dual-guidance policy network, an integration network, and an execution policy network; the decision-making module 302 is specifically configured to:

[0127] Input the current state into the dual-guidance policy network to obtain a plurality of first states output by the dual-guidance policy network;

[0128] Input each of the first states into the value network to obtain the first value corresponding to each of the first states output by the value network;

[0129] Input each of the first values into the integration network to obtain the target state output by the integration network;

[0130] Input the target state and the current state into the execution policy network to obtain the target decision output by the execution policy network.

[0131] Optionally, the autonomous decision-making device 300 based on dual-objective guidance further includes:

[0132] A second acquisition module, configured to acquire a training data set, where the training data set includes the multiple sample quadruples;

[0133] A training module, configured to train an initial dual-guidance policy network and an initial execution policy network respectively based on each of the sample quadruples to obtain the dual-guidance policy network and the execution policy network.

[0134] Optionally, the training module is specifically configured to:

[0135] Input the current sample state and the next sample state in each of the sample quadruples into the value network to obtain a first sample value corresponding to the current sample state and a second sample value corresponding to the next sample state output by the value network;

[0136] Input the current sample state into the initial dual-guidance policy network to obtain multiple first sample states corresponding to the current sample state output by the initial dual-guidance policy network;

[0137] Input the current sample state and the next sample state into the initial execution policy network to obtain a predicted action output by the initial execution policy network;

[0138] Determine the dual-guidance policy network and the execution policy network based on the first sample value, the second sample value, each of the first sample states, the sample reward, the next sample state, and the sample action.

[0139] Optionally, the training module is further configured to:

[0140] Update the parameters of the initial dual-guidance policy network and the initial execution policy network based on the first sample value, the second sample value, each of the first sample states, the sample reward, the next sample state, and the sample action to obtain an updated dual-guidance policy network and an updated execution policy network;

[0141] Determine the dual-guidance policy network and the execution policy network based on the updated dual-guidance policy network and the updated execution policy network.

[0142] Optionally, the training module is further configured to:

[0143] Determine a first optimization objective value of the initial dual-guidance policy network based on the first sample value, the second sample value, each of the first sample states, the sample reward, and the next sample state; the first optimization objective value is used to update the parameters of the initial dual-guidance policy network;

[0144] Determine a second optimization objective value of the initial execution policy network based on the first sample value, the second sample value, each of the first sample states, the sample reward, and the sample action; the second optimization objective value is used to update the parameters of the initial execution policy network;

[0145] Determine the updated dual-guidance policy network and the updated execution policy network based on the first optimization objective value and the second optimization objective value.

[0146] Figure 4 is a schematic physical structure diagram of an electronic device provided by the present invention, as Figure 4 shown, the electronic device 400 may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440. Among them, the processor 410, the communication interface 420, and the memory 430 complete mutual communication through the communication bus 440. The processor 410 may call logical instructions in the memory 430 to execute an autonomous decision-making method based on dual-objective guidance, and the method includes: obtaining the current state of a target object; inputting the current state into a dual-objective guidance decision-making model to obtain a target decision corresponding to the current state output by the dual-objective guidance decision-making model; the dual-objective guidance decision-making model is trained based on a plurality of sample quadruples, and the sample quadruple includes a current sample state, a sample action, a sample reward, and a next sample state; the dual-objective guidance decision-making model is used to determine the target decision corresponding to the current state.

[0147] In addition, when the logical instructions in the above-mentioned memory 430 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0148] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the autonomous decision-making method based on dual-objective guidance provided by the above-mentioned various methods. The method includes: obtaining the current state of a target object; inputting the current state into a dual-objective guidance decision model to obtain the target decision corresponding to the current state output by the dual-objective guidance decision model; the dual-objective guidance decision model is trained based on a plurality of sample quadruples, and the sample quadruple includes a current sample state, a sample action, a sample reward, and a next sample state; the dual-objective guidance decision model is used to determine the target decision corresponding to the current state.

[0149] In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the autonomous decision-making method based on dual-objective guidance provided by the above-mentioned various methods. The method includes: obtaining the current state of a target object; inputting the current state into a dual-objective guidance decision model to obtain the target decision corresponding to the current state output by the dual-objective guidance decision model; the dual-objective guidance decision model is trained based on a plurality of sample quadruples, and the sample quadruple includes a current sample state, a sample action, a sample reward, and a next sample state; the dual-objective guidance decision model is used to determine the target decision corresponding to the current state.

[0150] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0151] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An autonomous decision-making method based on dual-objective guidance, characterized in that, Applied to the scenario of the robot's autonomous decision-making task, including: Obtain the current state of the target object; the target object is the robot; the current state is the state of the target object in the current environment; Input the current state into the dual-goal guided decision-making model to obtain the target decision corresponding to the current state output by the dual-goal guided decision-making model; the dual-goal guided decision-making model is trained based on multiple sample quadruples of the robot, and the sample quadruple includes the current sample state, sample action, sample reward, and next sample state; the dual-goal guided decision-making model is used to determine the target decision corresponding to the current state; the target decision is the action to be executed; The dual-goal guided decision-making model includes a value network, a dual-guided policy network, an integration network, and an execution policy network; The step of inputting the current state into the dual-goal guided decision-making model to obtain the target decision corresponding to the current state output by the dual-goal guided decision-making model includes: Input the current state into the dual-guided policy network to obtain multiple first states output by the dual-guided policy network; Input each of the first states into the value network to obtain the first value corresponding to each of the first states output by the value network; Input each of the first values into the integration network to obtain the target state output by the integration network; Input the target state and the current state into the execution policy network to obtain the target decision output by the execution policy network; The dual-guided policy network and the execution policy network are trained based on the following steps: Obtain a training data set, and the training data set includes the multiple sample quadruples of the robot; Based on each of the sample quadruples, train the initial dual-guided policy network and the initial execution policy network respectively to obtain the dual-guided policy network and the execution policy network.

2. The autonomous decision-making method based on dual-goal guidance according to claim 1, characterized in that The step of training the initial dual-guided policy network and the initial execution policy network respectively based on each of the sample quadruples to obtain the dual-guided policy network and the execution policy network includes: Input the current sample state and the next sample state in each of the sample quadruples into the value network to obtain the first sample value corresponding to the current sample state output by the value network and the second sample value corresponding to the next sample state; Input the current sample state into the initial dual-guided policy network to obtain multiple first sample states corresponding to the current sample state output by the initial dual-guided policy network; Input the current sample state and the next sample state into the initial execution policy network to obtain the predicted action output by the initial execution policy network; Based on the first sample value, the second sample value, each of the first sample states, the sample reward, the next sample state, and the sample action, determine the dual-guided policy network and the execution policy network.

3. The autonomous decision-making method based on dual-objective guidance according to claim 2, wherein Determining the dual-guided policy network and the execution policy network based on the first sample value, the second sample value, each of the first sample states, the sample reward, and the sample action includes: Updating the parameters of the initial dual-guided policy network and the initial execution policy network based on the first sample value, the second sample value, each of the first sample states, the sample reward, the next sample state, and the sample action to obtain an updated dual-guided policy network and an updated execution policy network; Determining the dual-guided policy network and the execution policy network based on the updated dual-guided policy network and the updated execution policy network.

4. The autonomous decision-making method based on dual-objective guidance according to claim 3, wherein The updating the parameters of the initial dual-guided policy network and the initial execution policy network based on the first sample value, the second sample value, each of the first sample states, the sample reward, the next sample state, and the sample action to obtain an updated dual-guided policy network and an updated execution policy network includes: Determining a first optimization target value of the initial dual-guided policy network based on the first sample value, the second sample value, each of the first sample states, the sample reward, and the next sample state; the first optimization target value is used to update the parameters of the initial dual-guided policy network; Determining a second optimization target value of the initial execution policy network based on the first sample value, the second sample value, each of the first sample states, the sample reward, and the sample action; the second optimization target value is used to update the parameters of the initial execution policy network; Determining the updated dual-guided policy network and the updated execution policy network based on the first optimization target value and the second optimization target value.

5. An autonomous decision-making device based on dual-target guidance, characterized in that, Applied to the scenario of the robot's autonomous decision-making task, it includes: A first acquisition module, configured to acquire the current state of the target object; the target object is a robot; the current state is the state of the target object in the current environment; A decision-making module, configured to input the current state into the dual-target guided decision-making model to obtain the target decision corresponding to the current state output by the dual-target guided decision-making model; the dual-target guided decision-making model is trained based on multiple sample quadruples of the robot, and the sample quadruple includes the current sample state, the sample action, the sample reward, and the next sample state; the dual-target guided decision-making model is used to determine the target decision corresponding to the current state; the target decision is the action to be executed; The dual-target guided decision-making model includes a value network, a dual-guided policy network, an integration network, and an execution policy network; The decision-making module, specifically configured to: Input the current state into the dual-guided policy network to obtain multiple first states output by the dual-guided policy network; Input each of the first states into the value network to obtain the first value corresponding to each of the first states output by the value network; [[ID= ]]Input each of the first values into the integration network to obtain the target state output by the integration network; Input the target state and the current state into the execution policy network to obtain the target decision output by the execution policy network; The dual guidance policy network and the execution policy network are trained based on the following steps: Obtain a training data set, where the training data set includes the multiple sample quadruples of the robot; Based on each of the sample quadruples, train the initial dual guidance policy network and the initial execution policy network respectively to obtain the dual guidance policy network and the execution policy network.

6. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the dual-target-guided autonomous decision-making method according to any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the dual-target-guided autonomous decision-making method according to any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the dual-target-guided autonomous decision-making method according to any one of claims 1 to 4.

Images