Self-learning method and device for non-antagonistic task, equipment and medium

Abstract

The invention relates to the technical field of reinforcement learning, in particular to a self-learning method and device for a non-antagonistic task, equipment and a medium, and the method comprises the steps: 1, initializing a decision neural network of the non-antagonistic task; step 2, executing an MCTS state search algorithm based on the current decision neural network, obtaining a third-party efficiency score and recording path information; step 3, performing training optimization on a decision neural network according to the third-party efficiency score and the path information; and step 4, repeatedly executing the steps 2-3 in parallel until the decision neural network meets a trigger termination condition. The self-learning method for the non-antagonistic tasks provided by the invention is not limited to task scenes, can obtain better results for all the non-antagonistic tasks, and can perform adaptive modification according to the task scenes; the shortest solution can be quickly and gradually explored from 0 for different non-antagonistic tasks, repeated calculation is avoided to the greatest extent, and the exploration efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning technology, and in particular to a self-learning method, device, equipment and medium for non-adversarial tasks. Background Art

[0002] In the field of artificial intelligence and machine learning, Monte Carlo Tree Search (MCTS), as an efficient and flexible decision-making algorithm, has demonstrated its outstanding capabilities in many complex problems, especially in navigation and decision optimization. MCTS effectively balances the relationship between exploration and utilization by simulating possible future decision paths, combining statistical evaluation and pruning strategies, and thus finding the optimal or near-optimal solution in an uncertain environment. This feature makes MCTS an ideal choice for dealing with complex decision problems with a large number of states and uncertainties.

[0003] In recent years, with the rapid development of deep learning technology, the strategy of combining deep learning with MCTS has shown unprecedented potential. This fusion not only enhances MCTS's ability to evaluate strategies in complex environments, but also automatically learns feature representations through deep neural networks, further improving the algorithm's adaptability and generalization capabilities. Among them, algorithms represented by AlphaZero use deep learning networks to predict the value of the board state and possible actions for the next step, and use this to provide MCTS with high-quality prior knowledge, greatly accelerating the search process and improving decision quality. This close integration of deep learning and MCTS has not only promoted breakthrough progress in the algorithm in traditional adversarial tasks such as Go, but also inspired its application exploration in more adversarial tasks and self-learning systems.

[0004] In view of this, the deep integration of MCTS and deep learning technology to build an intelligent decision-making system with efficient self-learning capabilities has become one of the important directions of current artificial intelligence research. The system can automatically learn from data and optimize decision-making strategies, and can achieve efficient and accurate decision-making in complex and changing environments with little or no human intervention.

[0005] However, the existing combination method is only applicable to application scenarios that exhibit multi-agent adversarial characteristics, and there are no self-learning optimization cases for non-adversarial tasks, which leads to great limitations on the actual application of this algorithm. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present invention provides a self-learning method for non-adversarial tasks, aiming to expand the application scenarios of this type of algorithm for non-adversarial tasks, and to improve the self-learning ability and algorithm efficiency for such scenarios, so as to cope with more complex and changeable practical application needs.

[0007] The technical means adopted by the present invention to solve its technical problems are: a self-learning method for non-adversarial tasks, characterized in that the method includes: step 1: initializing the decision neural network of the non-adversarial task; step 2: executing the MCTS state search algorithm based on the current decision neural network, obtaining the third-party performance score and recording the path information; step 3: training and optimizing the decision neural network according to the third-party performance score and path information; step 4: repeating steps 2-3 in parallel until the decision neural network meets the trigger termination condition.

[0008] The non-adversarial tasks described in the above technical solution meet the following characteristics:

[0009] ① Satisfy all descriptions of the Markov decision process except the reward function;

[0010] ② The action space during task execution is discrete. For each state, the next optional action can only be selected from a limited number of values;

[0011] ③For a given state s and action a, a unique next action s' can always be generated;

[0012] ④ The decision black box that generates the next action has three outputs: the probability distribution of the output, the value of the output state, and the expected number of steps for the output to reach the ideal terminal state;

[0013] ⑤The task can always stop after a finite number of steps;

[0014] ⑥ It is possible to determine whether the termination status is a task success status within a limited time.

[0015] The MCTS state search algorithm described in the above technical solution includes a simulation phase, an update phase, a selection phase and an expansion phase, wherein:

[0016] The simulation stage selects actions through the current decision neural network until the task is completed, simulates a complete task process, and obtains the performance score;

[0017] The update phase updates the information of each state on the success path on the Monte Carlo tree according to the execution result of the task, including:

[0018] According to the execution result, update Found_victory(s)=1 for each state on the successful path;

[0019] Record the total length of the new path as length, and update EL(s) = min(EL(s), length);

[0020] Final calculation: V(s) = Found_victory(s) + 1 / EL(s);

[0021] At the same time, for the parent node of this state, update Q(s previous ,a)=v(s), when s previous When action a tends to state s; N(s previous ,a)+=1; where s previous is the state at the previous moment, used to distinguish it from the current state s;

[0022] The path includes a sequence consisting of s0, a0, s1, a1, sT, etc. Found_Victory(s) is a marker for whether a path that passes through state s and finally reaches the success state has been searched; EL(s): the length of the path that passes through s and reaches the terminal state, only the minimum value is recorded;

[0023] The selection phase selects the best child node for traversal on the explored node according to a certain strategy until an unexplored leaf node is reached, including:

[0024] At an explored node, choose the next action that maximizes the following formula:

[0025] preference(s,a)=H(*|s)+U(s,a)+Q(s,a)(1);

[0026] Among them, H(*|s) represents the entropy of selecting the next action in state S based on the current decision neural network. The larger the entropy, the more likely S is to be a key divergence point, and it is recommended that the next action be selected from the actions of S; U(s,a) represents the impact of the number of explorations N(s,a) on the selection of the next action;

[0027] The expansion phase adds new nodes at the selected leaf nodes to represent new possible states; the specific method is consistent with the original MCTS algorithm and is achieved by executing the actions selected in the selection phase.

[0028] Step 3 described in the above technical solution includes:

[0029] The back propagation algorithm based on the loss function optimizes the weights of the decision neural network, and the optimization objectives include:

[0030] Minimize the prediction error between v(s) and EL(s);

[0031] Minimize the error between p(s,a) and U(s,a)+Q(s,a), where p(s,a) represents the probability of taking action a in state s, as predicted by the model;

[0032] Maximize the entropy reduction effect of each execution step where Found_victory remains 1;

[0033] Minimize the KL divergence between the new model and the old model on the explored trajectory.

[0034] The non-adversarial task described in the above technical solution includes a large language model text generation task, wherein:

[0035] Simulation phase:

[0036] If the large language model executes MCTS for the first time, a text is directly generated according to the current parameters until the maximum length is reached or a termination character is generated;

[0037] If the large language model is not executing MCTS for the first time, that is, a character string already exists in the prompt word, a text is generated using the current character string as the prompt word;

[0038] After the text generation task is fully executed, feedback from a discriminator is obtained; the discriminator includes but is not limited to a code compilation discriminator, a task execution discriminator, and a human score;

[0039] Update phase:

[0040] According to the execution result of the large language model, update Found_victory(s)=1 for each state on the success path (each state corresponds to the prompt word + generated text);

[0041] Record the total length of the new path as length, and update EL(s) = min(EL(s), length). The length also records the number of generated tokens.

[0042] Calculate V(s) = Found_victory(s) + 1 / EL(s);

[0043] At the same time, for the parent node of this state, update Q(s previous ,a)=v(s), when s previous When action a tends to state s; N(s previous ,a)+=1; where s previous is the state at the previous moment;

[0044] Selection stage:

[0045] At an explored node, choose the next action that maximizes the following formula:

[0046] preference(s,a)=H(*|s)+U(s,a)+Q(s,a);

[0047] Among them, H(a|s) represents the entropy of selecting the next action in state S based on the current decision network. The larger the entropy, the more likely S is a semantic divergence point, and it is recommended that the next action be selected from the optional actions of S.

[0048] Expansion phase:

[0049] Expand the next node according to the selected action.

[0050] The non-adversarial task in the above technical solution includes a path planning algorithm, wherein:

[0051] Simulation phase:

[0052] Generate a path based on the current decision neural network and the current state until termination;

[0053] After a complete execution, the feedback of the discriminator is obtained, where 1 is passed and 0 is failed; the discriminator is used to detect whether a collision occurs during the simulation execution of the route;

[0054] Update phase:

[0055] According to the execution result, update Found_victory(s) = 1 for each state (anchor point coordinates + tree structure) on the successful path;

[0056] Record the total length of the new path as length, and update EL(s) = min(EL(s), length); the length also records the number of generated tokens;

[0057] Calculate V(s) = Found_victory(s) + 1 / EL(s);

[0058] At the same time, for the parent node of this state, update Q(s previous ,a)=v(s), when s previous When action a tends to state s; N(s previous ,a)+=1; where s previous is the state at the previous moment;

[0059] Selection stage:

[0060] At an explored node, choose the next action that maximizes the following formula:

[0061] preference(s,a)=H(*|s)+U(s,a)+Q(s,a);

[0062] Among them, H(a|s) represents the entropy of selecting the next action in state S based on the current decision network. The larger the entropy, the more the current network believes that it is possible to develop in any direction at this anchor point, and it is recommended that the next action be selected from the optional actions of S.

[0063] Use the above formula to determine where the current network thinks the path divergence point is, and select a new path based on the divergence point;

[0064] Expansion phase:

[0065] Expand the next node according to the selected action.

[0066] The non-adversarial task described in the above technical solution includes a robot evolution algorithm, wherein:

[0067] Simulation phase:

[0068] Directly generate a path based on the current network or current state until termination;

[0069] After a task is fully executed, feedback from the discriminator is obtained, where 1 is passed and 0 is failed; the discriminator is used to score the result of the evolution;

[0070] Update phase:

[0071] According to the execution result, update Found_victory(s) = 1 for each state on the success path (robot morphology or intelligence behind it, depending on the specific task);

[0072] Record the total length of the new path as length, and update EL(s) = min(EL(s), length); the length also records the number of generated tokens;

[0073] Calculate V(s) = Found_victory(s) + 1 / EL(s);

[0074] At the same time, for the parent node of this state, update Q(s previous ,a)=v(s), when s previous When action a tends to state s; N(s previous ,a)+=1; where s previous is the state at the previous moment;

[0075] Selection stage:

[0076] At an explored node, choose the next action that maximizes the following formula:

[0077] preference(s,a)=H(*|s)+U(s,a)+Q(s,a);

[0078] Among them, H(a|s) represents the entropy of selecting the next action in state S based on the current decision network. The larger the entropy, the more the current network believes that it is possible to mutate in any direction at this anchor point, and it is recommended that the next action be selected from the optional actions of S.

[0079] Use the above formula to determine which mutation step the network currently considers unreliable, and then try other mutation options;

[0080] Expansion phase:

[0081] Expand the next node according to the selected action.

[0082] The technical means adopted by the present invention to solve the technical problem is: a self-learning device for non-adversarial tasks, the device comprising:

[0083] Initialization module, used to initialize the decision neural network for non-adversarial tasks;

[0084] A state acquisition module, used to execute the MCTS state search algorithm based on the current decision neural network, obtain the third-party performance score and record the path information;

[0085] A training optimization module, used for training and optimizing the decision neural network according to the third-party performance score and path information;

[0086] The termination module is used to repeatedly execute steps 2-3 in parallel until the decision neural network meets the trigger termination condition.

[0087] The technical means adopted by the present invention to solve the technical problem is: an electronic device, characterized in that it includes: at least one processor and at least one memory, wherein:

[0088] The memory stores program instructions or codes;

[0089] The program instructions or codes are loaded and executed by the processor, so that the electronic device implements the self-learning method for non-adversarial tasks as described above.

[0090] The technical means adopted by the present invention to solve its technical problem is: a storage medium on which program instructions or codes are stored, characterized in that the program instructions or codes are loaded and executed by a processor to implement the self-learning method for non-adversarial tasks as described above.

[0091] The beneficial effects of the present invention are: the self-learning method for non-adversarial tasks provided by the present application is not limited to the task scenario, can achieve good results for all non-adversarial tasks, and can be adaptively modified according to the task scenario; for different non-adversarial tasks, the shortest solution can be quickly explored from 0 step by step, repeated calculations can be avoided to the greatest extent, and the exploration efficiency can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 A flowchart of a self-learning method for non-adversarial tasks shown in an embodiment of the present invention;

[0093] Figure 2 A structural block diagram of a self-learning device for non-adversarial tasks shown in an embodiment of the present invention;

[0094] Figure 3 The present invention is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0095] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0096] The following will clearly and completely describe the concept, specific structure and technical effects of the present invention in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present invention. In addition, all the connection / connection relationships involved in the patent do not refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features in the invention can be combined interchangeably without conflicting with each other.

[0097] The present application provides a self-learning method for non-adversarial tasks, wherein the general process of non-adversarial tasks includes:

[0098] Step 1: Get the current execution status of the non-adversarial task and save the corresponding action information.

[0099] Step 2: Input the action information of the current execution state into the decision neural network to generate the probability distribution of the next candidate action.

[0100] Step 3: Randomly sample from the probability distribution of the candidate actions and select an action to execute.

[0101] Step 4: Repeat steps 1-3 until the current execution state meets the trigger termination condition, and evaluate the performance score of the decision neural network through a third-party algorithm or neural network model.

[0102] In an exemplary embodiment, the termination condition may be a preset maximum number of iterations. Based on the termination condition trigger, the current state and execution trajectory of the task are transmitted to the evaluator or discriminator to obtain an evaluation of the quality of the decision neural network.

[0103] Based on the traditional MCTS algorithm, this paper develops a self-learning process for the non-adversarial reinforcement learning task described above, which autonomously updates the decision neural network weights during exploration. The specific steps are as follows:

[0104] Step 1: Initialize the decision neural network for the non-adversarial task.

[0105] In a possible implementation, the non-adversarial task needs to meet the following characteristics:

[0106] ① Satisfy all descriptions of the Markov decision process except the reward function;

[0107] The Markov decision process consists of:

[0108] Observe the current state: The decision maker first observes the current state s.

[0109] Select an action: Based on the current state s and the strategy π (a mapping function from state to action), select an action a. The strategy can be deterministic (Deterministic Policy) or stochastic (Stochastic Policy).

[0110] Take action and observe the result: Take the selected action a and observe the transition to the new state s' and the immediate reward R(s,a,s').

[0111] Update state: Update the current state to the new state s'.

[0112] Repeat process: Starting from the new state s', repeat the above process until the terminal state is reached or other termination conditions are met.

[0113] Among them, the state space (State Space): s, represents the set of all possible states. The state is the basic element in the decision-making process and represents the current environment of the decision maker.

[0114] Action Space: a, represents the set of all possible actions. Actions are operations that the decision maker can choose to perform in the current state.

[0115] Transition Probability: P represents the probability of transitioning to the next state after taking an action in the current state. It is denoted as P(s'|s,a), where s' represents the next state, s represents the current state, and a represents the current action.

[0116] ② The action space during task execution is discrete. For each state, the next optional action can only be selected from a limited number of values.

[0117] ③For a given state s and action a, a unique next action s' can always be generated.

[0118] ④ The decision black box that generates the next action has three outputs: the probability distribution of the output, the value of the output state, and the expected number of steps for the output to reach the ideal terminal state.

[0119] ⑤The task can always stop after a finite number of steps.

[0120] ⑥ It is possible to determine whether the termination status is a task success status within a limited time.

[0121] Step 2: Execute the MCTS state search algorithm based on the current decision neural network, obtain the third-party performance score and record the path information.

[0122] In a possible implementation, the MCTS state search algorithm includes a simulation phase, an update phase, a selection phase, and an expansion phase, wherein:

[0123] The simulation stage selects actions through the current decision neural network until the task is completed, simulates a complete task process, and obtains the performance score;

[0124] The update phase updates the information of each state on the success path on the Monte Carlo tree according to the execution result of the task, including:

[0125] According to the execution result, update Found_victory(s)=1 for each state on the successful path;

[0126] Record the total length of the new path as length, and update EL(s) = min(EL(s), length);

[0127] Final calculation: V(s) = Found_victory(s) + 1 / EL(s);

[0128] At the same time, for the parent node of this state, update Q(s previous ,a)=v(s), when s previous When action a tends to state s; N(s previous,a)+=1; where s previous is the state at the previous moment, used to distinguish it from the current state s;

[0129] The path includes a sequence consisting of s0, a0, s1, a1, sT, etc. Found_Victory(s) is a marker for whether a path that passes through state s and finally reaches the success state has been searched; EL(s): the length of the path that passes through s and reaches the terminal state, only the minimum value is recorded.

[0130] The selection phase selects the best child node for traversal on the explored node according to a certain strategy until an unexplored leaf node is reached, including:

[0131] At an explored node, choose the next action that maximizes the following formula:

[0132] preference(s,a)=H(*|s)+U(s,a)+Q(s,a)(1);

[0133] Among them, H(*|s) represents the entropy of selecting the next action in state S based on the current decision neural network. The larger the entropy, the more likely S is to be a key divergence point, and it is recommended that the next action be selected from the actions of S; U(s,a) represents the impact of the number of explorations N(s,a) on the selection of the next action.

[0134] The expansion phase adds new nodes at the selected leaf nodes to represent new possible states. The specific method is consistent with the original MCTS algorithm and is implemented by executing the actions selected in the selection phase.

[0135] Step 3: Train and optimize the decision neural network based on the third-party performance score and path information.

[0136] In a possible implementation, step 3 uses a back propagation algorithm based on a loss function to optimize the weights of the decision neural network, and the optimization objectives include:

[0137] ① Minimize the prediction error between v(s) and EL(s);

[0138] ② Minimize the error between p(s,a) and U(s,a)+Q(s,a), where p(s,a) represents the probability of taking action a in state s, as predicted by the model;

[0139] ③ Maximize the entropy reduction effect of each execution step where Found_victory remains 1;

[0140] ④ Minimize the KL divergence between the new model and the old model on the explored trajectory.

[0141] Among them, the above step 2 executes the MCTS algorithm based on the current decision neural network, which is responsible for exploring the position execution status. Step 3 is to train the decision neural network in parallel based on the data generated in step 2; the exploration record generated in step 2 is used to guide the optimization of the decision neural network in step 3, and the decision neural network optimized in step 3 further guides step 1 to generate better quality data. The two complement each other.

[0142] Step 4: Repeat steps 2-3 in parallel until the decision neural network meets the trigger termination condition.

[0143] In one possible implementation, taking the large language model text generation task as an example,

[0144] Simulate phase:

[0145] If the large language model executes MCTS for the first time, a text is directly generated according to the current parameters until the maximum length is reached or a termination character is generated.

[0146] If the large language model is not executing MCTS for the first time, that is, a character string already exists in the prompt word, a text is generated using the current character string as the prompt word;

[0147] After the text generation task is fully executed, feedback from the discriminator is obtained; the discriminator includes but is not limited to a code compilation discriminator, a task execution discriminator, and a human score.

[0148] Optionally, using a code compilation discriminator to check the syntax of the fine-tuned large language model generation program;

[0149] A task execution discriminator is used to determine whether the code generated by the fine-tuned large language model can successfully complete the task.

[0150] The code compilation discriminator has the following characteristics:

[0151] ①Can accept strings of any length as input;

[0152] ② The output can only be selected from two values, 0 and 1, 1 represents successful compilation, 0 represents unsuccessful compilation;

[0153] ③During the compilation process, you can selectively output the intermediate results of compilation or execution, but this step is not mandatory. Even if these intermediate compiled results are not output, the compilation process can continue and be completed.

[0154] In an exemplary embodiment, the code compilation discriminator may be a python compiler plus an error detector. If the error checker detects that there is an error in the compilation process, it means that the compilation has failed.

[0155] The task execution discriminator has the following characteristics:

[0156] ①Can accept strings of any length as input;

[0157] ② The output can only be selected from two values, 0 and 1. 1 represents that the task can be successfully completed, and 0 represents that the task cannot be successfully completed.

[0158] In an exemplary embodiment, the task execution discriminator may call open source simulation tools such as mujoco and ISAAC sim at the bottom layer to quickly determine whether the termination condition can be successfully simulated.

[0159] Update phase:

[0160] According to the execution result, update Found_victory(s) = 1 for each state on the success path (each state corresponds to the prompt word + generated text);

[0161] Record the total length of the new path as length, and update EL(s) = min(EL(s), length), where length also records the number of generated tokens;

[0162] Calculate V(s) = Found_victory(s) + 1 / EL(s);

[0163] At the same time, for the parent node of this state, update Q(s previous ,a)=v(s), when s previous When action a tends to state s; N(s previous ,a)+=1; where s previous is the state at the previous moment, used to distinguish it from the current state s.

[0164] Selection phase:

[0165] At an explored node, choose the next action that maximizes the following formula:

[0166] preference(s,a)=H(*|s)+U(s,a)+Q(s,a);

[0167] Among them, H(a|s) represents the entropy of selecting the next action in state S based on the current decision network. The larger the entropy, the more likely S is a semantic divergence point, and it is recommended that the next action be selected from the optional actions of S.

[0168] Of course, any combination of the above three items consisting of any non-negative numbers can be used as an exploration tendency function. The above formula is used to determine which token the large language text generation model is most likely to have problems when it wants to generate, and then replace this token with another token and regenerate it.

[0169] Expand:

[0170] Expand the next node according to the selected action.

[0171] In another possible implementation, taking the path planning algorithm as an example, for some path planning algorithms with obstacle avoidance as the goal, the program first generates a series of anchor points with close distances to each other on the map, and then uses the current position of the subject as the initial node, and each anchor point as a node of MCT to execute the MCTS search algorithm.

[0172] If the target range is reached without passing through the forbidden area during the simulation, Found_victory is recorded as 1, otherwise it is 0. Because the distance between each pair of adjacent nodes is similar, the execution step length of the Monte Carlo tree can be used to estimate the travel distance.

[0173] Simulate:

[0174] Generate a path based on the current decision neural network and the current state until termination;

[0175] After a complete execution, the feedback of the discriminator is obtained, where 1 is passed and 0 is failed; the discriminator is used to detect whether a collision occurs during the simulation execution of the route;

[0176] Update phase:

[0177] According to the execution result, update Found_victory(s) = 1 for each state (anchor point coordinates + tree structure) on the successful path;

[0178] Record the total length of the new path as length, and update EL(s) = min(EL(s), length); the length also records the number of generated tokens;

[0179] Calculate V(s) = Found_victory(s) + 1 / EL(s);

[0180] At the same time, for the parent node of this state, update Q(s previous ,a)=v(s), when s previous When action a tends to state s; N(s previous ,a)+=1; where s previous is the state at the previous moment;

[0181] Selection phase:

[0182] At an explored node, choose the next action that maximizes the following formula:

[0183] preference(s,a)=H(*|s)+U(s,a)+Q(s,a);

[0184] Among them, H(a|s) represents the entropy of selecting the next action in state S based on the current decision network. The larger the entropy, the more the current network believes that it is possible to develop in any direction at this anchor point, and it is recommended that the next action be selected from the optional actions of S.

[0185] The above formula is used to determine where the path divergence point is that the current network considers itself to be, and a new path is selected based on the divergence point.

[0186] Expand:

[0187] Expand the next node according to the selected action.

[0188] In another possible implementation, taking the robot evolution algorithm as an example, the evolutionary algorithm is a non-gradient optimization algorithm that simulates Darwin's biological natural selection process. The robot evolutionary algorithm has two main applications: robot morphological evolution and robot intelligence evolution. The robot evolutionary algorithm proposes a new morphology (or intelligence) through mutation, and then evaluates the evolved morphology (or intelligence) through robot tournaments or other automated means. In some robot evolutions, the choice of mutation is very limited (for example, a robot structure evolution can only add, delete, and modify components in a fixed parts group), and does not involve the phenomenon of "reversion" such as mutation back to the original state. The problems faced by this type of evolutionary algorithm can be solved by this patent.

[0189] State: the result of the current evolution, such as the current structure (or intelligence) of the robot;

[0190] Action: mutation, such as adding a component, changing the robot's thinking logic, etc.

[0191] Discriminator: An objective evaluation tool to determine whether the robot can adapt to the current complex task;

[0192] Step length: represents the number of steps taken to obtain the current evolution result. Generally speaking, the smaller the step length, the simpler the structure (over-intelligence) and the less subsequent development work.

[0193] Simulate:

[0194] Directly generate a path based on the current network or current state until termination;

[0195] After a task is completely executed, feedback from the discriminator is obtained, with 1 for passing and 0 for failing. The discriminator is used to score the results of the evolution.

[0196] Update phase:

[0197] According to the execution result, update Found_victory(s) = 1 for each state on the success path (robot morphology or intelligence behind it, depending on the specific task);

[0198] Record the total length of the new path as length, and update EL(s) = min(EL(s), length). The length also records the number of tokens generated;

[0199] Calculate V(s) = Found_victory(s) + 1 / EL(s);

[0200] At the same time, for the parent node of this state, update Q(s previous ,a)=v(s), when s previous When action a tends to state s; N(s previous ,a)+=1; where s previous is the state at the previous moment;

[0201] Selection phase:

[0202] At an explored node, choose the next action that maximizes the following formula:

[0203] preference(s,a)=H(*|s)+U(s,a)+Q(s,a);

[0204] Among them, H(a|s) represents the entropy of selecting the next action in state S based on the current decision network. The larger the entropy, the more the current network believes that it is possible to mutate in any direction at this anchor point, and it is recommended that the next action be selected from the optional actions of S.

[0205] Use the above formula to determine which mutation step the network currently considers unreliable, and then try other mutation options;

[0206] Expand:

[0207] Expand the next node according to the selected action.

[0208] The beneficial effects of the present invention are: the self-learning method for non-adversarial tasks provided is not limited to the task scenario, can achieve good results for all non-adversarial tasks, and can be adaptively modified according to the task scenario; for different non-adversarial tasks, it can quickly explore the shortest solution that can be accepted by the discriminator from 0, avoid repeated calculations to the greatest extent, and improve exploration efficiency.

[0209] It should be understood that, although the various steps in the above-mentioned flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0210] The following is an embodiment of the device of the present application, which can be used to perform the self-learning method for non-adversarial tasks involved in the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the method embodiment of the self-learning method for non-adversarial tasks involved in the present application.

[0211] See also Figure 2 In an embodiment of the present application, a self-learning device 20 for non-adversarial tasks is provided, and the device 20 includes: an initialization module 210, a state acquisition module 220, a training optimization module 230, and a termination module 240.

[0212] Wherein, the initialization module 210 is used to initialize the decision neural network of the non-adversarial task;

[0213] A state acquisition module 220 is used to execute the MCTS state search algorithm based on the current decision neural network, obtain the third-party performance score and record the path information;

[0214] A training optimization module 230, for training and optimizing the decision neural network according to the third-party performance score and path information;

[0215] The termination module 240 is used to repeatedly execute steps 2-3 in parallel until the decision neural network meets the trigger termination condition.

[0216] It should be noted that the self-learning device for non-adversarial tasks provided in the above embodiments only uses the division of the above-mentioned functional modules as an example when performing self-learning for non-adversarial tasks. In practical applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the self-learning device for non-adversarial tasks will be divided into different functional modules to complete all or part of the functions described above. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0217] In addition, the self-learning device for non-adversarial tasks and the method embodiments for the self-learning method for non-adversarial tasks provided in the above embodiments belong to the same concept, and the specific manner in which each module performs operations has been described in detail in the method embodiments and will not be repeated here.

[0218] See also Figure 3 , an electronic device 4000 is provided in an embodiment of the present application.

[0219] exist Figure 3 In the embodiment, the data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0220] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0221] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 400, but is not limited to this.

[0222] The memory 4003 stores program instructions or codes, and the processor 4001 can read the program instructions or codes stored in the memory 4003 through the communication bus 4002 .

[0223] When the program instructions or codes are executed by the processor 4001, the self-learning methods for non-adversarial tasks in the above-mentioned embodiments are implemented.

[0224] In addition, an embodiment of the present application provides a storage medium on which program instructions or codes are stored, and the program instructions or codes are loaded and executed by a processor to implement a self-learning method for non-adversarial tasks as described above.

[0225] In an embodiment of the present application, a computer program product is provided, which includes program instructions or codes, and the program instructions or codes are stored in a storage medium. A processor of an electronic device reads the program instructions or codes from the storage medium, loads and executes the program instructions or codes, so that the electronic device implements the self-learning method for non-adversarial tasks as described above.

[0226] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A self-learning method for non-adversarial tasks, characterized in that: The method comprises: Step 1: Initialize the decision neural network for non-adversarial tasks; Step 2: Execute the MCTS state search algorithm based on the current decision neural network, obtain the third-party performance score and record the path information; Step 3: training and optimizing the decision neural network according to the third-party performance score and path information; Step 4: Repeat steps 2-3 in parallel until the decision neural network meets the trigger termination condition.

2. The self-learning method for non-adversarial tasks according to claim 1, characterized in that: The non-adversarial task meets the following characteristics: ① Satisfy all descriptions of the Markov decision process except the reward function; ② The action space during task execution is discrete. For each state, the next optional action can only be selected from a limited number of values; ③For a given state s and action a, a unique next action s' can always be generated; ④ The decision black box that generates the next action has three outputs: the probability distribution of the output, the value of the output state, and the expected number of steps for the output to reach the ideal terminal state; ⑤The task can always stop after a finite number of steps; ⑥ It is possible to determine whether the termination status is a task success status within a limited time.

3. The self-learning method for non-adversarial tasks according to claim 1, characterized in that: The MCTS state search algorithm includes a simulation phase, an update phase, a selection phase, and an expansion phase, wherein: The simulation stage selects actions through the current decision neural network until the task is completed, simulates a complete task process, and obtains the performance score; The update phase updates the information of each state on the success path on the Monte Carlo tree according to the execution result of the task, including: According to the execution result, update Found_victory(s)=1 for each state on the successful path; Record the total length of the new path as length, and update EL(s) = min(EL(s), length); Final calculation: V(s) = Found_victory(s) + 1 / EL(s); At the same time, for the parent node of this state, update Q(s previous ,a)=v(s), when s previous When action a tends to state s; N(s previous ,a)+=1; where s previous is the state at the previous moment, used to distinguish it from the current state s; The path includes a sequence consisting of s0, a0, s1, a1, sT, etc. Found_Victory(s) is a marker for whether a path that passes through state s and finally reaches the success state has been searched; EL(s): the length of the path that passes through s and reaches the terminal state, only the minimum value is recorded; The selection phase selects the best child node for traversal on the explored node according to a certain strategy until an unexplored leaf node is reached, including: At an explored node, choose the next action that maximizes the following formula: preference(s,a)=H(*|s)+U(s,a)+Q(s,a)(1); Among them, H(*|s) represents the entropy of selecting the next action in state S based on the current decision neural network. The larger the entropy, the more likely S is to be a key divergence point, and it is recommended that the next action be selected from the actions of S; U(s,a) represents the impact of the number of explorations N(s,a) on the selection of the next action; The expansion phase adds new nodes at the selected leaf nodes to represent new possible states; the specific method is consistent with the original MCTS algorithm and is achieved by executing the actions selected in the selection phase.

4. The self-learning method for non-adversarial tasks according to claim 3, characterized in that: The step 3 comprises: The back propagation algorithm based on the loss function optimizes the weights of the decision neural network, and the optimization objectives include: Minimize the prediction error between v(s) and EL(s); Minimize the error between p(s,a) and U(s,a)+Q(s,a), where p(s,a) represents the probability of taking action a in state s, as predicted by the model; Maximize the entropy reduction effect of each execution step where Found_victory remains 1; Minimize the KL divergence between the new model and the old model on the explored trajectory.

5. The self-learning method for non-adversarial tasks according to claim 3, characterized in that: The non-adversarial task includes a large language model text generation task, where: Simulation phase: If the large language model executes MCTS for the first time, a text is directly generated according to the current parameters until the maximum length is reached or a termination character is generated; If the large language model is not executing MCTS for the first time, that is, a character string already exists in the prompt word, a text is generated using the current character string as the prompt word; After the text generation task is fully executed, feedback from a discriminator is obtained; the discriminator includes but is not limited to a code compilation discriminator, a task execution discriminator, and a human score; Update phase: According to the execution result of the large language model, update Found_victory(s)=1 for each state on the success path (each state corresponds to the prompt word + generated text); Record the total length of the new path as length, and update EL(s) = min(EL(s), length). The length also records the number of generated tokens. Calculate V(s) = Found_victory(s) + 1 / EL(s); At the same time, for the parent node of this state, update Q(s previous ,a)=v(s), when s previous When action a tends to state s; N(s previous ,a)+=1; where s previous is the state at the previous moment; Selection stage: At an explored node, choose the next action that maximizes the following formula: preference(s,a)=H(*|s)+U(s,a)+Q(s,a); Among them, H(a|s) represents the entropy of selecting the next action in state S based on the current decision network. The larger the entropy, the more likely S is a semantic divergence point, and it is recommended that the next action be selected from the optional actions of S. Expansion phase: Expand the next node according to the selected action.

6. The self-learning method for non-adversarial tasks according to claim 3, characterized in that: The non-adversarial task includes a path planning algorithm, where: Simulation phase: Generate a path based on the current decision neural network and the current state until termination; After a complete execution, the feedback of the discriminator is obtained, where 1 is passed and 0 is failed; the discriminator is used to detect whether a collision occurs during the simulation execution of the route; Update phase: According to the execution result, update Found_victory(s) = 1 for each state (anchor point coordinates + tree structure) on the successful path; Record the total length of the new path as length, and update EL(s) = min(EL(s), length); the length also records the number of generated tokens; Calculate V(s) = Found_victory(s) + 1 / EL(s); At the same time, for the parent node of this state, update Q(s previous ,a)=v(s), when s previous When action a tends to state s; N(s previous ,a)+=1; where s previous is the state at the previous moment; Selection stage: At an explored node, choose the next action that maximizes the following formula: preference(s,a)=H(*|s)+U(s,a)+Q(s,a); Among them, H(a|s) represents the entropy of selecting the next action in state S based on the current decision network. The larger the entropy, the more the current network believes that it is possible to develop in any direction at this anchor point, and it is recommended that the next action be selected from the optional actions of S. Use the above formula to determine where the current network thinks the path divergence point is, and select a new path based on the divergence point; Expansion phase: Expand the next node according to the selected action.

7. The self-learning method for non-adversarial tasks according to claim 3, characterized in that: The non-adversarial task includes a robotic evolution algorithm, wherein: Simulation phase: Directly generate a path based on the current network or current state until termination; After a task is fully executed, feedback from the discriminator is obtained, where 1 is passed and 0 is failed; the discriminator is used to score the result of the evolution; Update phase: According to the execution result, update Found_victory(s) = 1 for each state on the success path (robot morphology or intelligence behind it, depending on the specific task); Record the total length of the new path as length, and update EL(s) = min(EL(s), length); the length also records the number of generated tokens; Calculate V(s) = Found_victory(s) + 1 / EL(s); At the same time, for the parent node of this state, update Q(s previous ,a)=v(s), when s previous When action a tends to state s; N(s previous ,a)+=1; where s previous is the state at the previous moment; Selection stage: At an explored node, choose the next action that maximizes the following formula: preference(s,a)=H(*|s)+U(s,a)+Q(s,a); Among them, H(a|s) represents the entropy of selecting the next action in state S based on the current decision network. The larger the entropy, the more the current network believes that it is possible to mutate in any direction at this anchor point, and it is recommended that the next action be selected from the optional actions of S. Use the above formula to determine which mutation step the network currently considers unreliable, and then try other mutation options; Expansion phase: Expand the next node according to the selected action.

8. A self-learning device for non-adversarial tasks, characterized in that: The device comprises: Initialization module, used to initialize the decision neural network for non-adversarial tasks; A state acquisition module, used to execute the MCTS state search algorithm based on the current decision neural network, obtain the third-party performance score and record the path information; A training optimization module, used for training and optimizing the decision neural network according to the third-party performance score and path information; The termination module is used to repeatedly execute steps 2-3 in parallel until the decision neural network meets the trigger termination condition.

9. An electronic device, characterized in that: include: at least one processor, at least one memory, wherein: The memory stores program instructions or codes; The program instructions or codes are loaded and executed by the processor, so that the electronic device implements the self-learning method for non-adversarial tasks as described in any one of claims 1 to 7.

10. A storage medium having program instructions or codes stored thereon, characterized in that: The program instructions or codes are loaded and executed by a processor to implement the self-learning method for non-adversarial tasks as described in any one of claims 1 to 7.

Images