A self-learning method, device, equipment and medium for non-adversarial tasks

Abstract

The application relates to the technical field of reinforcement learning, and particularly relates to a self-learning method, device and equipment for a non-adversarial task and a medium, the method comprising the following steps: step 1: initializing a decision neural network of a non-adversarial task; step 2: performing an MCTS state search algorithm based on the current decision neural network, obtaining a third-party performance score and recording path information; step 3: training and optimizing the decision neural network according to the third-party performance score and the path information; and step 4: repeatedly and parallelly performing steps 2-3 until the decision neural network meets a triggering termination condition. The self-learning method for the non-adversarial task provided in the application is not limited to a task scene, can achieve good results for all non-adversarial tasks, can be modified according to the task scene, can quickly explore the shortest solution from 0 for different non-adversarial tasks, can avoid repeated calculation to the maximum extent, and can improve exploration efficiency.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning technology, and specifically to a self-learning method, apparatus, device, and medium for non-adversarial tasks. Background Technology

[0002] In the fields of artificial intelligence and machine learning, Monte Carlo Tree Search (MCTS), as an efficient and flexible decision-making algorithm, has demonstrated its superior capabilities in many complex problems, especially in navigation and decision optimization. By simulating possible future decision paths and combining statistical evaluation with pruning strategies, MCTS effectively balances exploration and exploitation, thereby finding optimal or near-optimal solutions in uncertain environments. This characteristic makes MCTS an ideal choice for handling complex decision problems with a large number of states and uncertainties.

[0003] In recent years, with the rapid development of deep learning technology, strategies combining deep learning with Match-Based Strategy Test (MCTS) have demonstrated unprecedented potential. This fusion not only enhances MCTS's policy evaluation capabilities in complex environments but also further improves the algorithm's adaptability and generalization ability through the automatic learning of feature representations by deep neural networks. Algorithms such as AlphaZero, for example, use deep learning networks to predict the value of the board state and possible next moves, providing MCTS with high-quality prior knowledge, significantly accelerating the search process and improving decision quality. This tight integration of deep learning and MCTS has not only driven breakthroughs in traditional adversarial tasks such as Go but has also inspired exploration of its applications in more adversarial tasks and self-learning systems.

[0004] Therefore, deeply integrating MCTS with deep learning technology to build an intelligent decision-making system with efficient self-learning capabilities has become one of the important directions of current artificial intelligence research. This system can automatically learn and optimize decision-making strategies from data, achieving efficient and accurate decision-making in complex and ever-changing environments with little or no human intervention.

[0005] However, the existing combination method is only applicable to application scenarios that exhibit multi-agent adversarial characteristics, and there are no self-learning optimization cases for non-adversarial tasks, which greatly limits the practical application of this algorithm. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention provides a self-learning method for non-adversarial tasks, aiming to expand the application scenarios of this type of algorithm for non-adversarial tasks and improve its self-learning ability and algorithm efficiency for such scenarios, so as to meet more complex and ever-changing practical application needs.

[0007] The technical means adopted by this invention to solve its technical problem is: a self-learning method for non-adversarial tasks, characterized in that the method includes: Step 1: initializing a decision neural network for the non-adversarial task; Step 2: executing the MCTS state search algorithm based on the current decision neural network to obtain a third-party performance score and record path information; Step 3: training and optimizing the decision neural network according to the third-party performance score and path information; Step 4: repeating steps 2-3 in parallel until the decision neural network meets the trigger termination condition.

[0008] The non-adversarial task described in the above technical solution satisfies the following characteristics:

[0009] ① It satisfies all descriptions in the Markov decision process except for the reward function;

[0010] ②The action space is discrete during task execution. For each state, the next action can only be selected from a finite number of values.

[0011] ③ For a given state s and action a, a unique next action s' can always be generated;

[0012] ④ The decision black box that generates the next action has three outputs: the probability distribution of the output, the value of the output state, and the expected number of steps to reach the ideal termination state.

[0013] ⑤ The task can always stop after a finite number of steps;

[0014] ⑥ It can determine whether the termination status is a successful status within a limited time.

[0015] The MCTS state search algorithm described in the above technical solution includes a simulation phase, an update phase, a selection phase, and an expansion phase, wherein...

[0016] The simulation phase selects actions through the current decision neural network until the task ends, simulating a complete task process and obtaining a performance score;

[0017] The update phase updates the information of each state on the successful path in the Monte Carlo tree according to the task execution result, including:

[0018] Based on the execution results, update Found_victory(s) = 1 for each state on the successful path;

[0019] Record the total length of the new path as length, and update EL(s) = min(EL(s), length);

[0020] Final calculation: V(s) = Found_victory(s) + 1 / EL(s);

[0021] At the same time, for the parent node of this state, update Q(s) previous ,a)=v(s), when s previous When action a tends towards state s; N(s) previous ,a)+=1; where, s previous This represents the state at the previous moment, used to distinguish it from the current state s;

[0022] The path includes a sequence of s0, a0, s1, a1, sT, etc. Found_Victory(s) is a marker indicating whether a path that passes through state s and finally reaches the success state has been found; EL(s) is the length of the path that passes through s and reaches the termination state, and only the minimum value is recorded.

[0023] The selection phase involves traversing the already explored nodes, selecting the optimal child node according to a certain strategy, until an unexplored leaf node is reached. This includes:

[0024] On the explored nodes, select the next action that maximizes the following formula:

[0025] preference(s,a)=H(*|s)+U(s,a)+Q(s,a)(1);

[0026] Where H(*|s) represents the entropy of selecting the next action in state S based on the current decision neural network. The larger the entropy, the more likely S is to be a critical divergence point, and it is recommended that the next action be selected from the actions in S; U(s,a) represents the influence of the number of explorations N(s,a) on the selection of the next action.

[0027] The expansion phase adds a new node at the selected leaf node to represent a new possible state; the specific method is the same as the original MCTS algorithm, which is achieved by executing the action selected in the selection phase.

[0028] Step 3 in the above technical solution includes:

[0029] The backpropagation algorithm based on the loss function optimizes the weights of the decision neural network, and the optimization objective includes:

[0030] Minimize the prediction error between v(s) and EL(s);

[0031] Minimize the error between p(s,a) and U(s,a)+Q(s,a), where p(s,a) represents the probability of choosing action a in state s, which is predicted by the model;

[0032] Maximize the entropy reduction effect of each execution step where Found_victory remains at 1;

[0033] Minimize the KL divergence between the new and old models on the explored trajectories.

[0034] The non-adversarial tasks described in the above technical solutions include large language model text generation tasks, wherein,

[0035] Simulation phase:

[0036] If the large language model executes MCTS for the first time, it will directly generate a text segment based on the current parameters until the maximum length is reached or a terminating character is generated.

[0037] If the large language model is not executing MCTS for the first time, that is, if the string already exists in the prompt word, then a piece of text is generated using the current string as the prompt word;

[0038] After the text generation task is completed, feedback from the discriminator is obtained; the discriminator includes, but is not limited to, a code compilation discriminator, a task execution discriminator, and human scoring.

[0039] Update phase:

[0040] Based on the execution results of the large language model, update Found_victory(s) = 1 for each state on the successful path (each state corresponds to the prompt word + the generated text);

[0041] Record the total length of the new path at this time as length, and update EL(s) = min(EL(s), length), where length also records the number of tokens generated;

[0042] Calculate V(s) = Found_victory(s) + 1 / EL(s);

[0043] At the same time, for the parent node of this state, update Q(s) previous ,a)=v(s), when s previous When action a tends towards state s; N(s) previous ,a)+=1; where, s previous This refers to the state at the previous moment;

[0044] Selection phase:

[0045] On the explored nodes, select the next action that maximizes the following formula:

[0046] preference(s,a)=H(*|s)+U(s,a)+Q(s,a);

[0047] Here, H(a|s) represents the entropy of selecting the next action in state S based on the current decision network. The larger the entropy, the more likely S is to be a semantic divergence point, and it is recommended that the next action be selected from the available actions in S.

[0048] Expansion phase:

[0049] Expand the next node based on the selected action.

[0050] The non-adversarial task described in the above technical solution includes a path planning algorithm, wherein,

[0051] Simulation phase:

[0052] A path is generated based on the current decision neural network and the current state, until termination;

[0053] After a complete execution, feedback is obtained from the discriminator: 1 for success and 0 for failure. The discriminator is used to detect whether a collision occurs during the simulated execution of the route.

[0054] Update phase:

[0055] Based on the execution results, update Found_victory(s) = 1 for each state (anchor coordinates + tree structure) on the successful path;

[0056] Record the total length of the new path as length, and update EL(s) = min(EL(s), length); the length also records the number of tokens generated.

[0057] Calculate V(s) = Found_victory(s) + 1 / EL(s);

[0058] At the same time, for the parent node of this state, update Q(s) previous ,a)=v(s), when s previous When action a tends towards state s; N(s) previous ,a)+=1; where, s previous This refers to the state at the previous moment;

[0059] Selection phase:

[0060] On the explored nodes, select the next action that maximizes the following formula:

[0061] preference(s,a)=H(*|s)+U(s,a)+Q(s,a);

[0062] Where H(a|s) represents the entropy of selecting the next action in state S based on the current decision network. The larger the entropy, the more the current network believes that it is possible to develop in any direction at this anchor point, and suggests that the next action be selected from the available actions in S.

[0063] Use the above formula to determine where the current network perceives the path divergence point, and select a new path based on the divergence point;

[0064] Expansion phase:

[0065] Expand the next node based on the selected action.

[0066] The non-adversarial task described in the above technical solution includes a robot evolutionary algorithm, wherein,

[0067] Simulation phase:

[0068] Generate a path directly based on the current network or current state until termination;

[0069] After a complete task is executed, feedback is obtained from the discriminator: 1 for success and 0 for failure; the discriminator is used to score the evolution results.

[0070] Update phase:

[0071] Based on the execution results, update Found_victory(s) = 1 for each state (robot form or underlying intelligence, depending on the specific task) on the successful path;

[0072] Record the total length of the new path as length, and update EL(s) = min(EL(s), length); the length also records the number of tokens generated.

[0073] Calculate V(s) = Found_victory(s) + 1 / EL(s);

[0074] At the same time, for the parent node of this state, update Q(s) previous ,a)=v(s), when s previous When action a tends towards state s; N(s) previous ,a)+=1; where, s previous This refers to the state at the previous moment;

[0075] Selection phase:

[0076] On the explored nodes, select the next action that maximizes the following formula:

[0077] preference(s,a)=H(*|s)+U(s,a)+Q(s,a);

[0078] Where H(a|s) represents the entropy of selecting the next action in state S based on the current decision network. The larger the entropy, the more the current network believes that it is possible to mutate in any direction at this anchor point, and suggests that the next action be selected from the available actions in S.

[0079] Use the above formula to determine which mutation the current network considers unreliable, and then try other mutation options;

[0080] Expansion phase:

[0081] Expand the next node based on the selected action.

[0082] The technical means adopted by this invention to solve its technical problem is: a self-learning device for non-adversarial tasks, the device comprising:

[0083] An initialization module is used to initialize the decision neural network for non-adversarial tasks;

[0084] The state acquisition module is used to execute the MCTS state search algorithm based on the current decision neural network, obtain third-party performance scores, and record path information;

[0085] The training optimization module is used to train and optimize the decision neural network based on the third-party performance score and path information.

[0086] The termination module is used to repeatedly execute steps 2-3 in parallel until the decision neural network meets the termination trigger condition.

[0087] The technical means adopted by this invention to solve its technical problem is: an electronic device, characterized in that it comprises: at least one processor and at least one memory, wherein,

[0088] The memory stores program instructions or code;

[0089] The program instructions or code are loaded and executed by the processor, enabling the electronic device to implement the self-learning method for non-adversarial tasks as described above.

[0090] The technical means adopted by the present invention to solve its technical problem is: a storage medium storing program instructions or code thereon, characterized in that the program instructions or code are loaded and executed by a processor to realize the self-learning method for non-adversarial tasks as described above.

[0091] The beneficial effects of this invention are: the self-learning method for non-adversarial tasks provided by this application is not limited by the task scenario, and can achieve good results for all non-adversarial tasks. It can be adapted and modified according to the task scenario. For different non-adversarial tasks, it can quickly explore the shortest solution from 0 step by step, avoid repeated calculations to the greatest extent, and improve exploration efficiency. Attached Figure Description

[0092] Figure 1 This is a flowchart illustrating a self-learning method for non-adversarial tasks according to an embodiment of the present invention;

[0093] Figure 2 This is a structural block diagram of a self-learning device for non-adversarial tasks, as shown in an embodiment of the present invention.

[0094] Figure 3 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0095] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0096] The following will clearly and completely describe the concept, specific structure, and technical effects of the present invention in conjunction with embodiments and accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are all within the scope of protection of the present invention. Furthermore, all connections / linkages involved in the patent do not simply refer to direct contact between components, but rather to the ability to form a better connection structure by adding or reducing connecting accessories according to specific implementation conditions. The various technical features in this invention can be combined interactively without contradicting each other.

[0097] This application provides a self-learning method for non-adversarial tasks, wherein the general process of a non-adversarial task includes:

[0098] Step 1: Obtain the current execution status of the non-confrontational task and save the corresponding action information.

[0099] Step 2: Input the action information of the current execution state into the decision neural network to generate the probability distribution of the next candidate action.

[0100] Step 3: Randomly sample from the probability distribution of the candidate actions and select one action to execute.

[0101] Step 4: Repeat steps 1-3 until the current execution state meets the termination trigger condition, and evaluate the performance score of the decision neural network through a third-party algorithm or neural network model.

[0102] In an exemplary embodiment, the termination condition may be a pre-set maximum number of iterations. Based on the termination condition, the current state of the task and the execution trajectory are transmitted to the evaluator or discriminator to obtain an evaluation of the merits of the decision neural network.

[0103] This invention, based on the traditional MCTS algorithm, develops a self-learning process for the non-adversarial reinforcement learning task described above, autonomously updating the weights of the decision neural network during exploration. The specific steps are as follows:

[0104] Step 1: Initialize the decision neural network for the non-adversarial task.

[0105] In one possible implementation, the non-adversarial task needs to satisfy the following characteristics:

[0106] ① It satisfies all descriptions in the Markov decision process except for the reward function;

[0107] The Markov decision-making process includes:

[0108] Observe the current state: The decision-maker first observes the current state s.

[0109] Choose an action: Based on the current state s and the policy π (a mapping function from state to action), choose an action a. The policy can be deterministic or stochastic.

[0110] Perform the action and observe the result: Perform the selected action a and observe the transition to the new state s' and the immediate reward R(s,a,s').

[0111] Update state: Update the current state to the new state s'.

[0112] Repeat the process: Starting from the new state s', repeat the above process until the termination state is reached or other termination conditions are met.

[0113] Here, the state space (s) represents the set of all possible states. States are the fundamental elements of the decision-making process, representing the current environmental situation of the decision-maker.

[0114] Action Space: 'a' represents the set of all possible actions. An action is an operation that a decision-maker can choose to perform in the current state.

[0115] Transition Probability: P represents the probability of transitioning to the next state after taking a certain action in the current state. It is denoted as P(s'|s,a), where s' represents the next state, s represents the current state, and a represents the current action.

[0116] ②The action space is discrete during task execution. For each state, the next action can only be selected from a limited set of values.

[0117] ③ For a given state s and action a, a unique next action s' can always be generated.

[0118] ④ The decision black box that generates the next action has three outputs: the probability distribution of the output, the value of the output state, and the expected number of steps to reach the ideal termination state.

[0119] ⑤ The task can always stop after a finite number of steps.

[0120] ⑥ It can determine whether the termination status is a successful status within a limited time.

[0121] Step 2: Execute the MCTS state search algorithm based on the decision neural network described above to obtain a third-party performance score and record path information.

[0122] In one possible implementation, the MCTS state search algorithm includes a simulation phase, an update phase, a selection phase, and an expansion phase, wherein...

[0123] The simulation phase selects actions through the current decision neural network until the task ends, simulating a complete task process and obtaining a performance score;

[0124] The update phase updates the information of each state on the successful path in the Monte Carlo tree according to the task execution result, including:

[0125] Based on the execution results, update Found_victory(s) = 1 for each state on the successful path;

[0126] Record the total length of the new path as length, and update EL(s) = min(EL(s), length);

[0127] Final calculation: V(s) = Found_victory(s) + 1 / EL(s);

[0128] At the same time, for the parent node of this state, update Q(s) previous ,a)=v(s), when s previous When action a tends towards state s; N(s) previous,a)+=1; where, s previous This represents the state at the previous moment, used to distinguish it from the current state s;

[0129] The path includes a sequence of s0, a0, s1, a1, sT, etc. Found_Victory(s) is a marker indicating whether a path that passes through state s and finally reaches the success state has been found; EL(s) is the length of the path that passes through s and reaches the termination state, and only the minimum value is recorded.

[0130] The selection phase involves traversing the already explored nodes, selecting the optimal child node according to a certain strategy, until an unexplored leaf node is reached. This includes:

[0131] On the explored nodes, select the next action that maximizes the following formula:

[0132] preference(s,a)=H(*|s)+U(s,a)+Q(s,a)(1);

[0133] Where H(*|s) represents the entropy of selecting the next action in state S based on the current decision neural network. The larger the entropy, the more likely S is to be a critical divergence point, and it is recommended that the next action be selected from the actions in S. U(s,a) represents the influence of the number of explorations N(s,a) on the selection of the next action.

[0134] The expansion phase adds a new node at the selected leaf node, representing a new possible state. The specific method is consistent with the original MCTS algorithm, implemented by executing the action selected in the selection phase.

[0135] Step 3: Train and optimize the decision neural network based on the third-party performance score and path information.

[0136] In one possible implementation, step 3 employs a backpropagation algorithm based on a loss function to optimize the weights of the decision neural network, wherein the optimization objective includes:

[0137] ① Minimize the prediction error between v(s) and EL(s);

[0138] ② Minimize the error between p(s,a) and U(s,a)+Q(s,a), where p(s,a) represents the probability of choosing action a in state s, which is predicted by the model;

[0139] ③ Maximize the entropy reduction effect of each execution step where Found_victory remains at 1;

[0140] ④ Minimize the KL divergence between the new model and the old model on the explored trajectory.

[0141] In this process, step 2 executes the MCTS algorithm based on the current decision neural network to explore the position execution state, while step 3 trains the decision neural network in parallel based on the data generated in step 2. The exploration records generated in step 2 are used to guide the optimization of the decision neural network in step 3, and the optimized decision neural network in step 3 further guides step 1 to generate higher quality data. The two complement each other.

[0142] Step 4: Repeat steps 2-3 in parallel until the decision neural network meets the termination trigger condition.

[0143] One possible implementation, taking the large language model text generation task as an example.

[0144] Simulation phase:

[0145] If the large language model executes MCTS for the first time, it will directly generate a text segment based on the current parameters until the maximum length is reached or a terminating character is generated.

[0146] If the large language model is not executing MCTS for the first time, that is, if the string already exists in the prompt word, then a piece of text is generated using the current string as the prompt word;

[0147] After the text generation task is completed, feedback from the discriminator is obtained; the discriminator includes, but is not limited to, a code compilation discriminator, a task execution discriminator, and human scoring.

[0148] Optionally, a code compilation discriminator can be used to check the syntax of the fine-tuned large language model generator.

[0149] The task execution discriminator is used to determine whether the code generated by the fine-tuned large language model can successfully complete the task.

[0150] The code compilation discriminator has the following characteristics:

[0151] ① It can accept strings of any length as input;

[0152] ② The output can only be selected from two values, 0 and 1. 1 means that the compilation was successful and 0 means that the compilation was unsuccessful.

[0153] ③ During the compilation process, intermediate compilation or execution results can be selectively output, but this step is not mandatory. Even if these intermediate compiled results are not output, the compilation process can continue and complete.

[0154] In one exemplary embodiment, the code compilation discriminator can be a Python compiler plus an error detector. If the error checker detects an error in the compilation process, it indicates that the compilation has failed.

[0155] The task execution discriminator has the following characteristics:

[0156] ① It can accept strings of any length as input;

[0157] ② The output can only be selected from two values, 0 and 1. 1 means that the task can be successfully completed, and 0 means that the task cannot be successfully completed.

[0158] In one exemplary embodiment, the task execution discriminator can call open-source simulation tools such as mujoco and ISAAC sim at the underlying level to quickly determine whether the termination condition can be successfully simulated.

[0159] Update phase:

[0160] Based on the execution results, update Found_victory(s) = 1 for each state on the successful path (each state corresponds to the prompt word + the generated text);

[0161] Record the total length of the new path as length, and update EL(s) = min(EL(s), length). Here, length also records the number of tokens generated.

[0162] Calculate V(s) = Found_victory(s) + 1 / EL(s);

[0163] At the same time, for the parent node of this state, update Q(s) previous ,a)=v(s), when s previous When action a tends towards state s; N(s) previous ,a)+=1; where, s previous This represents the state at the previous moment, used to distinguish it from the current state s.

[0164] Selection phase:

[0165] On the explored nodes, select the next action that maximizes the following formula:

[0166] preference(s,a)=H(*|s)+U(s,a)+Q(s,a);

[0167] Here, H(a|s) represents the entropy of selecting the next action in state S based on the current decision network. The larger the entropy, the more likely S is to be a semantic divergence point, and it is recommended that the next action be selected from the available actions in S.

[0168] Of course, any combination of the above three terms with any non-negative number can be used as an exploration tendency function. By using the above formula, we can determine which token the large language text generation model is most likely to encounter problems when it wants to generate it, and then replace this token with another token and regenerate it.

[0169] Expand phase:

[0170] Expand the next node based on the selected action.

[0171] In another possible implementation, taking path planning algorithms as an example, for some path planning algorithms with obstacle avoidance as the goal, the program first generates a series of anchor points that are close to each other on the map, and then uses the current position of the subject as the initial node, with each anchor point as a node of MCT, and executes the MCTS search algorithm.

[0172] If the target area is reached without traversing restricted areas during the simulation, Found_victory is recorded as 1; otherwise, it is 0. Because the distance between each pair of adjacent nodes is approximately equal, the execution step size of the Monte Carlo tree can be used to estimate the travel distance.

[0173] Simulation phase:

[0174] A path is generated based on the current decision neural network and the current state, until termination;

[0175] After a complete execution, feedback is obtained from the discriminator: 1 for success and 0 for failure. The discriminator is used to detect whether a collision occurs during the simulated execution of the route.

[0176] Update phase:

[0177] Based on the execution results, update Found_victory(s) = 1 for each state (anchor coordinates + tree structure) on the successful path;

[0178] Record the total length of the new path as length, and update EL(s) = min(EL(s), length); the length also records the number of tokens generated.

[0179] Calculate V(s) = Found_victory(s) + 1 / EL(s);

[0180] At the same time, for the parent node of this state, update Q(s) previous ,a)=v(s), when s previous When action a tends towards state s; N(s) previous ,a)+=1; where, s previous This refers to the state at the previous moment;

[0181] Selection phase:

[0182] On the explored nodes, select the next action that maximizes the following formula:

[0183] preference(s,a)=H(*|s)+U(s,a)+Q(s,a);

[0184] Where H(a|s) represents the entropy of selecting the next action in state S based on the current decision network. The larger the entropy, the more the current network believes that it is possible to develop in any direction at this anchor point, and suggests that the next action be selected from the available actions in S.

[0185] Use the above formula to determine where the current network perceives the path divergence point, and select a new path based on the divergence point.

[0186] Expand phase:

[0187] Expand the next node based on the selected action.

[0188] In another possible implementation, taking robot evolution algorithms as an example, evolutionary algorithms are non-gradient optimization algorithms that simulate Darwin's biological natural selection process. Robot evolution algorithms have two main applications: robot morphological evolution and robot intelligence evolution. Robot evolution algorithms propose new forms (or intelligences) through mutation, and then evaluate the evolved forms (or intelligences) through robot competitions or other automated means. In some robot evolutions, the choice of mutations is very limited (for example, the evolution of a robot structure can only perform addition, deletion, and modification operations on components within a fixed set of parts), and it does not involve the phenomenon of "reversion" where mutations return to the original state. The problems faced by such evolutionary algorithms can be solved by this patent.

[0189] State: The current evolutionary result, such as the robot's current structure (or intelligence);

[0190] Actions: Variations, such as adding a component or changing the robot's thinking logic;

[0191] Discriminator: An objective evaluation tool to determine whether a robot can adapt to a complex task.

[0192] Step size: This represents how many steps it took to arrive at the current evolutionary result. Generally speaking, the smaller the step size, the simpler the structure (over-intelligence), and the less work is required for further development.

[0193] Simulation phase:

[0194] Generate a path directly based on the current network or current state until termination;

[0195] After a complete task is executed, feedback is obtained from the discriminator: 1 for passing and 0 for failing; the discriminator is used to score the evolution results.

[0196] Update phase:

[0197] Based on the execution results, update Found_victory(s) = 1 for each state (robot form or underlying intelligence, depending on the specific task) on the successful path;

[0198] Record the total length of the new path as `length`, and update EL(s) = min(EL(s), length). The `length` also records the number of tokens generated.

[0199] Calculate V(s) = Found_victory(s) + 1 / EL(s);

[0200] At the same time, for the parent node of this state, update Q(s) previous ,a)=v(s), when s previous When action a tends towards state s; N(s) previous ,a)+=1; where, s previous This refers to the state at the previous moment;

[0201] Selection phase:

[0202] On the explored nodes, select the next action that maximizes the following formula:

[0203] preference(s,a)=H(*|s)+U(s,a)+Q(s,a);

[0204] Where H(a|s) represents the entropy of selecting the next action in state S based on the current decision network. The larger the entropy, the more the current network believes that it is possible to mutate in any direction at this anchor point, and suggests that the next action be selected from the available actions in S.

[0205] Use the above formula to determine which mutation the current network considers unreliable, and then try other mutation options;

[0206] Expand phase:

[0207] Expand the next node based on the selected action.

[0208] The beneficial effects of this invention are: the self-learning method for non-adversarial tasks provided is not limited by the task scenario, and can achieve good results for all non-adversarial tasks. It can be adapted and modified according to the task scenario. For different non-adversarial tasks, it can quickly explore from 0 to find the shortest solution that can be received by the discriminator, avoiding repeated calculations to the greatest extent and improving exploration efficiency.

[0209] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0210] The following are embodiments of the apparatus described in this application, which can be used to execute the self-learning method for non-adversarial tasks involved in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of the self-learning method for non-adversarial tasks involved in this application.

[0211] Please see Figure 2 This application provides a self-learning device 20 for non-adversarial tasks. The device 20 includes: an initialization module 210, a state acquisition module 220, a training optimization module 230, and a termination module 240.

[0212] Among them, the initialization module 210 is used to initialize the decision neural network for non-adversarial tasks;

[0213] The state acquisition module 220 is used to execute the MCTS state search algorithm based on the current decision neural network to obtain a third-party performance score and record path information;

[0214] The training optimization module 230 is used to train and optimize the decision neural network based on the third-party performance score and path information.

[0215] Termination module 240 is used to repeatedly execute steps 2-3 in parallel until the decision neural network meets the termination trigger condition.

[0216] It should be noted that the self-learning device for non-adversarial tasks provided in the above embodiments is only illustrated by the division of the above functional modules when performing self-learning for non-adversarial tasks. In practical applications, the above functions can be assigned to different functional modules as needed. That is, the internal structure of the self-learning device for non-adversarial tasks will be divided into different functional modules to complete all or part of the functions described above. The above modules can be embedded in hardware or independent of the processor in the computer device, or they can be stored in software in the memory of the computer device so that the processor can call and execute the operations corresponding to the above modules.

[0217] Furthermore, the self-learning device for non-adversarial tasks and the method embodiment for self-learning methods for non-adversarial tasks provided in the above embodiments belong to the same concept. The specific way in which each module performs its operation has been described in detail in the method embodiment, and will not be repeated here.

[0218] Please see Figure 3 This application provides an electronic device 4000.

[0219] exist Figure 3 In this design, data interaction between the processor 4001 and the memory 4003 can be achieved through at least one communication bus 4002. This communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0220] Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0221] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc. The memory 4003 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program instructions or code in the form of instructions or data structures and accessible by the electronic device 400, but not limited thereto.

[0222] The memory 4003 stores program instructions or code, and the processor 4001 can read the program instructions or code stored in the memory 4003 through the communication bus 4002.

[0223] When the program instructions or code are executed by the processor 4001, they implement the self-learning methods for non-adversarial tasks in the above embodiments.

[0224] Furthermore, this application provides a storage medium storing program instructions or code, which are loaded and executed by a processor to implement a self-learning method for non-adversarial tasks as described above.

[0225] This application provides a computer program product, which includes program instructions or code. The program instructions or code are stored in a storage medium. The processor of an electronic device reads the program instructions or code from the storage medium, loads and executes the program instructions or code, so that the electronic device can implement the self-learning method for non-adversarial tasks as described above.

[0226] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A self-learning method for non-adversarial tasks, characterized in that, The method includes: Step 1: Initialize the decision neural network for the non-adversarial task; Step 2: Execute the MCTS state search algorithm based on the aforementioned decision neural network to obtain third-party performance scores and record path information; Step 3: Train and optimize the decision neural network based on the third-party performance score and path information; Step 4: Repeat steps 2-3 in parallel until the decision neural network meets the termination trigger condition; The non-adversarial tasks include large language model text generation tasks, wherein... Simulation phase: If the large language model executes MCTS for the first time, it will directly generate a text segment based on the current parameters until the maximum length is reached or a terminating character is generated. If the large language model is not executing MCTS for the first time, that is, if the string already exists in the prompt word, then a piece of text is generated using the current string as the prompt word; After the text generation task is completed, feedback from the discriminator is obtained; the discriminator includes a code compilation discriminator, a task execution discriminator, and a human score. Update phase: Based on the execution result of the large language model, update Found_victory(s) = 1 for each state on the successful path, where each state corresponds to the prompt word + the generated text; Record the total length of the new path as length, and update EL(s) = min(EL(s), length), where length also records the number of tokens generated; Calculate V(s) = Found_victory(s) + 1 / EL(s); Found_victory(s) marks whether a path has been found that passes through state s and finally reaches the success state; EL(s): the length of the path that passes through s and reaches the termination state, only the minimum value is recorded; At the same time, for the parent node of this state, update Q(s) previous a) = V(s), when s previous When action a tends towards state s; N(s) previous ,a)+=1; where, s previous This refers to the state at the previous moment; Selection phase: On the explored nodes, select the next action that maximizes the following formula: preference(s,a)=H(*|s)+U(s,a)+Q(s,a); Where H(*|s) represents the entropy of selecting the next action in state s based on the current decision network. The larger the entropy, the more likely s is to be a semantic divergence point, and it is recommended that the next action be selected from the options of s. U(s,a) represents the upper bound confidence. Q(s,a) represents the tendency to select action a in state s based on the expected future returns. Expansion phase: Expand the next node based on the selected action.

2. The self-learning method for non-adversarial tasks according to claim 1, characterized in that, The non-adversarial task satisfies the following characteristics: ① It satisfies all descriptions in the Markov decision process except for the reward function; ②The action space is discrete during task execution. For each state, the next action can only be selected from a finite number of values. ③ For a given state s and action a, a unique next action s' can always be generated; ④ The decision black box that generates the next action has three outputs: the probability distribution of the output, the value of the output state, and the expected number of steps to reach the ideal termination state. ⑤ The task can always stop after a finite number of steps; ⑥ It can determine whether the termination status is a successful status within a limited time.

3. The self-learning method for non-adversarial tasks according to claim 1, characterized in that, Step 3 includes: The backpropagation algorithm based on the loss function optimizes the weights of the decision neural network, and the optimization objective includes: Minimize the prediction error between v(s) and EL(s); Minimize the error between p(s,a) and U(s,a)+Q(s,a), where p(s,a) represents the probability of choosing action a in state s, which is predicted by the model; Maximize the entropy reduction effect of each execution step where Found_victory remains at 1; Minimize the KL divergence between the new and old models on the explored trajectories.

4. A self-learning method for non-adversarial tasks, characterized in that, The method includes: Step 1: Initialize the decision neural network for the non-adversarial task; Step 2: Execute the MCTS state search algorithm based on the aforementioned decision neural network to obtain third-party performance scores and record path information; Step 3: Train and optimize the decision neural network based on the third-party performance score and path information; Step 4: Repeat steps 2-3 in parallel until the decision neural network meets the termination trigger condition; The non-adversarial task includes a path planning algorithm, wherein, Simulation phase: A path is generated based on the current decision neural network and the current state, until termination; After a complete execution, feedback from the discriminator is obtained: 1 for success and 0 for failure; the discriminator is used to detect whether a collision occurs during the simulated execution of the path. Update phase: Based on the execution result, update Found_victory(s) = 1 for each state on the successful path. This state is the anchor point coordinates + tree structure. Record the total length of the new path as length, and update EL(s) = min(EL(s), length); the length also records the number of tokens generated. Calculate V(s) = Found_victory(s) + 1 / EL(s); Found_victory(s) marks whether a path has been found that passes through state s and finally reaches the success state; EL(s): the length of the path that passes through s and reaches the termination state, only the minimum value is recorded; At the same time, for the parent node of this state, update Q(s) previous a) = V(s), when s previous When action a tends towards state s; N(s) previous ,a)+=1; where, s previous This refers to the state at the previous moment; Selection phase: On the explored nodes, select the next action that maximizes the following formula: preference(s,a)=H(*|s)+U(s,a)+Q(s,a); Where H(*|s) represents the entropy of selecting the next action in state s based on the current decision network. The larger the entropy, the more the current network believes that it is possible to develop in any direction at this anchor point, and suggests that the next action be selected from the available actions in s; U(s,a) represents the upper bound confidence; Q(s,a) represents the tendency to select action a based on the expected future returns in state s. Use the above formula to determine where the current network perceives the path divergence point, and select a new path based on the divergence point; Expansion phase: Expand the next node based on the selected action.

5. The self-learning method for non-adversarial tasks according to claim 4, characterized in that, The non-adversarial task satisfies the following characteristics: ① It satisfies all descriptions in the Markov decision process except for the reward function; ②The action space is discrete during task execution. For each state, the next action can only be selected from a finite number of values. ③ For a given state s and action a, a unique next action s' can always be generated; ④ The decision black box that generates the next action has three outputs: the probability distribution of the output, the value of the output state, and the expected number of steps to reach the ideal termination state. ⑤ The task can always stop after a finite number of steps; ⑥ It can determine whether the termination status is a successful status within a limited time.

6. The self-learning method for non-adversarial tasks according to claim 4, characterized in that, Step 3 includes: The backpropagation algorithm based on the loss function optimizes the weights of the decision neural network, and the optimization objective includes: Minimize the prediction error between v(s) and EL(s); Minimize the error between p(s,a) and U(s,a)+Q(s,a), where p(s,a) represents the probability of choosing action a in state s, which is predicted by the model; Maximize the entropy reduction effect of each execution step where Found_victory remains at 1; Minimize the KL divergence between the new and old models on the explored trajectories.

7. A self-learning method for non-adversarial tasks, characterized in that, The method includes: Step 1: Initialize the decision neural network for the non-adversarial task; Step 2: Execute the MCTS state search algorithm based on the aforementioned decision neural network to obtain third-party performance scores and record path information; Step 3: Train and optimize the decision neural network based on the third-party performance score and path information; Step 4: Repeat steps 2-3 in parallel until the decision neural network meets the termination trigger condition; The non-adversarial task includes a robot evolution algorithm, wherein, Simulation phase: Generate a path directly based on the current network or current state until termination; After a complete task is executed, feedback is obtained from the discriminator: 1 for success and 0 for failure; the discriminator is used to score the evolution results. Update phase: Based on the execution result, update Found_victory(s) = 1 for each state on the successful path, where each state includes the robot's physical structure or the underlying intelligence; Record the total length of the new path as length, and update EL(s) = min(EL(s), length); the length also records the number of tokens generated. Calculate V(s) = Found_victory(s) + 1 / EL(s); Found_victory(s) marks whether a path has been found that passes through state s and finally reaches the success state; EL(s): the length of the path that passes through s and reaches the termination state, only the minimum value is recorded; At the same time, for the parent node of this state, update Q(s) previous a) = V(s), when s previous When action a tends towards state s; N(s) previous ,a)+=1; where, s previous This refers to the state at the previous moment; Selection phase: On the explored nodes, select the next action that maximizes the following formula: preference(s,a)=H(*|s)+U(s,a)+Q(s,a); Where H(*|s) represents the entropy of selecting the next action in state s based on the current decision network. The larger the entropy, the more the current network believes that it is possible to mutate in any direction at this anchor point, and suggests that the next action be selected from the available actions in s; U(s,a) represents the upper bound confidence; Q(s,a) represents the tendency to select action a based on the expected future returns in state s. Use the above formula to determine which mutation the current network considers unreliable, and then try other mutation options; Expansion phase: Expand the next node based on the selected action.

8. The self-learning method for non-adversarial tasks according to claim 7, characterized in that, The non-adversarial task satisfies the following characteristics: ① It satisfies all descriptions in the Markov decision process except for the reward function; ②The action space is discrete during task execution. For each state, the next action can only be selected from a finite number of values. ③ For a given state s and action a, a unique next action s' can always be generated; ④ The decision black box that generates the next action has three outputs: the probability distribution of the output, the value of the output state, and the expected number of steps to reach the ideal termination state. ⑤ The task can always stop after a finite number of steps; ⑥ It can determine whether the termination status is a successful status within a limited time.

9. The self-learning method for non-adversarial tasks according to claim 7, characterized in that, Step 3 includes: The backpropagation algorithm based on the loss function optimizes the weights of the decision neural network, and the optimization objective includes: Minimize the prediction error between v(s) and EL(s); Minimize the error between p(s,a) and U(s,a)+Q(s,a), where p(s,a) represents the probability of choosing action a in state s, which is predicted by the model; Maximize the entropy reduction effect of each execution step where Found_victory remains at 1; Minimize the KL divergence between the new and old models on the explored trajectories.

10. A self-learning device for non-adversarial tasks, characterized in that, The device is used to implement the self-learning method for non-adversarial tasks as described in any one of claims 1-9, the device comprising: An initialization module is used to initialize the decision neural network for non-adversarial tasks; The state acquisition module is used to execute the MCTS state search algorithm based on the current decision neural network, obtain third-party performance scores, and record path information; The training optimization module is used to train and optimize the decision neural network based on the third-party performance score and path information. The termination module is used to repeatedly execute steps 2-3 in parallel until the decision neural network meets the termination trigger condition.

11. An electronic device, characterized in that, include: At least one processor, at least one memory, wherein, The memory stores program instructions or code; The program instructions or code are loaded and executed by the processor, enabling the electronic device to implement the self-learning method for non-adversarial tasks as described in any one of claims 1 to 9.

12. A storage medium storing program instructions or code thereon, characterized in that, The program instructions or code are loaded and executed by the processor to implement the self-learning method for non-adversarial tasks as described in any one of claims 1 to 9.

Images