Coach agent-based guidance action generation system and method

Abstract

The invention discloses a coach agent-based guidance action generation system and method. The coach agent-based guidance action generation system comprises an initial processing module and an iterative processing module, the initial processing module is used for acquiring target action data in real time, generating a current task state based on the student intelligent agent of the trained target system model, inputting the current task state to the coach intelligent agent, and obtaining and outputting a target optimal action; and the iterative processing module is used for acquiring an actual action corresponding to the target optimal action, acquiring target action data corresponding to the actual action, updating a task state, performing iterative processing according to the trained target system model until the current task is completed, and outputting the target action data obtained by each iterative processing. The training efficiency of the table tennis training process is improved.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning, and in particular to a coaching agent-based guidance action generation system, method, terminal and computer-readable storage medium. Background Art

[0002] Table tennis is a very popular sport, but currently in table tennis teaching, most teaching methods are direct teaching by coaches, which will undoubtedly lead to different teaching effects due to the coach's ability. In addition, students may not be able to perfectly implement instructions during actual execution due to limitations in their technical level or physical condition. In other words, there are often significant differences in students' ability to execute movements, resulting in low training efficiency.

[0003] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0004] The main purpose of the present invention is to provide a coaching intelligent agent-based guidance action generation system, method, terminal and computer-readable storage medium, aiming to solve the problem in the existing technology that in table tennis teaching, most teaching methods are direct teaching by coaches, which will undoubtedly lead to different teaching effects due to the coach's ability, and students may not be able to perfectly implement instructions during actual execution due to technical level or physical condition limitations, that is, students' action execution capabilities often vary significantly, resulting in low training efficiency.

[0005] To achieve the above object, the present invention provides a coaching action generation system based on a coaching agent, wherein the coaching action generation system based on a coaching agent includes an initial processing module and an iterative processing module;

[0006] an initial processing module for acquiring target action data in real time, generating a current task state based on the target action data based on the student agent of the trained target system model, inputting the current task state into the coach agent of the trained target system model, obtaining and outputting the target optimal action, wherein the target system model includes a coach agent and a student agent, and selecting a target data acquisition method from a direct traversal method and an octree-based search strategy, acquiring a training data set based on the target data acquisition, training the corresponding coach agent, and obtaining the trained target system model;

[0007] An iterative processing module is used to obtain the actual action corresponding to the target optimal action and the target action data corresponding to the actual action. The student intelligent agent based on the trained target system model updates the task status and performs iterative processing according to the trained target system model until the current task is completed, and outputs the target action data obtained in each iterative processing.

[0008] Optionally, selecting a target data acquisition method from a direct traversal method and an octree-based search strategy, acquiring a training data set according to the target data acquisition, training a corresponding coaching agent, and obtaining the trained target system model includes:

[0009] According to the pre-set, the target data acquisition method is selected from the direct traversal method and the octree-based search strategy;

[0010] Based on the target data acquisition method, generating a training data set;

[0011] The optimal strategies for different proficiency levels corresponding to each action in the training data set are stored in the coaching agent to obtain a trained target system model.

[0012] Optionally, generating a training data set based on the target data acquisition method includes:

[0013] When the target data acquisition method is a direct traversal method, based on the target system model, the optimal strategy corresponding to each action at different success rates is obtained and stored to obtain the corresponding training data set;

[0014] When the target data acquisition method is an octree-based search strategy, all strategies are initialized, and based on the target system model, the optimal strategy detection is performed cyclically within the set boundary to obtain the optimal strategy corresponding to each action under different success rates, and after storage, the corresponding training data set is obtained.

[0015] Optionally, the initial processing module includes:

[0016] A task state generation unit is configured to obtain target action data in real time, generate a strategy for the current student agent based on the trained student agent of the target system model, obtain a current execution success rate, and generate a current task state based on the target action data, the strategy for the current student agent, and the current execution success rate;

[0017] The target action data output unit is used to input the current task state into the coaching agent of the trained target system model, and the coaching agent of the trained target system model outputs the optimal target action data according to the current task state.

[0018] Optionally, the task status generating unit includes:

[0019] The first updating subunit is configured to obtain target action data in real time, generate a strategy based on the student agent of the trained target system model, obtain the number of executions of all target action data, and update the execution success rate of the current student agent based on the number of executions of all actions and a preset execution success rate calculation formula;

[0020] The second updating subunit is used to generate the current task state according to the target action data, the strategy of the current student agent and the execution success rate.

[0021] Optionally, the target action data output unit includes:

[0022] a discrete estimation subunit, configured to input the current task state into the trained coaching agent of the target system model, wherein the coaching agent obtains an action set corresponding to all target action data based on the target action data, and correspondingly calculates a discrete estimation of the execution success rate of each action in the action set;

[0023] The action output subunit is used to select the target optimal action according to all the discrete estimates and output it.

[0024] Optionally, the iterative processing module includes:

[0025] an execution success rate updating unit, configured to obtain an actual action corresponding to the target optimal action and update the corresponding execution success rate to obtain an updated current execution success rate;

[0026] an iterative processing unit, configured to obtain target action data corresponding to the actual action, update the task state based on the student agent of the trained target system model, and perform iterative processing according to the trained target system model;

[0027] The stop unit is used to end the iteration if all current tasks are completed and output the target action data obtained by each iteration.

[0028] In addition, to achieve the above-mentioned purpose, the present invention further provides a method for generating guidance actions based on a coaching agent, wherein the method for generating guidance actions based on a coaching agent comprises:

[0029] Construct a target system model based on the coach agent and student agent;

[0030] According to a pre-set setting, a target data acquisition method is selected from a direct traversal method and an octree-based search strategy, a training data set is acquired based on the target data acquisition method, and the coaching agent is trained according to the training data set to obtain a trained target system model;

[0031] Acquire target action data in real time, generate a current task state based on the target action data based on the student agent of the trained target system model, input the current task state into the coach agent of the trained target system model, obtain the target optimal action and output it;

[0032] Obtain the actual action corresponding to the target optimal action, and obtain the target action data corresponding to the actual action, update the task status based on the student intelligent agent of the trained target system model, and perform iterative processing according to the trained target system model until the current task is completed, and output the target action data obtained from each iterative processing.

[0033] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a coaching agent-based guidance action generation program stored on the memory and runnable on the processor, and when the coaching agent-based guidance action generation program is executed by the processor, the steps of the coaching agent-based guidance action generation method as described above are implemented.

[0034] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a coaching intelligent agent-based guidance action generation program, and when the coaching intelligent agent-based guidance action generation program is executed by the processor, the steps of the coaching intelligent agent-based guidance action generation method as described above are implemented.

[0035] In the present invention, the guidance action generation system based on the coaching agent includes an initial processing module and an iterative processing module; the initial processing module is used to obtain target action data in real time, and based on the student agent of the trained target system model, generates a current task state according to the target action data, inputs the current task state into the coaching agent of the trained target system model, obtains the target optimal action and outputs it, wherein the target system model includes a coaching agent and a student agent, and selects a target data acquisition method from a direct traversal method and an octree-based search strategy, obtains a training data set according to the target data, trains the corresponding coaching agent, and obtains the trained target system model; the iterative processing module is used to obtain the actual action corresponding to the target optimal action, and obtains the target action data corresponding to the actual action, updates the task state based on the student agent of the trained target system model, and performs iterative processing according to the trained target system model until the current task is completed, and outputs the target action data obtained from each iterative processing. The present invention uses a bi-level reinforcement learning (BiRL) framework. In the bottom-level interaction, the student agent continuously optimizes its cognitive strategy and improves its action execution ability through continuous interaction with the task environment. In the upper-level interaction, the coach agent adjusts the teaching strategy in real time by evaluating the success rate of the student's action execution. Specifically, the optimal action suggestion provided by the coach agent of the present invention for students of a specific skill level is equivalent to the optimal student action under the skill level parameter environment. When the task environment remains stable, the new environment still maintains stable characteristics after the skill level parameters are integrated. Furthermore, in the invention, the number of task environments caused by the student action set and its differentiated execution level grows exponentially. By combining the octree sampling method, repeated calculations are effectively avoided, the convergence speed of the coach strategy is significantly improved, and the obtained target system model can also efficiently perform corresponding processing on the target action data. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 1. It is a structural diagram of a preferred embodiment of the coaching action generation system based on the coaching agent of the present invention;

[0037] Figure 2 Schematic diagram of the target system model of the double-layer reinforcement learning in the coaching action generation system based on the coaching agent of the present invention;

[0038] Figure 3 This is a schematic diagram of the implementation of the direct traversal method in the coaching action generation method based on the coaching agent of the present invention;

[0039] Figure 4Schematic diagram of an implementation of an octree-based search strategy in a coaching agent-based guidance action generation method of the present invention;

[0040] Figure 5 Schematic diagram of a skill development curve showing the relationship between the success rate of action execution and the number of action executions in the coaching action generation method based on the coaching agent of the present invention;

[0041] Figure 6 is a flow chart of a preferred embodiment of the method for generating guiding actions based on a coaching agent of the present invention;

[0042] Figure 7 FIG. 4 is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0044] Table tennis is a very popular sport, but currently in table tennis teaching, most teaching methods are direct teaching by coaches, which will undoubtedly lead to different teaching effects due to the coach's ability. In addition, students may not be able to perfectly implement instructions during actual execution due to limitations in their technical level or physical condition. In other words, there are often significant differences in students' ability to execute movements, resulting in low training efficiency.

[0045] In response to one or more of the above problems, the present invention provides a guidance action generation system based on a coaching agent, which includes an initial processing module and an iterative processing module; the initial processing module is used to obtain target action data in real time, and based on the student agent of the trained target system model, generates a current task state according to the target action data, inputs the current task state into the coaching agent of the trained target system model, obtains the target optimal action and outputs it, wherein the target system model includes a coaching agent and a student agent, and selects a target data acquisition method from a direct traversal method and an octree-based search strategy, obtains a training data set according to the target data, trains the corresponding coaching agent, and obtains the trained target system model; the iterative processing module is used to obtain the actual action corresponding to the target optimal action, and obtains the target action data corresponding to the actual action, updates the task state based on the student agent of the trained target system model, and performs iterative processing according to the trained target system model until the current task is completed, and outputs the target action data obtained from each iterative processing.

[0046] The coaching action generation system based on the coaching agent in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the coaching action generation system based on the coaching agent includes an initial processing module 11 and an iterative processing module 12.

[0047] The target system model includes a coach agent and a student agent, and a target data acquisition method is selected from a direct traversal method and an octree-based search strategy. A training data set is acquired according to the target data acquisition method, and the corresponding coach agent is trained to obtain the trained target system model.

[0048] Specifically, under the BiRL (Bi-Level Reinforcement Learning) teaching framework, the target system model described in this paper is as follows: Figure 2 As shown in Figure 2, the student agent and coach agent are both built based on BiRL. The student agent with the execution level ρ, according to its strategy π l In task / environment state Report intent to the coaching agent in, represents the set of all actions, which in this invention is all ball-returning actions. This set can be further quantified by parameters such as shot type and force. The coaching agent follows its strategy π c and an estimate of the student's performance level. Provide students with suggested movements This invention mainly provides personalized guidance for students when their movements are not perfectly executed, and does not consider the students' deviation in understanding the instructions. Under this condition, the students fully understand the coach's suggestions and use them to replace the original brain intention. However, due to the existence of execution deviation, the actual movement a made by the students has only a certain probability of being consistent with the brain intention. Figure 1 To.

[0049] Therefore, it is necessary to conduct corresponding training on the coaching agent in advance so that the corresponding target system model can perform corresponding processing for situations with different execution success rates.

[0050] It should be noted that in the present invention, the student agent is modeled as an RL agent that interacts with the task environment to learn the best strategy. When the student agent receives a suggested action from the coach agent, it directly replaces the intended action generated by its own strategy with the suggested action. Traditional RL assumes that the agent can always perform actions perfectly without making mistakes. However, due to limited skill levels and proficiency, human students often have varying degrees of deviation in the execution of specific actions. This paper introduces the action execution success rate in the students. To model this execution deviation, the corresponding mathematical representation is the difference between the actual action a and the expected intention Same probability.

[0051] The coaching agent observes the student's behavior as they interact with the task environment and provides personalized guidance to promote effective learning. Unlike traditional AI coaches that only consider the timing of instruction, the coaching agent in this invention is designed to adjust its instruction strategy based on the student's evolving strategies and skill levels. This dynamic adjustment ensures that instruction aligns with the student's current abilities, promoting steady skill improvement while minimizing task failures.

[0052] In the present invention, for the student agent, when the student agent uses the actions performed by the actual body to update its strategy, it follows the standard MDP (Markov Decision Process, MDP) framework Similar to standard RL (Reinforcement Learning), where is the collection of all rewards, is the state transition function of the task environment in reinforcement learning. In this description, the student agent iterates the policy and converges by maximizing the task reward, and the actual performance of the converged policy depends on the degree of action execution. Here, imperfect physical execution can be viewed as a form of exploration, which is uncontrolled and has an exploration rate of ρ; in contrast, exploration in standard RL can be viewed as brain-level, where the exploration rate is controlled. The student policy update based on the physical execution of the action can be adopted by methods such as However, this strategy update method has certain problems. What the student can control is the brain intention, and the body execution is not controlled. Therefore, learning from the body execution will lead to inconsistency between the student's brain strategy and actual execution, especially when the success rate of the action execution is low. In one embodiment, when the student inadvertently performs an action a, he receives a generous task reward. Based on this, the corresponding intelligent agent believes that a is the best action, and updates the brain strategy accordingly. The next time the same state appears, As intended, but The actual execution is not ideal, resulting in losses. The bigger problem is that this deviation in actual execution does not affect the brain strategy, because the loss is caused by the deviation of the action a', the brain denies the deviation of the action a' rather than the intention. Therefore, no matter how many times the same state appears, the corresponding agent will try to execute regardless of the actual loss.

[0053] Therefore, while this approach can theoretically push students toward the global optimal strategy, it comes at the cost of slower convergence and frequent failures. Furthermore, because the execution of actions varies across different situations, this can lead to disconnects, further highlighting the inefficiency of relying solely on action execution updates, especially for learners with low action success rates.

[0054] Furthermore, the present invention updates the strategy based on the brain's expected intention. From the perspective of RL, learning from expected intention is equivalent to treating execution deviation as part of the environment dynamics. The original simple task environment Transformed into And generate a new MDP Therefore, different execution success rates correspond to different task environments and different optimal strategies, and these differentiated strategies are more consistent with the student's current skill proficiency. Updating the policy based on the brain's intended intention is a safer and more practical approach. By modeling execution errors as inherent randomness in the environment, the coaching agent accounts for the limitations of its physical execution and focuses on achieving the local optimal strategy for a specific skill level. This adaptability can lead to faster convergence to an actionable strategy and minimize failures during training, which is critical for maintaining human learner engagement and confidence. As students' skill level continues to develop through repeated practice, they will safely converge to better task strategies and ultimately approach the global optimum. When training the coaching agent, it is important to consider both of these student policy update methods, as they correspond to the two student populations, regardless of whether the policy update method is potentially used by the students or chosen by them. This consideration also ensures that the coaching agent's policy can adapt and generalize to different student behaviors in practice.

[0055] Furthermore, the target data acquisition method is selected from the direct traversal method and the octree-based search strategy, a training data set is acquired according to the target data acquisition, and a corresponding coaching agent is trained to obtain the trained target system model, including:

[0056] According to the pre-set, the target data acquisition method is selected from the direct traversal method and the octree-based search strategy;

[0057] Based on the target data acquisition method, generating a training data set;

[0058] The optimal strategies for different proficiency levels corresponding to each action in the training data set are stored in the coaching agent to obtain a trained target system model.

[0059] Specifically, in this invention, when training the target system model, the main focus is on training the coach agent, where the purpose of training is to obtain recommended actions with a success rate for all actions. In this invention, within the RL framework, we explore methods for training the coach agent to effectively guide the student agent. The key lies in the current task state s and the student's execution ability. Determine the best advice the coaching agent should give, or in other words, find the best actions for students with different skill levels. The complexity of this problem depends on two factors: the size of the action set and execution success rate Discretization granularity. Direct traversal methods are effective when the search space is relatively small; however, when faced with a larger or continuous execution rate space, direct traversal methods become impractical. Students with similar skill levels generally have similar optimal strategies. Accordingly, the dynamic programming algorithm is run only once. Therefore, the present invention develops an octree-based coaching strategy training algorithm; this algorithm can achieve efficient sampling in large discrete spaces, and the sampled data can also be used to fit the continuous space.

[0060] The present invention pre-judges the size of the search space of the environment to be applied and sets a corresponding threshold. When the size is lower than the threshold, a direct traversal method is used, and when the size exceeds the threshold, an octree-based search strategy is used.

[0061] After using the corresponding method to generate the training data set, the actions corresponding to each action are stored, and the corresponding execution success rate is also stored. The strategy with the highest execution success rate is taken as the best strategy. After storing these data in the coaching agent, the training of the coaching agent is completed, and the trained target system model can be obtained.

[0062] Furthermore, generating a training data set based on the target data acquisition method includes:

[0063] When the target data acquisition method is a direct traversal method, based on the target system model, the optimal strategy corresponding to each action at different success rates is obtained and stored to obtain the corresponding training data set;

[0064] When the target data acquisition method is an octree-based search strategy, all strategies are initialized, and based on the target system model, the optimal strategy detection is performed cyclically within the set boundary to obtain the optimal strategy corresponding to each action under different success rates, and after storage, the corresponding training data set is obtained.

[0065] It should be noted that the strategy of the coach agent in the direct traversal method can be derived by parallel training with different execution success rate parameters. This method involves training multiple student agents simultaneously, each student in the RL environment with different execution biases. Run in. Figure 3 As shown, the present invention further explains the detailed process, wherein ρ 1,a Equal to ρ init,a ,and Equal to ρ lim,a ; In this process, the best strategy corresponding to each action under different success rates is obtained and stored to obtain the corresponding training data set. Although these new environments introduce different degrees of randomness to the original tasks, if the basic task is stationary, that is, the state transition function Does not change over time, then the new environment after introducing action execution deviation It is also stable. Combining the local optimal strategy π of students at different action execution levels l,ρ , we can get the final coaching strategy π c . And for the explicit conversion function The original environment, the state transition function under the specific execution success rate parameter It is also explicit, and correspondingly, dynamic programming can be used to determine the local optimal strategy for students at each skill level, and then determine the optimal coaching strategy. Generally speaking, the state transition function of the task environment faced by traditional RL methods is implicit and unknowable. Therefore, it is necessary to balance two points in policy iteration: utilization (Exploitation), selecting the optimal action under the current strategy; and exploration (Exploration), considering that the current strategy is not good enough and trying new actions. Overemphasizing the utilization of existing strategies will cause the solution to fall into local optimality. Conversely, excessive exploration will lead to instability in the training process. Unlike general RL methods, after having an explicit task state transition function, the dynamic programming method does not need to deal with the balance between utilization and exploration, and can find the exact optimal solution through a finite number of iterations.

[0066] For the octree-based search strategy, in order to efficiently handle large search spaces, this invention implements an octree-based method that uses the assumption that adjacent points in the action execution space usually produce similar results. Figure 4 As shown, given a minimum action execution success rate ρ low and the maximum action execution success rate ρ high Definition dimensional space, if the local optimal strategies they generate after dynamic programming calculation are consistent It is considered that the action execution success rates corresponding to all points in the space share this strategy; otherwise, the space is divided into subspaces and checks whether the local optimal strategies corresponding to the minimum / maximum points used to define each subspace are consistent. This process continues until the strategies of all subspaces are solved or other preset stopping conditions are reached (such as the maximum recursion depth in the discrete case or the minimum space that can be divided). In the best case, the octree-based search strategy can reduce the computational complexity from Reduce to in, Expressing computational complexity, Indicates that N data require calculations of the order of N (2N, 0.5N both belong to this order of magnitude). In this invention, the traversal method requires calculating a strategy for each student's action execution, and then combining them together. The corresponding computational complexity is and that is The calculation of this order of magnitude is It is much smaller because the octree-based search strategy only calculates the boundary points of the execution success rate.

[0067] An initial processing module 11 is configured to obtain target action data in real time, generate a current task state based on the target action data based on the trained student agent of the target system model, input the current task state into the trained coach agent of the target system model, obtain the target optimal action, and output it;

[0068] Specifically, after completing the training of the coaching agent, the trained target system model is used to process specific actions.

[0069] The initial processing module includes:

[0070] A task state generation unit is configured to obtain target action data in real time, generate a strategy for the current student agent based on the trained student agent of the target system model, obtain a current execution success rate, and generate a current task state based on the target action data, the strategy for the current student agent, and the current execution success rate;

[0071] The target action data output unit is used to input the current task state into the coaching agent of the trained target system model, and the coaching agent of the trained target system model outputs the optimal target action data according to the current task state.

[0072] Specifically, in the present invention, for the student agent, the student agent consists of an action set A, a strategy π l And the execution success rate ρ of each action. The action set A specifies all possible expected intentions / actual actions of the student, which is generally related to the specific task and remains unchanged throughout the process. Strategy π lFrom task state s to intention The mapping of action a is known to the student and is constantly updated, with the goal of maximizing the cumulative reward. In table tennis, this is a mapping function between incoming balls and returned balls. The success rate ρ for each action is independent, unknown to the student, and increases with the number of attempts (corresponding to practice making perfect in real life). This value determines the probability of a successful execution of an intention.

[0073] In specific applications, the corresponding actions are obtained, and the student agent generates the strategy of the current student agent based on the corresponding actions, and obtains the current execution success rate. The corresponding tasks are sent to the coach agent based on these generated strategies.

[0074] Furthermore, the task status generating unit includes:

[0075] The first updating subunit is configured to obtain target action data in real time, generate a strategy based on the student agent of the trained target system model, obtain the number of executions of all target action data, and update the execution success rate of the current student agent based on the number of executions of all actions and a preset execution success rate calculation formula;

[0076] The second updating subunit is used to generate the current task state according to the target action data, the strategy of the current student agent and the execution success rate.

[0077] Specifically, in the present invention, in the student agent, specific actions are The success rate of action execution ρ a Will try the action with intention a as the number of times This relationship can be modeled by the monotone Sigmoid function as follows: init and ρ lim is the initial value and upper limit of the success rate of action execution, and N0 is a proportional factor, which is approximately equal to the number of intention attempts required to reach the midpoint of the skill level improvement curve. The specific formula is:

[0078]

[0079] This formula describes the process in which the success rate of an action gradually increases with the number of attempts. This process is described as a Sigmoid function curve. The only parameter contained in this function curve is N0. Specifically, for the execution success rate curve of an action a, the corresponding parameter is N 0,a The present invention uses this formula to update the execution success rate.

[0080] Furthermore, Figure 5The skill development curves for parameters N0 = 1000 / 2000 / 5000 are shown. The skill development curves may vary depending on the action and the student, reflecting the inherent difficulty of the action and individual differences between students. The set of execution success rates of all possible actions under a task constitutes the student's skill proficiency. The complete representation of .

[0081] Furthermore, the target action data output unit includes:

[0082] a discrete estimation subunit, configured to input the current task state into the trained coaching agent of the target system model, wherein the coaching agent obtains an action set corresponding to all target action data based on the target action data, and correspondingly calculates a discrete estimation of the execution success rate of each action in the action set;

[0083] The action output subunit is used to select the target optimal action according to all the discrete estimates and output it.

[0084] Specifically, in the present invention, after the task state is input into the trained coach agent of the target system model, the coach agent uses the student strategy π l Sampling As its state, a small correction to the student's last action is used as an instruction, and the student's current strategy is sampled (i.e., the expected action ) provides students with personalized semantic guidance. Students modify their actions based on the correction instructions and the content of the previous action, and perform the modified actions perfectly.

[0085] In this invention, the instruction space of the coach agent and the action space of the student agent are considered. The same situation, whereby the student agent can directly take the action suggested by the coach, rather than combining the corrective instruction with the last action performed as in the previous chapter. In this case, for an ideal student with perfect action execution (ρ = 1), it can be proved that a high-level or elite student can always provide the best action (suggestion) in any task state and produce the best learning results. That is, the best instruction is only related to the state s and has nothing to do with the student's policy π l It is irrelevant. For students who have deviations in the execution of the action The coach’s optimal strategy depends on the state of the environment s and the student’s skill level ρ, and is independent of the student’s strategy π l Therefore, the state representation of the coaching model in this paper can be expressed by Simplified to {s,ρ}. The coach's guidance focuses not only on what to do, but also on what the student can successfully perform based on his or her current skill level.

[0086] As part of the state, the coach agent needs to estimate the success rate of the student's action execution by monitoring the difference in the student's "intention→execution" over time. In the present invention, for each action in the set A sliding window method is used to obtain a discrete estimate of its execution success rate As shown in the following formula:

[0087]

[0088] Where W is the sliding window width, h is the discretization function, is a list of boolean values representing the results of attempting action a, where b a,j = 1 indicates that the intention was successfully executed on the jth attempt. The sliding window design helps maintain a reasonable frequency of skill proficiency updates. Too frequent updates (e.g., window width W = 1) can cause jumps in proficiency and prevent effective strategy updates. Infrequent updates (e.g., window width of one scene) cannot reflect changes in the student's skill level in a timely manner.

[0089] The iterative processing module 12 is used to obtain the actual action corresponding to the target optimal action, and obtain the target action data corresponding to the actual action. Based on the student intelligent agent of the trained target system model, the task status is updated, and iterative processing is performed according to the trained target system model until the current task is completed, and the target action data obtained by each iterative processing is output.

[0090] After completing the corresponding initial processing, each action is iteratively processed until the end, and the corresponding target action data is stored after the end for subsequent training of the coaching agent at predetermined intervals.

[0091] The iterative processing module includes:

[0092] an execution success rate updating unit, configured to obtain an actual action corresponding to the target optimal action and update the corresponding execution success rate to obtain an updated current execution success rate;

[0093] an iterative processing unit, configured to obtain target action data corresponding to the actual action, update the task state based on the student agent of the trained target system model, and perform iterative processing according to the trained target system model;

[0094] The stop unit is used to end the iteration if all current tasks are completed and output the target action data obtained by each iteration.

[0095] Specifically, during the iterative processing, the execution success rate is updated each time according to the actual action of the last specific execution, thereby ensuring that the execution success rate can be dynamically updated according to the specific number of executions.

[0096] The coaching action generation system based on the coaching agent of the present invention includes an initial processing module and an iterative processing module; the initial processing module is used to obtain target action data in real time, generate a current task state according to the target action data based on the student agent of the trained target system model, input the current task state into the coaching agent of the trained target system model, obtain the target optimal action and output it, wherein the target system model includes a coaching agent and a student agent, and selects a target data acquisition method from a direct traversal method and an octree-based search strategy, obtains a training data set according to the target data, trains the corresponding coaching agent, and obtains the trained target system model; the iterative processing module is used to obtain the actual action corresponding to the target optimal action, and obtain the target action data corresponding to the actual action, update the task state based on the student agent of the trained target system model, and perform iterative processing according to the trained target system model until the current task is completed, and output the target action data obtained from each iterative processing. The present invention uses a two-layer reinforcement learning framework. In the bottom-level interaction, the student agent continuously optimizes its cognitive strategy and improves its action execution ability through continuous interaction with the task environment. In the upper-level interaction, the coach agent adjusts the teaching strategy in real time by evaluating the success rate of the student's action execution. Specifically, the optimal action suggestion provided by the coach agent of the present invention for students of a specific skill level is equivalent to the optimal student action under the skill level parameter environment. When the task environment remains stable, the new environment still maintains stable characteristics after the skill level parameters are integrated. Furthermore, in the present invention, the number of task environments caused by the student action set and its differentiated execution level grows exponentially. By combining the octree sampling method, repeated calculations are effectively avoided, the convergence speed of the coach strategy is significantly improved, and the obtained target system model can also efficiently perform corresponding processing on the target action data.

[0097] Furthermore, if Figure 6 As shown, based on the above-mentioned coaching agent-based guidance action generation system, the present invention also provides a coaching agent-based guidance action generation method, wherein the coaching agent-based guidance action generation method includes:

[0098] Step S61: acquiring target action data in real time, generating a current task state based on the target action data based on the student agent of the trained target system model, inputting the current task state into the coach agent of the trained target system model, obtaining and outputting the target optimal action, wherein the target system model includes a coach agent and a student agent, and selecting a target data acquisition method from a direct traversal method and an octree-based search strategy, acquiring a training data set based on the target data acquisition method, training the corresponding coach agent, and obtaining the trained target system model;

[0099] Step S62: obtain the actual action corresponding to the target optimal action, and obtain the target action data corresponding to the actual action. Based on the student agent of the trained target system model, update the task status, and perform iterative processing according to the trained target system model until the current task is completed, and output the target action data obtained in each iterative processing.

[0100] Furthermore, the real-time acquisition of target action data, generating a current task state based on the target action data based on the student agent of the trained target system model, inputting the current task state into the coach agent of the trained target system model, obtaining and outputting the target optimal action, includes:

[0101] Acquire target action data in real time, generate a strategy for the current student agent based on the trained target system model, obtain a current execution success rate, and generate a current task state based on the target action data, the strategy for the current student agent, and the current execution success rate;

[0102] The current task state is input to the coaching agent of the trained target system model, and the coaching agent of the trained target system model outputs optimal target action data based on the current task state.

[0103] The real-time acquisition of target action data, generating a strategy for the current student agent based on the trained target system model, and obtaining a current execution success rate, and generating a current task state based on the target action data, the strategy for the current student agent, and the current execution success rate, include:

[0104] Acquire target action data in real time, generate strategies based on the student agent of the trained target system model, obtain the number of executions of all target action data, and update the execution success rate of the current student agent based on the number of executions of all actions and a preset execution success rate calculation formula;

[0105] The current task state is generated according to the target action data, the strategy of the current student agent and the execution success rate.

[0106] The step of inputting the current task state to the trained coaching agent of the target system model, and the trained coaching agent of the target system model outputting optimal target action data according to the current task state, comprises:

[0107] Inputting the current task state into the coaching agent of the trained target system model, the coaching agent obtaining an action set corresponding to all target action data based on the target action data, and correspondingly calculating a discrete estimate of the execution success rate of each action in the action set;

[0108] Based on all the discrete estimates, the target best action is selected and output.

[0109] The step of obtaining the actual action corresponding to the target optimal action and obtaining the target action data corresponding to the actual action, updating the task state based on the student agent of the trained target system model, and performing iterative processing according to the trained target system model until the current task is completed, and outputting the target action data obtained from each iterative processing, includes:

[0110] Obtaining the actual action corresponding to the target optimal action, and updating the corresponding execution success rate to obtain an updated current execution success rate;

[0111] Obtain target action data corresponding to the actual action, update the task state based on the student agent of the trained target system model, and perform iterative processing according to the trained target system model;

[0112] If all current tasks are completed, the iteration ends and the target action data obtained from each iteration is output.

[0113] Furthermore, if Figure 7 As shown, based on the above-mentioned coaching agent-based guidance action generation method and system, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 7 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0114] In some embodiments, the memory 20 can be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 can also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 can also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 can also be used to temporarily store data that has been output or is to be output. In one embodiment, a coaching intelligent agent-based guidance action generation program 40 is stored on the memory 20, and the coaching intelligent agent-based guidance action generation program 40 can be executed by the processor 10, thereby realizing the coaching intelligent agent-based guidance action generation method of the present invention.

[0115] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program code or process data stored in the memory 20, such as executing the coaching agent-based guidance action generation method.

[0116] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch screen, etc. The display 30 is used to display information on the terminal and to display a visual user interface.

[0117] In one embodiment, when the processor 10 executes the coaching action generation program 40 based on the coaching agent in the memory 20, the steps of the above coaching action generation method based on the coaching agent are implemented.

[0118] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a coaching action generation program based on a coaching agent, and when the coaching action generation program based on a coaching agent is executed by a processor, the steps of the above coaching action generation method based on a coaching agent are implemented.

[0119] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal comprising the element.

[0120] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When the program is executed, it can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0121] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A coaching action generation system based on a coaching agent, characterized in that: The coaching action generation system based on the coaching agent includes an initial processing module and an iterative processing module; an initial processing module for acquiring target action data in real time, generating a current task state based on the target action data based on the student agent of the trained target system model, inputting the current task state into the coach agent of the trained target system model, obtaining and outputting the target optimal action, wherein the target system model includes a coach agent and a student agent, and selecting a target data acquisition method from a direct traversal method and an octree-based search strategy, acquiring a training data set based on the target data acquisition, training the corresponding coach agent, and obtaining the trained target system model; An iterative processing module is used to obtain the actual action corresponding to the target optimal action and the target action data corresponding to the actual action. The student intelligent agent based on the trained target system model updates the task status and performs iterative processing according to the trained target system model until the current task is completed, and outputs the target action data obtained in each iterative processing.

2. The coaching action generation system based on the coaching agent according to claim 1 is characterized in that: The target data acquisition method is selected from the direct traversal method and the octree-based search strategy, a training data set is obtained according to the target data acquisition, a corresponding coaching agent is trained, and the trained target system model is obtained, including: According to the pre-set, the target data acquisition method is selected from the direct traversal method and the octree-based search strategy; Based on the target data acquisition method, generating a training data set; The optimal strategies for different proficiency levels corresponding to each action in the training data set are stored in the coaching agent to obtain a trained target system model.

3. The coaching action generation system based on the coaching agent according to claim 2 is characterized in that: The step of generating a training data set based on the target data acquisition method includes: When the target data acquisition method is a direct traversal method, based on the target system model, the optimal strategy corresponding to each action at different success rates is obtained and stored to obtain the corresponding training data set; When the target data acquisition method is an octree-based search strategy, all strategies are initialized, and based on the target system model, the optimal strategy detection is performed cyclically within the set boundary to obtain the optimal strategy corresponding to each action under different success rates, and after storage, the corresponding training data set is obtained.

4. The coaching action generation system based on the coaching agent according to claim 1 is characterized in that: The initial processing module includes: A task state generation unit is configured to obtain target action data in real time, generate a strategy for the current student agent based on the trained student agent of the target system model, obtain a current execution success rate, and generate a current task state based on the target action data, the strategy for the current student agent, and the current execution success rate; The target action data output unit is used to input the current task state into the coaching agent of the trained target system model, and the coaching agent of the trained target system model outputs the optimal target action data according to the current task state.

5. The coaching action generation system based on the coaching agent according to claim 4 is characterized in that: The task status generating unit includes: The first updating subunit is configured to obtain target action data in real time, generate a strategy based on the student agent of the trained target system model, obtain the number of executions of all target action data, and update the execution success rate of the current student agent based on the number of executions of all actions and a preset execution success rate calculation formula; The second updating subunit is used to generate the current task state according to the target action data, the strategy of the current student agent and the execution success rate.

6. The coaching action generation system based on the coaching agent according to claim 4 is characterized in that: The target action data output unit includes: a discrete estimation subunit, configured to input the current task state into the trained coaching agent of the target system model, wherein the coaching agent obtains an action set corresponding to all target action data based on the target action data, and correspondingly calculates a discrete estimation of the execution success rate of each action in the action set; The action output subunit is used to select the target optimal action according to all the discrete estimates and output it.

7. The coaching action generation system based on the coaching agent according to claim 1 is characterized in that: The iterative processing module includes: an execution success rate updating unit, configured to obtain an actual action corresponding to the target optimal action and update the corresponding execution success rate to obtain an updated current execution success rate; an iterative processing unit, configured to obtain target action data corresponding to the actual action, update the task state based on the student agent of the trained target system model, and perform iterative processing according to the trained target system model; The stop unit is used to end the iteration if all current tasks are completed and output the target action data obtained by each iteration.

8. A method for generating guidance actions based on a coaching agent, characterized in that: The coaching action generation method based on the coaching agent specifically includes: Acquire target action data in real time, generate a current task state based on the target action data based on the student agent of the trained target system model, input the current task state into the coach agent of the trained target system model, obtain and output the target optimal action, wherein the target system model includes a coach agent and a student agent, and select a target data acquisition method from a direct traversal method and an octree-based search strategy, acquire a training data set based on the target data acquisition, train the corresponding coach agent, and obtain the trained target system model; Obtain the actual action corresponding to the target optimal action, and obtain the target action data corresponding to the actual action, update the task status based on the student intelligent agent of the trained target system model, and perform iterative processing according to the trained target system model until the current task is completed, and output the target action data obtained from each iterative processing.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a coaching agent-based guidance action generation program stored in the memory and runnable on the processor. When the coaching agent-based guidance action generation program is executed by the processor, the steps of the coaching agent-based guidance action generation method as described in claim 8 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a coaching action generation program based on a coaching agent, and when the coaching action generation program based on a coaching agent is executed by a processor, the steps of the coaching action generation method based on a coaching agent as described in claim 8 are implemented.

Images