User guidance method and system based on teaching scene, and storage medium

Abstract

The invention provides a user guidance method and system based on a teaching scene and a storage medium, and belongs to the technical field of artificial intelligence, and the method comprises the steps: determining the type of the teaching scene according to a teaching guidance request sent by a user; if the category is that guidance takes effect immediately, acquiring first observation information, processing the first observation information, and inputting the processed first observation information into a trained first coach agent to obtain first guidance information for the user; and if the category indicates that the guidance delay takes effect, acquiring second observation information, processing the second observation information, and inputting the processed second observation information into a trained second coach agent to obtain second guidance information for the user. According to the invention, the corresponding coach agents are used for processing different teaching scenes, the guidance information for the user can be automatically obtained, the labor cost is saved, and the response speed of guidance is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a user guidance method, system and storage medium based on a teaching scenario. Background Art

[0002] In many fields requiring skill guidance (such as rehabilitation therapy and sports training), traditional approaches rely primarily on on-site demonstrations and real-time corrections by human coaches or instructors. For example, in rehabilitation therapy, physical therapists provide one-on-one guidance to patients on how to complete rehabilitation exercises, a time-consuming process that requires constant observation of the patient's movements. In sports training, coaches individually correct students' playing form and power techniques, limiting their efficiency due to manual effort and time. These traditional approaches result in high costs for manual guidance and limited response times.

[0003] Therefore, the existing technology has defects and needs to be improved and developed. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a user guidance method, device, terminal and storage medium based on teaching scenarios in response to the above-mentioned defects of the existing technology, aiming to solve the problems of high cost and limited response speed of human guidance in the existing technology.

[0005] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0006] In a first aspect, an embodiment of the present invention provides a user guidance method based on a teaching scenario, the method comprising:

[0007] Determine the category of the teaching scenario based on the teaching guidance request issued by the user;

[0008] If the category is immediate guidance, obtaining first observation information, processing it and inputting it into a trained first coaching agent to obtain first guidance information for the user;

[0009] If the category is delayed guidance, second observation information is obtained, processed and input into a trained second coaching agent to obtain second guidance information for the user.

[0010] In one embodiment, the first observation information includes user status data, teaching scene environment data, and user intended motion data; obtaining the first observation information, processing it and inputting it into a trained first coaching agent to obtain first guidance information for the user, including:

[0011] Obtain user status data, teaching scene environment data and user intention movement data;

[0012] Processing the user status data, the teaching scenario environment data and the user intention operation data into a first joint feature vector;

[0013] The first joint feature vector is input into the trained first coaching agent, and the action correction instruction with the highest reward value is selected and output as the first guidance information through the deep Q network inside the coaching agent.

[0014] In one embodiment, the second observation information includes user status data, teaching scene environment data, and user current motion data; obtaining the second observation information, processing it and inputting it into the second coaching agent to obtain second guidance information for the user includes:

[0015] Obtain user status data, teaching scene environment data, and user current motion data;

[0016] Processing the user state data, the teaching scene environment data and the user's current motion data into a second joint feature vector;

[0017] The second joint feature vector is input into the trained second coaching agent, and the action correction instruction with the highest reward value is selected and output as the second guidance information through the deep Q network inside the coaching agent.

[0018] In one embodiment, the training step of the first coach agent includes:

[0019] Constructing a first collaborative training framework comprising a first coach agent and a student agent, wherein the student agent and the first coach agent both comprise a deep Q-network;

[0020] Initializing a first task environment, where the first task environment is a simulation environment in which the first coach agent and its student agents interact;

[0021] Initializing the parameters of the deep Q network of the first coach agent and the parameters of the deep Q network in the student agent;

[0022] Iteratively training the first coach agent and the student agent within the first task environment, and resetting parameters of the deep Q network within the student agent after each predetermined number of training episodes;

[0023] The training is repeated until the preset total number of training episodes is reached, and a trained first coaching agent is obtained.

[0024] In one embodiment, iteratively performing training of the first coach agent and the student agent in the first task environment includes:

[0025] Iteratively performing training of the first coach agent and the student agent within the first task environment includes:

[0026] The trainee agent obtains the current task state of the first task environment, generates an intended action based on the current task state using its internal deep Q network, and reports a first state-action pair consisting of the current task state and the intended action to the first coach agent;

[0027] The first coaching agent generates guidance instructions based on the first state-action pair through its internal deep Q network, wherein the guidance instructions include action correction instructions and evaluation rewards;

[0028] The student agent integrates the intended action with the guidance instruction, generates the actual execution action and executes it;

[0029] The first task environment generates a task reward based on the actual execution action of the student agent and transfers to a new task state;

[0030] The student agent superimposes the task reward and the evaluation reward to update the parameters of its internal deep Q network, and the first coach agent updates the parameters of its internal deep Q network according to the task reward.

[0031] In one embodiment, the training step of the second coach agent includes:

[0032] Constructing a second collaborative training framework comprising a second coach agent and a student agent, wherein the student agent and the second coach agent both comprise a deep Q-network;

[0033] Initializing a second task environment, where the second task environment is a simulation environment in which the second coach agent and the student agent interact;

[0034] Initializing the parameters of the deep Q network of the second coach agent and the parameters of the deep Q network in the student agent;

[0035] Iteratively training the second coach agent and the student agent in the second task environment, resetting the policy parameters of the deep Q network in the student agent and clearing the historical instruction cache of the student agent after completing a preset number of training episodes;

[0036] The training is repeated until the preset total number of training episodes is reached, and a trained second coaching agent is obtained.

[0037] In one embodiment, iteratively performing training of the second coach agent and the student agent in the second task environment includes:

[0038] The student agent obtains the current task state of the second task environment, generates a state cluster based on the current task state, and determines whether a corresponding instruction snapshot exists in the historical instruction cache of the student agent based on the state cluster, wherein the instruction snapshot includes the state cluster, actions, and historical guidance instructions, and the historical guidance instructions include historical action correction instructions and historical evaluation rewards;

[0039] If there is no instruction snapshot corresponding to the state cluster, the student agent generates and executes a current action based on the state cluster, and reports a second state-action pair consisting of the state cluster and the current action to the second coach agent;

[0040] The second coach agent generates guidance instructions based on the second state-action pair through its internal deep Q network, and stores the state cluster and guidance instructions in a historical instruction cache of the trainee agent, wherein the guidance instructions include action correction instructions and evaluation rewards;

[0041] The second task environment generates a task reward according to the current action performed by the student agent and transfers to a new task state;

[0042] The student agent updates the parameters of its internal deep Q network based on the task reward, and the deep Q network in the second coach agent maintains the current parameters until the student agent finds the guidance instructions corresponding to the current state cluster in its historical instruction cache and executes them.

[0043] In one embodiment, iteratively performing training of the second coach agent and the student agent in the second task environment further includes:

[0044] If there is an instruction snapshot corresponding to the state cluster, the student agent obtains and executes a current action based on the corresponding instruction snapshot, and the second task environment generates a task reward according to the student agent's execution of the current action and transfers to a new task state;

[0045] The second coach agent updates the parameters of its internal deep Q network according to the task reward, and the student agent updates the parameters of its internal deep Q network according to the task reward and the historical evaluation reward in the historical guidance instructions.

[0046] In a second aspect, an embodiment of the present invention further provides a user guidance system based on a teaching scenario, the system comprising:

[0047] A category determination module is used to determine the category of the teaching scenario based on the teaching guidance request issued by the user;

[0048] a first guidance module, configured to obtain first observation information if the category is immediate guidance, process the first observation information and input it into a trained first coaching agent to obtain first guidance information for the user;

[0049] The second guidance module is used to obtain second observation information if the category is delayed guidance, process the second observation information and input it into the trained second coaching agent to obtain second guidance information for the user.

[0050] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium, which stores a user guidance program based on a teaching scenario, and the user guidance program based on a teaching scenario can be executed to implement the steps of the user guidance method based on a teaching scenario as described above.

[0051] The present invention has the following beneficial effects: Based on a user's instructional guidance request, the present invention determines the type of instructional scenario. If the instruction is immediately effective, first observation information is obtained, processed, and input into a trained first coaching agent to obtain first instructional information specific to the user. If the instruction is delayed effective, second observation information is obtained, processed, and input into a trained second coaching agent to obtain second instructional information specific to the user. By utilizing corresponding coaching agents for different instructional scenarios, the present invention automatically obtains instructional information specific to the user, saving labor costs and improving instructional response speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flowchart of a preferred embodiment of the user guidance method based on teaching scenarios in the present invention.

[0053] Figure 2 The first collaborative training framework in the present invention is a schematic diagram.

[0054] Figure 3 It is an interactive schematic diagram of the first collaborative training framework in the present invention.

[0055] Figure 4 It is a data interaction diagram of the first collaborative training framework in the present invention.

[0056] Figure 5 It is an interactive schematic diagram of the second collaborative training framework in the present invention.

[0057] Figure 6 It is a data interaction diagram of the second collaborative training framework in the present invention.

[0058] Figure 7 It is a structural diagram of a preferred embodiment of the user guidance system based on teaching scenarios in the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0060] In many fields requiring skill guidance (such as rehabilitation therapy and sports training), traditional approaches rely primarily on on-site demonstrations and real-time corrections by human coaches or instructors. For example, in rehabilitation therapy, physical therapists provide one-on-one guidance to patients on how to complete rehabilitation exercises, a time-consuming process that requires constant observation of the patient's movements. In sports training, coaches individually correct students' playing form and power techniques, limiting their efficiency due to manual effort and time. These traditional approaches result in high costs for manual guidance and limited response times.

[0061] To address the aforementioned shortcomings of the prior art, the present invention provides a user guidance method, system, and storage medium based on teaching scenarios. The method comprises: determining the category of the teaching scenario based on a teaching guidance request issued by a user; if the category indicates immediate guidance, obtaining first observation information, processing it, and inputting it into a trained first coaching agent to obtain first guidance information for the user; if the category indicates delayed guidance, obtaining second observation information, processing it, and inputting it into a trained second coaching agent to obtain second guidance information for the user. By utilizing corresponding coaching agents for different teaching scenarios, the present invention automatically obtains guidance information tailored to the user, saving labor costs and improving the response speed of guidance.

[0062] See Figure 1 The user guidance method based on the teaching scenario according to the embodiment of the present invention includes the following steps:

[0063] Step S100: Determine the category of the teaching scenario according to the teaching guidance request sent by the user.

[0064] Specifically, the trained first coaching agent and the trained second coaching agent are both deployed in the CPU or GPU of the target robot. The target robot can support calling corresponding coaching agents for different teaching scenarios to provide user guidance. A teaching guidance request can be in voice or text form. When the target robot receives a coaching guidance request from a user, it first determines the category of the teaching guidance request. If it is in voice form, the request is converted into text and then analyzed using a pre-trained language model to determine the user's request intent. If it is in text form, the pre-trained language model is directly used to analyze the user's request intent to determine the teaching scenario category. Language models can be BERT, GPT, etc. For example, if the request content includes "lower limb rehabilitation training guidance" or "balance training guidance guidance," the user classifies it as immediate guidance. If the request content includes ball sports such as "table tennis," "tennis," or "football," the user classifies it as delayed guidance. The difference between the two categories lies in the time it takes for the guidance to take effect. Immediate guidance means that the user can only take further action based on the guidance provided by the agent. The delayed effectiveness of guidance means that the user has currently completed the relevant action, and the guidance information will be used as a reference when performing the same action next time.

[0065] See Figure 1 The user guidance method based on the teaching scenario according to the embodiment of the present invention further includes the following steps:

[0066] Step S200: If the category is immediate guidance, first observation information is obtained, and after processing, it is input into a trained first coaching agent to obtain first guidance information for the user.

[0067] Specifically, the first observation information is generated based on the image collected by the target robot and the received teaching guidance request.

[0068] In one implementation, the first observation information includes user status data, teaching scene environment data, and user intended motion data; the first observation information is obtained, processed, and input into a trained first coaching agent to obtain first guidance information for the user, including:

[0069] Obtain user status data, teaching scene environment data and user intention movement data;

[0070] Processing the user status data, the teaching scenario environment data and the user intention operation data into a first joint feature vector;

[0071] The first joint feature vector is input into the trained first coaching agent, and the action correction instruction with the highest reward value is selected and output as the first guidance information through the deep Q network inside the coaching agent.

[0072] Specifically, user status data refers to the user's physiological status data, such as height and the range of motion angles of each joint. After the target robot captures an image of the user, it can use computer vision algorithms to estimate human posture to generate the user's physiological status data. Teaching scene environment data refers to data on facilities in the teaching scene. After the target robot captures an image of the teaching scene, it can use object detection algorithms to determine the location, dimensions, and other data of each facility in the teaching scene to generate the teaching scene environment data. When the present invention is classified as immediate guidance, it can be applied to user rehabilitation training scenarios. In this scenario, the teaching scene environment data can include the location and dimensions of rehabilitation equipment. User intended motion data refers to the user's intended action, i.e., the action the user plans to perform but has not yet performed. User intended motion data can be obtained from a teaching guidance request. If the user's teaching guidance request includes the content "perform lower limb rehabilitation exercises," the user intended motion data is determined to be action data related to lower limb movements, such as kicking the left leg forward 30 degrees. The user status data, teaching scene environment data, and user intended motion data are normalized and mapped into the same vector space to obtain a first joint feature vector. The first coaching agent includes a deep Q network (DQN). Due to its training, the DQN selects the corrective action with the highest reward (i.e., Q-value) as the first guidance information. This first guidance information can be output in voice or text form for the user's reference, such as "Please slightly bend your right knee to maintain balance, kick your left leg forward 30 degrees, and hold for 5 seconds."

[0073] See Figure 1 The user guidance method based on the teaching scenario according to the embodiment of the present invention further includes the following steps:

[0074] Step S300: If the category is delayed guidance, second observation information is obtained, and after processing, it is input into a trained second coaching agent to obtain second guidance information for the user.

[0075] Specifically, the second observation information includes user status data, teaching scene environment data, and user current motion data.

[0076] In one implementation, obtaining the second observation information, processing the second observation information and inputting the second observation information into the second coaching agent to obtain the second guidance information for the user includes:

[0077] Obtain user status data, teaching scene environment data, and user current motion data;

[0078] Processing the user state data, the teaching scene environment data and the user's current motion data into a second joint feature vector;

[0079] The second joint feature vector is input into the trained second coaching agent, and the action correction instruction with the highest reward value is selected and output as the second guidance information through the deep Q network inside the coaching agent.

[0080] Specifically, the present invention can be applied to scenarios where users are playing table tennis or tennis when the category is delayed guidance. Taking table tennis as an example, user status data may include the user's height, range of motion of each joint, etc. The teaching scene environment data may include the position of the table, the relative position of the user and the table, the relative position of the opponent and the table, etc. The user's current motion data can be calculated by the target robot using a visual tracking algorithm, specifically including the user's serve speed, serve type, serve angle, return type, return speed, return angle, etc. The user status data, teaching scene environment data, and user intention operation data are normalized and mapped to the same vector space to obtain a second joint feature vector. The second coaching agent also includes a deep Q network (DQN). Due to training, the deep Q network can select the action correction instruction with the highest reward value (i.e., Q value) as the second guidance information. The second guidance information can be output in the form of voice or text for the user's reference. For example, for the scenario where the user hits the ball with his forehand, the second guidance information generated may be: "Accelerate the forearm to exert force when hitting the ball, increase the proportion of forearm strength, relax the upper arm appropriately, and keep the arc low and flat."

[0081] In one implementation, the training step of the first coach agent includes:

[0082] Constructing a first collaborative training framework comprising a first coach agent and a student agent, wherein the student agent and the first coach agent both comprise a deep Q-network;

[0083] Initializing a first task environment, where the first task environment is a simulation environment in which the first coach agent and its student agents interact;

[0084] Initializing the parameters of the deep Q network of the first coach agent and the parameters of the deep Q network in the student agent;

[0085] Iteratively training the first coach agent and the student agent within the first task environment, and resetting parameters of the deep Q network within the student agent after each predetermined number of training episodes;

[0086] The training is repeated until the preset total number of training episodes is reached, and a trained first coaching agent is obtained.

[0087] Specifically, the present invention pre-constructs a first collaborative training framework in the training, which includes a first coaching agent and a student agent. Figure 2As shown in Figure 2. The student agent is located at the bottom interaction layer of the framework, simulating the student's self-learning process, and is the interaction between the student agent and the task environment. The first coach agent is located at the upper interaction layer of the framework, observing the interaction process between the student agent and the task environment and providing guidance to improve the student's action strategy. The interaction between the first coach agent and the student agent both includes a deep Q network. In the interaction between the first coach agent and the student agent, the student strategy π l Parameterized by its internal deep Q network, under the task state S, the student strategy π l Generate the intended action by selecting the action with the highest Q value And report the state-action pair consisting of task state and intended action to the first coaching agent. The coaching strategy π of the first coaching agent c It is parameterized by its internal deep Q network and generates guidance instructions based on state-action pairs And transmit it to the student agent. The student agent understands the instruction and updates the intention action, and obtains This process can be expressed as ). Then execute the new action. This process can be expressed as

[0088] The first task environment is a simulation environment that is consistent with the category of teaching scenarios with immediate effect of guidance. In this simulation environment, the task state S is set to user state data and teaching scenario environment data. The user state data can be user physiological state data, such as height, range of motion angles of each joint, etc. The teaching scenario environment data can be the position and size of rehabilitation training equipment, etc. The intended action is set to an action related to rehabilitation training. These can be obtained by collecting data from rehabilitation training users in real life. The instructions generated by the first teacher agent are instructions related to rehabilitation training.

[0089] After initializing the first task environment, the first coach agent and the student agent are trained alternately, and the parameters of the deep Q network in the student agent are reset every time a preset number of episodes of training are completed.

[0090] In one implementation, iteratively performing training of the first coach agent and the student agent in the first task environment includes:

[0091] The trainee agent obtains the current task state of the first task environment, generates an intended action based on the current task state using its internal deep Q network, and reports a first state-action pair consisting of the current task state and the intended action to the first coach agent;

[0092] The first coaching agent generates guidance instructions based on the first state-action pair through its internal deep Q network, wherein the guidance instructions include action correction instructions and evaluation rewards;

[0093] The student agent integrates the intended action with the guidance instruction, generates the actual execution action and executes it;

[0094] The first task environment generates a task reward based on the actual execution action of the student agent and transfers to a new task state;

[0095] The student agent superimposes the task reward and the evaluation reward to update the parameters of its internal deep Q network, and the first coach agent updates the parameters of its internal deep Q network according to the task reward.

[0096] Specifically, in the traditional teaching framework, the coach and the student agent usually share the same action set A. The coach only provides the optimal action based on the task state, and is essentially an elite student under the task. This study focuses on more general teaching scenarios, and the scope and methods of coach guidance have also been expanded. First, the content of guidance Includes evaluation information and corrective suggestions for the student's current actions / strategies as additional rewards, rather than directly providing the best action the student can perform. * ∈A. Second, the coach’s guidance should be consistent with the trainee’s current skill level (i.e., strategy π l ) to improve the efficiency of policy improvement, which is fundamentally different from existing work that provides the same guidance for the same task / environment state s regardless of the actual skill level of the individual learner. Therefore, the present invention defines the state of the upper-level reinforcement learning interaction as the "state-intention-action" pair in the lower-level interaction.<s,a> , which can be regarded as a sample of the student’s strategy under the current task. The coach agent uses its strategy π c :<s,a> →i provides guidance to students when necessary, and its state transition corresponds to the improvement of teaching strategies.

[0097] In the interactive teaching scenario, the self-learning of the student agent can be regarded as a traditional reinforcement learning problem, where the student improves its strategy π according to the environment reward l In this process, at each time step t, the student agent observes the current state of the environment Select Action When the student agent performs action a t Afterwards, the environment transitions to the new state and provides a scalar reward As an evaluation of the quality of the action. The ultimate goal of the student agent is to learn an optimal strategy To maximize the expected cumulative return The formula can be expressed as:

[0098]

[0099] Where τ = {(s0, a0, r0), (s1, a1, r1), ...} represents the trajectory generated by the agent following the strategy π, and p π represents the steady-state distribution of the state after the interaction between the strategy π and the environment, s0 follows p π The initial state of the distribution, V π (s) and Q π (s,a) are the state value and action value of strategy π respectively, is the expectation operator.

[0100] The goal of the coach agent is to guide students with different initial skill levels (i.e., student strategies) l,0 After a certain number of time steps T of training, the students converge to the optimal strategy π l,T The learning objective of the coaching agent is expressed as follows:

[0101]

[0102] Where, π l,0 Initial student strategy for students, is the expectation operator, π c is the teacher's strategy, π l,T is the student strategy after T time steps of guidance, d represents the evolution of the student strategy, s0 is the initial state of the task environment, is the value of the initial state, and the goal of the teacher agent is to maximize

[0103] The initial strategy π of the student agent l,0 Follow the prior distribution After the coaching agent uses the policy π c After training, the student’s final strategy is π l,T Similar to standard reinforcement learning, the training results of the student agent are determined by the state value under the final strategy. The goal of the coaching agent is to maximize the performance of various trainees after training. c , regardless of its initial policy before training. Similar to the student agent, the coach agent also learns from the task reward r. The improvement of the student agent's performance indirectly reflects the effectiveness of teaching.

[0104] The interactive diagram of the first collaborative training framework is as follows Figure 3As shown. The student agent declares its intended action to the coach agent before taking action. After receiving the student agent's intended action, the coach agent provides immediate instructions to the student, and the student agent can then immediately perform the corrected action according to the instructions. In this case, the effectiveness of the coach's instructions can be immediately evaluated based on the rewards obtained at the end of this round of training. Specifically, each iterative training includes the following steps: 1. The student agent and the coach agent first observe the current task state S t ; 2. Students follow the current strategy π l,t Report intended movement to the coach 3. The coach follows the current strategy π c,t Provide Instructions I t , including evaluation and reward of intended action and corrective action instruction; 4. The trainees understand the instruction and update the intention Without considering the execution deviation, the actual action 5. After the student performs the action, the task environment generates a reward R t And transition to the next state S t+1 6. The trainee updates his strategy based on the task reward and the coach’s evaluation reward for the intended action, and obtains π l,t+1 ; 7. The coach updates his strategy based on the task reward and obtains π c,t+1 The above steps complete one iteration of training. In the next iteration of training, the student agent and the coach agent continue the above interaction process, gradually optimizing the parameters of their respective deep Q networks, that is, optimizing their respective strategies.

[0105] The student agent learns from the task reward R and the evaluation reward ΔR, which can be expressed as: It should be noted that the task reward R is the evaluation of the first task environment on the action A actually performed by the student agent, while the evaluation reward ΔR is the evaluation of the first coach agent on the student's intended action Evaluation of. Experience tuple<S,A,R,S′> and is saved to the replay buffer D of the student agent l is used for policy update, where the symbol ′ represents the variable for the next step. S′ represents the next state that the task state S transitions to after action a. When the student agent adopts the instruction generated by the coach at a certain step, the coach will share the same task reward R as the evaluation of the coach's strategy. The coach's experience tuple is represented as When the next intention is observed After that, store the experience tuple into the replay buffer D c is used for coaching strategy update. The coaching agent’s strategy update follows the formula:

[0106]

[0107] In this case, the coach's instructions are based entirely on the student's expected movements. Rather than the actual action A. Figure 4 The data interaction diagram of the first collaborative training framework is shown. The ⊕ symbol in the figure is used to represent the trainee's understanding process of the instruction. It can be understood that Figure 4 The subscript t of each parameter in represents the parameter at time step t.

[0108] In addition, in order to balance the student's self-learning and teaching to make it closer to the real situation, the number of instructions in the strategy evaluation of the coaching agent is set to be limited. However, there is no such restriction in the training of the coaching agent (policy iteration), so that the coaching agent can update the strategy more efficiently. As the training progresses, the student's strategy π l Tends to converge, continuing to provide instructions to students whose strategies have converged may cause the coaching strategy to converge to a local optimum. To solve this problem, the problem of resetting the parameters of the deep Q network in the student agent can be solved by setting the hyperparameter ρ. ρ = 1 means that the parameters of the deep Q network in the student agent (i.e., the student strategy) will be reset in each scene, and the expected intentions in the corresponding situations are almost randomly selected at each step. The more frequently the student strategy is reset, the more diverse the "state-expected intention" that the student can propose, thereby enabling the coaching agent to explore more. When the training is repeated until the preset total number of training scenes E is reached, the coaching agent can explore more. c After that, the training process will terminate.

[0109] In one implementation, the training step of the second coach agent includes:

[0110] Constructing a second collaborative training framework comprising a second coach agent and a student agent, wherein the student agent and the second coach agent both comprise a deep Q-network;

[0111] Initializing a second task environment, where the second task environment is a simulation environment in which the second coach agent and the student agent interact;

[0112] Initializing the parameters of the deep Q network of the second coach agent and the parameters of the deep Q network in the student agent;

[0113] Iteratively training the second coach agent and the student agent in the second task environment, resetting the policy parameters of the deep Q network in the student agent and clearing the historical instruction cache of the student agent after completing a preset number of training episodes;

[0114] The training is repeated until the preset total number of training episodes is reached, and a trained second coaching agent is obtained.

[0115] Specifically, the structure of the second collaborative training framework is similar to that of the first coordinated training framework, except that the first coach agent is replaced by a second coach agent. This description will not be repeated here. The student agent, located at the bottom interaction layer of the framework, simulates the student's self-learning process and represents the interaction between the student agent and the task environment. The second coach agent, located at the upper interaction layer of the framework, observes the interaction between the student agent and the task environment and provides guidance to improve the student's action strategy. Both the second coach agent and the student agent's interactions involve a deep Q-network. Similar to the immediate instruction implementation, the hyperparameter ρ is designed to generate a more diverse set of experience tuples. The student's historical instruction cache M and the parameters of the deep Q-network within it are reset every p episodes. The second task environment is a simulation environment consistent with the teaching scenario category of delayed instruction implementation. In this simulation environment, the task state S is set to user state data and teaching scenario environment data. The user state data can be user physiological state data, such as the range of joint motion angles. The teaching scenario environment data can be the position of the ping-pong ball, the table, and the relative position of the user and the table. The actions are set to be table tennis-related actions. These can be obtained by collecting data from real-life table tennis users. The instructions generated by the second teacher agent are instructions related to table tennis.

[0116] In one implementation, iteratively performing training of the second coach agent and the student agent in the second task environment includes:

[0117] The student agent obtains the current task state of the second task environment, generates a state cluster based on the current task state, and determines whether a corresponding instruction snapshot exists in the historical instruction cache of the student agent based on the state cluster, wherein the instruction snapshot includes the state cluster, actions, and historical guidance instructions, and the historical guidance instructions include historical action correction instructions and historical evaluation rewards;

[0118] If there is no instruction snapshot corresponding to the state cluster, the student agent generates and executes a current action based on the state cluster, and reports a second state-action pair consisting of the state cluster and the current action to the second coach agent;

[0119] The second coach agent generates guidance instructions based on the second state-action pair through its internal deep Q network, and stores the state cluster and guidance instructions in a historical instruction cache of the trainee agent, wherein the guidance instructions include action correction instructions and evaluation rewards;

[0120] The second task environment generates a task reward according to the current action performed by the student agent and transfers to a new task state;

[0121] The student agent updates the parameters of its internal deep Q network based on the task reward, and the deep Q network in the second coach agent maintains the current parameters until the student agent finds the guidance instructions corresponding to the current state cluster in its historical instruction cache and executes them.

[0122] Specifically, the mechanism of delayed instruction adoption brings two major challenges to student teaching: low instruction utilization efficiency and outdated instructions. First, the requirement of state rematching reduces the efficiency of evaluating and updating the coaching strategy, especially for large-scale and continuous problems. To this end, methods such as state clustering are needed to classify the state s into a coarser-grained discrete state space φ(s) so that state clusters can share the same instructions and increase the probability of instruction replay. Second, when returning to the same state cluster φ(S t )=φ(S j ), the memorized instruction information is for A j , which may be related to the current student strategy or action No match. This means that past recommendations I j It cannot be used directly to correct the current action of the student agent. Instead, the complete history information including actions and instructions should be used to deduce the current action A t =A j ⊕I j , to achieve more reasonable instruction adoption and evaluation.

[0123] Specifically, the interaction diagram of the second collaborative training framework is as follows: Figure 5 As shown, the data interaction diagram of the second collaborative training framework is as follows Figure 6 As shown, Figure 6The subscript t in each parameter represents that parameter at time step t. After the student agent obtains the current task state of the second task environment, it maps this current task state to discrete categories, obtaining the state cluster φ(s) corresponding to this task state. Then, using this state cluster as a query, it searches the student agent's historical instruction cache M to determine whether a corresponding instruction snapshot exists. The instruction snapshot contains the state cluster φ(s), action A, and historical guidance instructions I. If no corresponding instruction snapshot exists, the student agent generates and executes the current action based on the task cluster, reporting the second state-action pair formed by the current state cluster and the current action to the second coach agent. The second coach agent generates guidance instructions based on this second state-action pair using its internal deep Q-network and stores the state cluster, current action, and guidance instructions as instruction snapshots in the student agent's historical instruction cache M. The second task environment generates a task reward based on the student agent's execution of the current action and transitions to a new task state. Based on the new task state, the student agent obtains a new task cluster corresponding to the task state, stores the experience tuple <φ(S), A, ΔR, φ(S′)> in its internal replay buffer, and updates the parameters of its internal deep Q-network based on the task reward. The parameters of the deep Q-network in the teacher agent are not updated at this time. They need to wait until the instructions generated by the teacher are in the history buffer and are called by the student agent before they can be updated.

[0124] In one implementation, iteratively performing training of the second coach agent and the student agent in the second task environment further includes:

[0125] If there is an instruction snapshot corresponding to the state cluster, the student agent obtains and executes a current action based on the corresponding instruction snapshot, and the second task environment generates a task reward according to the student agent's execution of the current action and transfers to a new task state;

[0126] The second coach agent updates the parameters of its internal deep Q network according to the task reward, and the student agent updates the parameters of its internal deep Q network according to the task reward and the historical evaluation reward in the historical guidance instructions.

[0127] Specifically, if there is an instruction snapshot associated with the state cluster, the current action is obtained and executed based on the actions and historical guidance instructions in all associated instruction snapshots. The second task environment generates a task reward based on the student agent's execution of the current action and transfers to a new task state; since the guidance instruction generated by the second coach agent is executed at this time, the experience tuple <φ(S), A, I, R, φ(S′), A′> is saved to the replay buffer D of the second coach agent. cThe second trainer agent updates the parameters of its internal deep Q-network based on the task reward. The student agent updates the parameters of its internal deep Q-network based on the task reward and the historical evaluation rewards in the historical guidance instructions.

[0128] In summary, the present invention determines the type of teaching scenario based on a user's instructional guidance request. If the instruction is immediate, first observation information is obtained, processed, and input into a trained first coaching agent to obtain first instructional information specific to the user. If the instruction is delayed, second observation information is obtained, processed, and input into a trained second coaching agent to obtain second instructional information specific to the user. By utilizing corresponding coaching agents for different instructional scenarios, the present invention automatically obtains instructional information specific to the user, saving labor costs and improving instructional response speed.

[0129] In one embodiment, if Figure 7 As shown, based on the above-mentioned user guidance method based on teaching scenarios, the present invention also provides a user guidance system based on teaching scenarios, the system comprising:

[0130] A category determination module is used to determine the category of the teaching scenario based on the teaching guidance request issued by the user;

[0131] a first guidance module, configured to obtain first observation information if the category is immediate guidance, process the first observation information and input it into a trained first coaching agent to obtain first guidance information for the user;

[0132] The second guidance module is used to obtain second observation information if the category is delayed guidance, process the second observation information and input it into the trained second coaching agent to obtain second guidance information for the user.

[0133] It should be noted that the aforementioned explanation of the embodiment of the user guidance method based on the teaching scenario is also applicable to the user guidance system based on the teaching scenario of this embodiment, and will not be repeated here.

[0134] An embodiment of the present invention also provides a computer-readable storage medium, on which a user guidance program based on a teaching scenario is stored. When the user guidance program based on a teaching scenario is executed by a processor, the steps of any one of the user guidance methods based on a teaching scenario provided in an embodiment of the present invention are implemented.

[0135] It should be understood that the sequence numbers of the steps in the above embodiments do not imply a specific order of execution; the order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0136] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0137] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0138] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0139] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / terminal device and method can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For example, the division of the modules or units described above is merely a logical functional division. In actual implementation, other division methods may be used. For example, multiple units or components may be combined or integrated into another device, or some features may be omitted or not implemented.

[0140] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A user guidance method based on teaching scenarios, characterized in that: The method comprises: Determine the category of the teaching scenario based on the teaching guidance request issued by the user; If the category is immediate guidance, obtaining first observation information, processing it and inputting it into a trained first coaching agent to obtain first guidance information for the user; If the category is delayed guidance, second observation information is obtained, processed and input into a trained second coaching agent to obtain second guidance information for the user.

2. The user guidance method based on teaching scenarios according to claim 1, characterized in that: The first observation information includes user status data, teaching scene environment data and user intended motion data; Obtaining first observation information, processing it and inputting it into a trained first coaching agent to obtain first guidance information for the user, including: Obtain user status data, teaching scene environment data and user intention movement data; Processing the user status data, the teaching scenario environment data and the user intention operation data into a first joint feature vector; The first joint feature vector is input into the trained first coaching agent, and the action correction instruction with the highest reward value is selected and output as the first guidance information through the deep Q network inside the coaching agent.

3. The user guidance method based on teaching scenarios according to claim 2, characterized in that: The second observation information includes user status data, teaching scene environment data, and user current motion data; Obtaining the second observation information, processing it and inputting it into the second coaching agent to obtain second guidance information for the user, including: Obtain user status data, teaching scene environment data, and user current motion data; Processing the user state data, the teaching scene environment data and the user's current motion data into a second joint feature vector; The second joint feature vector is input into the trained second coach agent, and the action correction instruction with the highest reward value is selected and output as the second guidance information through the deep Q network inside the coach agent.

4. The user guidance method based on teaching scenarios according to claim 1, characterized in that: The training steps of the first coach agent include: Constructing a first collaborative training framework comprising a first coach agent and a student agent, wherein the student agent and the first coach agent both comprise a deep Q-network; Initializing a first task environment, where the first task environment is a simulation environment in which the first coach agent and its student agents interact; Initializing the parameters of the deep Q network of the first coach agent and the parameters of the deep Q network in the student agent; Iteratively training the first coach agent and the student agent within the first task environment, and resetting parameters of the deep Q network within the student agent after each predetermined number of training episodes; The training is repeated until the preset total number of training episodes is reached, and a trained first coaching agent is obtained.

5. The user guidance method based on teaching scenarios according to claim 4 is characterized in that: Iteratively performing training of the first coach agent and the student agent within the first task environment includes: The trainee agent obtains the current task state of the first task environment, generates an intended action based on the current task state using its internal deep Q network, and reports a first state-action pair consisting of the current task state and the intended action to the first coach agent; The first coaching agent generates guidance instructions based on the first state-action pair through its internal deep Q network, wherein the guidance instructions include action correction instructions and evaluation rewards; The student agent integrates the intended action with the guidance instruction, generates the actual execution action and executes it; The first task environment generates a task reward based on the actual execution action of the student agent and transfers to a new task state; The student agent superimposes the task reward and the evaluation reward to update the parameters of its internal deep Q network, and the first coach agent updates the parameters of its internal deep Q network according to the task reward.

6. The user guidance method based on teaching scenarios according to claim 1, characterized in that: The training steps of the second coach agent include: Constructing a second collaborative training framework comprising a second coach agent and a student agent, wherein the student agent and the second coach agent both comprise a deep Q-network; Initializing a second task environment, where the second task environment is a simulation environment in which the second coach agent and the student agent interact; Initializing the parameters of the deep Q network of the second coach agent and the parameters of the deep Q network in the student agent; Iteratively training the second coach agent and the student agent in the second task environment, resetting the policy parameters of the deep Q network in the student agent and clearing the historical instruction cache of the student agent after completing a preset number of training episodes; The training is repeated until the preset total number of training episodes is reached, and a trained second coaching agent is obtained.

7. The user guidance method based on teaching scenarios according to claim 6, characterized in that: Iteratively performing training of a second coach agent and the student agent in the second task environment includes: The student agent obtains the current task state of the second task environment, generates a state cluster based on the current task state, and determines whether a corresponding instruction snapshot exists in the historical instruction cache of the student agent based on the state cluster, wherein the instruction snapshot includes the state cluster, actions, and historical guidance instructions, and the historical guidance instructions include historical action correction instructions and historical evaluation rewards; If there is no instruction snapshot corresponding to the state cluster, the student agent generates and executes a current action based on the state cluster, and reports a second state-action pair consisting of the state cluster and the current action to the second coach agent; The second coach agent generates guidance instructions based on the second state-action pair through its internal deep Q network, and stores the state cluster and guidance instructions in a historical instruction cache of the trainee agent, wherein the guidance instructions include action correction instructions and evaluation rewards; The second task environment generates a task reward according to the current action performed by the student agent and transfers to a new task state; The student agent updates the parameters of its internal deep Q network based on the task reward, and the deep Q network in the second coach agent maintains the current parameters until the student agent finds the guidance instructions corresponding to the current state cluster in its historical instruction cache and executes them.

8. The user guidance method based on teaching scenarios according to claim 7, characterized in that: Iteratively performing training of the second coach agent and the student agent in the second task environment further includes: If there is an instruction snapshot corresponding to the state cluster, the student agent obtains and executes a current action based on the corresponding instruction snapshot, and the second task environment generates a task reward according to the student agent's execution of the current action and transfers to a new task state; The second coach agent updates the parameters of its internal deep Q network according to the task reward, and the student agent updates the parameters of its internal deep Q network according to the task reward and the historical evaluation reward in the historical guidance instructions.

9. A user guidance system based on teaching scenarios, characterized in that: include: A category determination module is used to determine the category of the teaching scenario based on the teaching guidance request issued by the user; a first guidance module, configured to obtain first observation information if the category is immediate guidance, process the first observation information and input it into a trained first coaching agent to obtain first guidance information for the user; The second guidance module is used to obtain second observation information if the category is delayed guidance, process the second observation information and input it into the trained second coaching agent to obtain second guidance information for the user.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a user guidance program based on a teaching scenario. When the user guidance program based on a teaching scenario is executed by a processor, the steps of the user guidance method based on a teaching scenario as described in any one of claims 1 to 8 are implemented.

Images