Model training method and virtual coach guidance strategy generation method

Abstract

The application discloses a model training method and a virtual coach guidance strategy generation method, and relates to the technical field of computers. The method comprises the following steps: acquiring a current state of a user after the user performs a current training action; wherein the current state comprises any multiple items of an action quality score, a fatigue degree score, a self-satisfaction score, an understanding degree score and sensor data of a sports device; inputting the current state into a DQN model to obtain Q values of each strategy in a preset action space; selecting a current strategy from the action space based on the Q values of each strategy; performing the current strategy; acquiring a next state of the user after the user performs a next training action in response to the execution of the current strategy; and adjusting parameters of the DQN model based on the current state, the next state and the current strategy. The embodiment can provide more targeted training guidance for the user and improve the training quality of the user.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a model training method and a virtual coach guidance strategy generation method. BACKGROUND

[0002] In the fields of sports training and bodybuilding, scientific and personalized training guidance is crucial for improving results, avoiding injuries and maintaining motivation. Traditional face-to-face coaching mode is limited by time, region, cost and coach resources, and it is difficult to meet the growing demand for personalized and convenient training. In recent years, virtual coach systems based on artificial intelligence, sensor technology and human-computer interaction have developed rapidly, providing an accessible, scalable and cost-effective alternative or complementary solution for users.

[0003] Existing methods generally provide training guidance for users based on pre-set and universal training rule libraries. Users usually choose a fixed set of rules according to goals (such as fat loss, muscle gain and endurance improvement) or experience levels. These rules lack adaptability to individual conditions and dynamic changes, affecting the training quality of users. SUMMARY

[0004] Therefore, the embodiments of the present application provide a model training method and a virtual coach guidance strategy generation method, which can provide more targeted training guidance for users based on their specific training conditions and improve the training quality of users.

[0005] In a first aspect, the embodiments of the present application provide a model training method, comprising:

[0006] obtaining a current state of a user after performing a current training action; wherein the current state comprises any one or more of an action quality score, a fatigue level score, a self-satisfaction score, an understanding level score and sensor data of a sports device, the action quality score is used to represent the execution quality of the current training action, the fatigue level score is used to represent the fatigue level of the user, the self-satisfaction score is used to represent the satisfaction of the user for performing the current training action, and the understanding level score is used to represent the understanding level of the user for the current training action;

[0007] inputting the current state into a DQN (DeepQ-Network, DeepQ-Network) model to obtain Q values of each strategy in a preset action space; wherein the action space comprises any one or more of outputting a preset encouraging feedback, outputting a correction instruction, outputting a rest suggestion, outputting a query question and no output, which are used as strategies for the virtual coach to guide the user training;

[0008] selecting a current strategy from the action space based on the Q values of each strategy;

[0009] execute the current policy;

[0010] after the user responds to the current policy to execute a next training action, obtaining a next state of the user;

[0011] based on the current state, the next state and the current policy, adjusting parameters of the DQN model.

[0012] In a second aspect, the embodiments of the present application provide a method for generating a virtual trainer guidance strategy, comprising:

[0013] Based on the DQN model trained by the method described in the above embodiments, a strategy for the virtual trainer to guide the target user in training is generated.

[0014] In a third aspect, the embodiments of the present application provide a computer program product comprising a computer program, which, when executed by a processor, implements the method according to any of the above embodiments.

[0015] One embodiment of the above invention has the following advantages or beneficial effects: The DQN model is trained based on multiple dimension data such as action quality and fatigue degree, so that it can learn the optimal response strategy for multiple dimensions of the user, and adjust the guidance strategy in time according to the change of the user state, to provide more personalized training guidance for the user. For example, if the user easily completes the set task, the DQN model can automatically increase the difficulty by adjusting the strategy to maintain effective stimulation; if the user is obviously struggling or the action is deformed, it can timely reduce the difficulty or suggest a rest to prevent ineffective training or injury. In addition, the training process also considers the internal state of the user such as fatigue degree, self-satisfaction degree and understanding degree, which can further improve the guidance quality of the virtual trainer. The data and experience accumulated by the DQN model will continuously optimize its decision-making strategy, so that the guidance ability of the virtual trainer becomes stronger and more accurate.

[0016] The further effects of the above non-conventional optional mode will be described in the following combined with the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings are used to better understand the present application and do not constitute an improper limitation on the present application. Among them:

[0018] Figure 1 is a flowchart of a model training method provided by one embodiment of the present application;

[0019] Figure 2 is a schematic diagram of a model training device provided by one embodiment of the present application. DETAILED DESCRIPTION

[0020] Exemplary embodiments of the present application are described herein with reference to the accompanying drawings, which are cited by way of example only. The detailed description of embodiments of the present application is not intended to limit the scope of the present application as this is described by the appended claims. It should be understood that various modifications can be made to the embodiments described herein, and the general principles described herein can be applied to other embodiments and applications without departing from the scope of the present application. For the purposes of simplicity and clarity, detailed descriptions of well-known methods, techniques, and apparatuses are omitted so as not to unnecessarily obscure the embodiments of the present application.

[0021] As shown in Figure 1 The present application provides a model training method, comprising:

[0022] Step 101: Obtain the current state of the user after performing the current training action.

[0023] The method is applied to a virtual coach system. The current state includes any one or more of an action quality score, a fatigue level score, a self-satisfaction score, an understanding level score, and sensor data of a sports device. The action quality score is used to represent the execution quality of the current training action. The fatigue level score is used to represent the fatigue level of the user. The self-satisfaction score is used to represent the satisfaction level of the user for performing the current training action. The understanding level score is used to represent the understanding level of the user for the current training action.

[0024] The state of the user, including the current state or the next state, can be obtained through visual data, voice data, sensor data of a sports device, and interface interaction data. The visual data is obtained by capturing the motion video stream of the user through a camera. The voice data refers to the voice interaction content between the user and the virtual coach, including answers to specific questions and spontaneous speech, etc. The sensor data refers to the kinematics or dynamics data collected by a wearable device or a training instrument. The interface interaction data refers to the data generated by the user operating the software interface provided by the virtual coach system.

[0025] The action quality score can include the score of one or more action dimensions. For example, for a deep squat, there are 5 working dimensions: squat depth, knee trajectory, back posture, hip joint angle, and action speed, which evaluate the training action of deep squat from different dimensions.

[0026] For example, the score of the action dimension has three types. If the quality of the action dimension is poor or needs to be corrected, the score is 0. If the quality of the action dimension is medium or acceptable, the score is 1. If the quality of the action dimension is excellent or standard, the score is 2. The quality or score of the action dimension can be determined by comparing with a preset standard action, and the calculation method of the action quality score is not limited herein. In actual application scenarios, the method can directly obtain the scores of each item of the current state, and can also calculate the scores of each item through the obtained video data, etc.

[0027] The fatigue degree score can be obtained through voice data and interface interaction data between the user and the virtual trainer, for example, the virtual trainer actively asks the user about the current fatigue degree, if it is low, the fatigue degree score is 0, if it is medium, the fatigue degree score is 1, and if it is high, the fatigue degree score is 2.

[0028] The self-satisfaction score can be obtained through voice data and interface interaction data between the user and the virtual trainer, for example, the user actively expresses the satisfaction degree of performing the current training action, if not satisfied, the self-satisfaction score is 0, if general, the self-satisfaction score is 1, and if satisfied, the self-satisfaction score is 2.

[0029] The understanding degree score can be obtained through voice data and action quality scores between the user and the virtual trainer, for example, if the user expresses misunderstanding or the action essentials repeatedly appear errors, the understanding degree is low, the understanding degree score is 0, if the user has partial uncertainty or the action essentials occasionally appear errors, the understanding degree is medium, the understanding degree score is 1, and if the action essentials do not appear errors or the user does not express doubts, the understanding degree is high, the understanding degree score is 2.

[0030] The sensor data corresponding to the sports equipment can include a plurality of continuous numerical values, such as a peak force, i.e., a peak value of the force, an average angular velocity, a maximum range of activity, etc.

[0031] The current training action can be an action or an action unit of repeating the same action, for example, the action unit includes 3 times of deep squats.

[0032] Step 102: input the current state into the DQN model to obtain the Q values of each policy in the preset action space.

[0033] The action space includes any one or more of the following strategies for the virtual trainer to guide the user in training: outputting a preset encouraging feedback, outputting a correction instruction, outputting a rest suggestion, outputting a query question, and no output.

[0034] The current state is input into the DQN model in the form of a vector, and if the current state is not in the form of a vector, it needs to be preprocessed, and the specific process will be further described in subsequent embodiments.

[0035] The encouraging feedback is used to encourage the user to maintain the action execution quality, for example, outputting encouraging speeches such as “well done” and “hold on”.

[0036] The correction instruction is used to correct the training action, for example, outputting voice correction instructions such as “pay attention to not to buckle the knees” and “keep the core tight and the back straight”.

[0037] The rest suggestion is used to suggest the user to take a short break, such as "please take a break for a while".

[0038] The inquiry question is used to inquire the user to know the feeling or understanding degree of the user, such as "how do you feel now", "is the action point clear".

[0039] The encouraging feedback, the correction instruction, the rest suggestion and the inquiry question can be output in the form of text or voice.

[0040] No output means that the virtual trainer does not need to intervene at present, and can keep silent to continue to observe the action of the user.

[0041] Step 103: selecting the current strategy from the action space based on the Q value of each strategy.

[0042] The embodiment of the application adopts the ε-greedy strategy, and selects the strategy based on the Q value output by the current network in the DQN model.

[0043] Step 104: executing the current strategy.

[0044] The virtual trainer as an agent can execute the current strategy, and the user can execute the next training action according to the current strategy.

[0045] For example, the user performs 5 times of deep squats in succession, and the virtual trainer gives the correction instruction "straighten the back", and the user adjusts the posture and performs 5 times of deep squats again, wherein the 5 times of deep squats are an action unit.

[0046] Step 105: obtaining the next state of the user after the user responds to the execution of the next training action of the current strategy.

[0047] Step 106: adjusting the parameters of the DQN model based on the current state, the next state and the current strategy.

[0048] The embodiment of the application trains the DQN model based on multiple dimension data such as action quality and fatigue degree, so that it can learn the optimal response strategy for multiple dimensions of the user, and timely adjust the guidance strategy according to the change of the state of the user, to provide more personalized training guidance for the user. For example, if the user easily completes the set task, the DQN model can automatically increase the difficulty by adjusting the strategy to maintain effective stimulation; if the user is obviously tired or the action is deformed, it can timely reduce the difficulty or suggest to take a break to prevent ineffective training or injury. In addition, the training process also considers the internal state of the user such as fatigue degree, self-satisfaction degree and understanding degree, which can further improve the guidance quality of the virtual trainer. The data and experience accumulated by the DQN model will continuously optimize its decision strategy, so that the guidance ability of the virtual trainer is stronger and more accurate.

[0049] In an embodiment of the present application, the method further comprises:

[0050] In response to the current item being a discrete value, converting the discrete value into a numerical vector through one-hot encoding or an embedding layer; wherein the action quality score, the fatigue level score, the self-satisfaction score, and the understanding level score are discrete values, and the sensor data of the sports equipment is a continuous value;

[0051] In response to the current item being a continuous value, normalizing the current item;

[0052] Inputting the current state into the DQN model, including:

[0053] After concatenating the vectors corresponding to each item in the current state, inputting the DQN model.

[0054] For example, the current state includes the action quality score, the fatigue level score, and the sensor data of the sports equipment, converting the action quality score and the fatigue level score into numerical vectors, normalizing the sensor data of the sports equipment, and inputting the processed items into the DQN model after concatenation.

[0055] The current state s obtained after preprocessing can be a mixed type vector s=[q1,...q M , f, sat, u], wherein q1...q M are M continuous values representing scores of the same training action in M different action dimensions. For example, after normalization processing to the [0, 1] interval, the higher the value, the better the quality. F is used to represent the fatigue level score, which is a discrete value, sat is used to represent the self-satisfaction score, which is a discrete value, and u is used to represent the understanding level score, which is a discrete value.

[0056] In an embodiment of the present application, based on the current state, the next state, and the current policy, adjusting the parameters of the DQN model, including:

[0057] Based on the current state, the next state, and the current policy, calculating the immediate reward;

[0058] Storing the experience composed of the current state, the current policy, the immediate reward, and the next state into the experience replay buffer;

[0059] Sampling a preset number of target experiences from the experience replay buffer;

[0060] Based on the target network in the DQN model, calculating the target Q value of each target experience;

[0061] Based on the current network in the DQN model, calculating the predicted Q value of each target experience;

[0062] Based on the target Q value and the predicted Q value of each target experience, calculating the value of the loss function;

[0063] The parameters of the DQN model are adjusted based on the value of the loss function.

[0064] The calculation process of the target Q value and the predicted Q value can refer to the training process of the existing DQN model, which will not be described here. In this embodiment of the application, mean square error calculation can be used.

[0065] In this embodiment of the application, the action space is constructed based on the training construction state of the user and the guidance strategy of the virtual coach, and the DQN model is trained, so that the model can learn a personalized strategy, and then high-quality training guidance can be provided for the user.

[0066] In an embodiment of the application, the immediate reward is calculated based on the current state, the next state and the current strategy, including:

[0067] The action quality reward is calculated based on the action quality score in the current state and the action quality score in the next state;

[0068] The user state reward is calculated based on the self-satisfaction score, the fatigue degree score and the understanding degree score in the next state, and the understanding degree score in the current state;

[0069] The training progress reward is calculated based on the sensor data of the exercise equipment in the next state;

[0070] The strategy cost penalty is calculated based on the current strategy;

[0071] The action quality reward, the user state reward, the training progress reward and the strategy cost penalty are weighted and summed to obtain the immediate reward.

[0072] For example, the immediate reward is calculated by using formula (1).

[0073] r t =w q ×R quality +w us ×R user_state +w p ×R progression +w c ×R action_cost (1)

[0074] Wherein, r t is used to represent the immediate reward, R quality is used to represent the action quality reward, R user_state is used to represent the user state reward, R progression is used to represent the training progress reward, R action_cost is used to represent the strategy cost penalty, w q , w us , and w pw c are corresponding weights respectively.

[0075] The embodiment of the present application calculates the instant reward generated by the current strategy of the virtual trainer from multiple dimensions such as action quality, user state, etc. Since the instant reward considers multiple different dimensions, it can more comprehensively and accurately measure the current strategy, select a more suitable strategy for the user, and further improve the quality of training guidance. In actual application scenarios, the calculation method of the instant reward can also be adjusted according to business needs, such as calculating the instant reward based on action quality reward and user state reward.

[0076] In addition, the embodiment of the present application calculates the instant reward by using the weighted summation method, which can further reflect the importance difference of each dimension and improve the accuracy of instant reward calculation. The rewards or penalties such as action quality reward can also be calculated in different ways, and only one of the calculation methods is described in detail below.

[0077] In an embodiment of the present application, the action quality score in the current state and the action quality score in the next state each include scores of multiple action dimensions.

[0078] Based on the action quality score in the current state and the action quality score in the next state, the action quality reward is calculated, including:

[0079] Based on the scores of multiple action dimensions in the next state, a basic action quality reward is calculated.

[0080] Based on the score difference between the next state and the current state in each action dimension, an action quality reward increment is calculated.

[0081] Based on the basic action quality reward and the action quality reward increment, the action quality reward is calculated.

[0082] The action quality reward can be calculated by formula (2).

[0083]

[0084] wherein q t+1，i is the score of action dimension i in the next training action, q t，i is the score of action dimension i in the current training action, w q1 is a weight for representing the basic action quality reward, w q2 is a weight for representing the action quality reward increment.

[0085] The embodiment of the present application encourages the user to improve the quality of the training action through the action quality reward. The embodiment

[0086] The action quality reward is more accurate to reflect the quality of the current policy by considering not only the quality of the next state but also the change of the next state relative to the current state.

[0087] In an embodiment of the present application, the training progress reward is calculated based on the sensor data of the sports equipment in the next state, comprising:

[0088] Based on the sensor data of the sports equipment in the next state, it is determined whether the user completes the next training action;

[0089] In response to completing the next training action, the training progress reward is determined as a preset first threshold value;

[0090] In response to not completing the next training action, the training progress reward is determined as a preset second threshold value;

[0091] Wherein, the first threshold value is greater than the second threshold value, and the first threshold value is a positive number.

[0092] In order to encourage the user to complete the training action, the present application gives greater reward to the completed training action, for example, completing the next training action, R progression =C prog , wherein C prog is the first threshold value. In actual application scenarios, not completing the next training action can not encourage or punish the user, for example, the second threshold value is 0 or negative.

[0093] In an embodiment of the present application, wherein in the case that the action quality reward, the user state reward, and the training progress reward are fixed, the instant reward corresponding to any of the strategies of outputting the preset encouraging feedback, outputting the correction instruction, outputting the rest suggestion, and outputting the inquiry question is less than the instant reward corresponding to no output.

[0094] No output means no need for virtual trainer intervention. In order to avoid too much intervention, the embodiment of the present application encourages the strategy of no output.

[0095] For example, for outputting the preset encouraging feedback, outputting the correction instruction, outputting the rest suggestion, and outputting the inquiry question, R action_cost =-C cost , for no output, R action_cost =0, and C cost is used to represent the fixed cost of each active intervention of the virtual trainer.

[0096] In an embodiment of the present application, the user state reward is calculated based on the self-satisfaction score, the fatigue degree score, and the understanding degree score in the next state, and the understanding degree score in the current state, comprising:

[0097] determine a self-satisfaction reward based on the self-satisfaction score in the next state;

[0098] determine a fatigue degree reward based on the fatigue degree score in the next state;

[0099] determine an understanding degree reward based on the understanding degree score in the next state;

[0100] determine an understanding degree reward increment based on a difference between the understanding degree score in the next state and the understanding degree score in the current state;

[0101] calculate the user state reward based on the self-satisfaction reward, the fatigue degree reward, the understanding degree reward, and the understanding degree reward increment.

[0102] For example, if the self-satisfaction score is 2, the self-satisfaction reward is C sat , otherwise, the self-satisfaction reward is 0.

[0103] In the embodiments of the present application, the user state reward can also be directly calculated by formula (3).

[0104] R user_state = C sat × Π(sat t+1 = High) - P fat × Π(f t+1 = High) - P und × Π(u t+1 = Low) + C und_inc × Π(u t+1 > U t ) (3)

[0106] wherein Π(·) is an indicator function, outputting 1 when the condition in the parentheses is true, and 0 otherwise; and C sat , P fat , P und , C und_inc are preset positive numbers, respectively representing the rewards corresponding to the user reaching high satisfaction, being in a high fatigue state, being in a low understanding degree, and the understanding degree being improved compared with the current state, and the reward being negative is a punishment, sat t+1 is used to represent the self-satisfaction of the next state, f t+1 is used to represent the fatigue degree of the next state, u t+1 is used to represent the understanding degree of the next state, U t is used to represent the understanding degree of the current state, and High and Low respectively correspond to high and low degrees, and can also be replaced by scores, which are 2 and 0.

[0107] The embodiment of the application can encourage positive user states such as high understanding degree, punish negative states such as high fatigue degree, and encourage positive changes in states through understanding degree reward increments, thereby improving the training quality of the user.

[0108] The embodiment of the application provides a method for generating a virtual coach guidance strategy, comprising:

[0109] Based on the DQN model trained by any of the above embodiments, a strategy for guiding the target user to train by the virtual coach is generated.

[0110] Specifically, the state of the target user after performing the target training action is obtained, and the state is input into the trained DQN model to obtain the strategy for guiding the target user to train.

[0111] As shown in Figure 2 The embodiment of the application provides a model training device, comprising:

[0112] The acquisition module 201 is configured to acquire a current state of a user after performing a current training action; wherein the current state comprises any multiple of an action quality score, a fatigue degree score, a self-satisfaction score, an understanding degree score, and sensor data of a sports device, the action quality score is used to represent the execution quality of the current training action, the fatigue degree score is used to represent the fatigue degree of the user, the self-satisfaction score is used to represent the satisfaction degree of the user to the execution of the current training action, and the understanding degree score is used to represent the understanding degree of the user to the current training action;

[0113] The input module 202 is configured to input the current state into a deep Q network (DQN) model to obtain Q values of each strategy in a preset action space; wherein the action space comprises any multiple of outputting a preset encouraging feedback, outputting a correction instruction, outputting a rest suggestion, outputting a query question, and no output for a strategy for guiding the user to train by the virtual coach;

[0114] The processing module 203 is configured to select a current strategy from the action space based on the Q values of each strategy; and execute the current strategy.

[0115] The adjustment module 204 is configured to acquire a next state of the user after the user responds to the execution of the next training action by the current strategy; and adjust the parameters of the DQN model based on the current state, the next state, and the current strategy.

[0116] In an embodiment of the present application, the obtaining module 201 is configured to, in response to the current item being a discrete value, convert the discrete value into a numerical vector through one-hot encoding or an embedding layer; wherein the action quality score, the fatigue level score, the self-satisfaction score, and the understanding level score are discrete values, and the sensor data of the sports equipment is a continuous value; and in response to the current item being a continuous value, normalize the current item.

[0117] The input module 202 is configured to splice the vectors corresponding to the items in the current state and input the spliced vectors into the DQN model.

[0118] In an embodiment of the present application, the adjusting module 204 is configured to calculate an immediate reward based on the current state, the next state, and the current policy; store the experience composed of the current state, the current policy, the immediate reward, and the next state into an experience replay buffer; sample a preset number of target experiences from the experience replay buffer; calculate target Q values of the target experiences based on a target network in the DQN model; calculate predicted Q values of the target experiences based on a current network in the DQN model; calculate a value of a loss function based on the target Q values and the predicted Q values of the target experiences; and adjust the parameters of the DQN model based on the value of the loss function.

[0119] In an embodiment of the present application, the adjusting module 204 is configured to calculate an action quality reward based on the action quality score in the current state and the action quality score in the next state; calculate a user state reward based on the self-satisfaction score, the fatigue level score, and the understanding level score in the next state and the understanding level score in the current state; calculate a training progress reward based on the sensor data of the sports equipment in the next state; calculate a policy cost penalty based on the current policy; and obtain the immediate reward by weighted sum of the action quality reward, the user state reward, the training progress reward, and the policy cost penalty.

[0120] In an embodiment of the present application, the adjusting module 204 is configured to calculate a basic action quality reward based on the scores of the multiple action dimensions in the next state; calculate an action quality reward increment based on the score differences between the next state and the current state in each action dimension; and calculate the action quality reward based on the basic action quality reward and the action quality reward increment; wherein the action quality score in the current state and the action quality score in the next state both include the scores of the multiple action dimensions.

[0121] In an embodiment of the present application, the adjusting module 204 is configured to determine whether the user completes the next training action based on the sensor data of the sports equipment in the next state; in response to completing the next training action, determine that the training progress reward is a preset first threshold value; and in response to not completing the next training action, determine that the training progress reward is a preset second threshold value; wherein the first threshold value is greater than the second threshold value, and the first threshold value is a positive number.

[0122] In an embodiment of the present application, the adjustment module 204 is configured to determine a self-satisfaction reward based on the self-satisfaction score in the next state, determine a fatigue degree reward based on the fatigue degree score in the next state, determine an understanding degree reward based on the understanding degree score in the next state, determine an understanding degree reward increment based on the difference between the understanding degree score in the next state and the understanding degree score in the current state, and calculate the user state reward based on the self-satisfaction reward, the fatigue degree reward, the understanding degree reward, and the understanding degree reward increment.

[0123] An embodiment of the present application provides a virtual coach system, comprising:

[0124] The guidance module is configured to generate a strategy for guiding the target user to train based on the DQN model trained by the method in any of the above embodiments.

[0125] Specifically, the guidance module is configured to obtain a state of the target user after performing a target training action, input the state into the trained DQN model, and obtain a strategy for guiding the target user to train.

[0126] The virtual coach system further comprises a model training device, and the guidance module generates a strategy for guiding the target user to train based on the DQN model trained by the model training device.

[0127] An embodiment of the present application provides an electronic device, comprising:

[0128] one or more processors;

[0129] a storage device configured to store one or more programs,

[0130] When the one or more programs are executed by the one or more processors, the one or more processors implement the method in any of the above embodiments.

[0131] An embodiment of the present application provides a computer readable medium having a computer program stored thereon, and the program is executed by a processor to implement the method in any of the above embodiments.

[0132] It should be noted that computer-readable media in this disclosure can be computer-readable storage media, or computer-readable signal media, or any combination thereof. Computer-readable storage media can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this disclosure, computer-readable storage media can be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a computer-readable storage medium, or any computer-readable medium that can transmit or propagate code in the form of computer-readable instructions or program code, or any combination of the above. The computer-readable medium can be transmitted in baseband or as part of a carrier wave over a transmission medium, including a wired medium, or a wireless medium, or any suitable combination of the above. Computer-readable media can also be any medium that can be used to store or transfer a program for use by or in connection with an instruction execution system, apparatus, or device.

[0133] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functional processes, and operational processes, according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0134] The foregoing detailed description has set forth various embodiments of the devices and / or processes via the use of specific terminology. However, embodiments thereof can be practiced with the exact description not being set forth but with the same essence. Therefore, embodiments cannot be limited to the specific details or the exact implementation set forth above. Rather, it encompasses numerous modifications, alterations, and changes in the described embodiments.

Claims

1. A model training method, characterized in that, The method comprises: obtaining a current state of a user after performing a current training action, wherein the current state comprises any one or more of an action quality score, a fatigue degree score, a self-satisfaction score, an understanding degree score, and sensor data of a sports device, the action quality score is used to represent the execution quality of the current training action, the fatigue degree score is used to represent the fatigue degree of the user, the self-satisfaction score is used to represent the satisfaction degree of the user on performing the current training action, and the understanding degree score is used to represent the understanding degree of the user on the current training action; inputting the current state into a deep Q network (DQN) model to obtain Q values of each strategy in a preset action space, wherein the action space comprises any one or more of outputting a preset encouraging feedback, outputting a correction instruction, outputting a rest suggestion, outputting a query question, and no output, which are used for a virtual trainer to guide the user in training; selecting a current strategy from the action space based on the Q values of each strategy; executing the current strategy; obtaining a next state of the user after the user responds to the execution of the current strategy to perform a next training action; adjusting parameters of the DQN model based on the current state, the next state, and the current strategy; adjusting parameters of the DQN model based on the current state, the next state, and the current strategy, comprising: calculating an immediate reward based on the current state, the next state, and the current strategy; storing experience composed of the current state, the current strategy, the immediate reward, and the next state into an experience replay buffer; sampling a preset number of target experience from the experience replay buffer; calculating target Q values of each target experience based on a target network in the DQN model; calculating predicted Q values of each target experience based on a current network in the DQN model; calculating a value of a loss function based on the target Q values and the predicted Q values of each target experience; adjusting parameters of the DQN model based on the value of the loss function; calculating an immediate reward based on the current state, the next state, and the current strategy, comprising: calculating an action quality reward based on the action quality score in the current state and the action quality score in the next state; calculating a user state reward based on the self-satisfaction score, the fatigue degree score, and the understanding degree score in the next state, and the understanding degree score in the current state; calculating a strategy cost penalty based on the current strategy; calculating the immediate reward based on the action quality reward, the user state reward, and the strategy cost penalty; calculating an immediate reward based on the current state, the next state, and the current strategy, comprising: calculating an action quality reward based on the action quality score in the current state and the action quality score in the next state; calculating a user state reward based on the self-satisfaction score, the fatigue degree score, and the understanding degree score in the next state, and the understanding degree score in the current state; compute a training progress reward based on sensor data of the motion device in the next state; compute a policy cost penalty based on the current policy; weight-sum the action quality reward, the user state reward, the training progress reward, and the policy cost penalty to obtain the immediate reward.

2. The method of claim 1, wherein, Further comprising: in response to the current item being a discrete value, convert the discrete value into a numerical vector through one-hot encoding or an embedding layer; wherein the action quality score, the fatigue level score, the self-satisfaction score, and the understanding level score are the discrete values, and the sensor data of the motion device are continuous values; in response to the current item being a continuous value, normalize the current item; input the current state into a DQN model, comprising: concatenate the vectors corresponding to each item in the current state, and input the concatenated vectors into the DQN model.

3. The method of claim 1, characterized in that wherein the action quality score in the current state and the action quality score in the next state each include scores of multiple action dimensions; compute an action quality reward based on the action quality score in the current state and the action quality score in the next state, comprising: compute a basic action quality reward based on the scores of the multiple action dimensions in the next state; compute an action quality reward increment based on the score difference between the next state and the current state in each of the action dimensions; compute the action quality reward based on the basic action quality reward and the action quality reward increment.

4. The method of claim 1, wherein compute a training progress reward based on sensor data of the motion device in the next state, comprising: determine whether the user completes the next training action based on the sensor data of the motion device in the next state; in response to completing the next training action, determine that the training progress reward is a preset first threshold value; in response to not completing the next training action, determine that the training progress reward is a preset second threshold value; wherein the first threshold value is greater than the second threshold value, and the first threshold value is a positive number.

5. The method of claim 1, wherein wherein in the case where the action quality reward, the user state reward, and the training progress reward are fixed, the immediate reward corresponding to any one of the strategies of outputting a preset encouraging feedback, outputting a correction instruction, outputting a rest suggestion, and outputting a query question is less than the immediate reward corresponding to no output.

6. The method of claim 1, wherein compute a user state reward based on the self-satisfaction score, the fatigue level score, and the understanding level score in the next state, and the understanding level score in the current state, comprising: determine a self-satisfaction reward based on the self-satisfaction score in the next state; determine a fatigue level reward based on the fatigue level score in the next state; determine an understanding level reward based on the understanding level score in the next state; determine an understanding level reward increment based on the difference between the understanding level score in the next state and the understanding level score in the current state; Based on the self-satisfaction reward, the fatigue degree reward, the understanding degree reward and the understanding degree reward increment, the user state reward is calculated.

7. A method for generating a virtual coach guidance strategy, characterized in that The method comprises the following steps: The DQN model trained based on the method of any one of claims 1-6 is used to generate a strategy for a virtual coach to guide a target user in training.

8. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method of any one of claims 1-6.

Images