Methods, devices, equipment, and storage media for providing feedback on motion performance

Abstract

This application discloses a method, apparatus, device, and storage medium for providing feedback on motion performance, belonging to the field of machine learning technology. The method includes: acquiring a user's motion video; calling a motion feedback model to predict motion features in the (k+1)th frame of the motion video based on the motion feature vector of the kth frame, obtaining the predicted feature vector of the (k+1)th frame; calling the motion feedback model to extract motion features from the (k+1)th frame of the motion image, obtaining the motion feature vector of the (k+1)th frame; calculating a motion performance value based on the difference between the predicted feature vector of the (k+1)th frame and the motion feature vector of the (k+1)th frame, generating motion performance information for the (k+1)th frame of the motion image; and displaying the (k+1)th frame of the motion image and the corresponding motion performance information on a first user interface. This method can provide feedback on the user's motion performance based on the motion image using a motion feedback model, achieving motion performance feedback supported solely by a camera, and is suitable for daily life.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to a method, apparatus, device, and storage medium for providing feedback on motion performance. Background Technology

[0002] With social progress and development, more and more people are paying attention to fitness in order to improve their health and relieve stress. As a result, many fitness apps have emerged.

[0003] People can choose exercise videos that interest them on fitness apps and learn while watching, but this method lacks proper guidance. This led to the development of specialized software for exercise on fitness equipment equipped with pressure sensors, cameras, and other sensors. These sensors collect user exercise videos and force data, and the software compares this data with that of professional athletes to provide guidance.

[0004] However, the aforementioned professional software for providing exercise guidance relies on too many sensor devices, making it difficult to popularize in daily life. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for providing feedback on motion performance. The technical solution is as follows:

[0006] According to one aspect of this application, a method for providing feedback on athletic performance is provided, the method comprising:

[0007] Obtain the user's exercise videos;

[0008] The motion feedback model is invoked to predict the motion features in the (k+1)th frame of the motion video based on the motion feature vector of the kth frame, and the predicted feature vector of the (k+1)th frame is obtained. The motion feature vector of the kth frame is extracted from the kth frame of the motion video, where k is a positive integer.

[0009] The motion feedback model is invoked to extract motion features from the (k+1)th frame motion image, resulting in the (k+1)th frame motion feature vector;

[0010] Based on the difference between the predicted feature vector of the (k+1)th frame and the motion feature vector of the (k+1)th frame, a motion performance value is calculated to generate motion performance information of the (k+1)th frame motion image. The motion performance information is used to indicate the difference between the actual motion performance and the standard motion performance in the (k+1)th frame motion image.

[0011] The (k+1)th frame motion image and the motion performance information corresponding to the (k+1)th frame motion image are displayed on the first user interface.

[0012] According to another aspect of this application, a device for providing feedback on motion performance is provided, the device comprising:

[0013] The acquisition module is used to acquire the user's motion videos;

[0014] The motion feedback model module is used to call the motion feedback model to predict the motion features in the (k+1)th frame of the motion video based on the motion feature vector of the kth frame, and to obtain the predicted feature vector of the (k+1)th frame. The motion feature vector of the kth frame is extracted from the kth frame of the motion video, where k is a positive integer.

[0015] The motion feedback model module is used to call the motion feedback model to extract motion features from the (k+1)th frame motion image and obtain the (k+1)th frame motion feature vector.

[0016] The generation module is used to calculate the motion performance value based on the difference between the predicted feature vector of the (k+1)th frame and the motion feature vector of the (k+1)th frame, and generate the motion performance information of the (k+1)th frame motion image. The motion performance information is used to indicate the difference between the actual motion performance and the standard motion performance in the (k+1)th frame motion image.

[0017] The display module is used to display the (k+1)th frame motion image and the motion performance information corresponding to the (k+1)th frame motion image on the first user interface.

[0018] According to another aspect of this application, a terminal is provided, the terminal including a processor and a memory coupled to the processor, and program instructions stored in the memory, wherein the processor, when executing the program instructions, implements the methods provided in various aspects of this application.

[0019] According to another aspect of this application, a computer-readable storage medium is provided, wherein program instructions are stored therein, which, when executed by a processor, implement the methods provided in various aspects of this application.

[0020] According to one aspect of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the methods provided in various alternative implementations of the above-described method.

[0021] The beneficial effects of the technical solutions provided in this application embodiment may include:

[0022] This method for providing feedback on athletic performance calls a motion feedback model to provide feedback on the motion performance in motion images. Furthermore, on hardware devices, the terminal can support the implementation of this method using only a camera, eliminating the need for multiple cameras and sensors to assist in motion performance feedback. This makes it more suitable for daily life and reduces application costs, especially since it requires fewer participants. Compared to scenarios requiring coaches, analysts, researchers, and other personnel for athletic performance evaluation, it saves significant human resources, thereby reducing equipment and human resource costs. Secondly, this method can be applied to various types of sports, and the feedback on athletic performance is not limited to any particular sport. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a structural block diagram of a terminal provided in an exemplary embodiment of this application;

[0025] Figure 2 This is a flowchart of a method for providing feedback on motion performance provided in an exemplary embodiment of this application;

[0026] Figure 3 This is a flowchart of a training method for a motion feedback model provided in an exemplary embodiment of this application;

[0027] Figure 4 This is a flowchart of a training method for a motion feedback model provided in another exemplary embodiment of this application;

[0028] Figure 5 This is a schematic diagram of an interface for displaying reference motion performance information provided in an exemplary embodiment of this application;

[0029] Figure 6 This is a schematic diagram of an interface for displaying calibrated motion performance information provided in an exemplary embodiment of this application;

[0030] Figure 7 This is a flowchart of a training method for a motion feedback model provided in another exemplary embodiment of this application;

[0031] Figure 8 This is a schematic diagram of an interface for editing moving images, provided in an exemplary embodiment of this application;

[0032] Figure 9This is a flowchart of a training method for a pre-trained model provided in an exemplary embodiment of this application;

[0033] Figure 10 This is a framework for the training process of a pre-trained model provided in an exemplary embodiment of this application;

[0034] Figure 11 This is a framework for the training process of a motion feedback model provided in an exemplary embodiment of this application;

[0035] Figure 12 This is a block diagram of a device for providing feedback on motion performance, provided in an exemplary embodiment of this application.

[0036] Figure 13 This is a block diagram of a device for providing feedback on motion performance, provided in another exemplary embodiment of this application;

[0037] Figure 14 This is a block diagram of a device for providing feedback on motion performance, provided in another exemplary embodiment of this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0039] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0040] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0041] To facilitate understanding of the solutions shown in the embodiments of this application, several terms appearing in the embodiments of this application will be introduced below.

[0042] Artificial Intelligence (AI) is a branch of computer science that studies the theories, methods, technologies, and application systems used to simulate, extend, and expand human intelligence. It attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. The field of AI includes machine learning, natural language processing, image recognition, speech recognition, vision technology, and robotics.

[0043] Machine learning is a multidisciplinary field that involves probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance.

[0044] In the field of sports, machine learning can be used to evaluate a user's athletic performance. For example, sports equipment can be equipped with a series of sensors such as cameras, pressure sensors, and heart rate monitors. These sensors collect data on the user's exercise videos, heart rate, and force exertion during exercise. Specialized software can then compare the user's data with that of professional athletes to provide guidance. However, this method of sports guidance relies on too many sensor devices, making it difficult to popularize in daily life.

[0045] Therefore, this application proposes a method for providing feedback on motion performance, which can evaluate the user's motion performance by capturing motion video through a camera on the terminal. For detailed implementation details of this method, please refer to the following embodiments.

[0046] For example, the method for providing feedback on motion performance and the training method for the motion feedback model shown in the embodiments of this application can be applied to a terminal that has a video acquisition device and video acquisition function. The terminal may include a mobile phone, tablet computer, laptop computer, desktop computer, all-in-one computer, smart glasses, smartwatch, digital camera, Moving Picture Experts Group Audio Layer IV (MP4) terminal, Moving Picture Experts Group Audio Layer V (MP5) terminal, learning machine, e-reader, e-book, electronic dictionary, Virtual Reality (VR) playback terminal, or Augmented Reality (AR) playback terminal, etc.

[0047] Figure 1 This is a structural block diagram of a terminal provided in an exemplary embodiment of this application, such as... Figure 1 As shown, the terminal includes a processor 120 and a memory 140. The memory 140 stores at least one instruction, which is loaded and executed by the processor 120 to implement the training method of the motion feedback model and the method for providing feedback on motion performance as described in the various method embodiments of this application.

[0048] In this application, terminal 100 is an electronic device with video capture capabilities. For example, when terminal 100 runs an application with a pre-trained model, terminal 100 can perform personalized training on the pre-trained model based on the user's motion video captured by the camera, so as to obtain a motion feedback model that conforms to the user's personal motion characteristics.

[0049] For example, terminal 100 acquires a pre-trained model and user motion video samples. The pre-trained model is obtained by pre-training a machine learning model using unlabeled motion videos. On the second user interface, motion image samples and editable controls are displayed in the motion video samples. The editable controls are used to edit the user's calibrated motion performance information in the motion image samples. The calibrated motion performance information is determined as the label of the motion image samples. Based on the motion image samples and labels, the pre-trained model is trained to obtain a motion feedback model for feeding back the user's motion performance.

[0050] For example, after completing the above model training, the terminal 100 can also call the motion feedback model based on the user's motion video captured by the camera to provide feedback on the user's motion performance in the motion image of the motion video, so as to provide the user with a reference for improving their exercise.

[0051] For example, terminal 100 acquires the user's motion video; it calls a motion feedback model to predict the motion features in the (k+1)th frame of the motion video based on the motion feature vector of the kth frame, obtaining the predicted feature vector of the (k+1)th frame. The motion feature vector of the kth frame is extracted from the motion image of the kth frame of the motion video, where k is a positive integer; it calls a motion feedback model to extract the motion features in the (k+1)th frame of the motion image, obtaining the motion feature vector of the (k+1)th frame; it calculates the motion performance value based on the difference between the predicted feature vector of the (k+1)th frame and the motion feature vector of the (k+1)th frame, generating motion performance information for the (k+1)th frame of the motion image. The motion performance information is used to indicate the difference between the actual motion performance and the standard motion performance in the (k+1)th frame of the motion image; and it displays the (k+1)th frame of the motion image and the corresponding motion performance information on the first user interface.

[0052] Processor 120 may include one or more processing cores. Processor 120 connects to various parts within terminal 100 using various interfaces and lines, and performs various functions and processes data of terminal 100 by running or executing instructions, programs, code sets, or instruction sets stored in memory 140, and by calling data stored in memory 140. Optionally, processor 120 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 120 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display on the screen; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 120 and may be implemented as a separate chip.

[0053] The memory 140 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 140 may include a non-transitory computer-readable storage medium. The memory 140 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 140 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the various method embodiments described below, etc.; the data storage area may store data involved in the various method embodiments described below, etc.

[0054] In an optional embodiment, the terminal 100 further includes an AI processing unit 160; the AI ​​processing unit 160 is specifically designed to handle large computational tasks in artificial intelligence applications. Optionally, the AI ​​processing unit 160 may be implemented using one or more hardware forms selected from graphics processing units (GPUs), field-programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs). Instructions in the memory 140 may be loaded and executed by the AI ​​processing unit 160 to implement the training method for the motion feedback model and the method for providing feedback on motion performance as described in the various method embodiments of this application.

[0055] In another alternative embodiment, the terminal 100 may also work in concert with the processor 140 and the AI ​​processing unit 160 to implement the training method of the motion feedback model and the method for providing feedback on motion performance as described in the various method embodiments of this application.

[0056] Please see Figure 2 This is a flowchart illustrating a method for providing feedback on motion performance according to an exemplary embodiment of this application. This method for providing feedback on motion performance can be applied to the above-mentioned... Figure 1 In the terminal shown. Figure 2 In China, methods used to provide feedback on athletic performance include:

[0057] Step 210: Obtain the user's motion video.

[0058] In an optional embodiment, the terminal is equipped with a camera; the terminal uses the camera to capture video of the user's movements.

[0059] In another alternative embodiment, the terminal stores the user's motion video in its memory; the terminal can also retrieve the user's motion video from the memory.

[0060] In another optional embodiment, a network connection exists between the terminal and the camera device; the terminal can also acquire motion videos of the user captured by the camera device via the network connection. For example, the network connection can be a wired network or a wireless network connection.

[0061] After obtaining the user's motion video, the terminal first performs frame splitting processing on the motion video to obtain at least two motion images. For example, the terminal is configured with a motion feedback model; the terminal calls the motion feedback model to perform frame splitting processing on the user's motion video. For example, the motion feedback model is set with a specified frame number as a frame splitting parameter. The motion feedback model performs frame splitting processing on the motion video according to the specified frame number, extracting motion image frames at specified intervals, or extracting keyframes of the motion images at specified intervals; for example, if the specified frame number is 2 frames, the terminal extracts one motion image frame from the motion video every 2 frames.

[0062] After the frame splitting is completed, the terminal executes steps 220 to 240 to call the motion feedback model to calculate the motion performance information in each of the at least two motion images, as shown below.

[0063] Step 220: Call the motion feedback model to predict the motion features in the (k+1)th frame of the motion video based on the motion feature vector of the kth frame, and obtain the predicted feature vector of the (k+1)th frame.

[0064] Here, the motion feature vector of the k-th frame is extracted from the k-th frame of the motion video, where k is a positive integer.

[0065] For example, in the process of calculating motion performance information, the terminal calls the motion feedback model to extract the motion feature vector of the kth frame from the kth frame of the motion image of the motion video, calls the motion feedback model to predict the predicted feature vector of the k+1th frame of the motion image of the motion video based on the motion feature vector of the kth frame, and caches the predicted feature vector of the k+1th frame.

[0066] Optionally, the motion feedback model includes a motion capture sub-model and a dynamic prediction sub-model for human body posture; the terminal calls the motion capture sub-model to extract motion features from k frames of motion images to obtain the motion feature vector of the kth frame; and calls the dynamic prediction sub-model to predict the motion features in the (k+1)th frame of motion images based on the motion feature vector of the kth frame to obtain the predicted feature vector of the (k+1)th frame.

[0067] Step 230: Call the motion feedback model to extract motion features from the (k+1)th frame motion image to obtain the (k+1)th frame motion feature vector.

[0068] The terminal invokes the motion feedback model to extract the motion feature vector of the (k+1)th frame from the motion image. Optionally, the motion feedback model includes a human body posture motion capture sub-model; the terminal invokes the motion capture sub-model to extract motion features from the (k+1)th frame of the motion image, thus obtaining the motion feature vector of the (k+1)th frame.

[0069] Step 240: Calculate the motion performance value based on the difference between the predicted feature vector of frame k+1 and the motion feature vector of frame k+1, and generate the motion performance information of the motion image of frame k+1.

[0070] The above motion performance information is used to indicate the difference between the actual motion performance and the standard motion performance in the (k+1)th frame of the motion image.

[0071] Optionally, the aforementioned motion performance information includes at least one of a score for the human posture in the image and guidance information. The guidance information refers to instructions given to the user while performing an action. For example, "Continue raising your arm, until it forms a 45-degree angle with your torso"; or even more detailed instructions, such as "Continue raising your arm 30 degrees, until it forms a 45-degree angle with your torso."

[0072] After the terminal calculates the predicted feature vector of the (k+1)th frame, it obtains the cached motion feature vector of the (k+1)th frame through the motion feedback model, calculates the difference vector between the predicted feature vector of the (k+1)th frame and the motion feature vector of the (k+1)th frame, calculates the user's motion performance value in the motion image of the (k+1)th frame based on the difference vector, and then generates the motion performance information of the (k+1)th frame based on the motion performance value.

[0073] Step 250: Display the (k+1)th frame motion image and the motion performance information corresponding to the (k+1)th frame motion image on the first user interface.

[0074] The terminal displays a motion image and corresponding motion performance information on a first user interface. For example, the terminal displays a motion image on the first user interface and overlays the corresponding motion performance information on top of the motion image. Alternatively, the terminal displays a motion image and displays the corresponding motion performance information to the left, right, top, or bottom of the motion image.

[0075] For example, the terminal displays the (k+1)th frame of motion images and the motion performance information corresponding to the (k+1)th frame of motion images; in response to a sliding operation in a first direction on the terminal screen, the terminal displays the (k+2)th frame of motion images and the motion performance information corresponding to the (k+2)th frame of motion images; in response to a sliding operation in a second direction on the terminal screen, the terminal displays the (k)th frame of motion images and the motion performance information corresponding to the (k)th frame of motion images, where the first direction and the second direction are different sliding directions.

[0076] It should be noted that, based on motion images, physical health data can also be combined to evaluate athletic performance. For example, the terminal acquires the user's exercise video and physical health data; it calls a motion feedback model to predict the predicted feature vector at the (k+1)th moment of exercise based on the motion feature vector at the kth moment of exercise. The motion feature vector at the kth moment of exercise is extracted from the motion image of the exercise video and the physical health data at the kth moment of exercise; it calls another motion feedback model to extract the motion feature vector at the (k+1)th moment of exercise based on the motion image of the exercise video and the physical health data at the (k+1)th moment of exercise; it calculates the athletic performance value based on the difference between the predicted feature vector and the motion feature vector at the (k+1)th moment of exercise, generating the user's athletic performance information at the (k+1)th moment of exercise. This athletic performance information is used to indicate the difference between the actual athletic performance at the (k+1)th moment of exercise and the standard athletic performance; and it displays the motion image, physical health data, and athletic performance information at the (k+1)th moment of exercise on the first user interface.

[0077] For example, the aforementioned health data includes at least one of heart rate location, heart rate, and exertion during exercise. This health data may be obtained through at least one smart terminal accessory, such as a smartwatch, sensor, or heart rate location detector, and the smart terminal accessory has a communication connection with the terminal.

[0078] In summary, the method for providing feedback on motion performance provided in this embodiment calls a motion feedback model to provide feedback on motion performance in motion images. Therefore, on hardware devices, the terminal can support the implementation of this method for providing feedback on motion performance using only a camera, eliminating the need for multiple cameras and sensors to assist in motion performance feedback. This is more suitable for daily life and reduces application costs, especially since it requires fewer participants. Compared to motion performance evaluation scenarios that require the participation of coaches, analysts, researchers, and other personnel, it saves a significant amount of human resources, thereby reducing equipment and human resource costs. Furthermore, this method can be applied to various types of sports, and the feedback on motion performance is not limited to any particular type of sport.

[0079] For the training method of the motion feedback model in the above embodiments, please refer to the following embodiments.

[0080] Please see Figure 3 This is a flowchart illustrating a training method for a motion feedback model provided in an exemplary embodiment of this application. This training method for the motion feedback model can be applied to the above-mentioned... Figure 1 In the terminal shown. Figure 3 The training method for this motion feedback model includes:

[0081] Step 310: Obtain the pre-trained model and obtain motion video samples from the user.

[0082] The aforementioned pre-trained model is obtained by pre-training a machine learning model using unlabeled motion videos. If the pre-trained model exists in the terminal, the terminal directly obtains the pre-trained model to execute steps 320 to 340 below; if the pre-trained model does not exist in the terminal, the terminal loads the pre-trained model from the server to execute steps 320 to 340 below.

[0083] In an optional embodiment, the terminal is equipped with a camera, which captures motion video samples of the user.

[0084] In another alternative embodiment, the terminal stores motion video samples of the user in its memory; the terminal retrieves the user's motion video samples from the memory.

[0085] In another optional embodiment, a network connection exists between the terminal and the camera device; the terminal acquires motion video samples of the user captured by the camera device through the network connection. For example, the network connection can be a wired network or a wireless network connection.

[0086] Step 320: Display motion image samples and editable controls in the motion video samples on the second user interface. The editable controls are used to edit the user's calibration motion performance information in the motion image samples.

[0087] For example, the terminal displays motion image samples from a motion video sample on a second user interface, and each frame of the motion image sample corresponds to at least one editable control. The at least one editable control is used to edit the user's calibrated motion performance information in the motion image sample. This calibrated motion performance information indicates the difference between the actual motion performance in the motion image sample and the standard motion performance. Specifically, in terms of action guidelines, the standard action for each movement is the standard motion performance; in terms of scoring, a perfect score is the standard motion performance.

[0088] Optionally, in response to the first invocation of the pre-trained model, the terminal executes the step of displaying motion image samples and editable controls from the motion video samples on the second user interface. That is, when the pre-trained model is used for the first time on the terminal, it is necessary to first collect the user's personal motion video to retrain the pre-trained model. Therefore, when the pre-trained model is invoked for the first time, the terminal displays motion image samples and editable controls from the motion video samples on the second user interface to edit the calibration motion performance information of the motion image samples.

[0089] For example, the terminal performs frame splitting on the motion video sample to obtain at least two motion image samples from the motion video sample. For example, the terminal can invoke a pre-trained model to perform frame splitting on the user's motion video sample. For example, the pre-trained model is set with a specified frame splitting parameter; the pre-trained model performs frame splitting on the motion video sample according to the specified frame number, extracting motion image frames at specified intervals as samples, or extracting keyframes of the motion images at specified intervals as samples; for example, if the specified frame number is 2 frames, the terminal extracts one motion image sample from the motion video sample every 2 frames.

[0090] The terminal displays at least two frames of motion image samples obtained from frame splitting, and displays the (i+1)th frame motion image sample and at least one editable control corresponding to the (i+1)th frame motion image sample on the terminal; in response to a click operation on the first boundary of the terminal screen, it displays the (i+2)th frame motion image sample and at least one editable control corresponding to the (i+2)th frame motion image sample; in response to a click operation on the second boundary of the terminal screen, it displays the (i)th frame motion image sample and at least one editable control corresponding to the (i)th frame motion image sample, wherein the first boundary and the second boundary are different screen boundaries, and i is a positive integer.

[0091] Step 330: Determine the calibration motion performance information as the label for the motion image sample.

[0092] For example, in response to the editing operation on the editable control of the calibration motion performance information of the (i+1)th motion image sample, the terminal displays the edited calibration motion performance information of the (i+1)th motion image sample; in response to the editing completion operation on the terminal, the (i+1)th calibration motion performance information is determined as the label of the (i+1)th motion image sample.

[0093] Step 340: Train the pre-trained model based on motion images and labels to obtain a motion feedback model for feedback on the user's motion performance.

[0094] The terminal calls the pre-trained model to predict the motion features in the (i+1)th frame of the motion video sample based on the motion feature vector of the i-th frame, and obtains the predicted feature vector of the (i+1)th frame. The motion feature vector of the i-th frame is extracted from the i-th frame of the motion video sample. Based on the predicted feature vector of the (i+1)th frame and the (i+1)th label vector, the prediction error of the motion performance is calculated. The (i+1)th label vector is generated by the label of the (i+1)th frame of the motion image sample, where i is a positive integer. Based on the prediction error, the model parameters in the pre-trained model are adjusted to obtain the motion feedback model.

[0095] Optionally, the pre-trained model includes a motion capture sub-model for human body posture and a dynamic prediction sub-model. For the prediction of the predicted feature vector, the terminal calls the motion capture sub-model to extract the motion features in the i-th frame motion image sample to obtain the i-th frame motion feature vector. The terminal then calls the dynamic prediction sub-model to predict the motion features in the (i+1)-th frame motion image sample based on the i-th frame motion feature vector to obtain the (i+1)-th frame predicted feature vector.

[0096] For example, the label vectors described above are generated by encoding the labels of moving image samples using a pre-trained model. Alternatively, the label vectors described above are generated by word embedding of the labels of moving image samples using a pre-trained model.

[0097] For example, the terminal calculates the cross-entropy loss between the predicted feature vector of the (i+1)th frame and the (i+1)th label vector as the prediction loss. Alternatively, the terminal calculates the mean squared error between the predicted feature vector of the (i+1)th frame and the (i+1)th label vector as the prediction loss.

[0098] In summary, the training method for the motion feedback model provided in this embodiment collects the user's personal motion videos to adjust the parameters of the pre-trained model. That is, it uses a generalized motion feedback model to train a motion feedback model suitable for the individual, making the motion feedback model more consistent with the individual's motion characteristics. This allows the model to accurately provide personalized guidance during the user's exercise, thereby improving the user's training effect and enhancing the user experience.

[0099] Based on the methods shown in the above embodiments, this application also provides a training method for a motion feedback model. Before editing the calibration motion performance information, reference motion performance information is displayed in an editable control to provide a reference for the user. Please refer to the following embodiments.

[0100] Please refer to Figure 4 It is based on Figure 3 The flowchart shown provides some steps of the training method for the motion feedback model. Figure 4 After completing step 320, the terminal can execute steps 322 to 328, and the execution process is described below:

[0101] Step 322: Call the pre-trained model to predict the motion features in the (i+1)th frame of the motion video sample based on the motion feature vector of the i-th frame, and obtain the predicted feature vector of the (i+1)th frame.

[0102] The motion feature vector of the i-th frame is extracted from the i-th frame motion image sample of the motion video sample.

[0103] For example, the pre-trained model includes a motion capture sub-model for human body posture and a dynamic prediction sub-model; the terminal calls the motion capture sub-model to extract motion features from the i-th frame motion image sample to obtain the i-th frame motion feature vector; and calls the dynamic prediction sub-model to predict the motion features in the (i+1)-th frame motion image sample based on the i-th frame motion feature vector to obtain the (i+1)-th frame prediction feature vector.

[0104] Step 324: Call the pre-trained model to extract motion features from the (i+1)th frame motion image sample to obtain the (i+1)th frame motion feature vector.

[0105] For example, the terminal calls the motion capture sub-model to extract motion features from the motion image sample of the (i+1)th frame, and obtains the motion feature vector of the (i+1)th frame.

[0106] Step 326: Calculate the motion performance value based on the difference between the predicted feature vector of frame i+1 and the motion feature vector of frame i+1, and generate reference motion performance information for the motion image sample of frame i+1.

[0107] For example, the terminal calculates the difference vector between the predicted feature vector of the (i+1)th frame and the motion feature vector of the (i+1)th frame, calculates the motion performance value based on the difference vector, and then generates reference motion performance information for the (i+1)th motion image sample based on the difference performance value.

[0108] Step 328: Display the reference motion performance information corresponding to the motion image sample of frame i+1 in the editable control.

[0109] For example, such as Figure 5 Before performing steps 322 to 328, a motion image sample 12 and an editable control 13 are displayed on the second user interface 11. After performing steps 322 to 328, reference motion performance information is displayed in the editable control 13.

[0110] It should be noted that the determination of the label in step 330 can also be achieved in this embodiment through the following steps:

[0111] In response to the editing operation on the editable control, the terminal displays the (i+1)th calibration motion performance information; in response to the editing completion operation on the terminal, the (i+1)th calibration motion performance information is determined as the label of the (i+1)th frame motion image sample.

[0112] For example, the reference athletic performance information includes at least two reference athletic performance sub-information pieces, and the calibration athletic performance information includes at least two calibration athletic performance sub-information pieces; such as Figure 6 The reference sports performance sub-information "Posture completion rate: 60%" was changed to the calibration sports performance sub-information "Posture completion rate: 80%", and the reference sports performance sub-information "both arms raised to the sides" was changed to the calibration sports performance sub-information "both arms raised to the sides downwards".

[0113] In summary, the motion feedback model training method provided in this embodiment generates reference motion performance information corresponding to motion image samples by applying the pre-trained model. This provides a reference for users to edit the calibration motion performance information of motion image samples on the terminal, guides the user's editing method, and improves the user's editing efficiency. It can also unify the editing format of calibration motion performance information to a high extent, which is conducive to rapid calculation during model training.

[0114] In subsequent applications of the motion feedback model, if the user is not satisfied with the training effect of the motion feedback model once, the motion feedback model can be trained multiple times. Please refer to the following examples.

[0115] Please refer to Figure 7 It is based on Figure 2 The flowchart shown provides some steps of the motion feedback method. Figure 7 In the process, after the terminal completes step 250, the first user interface displays functional controls for triggering local model training, and then steps 410 to 420 can be executed. The execution process is described below:

[0116] Step 410: In response to the trigger operation on the function control, display the motion performance information corresponding to the (k+1)th frame motion image in the editable control, and display a confirmation control on the first user interface. The editable control is used to edit the calibration motion performance information of the user in the (k+1)th frame motion image.

[0117] The terminal displays motion performance information corresponding to the motion image on the first user interface. After receiving a trigger operation on a function control, it uses an editable control to display the motion performance information. For example, such as... Figure 8 The motion performance information 22 corresponding to the motion image is displayed on the first user interface 21. After receiving the trigger operation on the function control 23, the motion performance information 22 is displayed using the editable control 24.

[0118] The motion performance information is edited on the first user interface, and the edited motion performance information is used as the calibration motion performance information in the (k+1)th frame motion image, so as to use the calibration motion performance information as a label.

[0119] Step 420: In response to the confirmation operation on the confirmation control, confirm the calibration motion performance information as the label of the (k+1)th frame motion image, and retrain the motion feedback model based on the (k+1)th frame motion image and the label.

[0120] After receiving the confirmation operation from the confirmation control, the terminal confirms the calibration motion performance information of the (k+1)th frame motion image as the label of the (k+1)th frame motion image, and retrains the motion feedback model based on the (k+1)th frame motion image and the label. For example, as shown... Figure 8 When the terminal receives the confirmation operation on the confirmation control 25, it will confirm the calibration motion performance information as a label for the motion image.

[0121] For example, the terminal calls the motion feedback model to predict the motion features in the (k+1)th frame of the motion video based on the motion feature vector of the kth frame, and obtains the predicted feature vector of the (k+1)th frame. The motion feature vector of the kth frame is extracted from the motion image of the kth frame of the motion video. Based on the predicted feature vector of the (k+1)th frame and the (k+1)th label vector, the prediction error of the motion performance is calculated. The (k+1)th label vector is generated by the label of the motion image of the (k+1)th frame, and k is a positive integer. Based on the prediction error, the model parameters in the motion feedback model are adjusted to obtain the retrained motion feedback model.

[0122] For example, the motion feedback model includes a motion capture sub-model for human body posture and a dynamic prediction sub-model. For the prediction of the predicted feature vector, the terminal calls the motion capture sub-model to extract the motion features in the k-th frame motion image to obtain the k-th frame motion feature vector. The dynamic prediction sub-model is then called to predict the motion features in the (k+1)-th frame motion image based on the k-th frame motion feature vector to obtain the (k+1)-th frame predicted feature vector.

[0123] For example, the terminal calculates the mean squared error between the predicted feature vector of the (k+1)th frame and the (k+1)th label vector to obtain the prediction error of the motion performance. Alternatively, the terminal calculates the cross-entropy loss between the predicted feature vector of the (k+1)th frame and the (k+1)th label vector as the prediction error.

[0124] In summary, the training method for the motion feedback model provided in this embodiment can determine the number of local training sessions based on the user's individual accuracy requirements for the motion feedback model, thereby continuously improving the feedback accuracy of the motion feedback model and enhancing the user experience.

[0125] Based on the methods shown in the above embodiments, the motion feedback model is obtained by retraining the pre-trained model as the initial model. This application also provides a method for training the pre-trained model. This method allows the server to train a machine learning model using publicly available samples, and then distribute the training data to the terminal. Please refer to [link / reference]. Figure 9 This is a flowchart illustrating a training method for a pre-trained model provided in an exemplary embodiment of this application. Figure 9 The training methods for this pre-trained model include:

[0126] Step 510: Obtain unlabeled motion videos and machine learning models.

[0127] For example, the server can retrieve unlabeled sports videos from a database, which can be sports videos of various sports crawled from the Internet using a web crawler; the server is equipped with a machine learning model.

[0128] For example, after obtaining an unlabeled motion video, the server performs frame splitting on the unlabeled motion video to obtain at least two motion images. For example, the server can invoke a machine learning model to perform frame splitting on the unlabeled motion video to obtain at least two motion images.

[0129] Step 520: Call the machine learning model to predict the motion features in the (j+1)th frame of the unlabeled motion video based on the motion feature vector of the jth frame, and obtain the predicted feature vector of the (j+1)th frame.

[0130] Here, the motion feature vector of the j-th frame is extracted from the j-th frame motion image of the unlabeled motion video, where j is a positive integer.

[0131] For example, the machine learning model includes a motion capture sub-model for human body posture and a dynamic prediction sub-model; the server calls the motion capture sub-model to extract motion features from the j-th frame motion image to obtain the motion feature vector of the j-th frame; the server calls the dynamic prediction sub-model to predict the motion features in the (j+1)-th frame motion image based on the motion feature vector of the j-th frame to obtain the predicted feature vector of the (j+1)-th frame.

[0132] Step 530: Call the machine learning model to extract motion features from the (j+1)th frame of the unlabeled motion video, and obtain the motion feature vector of the (j+1)th frame.

[0133] For example, the server calls the motion capture sub-model to extract motion features from the (j+1)th frame of the motion image, and obtains the motion feature vector of the (j+1)th frame.

[0134] Step 540: Calculate the prediction error of the machine learning model based on the predicted feature vector of frame j+1 and the motion feature vector of frame j+1.

[0135] For example, the server calculates the mean square error of the predicted feature vector and the motion feature vector of the (j+1)th frame to obtain the error of the machine learning model prediction.

[0136] For example, the server calculates the cross-entropy loss between the predicted feature vector of frame j+1 and the motion feature vector of frame j+1 to obtain the error in the prediction by the machine learning model.

[0137] Step 550: Adjust the model parameters in the machine learning model using the error to train and obtain the pre-trained model.

[0138] For example, such as Figure 10 The diagram illustrates the training of the pre-trained model. The server splits the unlabeled motion video 31 into frames to obtain a single-frame motion image 32. The motion image 32 is used to pre-train the machine learning model to obtain a pre-trained model 33. The pre-trained model 33 includes a motion capture sub-model 34 and a dynamic prediction sub-model 35.

[0139] In summary, the training method of the pre-trained model provided in this embodiment uses a large number of publicly available sports videos on the Internet as samples to train the machine learning model, thereby obtaining a pre-trained model with sports performance feedback function, so as to provide users with an initial model, so that users can use the initial model to customize a sports performance feedback model that meets their own needs.

[0140] The training method for the motion feedback model provided in the embodiments of this application, such as... Figure 11 As shown, the server first pre-trains the machine learning model 51 using unlabeled motion video 41 to obtain a pre-trained model 52, and then sends the pre-trained model 52 to the terminal. The user sets a labeled motion video through the terminal, and the terminal uses the labeled motion video 42 to adjust the parameters of the pre-trained model 52, finally obtaining a motion feedback model 53 that meets the user's personal needs, thus maximizing the satisfaction of the user's personalized needs.

[0141] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0142] Please refer to Figure 12 This illustration shows a structural block diagram of a device for providing feedback on motion performance, provided in an exemplary embodiment of this application. The device for providing feedback on motion performance can be implemented as all or part of a terminal through software, hardware, or a combination of both. The device includes:

[0143] The acquisition module 610 is used to acquire the user's motion video;

[0144] The motion feedback model module 620 is used to predict the motion features in the (k+1)th frame of the motion video based on the motion feature vector of the kth frame, and to obtain the predicted feature vector of the (k+1)th frame. The motion feature vector of the kth frame is extracted from the kth frame of the motion video, where k is a positive integer.

[0145] The motion feedback model module 620 is used to extract motion features from the (k+1)th frame motion image to obtain the (k+1)th frame motion feature vector.

[0146] The generation module 630 is used to calculate the motion performance value based on the difference between the predicted feature vector of the (k+1)th frame and the motion feature vector of the (k+1)th frame, and generate motion performance information of the (k+1)th frame motion image. The motion performance information is used to indicate the difference between the actual motion performance and the standard motion performance in the (k+1)th frame motion image.

[0147] The display module 640 is used to display the (k+1)th frame motion image and the motion performance information corresponding to the (k+1)th frame motion image on the first user interface.

[0148] In an optional embodiment, the motion feedback model module 620 includes a motion capture sub-model module 622 for human body posture and a dynamic prediction sub-model module 624.

[0149] The motion capture sub-model module 622 is used to extract motion features from k frames of motion images to obtain the motion feature vector of the kth frame.

[0150] The dynamic prediction sub-model module 624 is used to predict the motion features in the (k+1)th frame motion image based on the motion feature vector of the kth frame, and obtain the predicted feature vector of the (k+1)th frame.

[0151] In an optional embodiment,

[0152] The motion capture sub-model module 622 is used to extract motion features from the (k+1)th frame motion image to obtain the (k+1)th frame motion feature vector.

[0153] In an optional embodiment, the device further includes a training module 650, such as Figure 13 As shown, the training of the motion feedback model can be performed by the application-side terminal or another training-side terminal.

[0154] The acquisition module 610 is also used to acquire a pre-trained model and to acquire motion video samples of the user. The pre-trained model is obtained by pre-training a machine learning model using unlabeled motion videos.

[0155] The display module 640 is also used to display motion image samples and editable controls in the motion video samples on the second user interface. The editable controls are used to edit the calibration motion performance information of the user in the motion image samples.

[0156] Training module 650 is used to determine the labels of motion image samples based on the calibration motion performance information;

[0157] Training module 650 is used to train a pre-trained model based on motion image samples and labels to obtain a motion feedback model for feedback on the user's motion performance.

[0158] In an optional embodiment,

[0159] Training module 650 is used to call the pre-trained model to predict the motion features in the (i+1)th frame motion image sample of the motion video sample based on the motion feature vector of the i-th frame, and obtain the predicted feature vector of the (i+1)th frame. The motion feature vector of the i-th frame is extracted from the i-th frame motion image sample of the motion video sample.

[0160] Training module 650 is used to calculate the prediction error of motion performance based on the predicted feature vector of the (i+1)th frame and the (i+1)th label vector. The (i+1)th label vector is generated from the labels of the motion image samples of the (i+1)th frame, where i is a positive integer.

[0161] Training module 650 is used to adjust the model parameters in the pre-trained model based on the prediction error to obtain the motion feedback model.

[0162] In an optional embodiment, the pre-trained model includes a motion capture sub-model of human body posture and a dynamic prediction sub-model; the training module 650 is used for:

[0163] The motion capture sub-model is invoked to extract motion features from the motion image sample of the i-th frame, resulting in the motion feature vector of the i-th frame.

[0164] The dynamic prediction sub-model is invoked to predict the motion features in the (i+1)th frame motion image sample based on the motion feature vector of the i-th frame, thus obtaining the predicted feature vector of the (i+1)th frame.

[0165] In an optional embodiment, the display module 640 is further configured to:

[0166] In response to the first invocation of the pre-trained model, the steps of displaying motion image samples and editable controls in the motion video samples on the second user interface are performed.

[0167] In an optional embodiment,

[0168] The training module 650 is also used to call the pre-trained model to predict the motion features in the (i+1)th frame motion image sample of the motion video sample based on the motion feature vector of the i-th frame, and to obtain the predicted feature vector of the (i+1)th frame. The motion feature vector of the i-th frame is extracted from the i-th frame motion image sample of the motion video sample.

[0169] The training module 650 is also used to call the pre-trained model to extract motion features from the motion image samples of the (i+1)th frame and obtain the motion feature vector of the (i+1)th frame.

[0170] The training module 650 is also used to calculate the motion performance value based on the difference between the predicted feature vector of the (i+1)th frame and the motion feature vector of the (i+1)th frame, and to generate reference motion performance information for the motion image sample of the (i+1)th frame.

[0171] Display module 640 is used to display reference motion performance information corresponding to the (i+1)th frame motion image sample in an editable control, where i is a positive integer.

[0172] In an optional embodiment, the training module 650 is used for:

[0173] In response to editing operations on the editable control on the reference motion performance information, display the (i+1)th piece of calibrated motion performance information that has been edited;

[0174] In response to the editing completion operation on the terminal, the (i+1)th calibration motion performance information is determined as the label of the (i+1)th frame motion image sample.

[0175] In an optional embodiment, the device further includes a pre-training module 660, such as Figure 14 As shown, the training of the pre-trained model can be performed by a terminal or a server. For example, the terminal can be an application-side terminal or another training-side terminal.

[0176] The acquisition module 610 is also used to acquire unlabeled motion videos and machine learning models;

[0177] The pre-training module 660 is used to call the machine learning model to predict the motion features in the (j+1)th frame of the unlabeled motion video based on the motion feature vector of the jth frame, and to obtain the predicted feature vector of the (j+1)th frame. The motion feature vector of the jth frame is extracted from the jth frame of the unlabeled motion video.

[0178] The pre-training module 660 is used to call the machine learning model to extract motion features from the (j+1)th frame of the unlabeled motion video and obtain the motion feature vector of the (j+1)th frame.

[0179] The pre-training module 660 is used to calculate the prediction error of the machine learning model based on the predicted feature vector of the (j+1)th frame and the motion feature vector of the (j+1)th frame, where j is a positive integer; the model parameters in the machine learning model are adjusted by the error to train the pre-trained model.

[0180] In an optional embodiment, the first user interface includes functional controls for triggering local training of the model;

[0181] The display module 640 is used to respond to the trigger operation on the function control, display the motion performance information corresponding to the motion image of the (k+1)th frame in the editable control, and display the confirmation control on the first user interface. The editable control is used to edit the calibration motion performance information of the user in the motion image of the (k+1)th frame.

[0182] The training module 650 is used to confirm the calibration motion performance information as the label of the (k+1)th frame motion image in response to the confirmation operation on the confirmation control;

[0183] Training module 650 is used to retrain the motion feedback model based on the motion image and label of frame k+1.

[0184] In summary, the device for providing feedback on motion performance provided in this embodiment calls a motion feedback model to provide feedback on motion performance in motion images. Therefore, on the hardware device, the terminal can support the implementation of the method for providing feedback on motion performance using only a camera, eliminating the need for multiple cameras and sensors to assist in motion performance feedback. This is more suitable for daily life and reduces application costs, especially since it requires fewer participants. Compared to motion performance evaluation scenarios that require the participation of coaches, analysts, researchers, and other personnel, it saves a significant amount of human resources, thereby reducing equipment and human resource costs. Furthermore, this device can be applied to various types of sports, and the feedback on motion performance is not limited to any particular type of sport.

[0185] This application also provides a computer-readable medium storing at least one instruction, which is loaded and executed by a processor to implement the methods for providing feedback on motion performance and the training methods for motion feedback models as described in the above embodiments.

[0186] It should be noted that the above embodiments of the device for providing feedback on motion performance, when executing the method for providing feedback on motion performance and the training method for the motion feedback model, are only illustrative examples of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the device for providing feedback on motion performance provided in the above embodiments, the method embodiments for providing feedback on motion performance, and the training method embodiments for the motion feedback model all belong to the same concept. The specific implementation process is detailed in the method embodiments and will not be repeated here.

[0187] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0188] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0189] The above description is merely an exemplary embodiment that can be implemented in this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for providing feedback on motion performance, characterized in that, The method includes: Acquire user's exercise videos and physical health data, wherein the physical health data includes at least one of the following: heart rate location, heart rate, and exercise exertion; The motion feedback model is invoked to predict the motion features in the (k+1)th frame of the motion video based on the motion feature vector of the kth frame, and the predicted feature vector of the (k+1)th frame is obtained. The motion feature vector of the kth frame is extracted from the kth frame of the motion video and the body health data, where k is a positive integer. The motion feedback model is invoked to extract motion features from the (k+1)th frame motion image, resulting in the (k+1)th frame motion feature vector; Based on the difference between the predicted feature vector of the (k+1)th frame and the motion feature vector of the (k+1)th frame, a motion performance value is calculated to generate motion performance information of the (k+1)th frame motion image. The motion performance information of the (k+1)th frame motion image is used to indicate the difference between the actual motion performance and the standard motion performance in the (k+1)th frame motion image. The first user interface displays the (k+1)th frame motion image, the body health data, and the motion performance information corresponding to the (k+1)th frame motion image; The training process of the motion feedback model includes: Obtain a pre-trained model and obtain motion video samples of the user. The pre-trained model is obtained by pre-training a machine learning model using unlabeled motion videos. The second user interface displays motion image samples and editable controls from the motion video samples. The editable controls are used to edit the user's calibration motion performance information in the motion image samples. The calibration motion performance information is used to indicate the difference between the actual motion performance in the motion image samples and the standard motion performance. The calibrated motion performance information is used as the label for the motion image sample; The pre-trained model is trained based on the motion image samples and the labels to obtain the motion feedback model used to provide feedback on the user's motion performance; The pre-trained model is invoked to predict the motion features in the (i+1)th frame motion image sample of the motion video sample based on the motion feature vector of the i-th frame, thereby obtaining the (i+1)th frame predicted feature vector. The i-th frame motion feature vector is extracted from the i-th frame motion image sample of the motion video sample. The pre-trained model is invoked to extract motion features from the (i+1)th frame motion image sample, resulting in the (i+1)th frame motion feature vector. The motion performance value is calculated based on the difference between the predicted feature vector of the (i+1)th frame and the motion feature vector of the (i+1)th frame, and reference motion performance information of the motion image sample of the (i+1)th frame is generated. The editable control displays reference motion performance information corresponding to the (i+1)th frame motion image sample, where i is a positive integer; Wherein, determining the calibrated motion performance information as the label of the motion image sample includes: In response to the editing operation on the editable control on the reference motion performance information, the (i+1)th calibrated motion performance information is displayed. In response to the editing completion operation on the terminal, the (i+1)th calibration motion performance information is determined as the label of the (i+1)th frame motion image sample.

2. The method according to claim 1, characterized in that, The motion feedback model includes a motion capture sub-model of human body posture and a dynamic prediction sub-model; The motion feedback model is invoked to predict the motion features in the (k+1)th frame of the motion video based on the motion feature vector of the kth frame, resulting in the predicted feature vector of the (k+1)th frame, including: The motion capture sub-model is invoked to extract motion features from the k-frame motion images, resulting in the k-th frame motion feature vector; The dynamic prediction sub-model is invoked to predict the motion features in the (k+1)th frame motion image based on the motion feature vector of the kth frame, thereby obtaining the predicted feature vector of the (k+1)th frame.

3. The method according to claim 2, characterized in that, The step of calling the motion feedback model to extract motion features from the (k+1)th frame motion image to obtain the (k+1)th frame motion feature vector includes: The motion capture sub-model is invoked to extract motion features from the (k+1)th frame motion image, thereby obtaining the (k+1)th frame motion feature vector.

4. The method according to claim 1, characterized in that, The step of training the pre-trained model based on the motion image samples and the labels to obtain the motion feedback model used to provide feedback on the user's motion performance includes: The pre-trained model is invoked to predict the motion features in the (i+1)th frame motion image sample of the motion video sample based on the motion feature vector of the i-th frame, thereby obtaining the (i+1)th frame predicted feature vector. The i-th frame motion feature vector is extracted from the i-th frame motion image sample of the motion video sample. Based on the predicted feature vector of the (i+1)th frame and the (i+1)th label vector, the prediction error of motion performance is calculated. The (i+1)th label vector is generated from the labels of the motion image samples of the (i+1)th frame, where i is a positive integer. The model parameters in the pre-trained model are adjusted based on the prediction error to obtain the motion feedback model.

5. The method according to claim 4, characterized in that, The pre-trained model includes a motion capture sub-model for human body posture and a dynamic prediction sub-model. The step of calling the pre-trained model to predict the motion features in the (i+1)th frame of the motion video sample based on the motion feature vector of the i-th frame, and obtaining the predicted feature vector of the (i+1)th frame, includes: The motion capture sub-model is invoked to extract motion features from the i-th frame motion image sample, resulting in the i-th frame motion feature vector; The dynamic prediction sub-model is invoked to predict the motion features in the (i+1)th frame motion image sample based on the motion feature vector of the i-th frame, thereby obtaining the predicted feature vector of the (i+1)th frame.

6. The method according to claim 1, characterized in that, The display of motion image samples and editable controls in the motion video samples on the second user interface includes: In response to the first invocation of the pre-trained model, the step of displaying motion image samples and editable controls in the motion video samples on the second user interface is performed.

7. The method according to claim 1, characterized in that, The training process of the pre-trained model includes: Acquire unlabeled motion videos and machine learning models; The machine learning model is invoked to predict the motion features in the (j+1)th frame of the unlabeled motion video based on the motion feature vector of the jth frame, and the predicted feature vector of the (j+1)th frame is obtained. The motion feature vector of the jth frame is extracted from the jth frame of the unlabeled motion video. The machine learning model is invoked to extract motion features from the (j+1)th frame of the unlabeled motion video, resulting in the motion feature vector of the (j+1)th frame. Based on the predicted feature vector of the (j+1)th frame and the motion feature vector of the (j+1)th frame, the prediction error of the machine learning model is calculated, where j is a positive integer; The model parameters in the machine learning model are adjusted using the error to train the pre-trained model.

8. The method according to any one of claims 1 to 3, characterized in that, The first user interface includes functional controls for triggering local training of the model; The method further includes: In response to a trigger operation on the function control, the motion performance information corresponding to the (k+1)th frame motion image is displayed in the editable control, and a confirmation control is displayed on the first user interface. The editable control is used to edit the calibration motion performance information of the user in the (k+1)th frame motion image. In response to the confirmation operation on the confirmation control, the calibration motion performance information is confirmed as the label of the (k+1)th frame motion image, and the motion feedback model is retrained based on the (k+1)th frame motion image and the label.

9. A device for providing feedback on motion performance, characterized in that, The device includes: The acquisition module is used to acquire the user's exercise videos and physical health data, wherein the physical health data includes at least one of the following: heart rate location, heart rate, and exercise exertion. The motion feedback model module is used to predict the motion features in the (k+1)th frame of the motion video based on the motion feature vector of the kth frame, and to obtain the predicted feature vector of the (k+1)th frame. The motion feature vector of the kth frame is extracted from the kth frame of the motion video and the body health data, where k is a positive integer. The motion feedback model module is used to extract motion features from the (k+1)th frame motion image to obtain the (k+1)th frame motion feature vector. The generation module is used to calculate the motion performance value based on the difference between the predicted feature vector of the (k+1)th frame and the motion feature vector of the (k+1)th frame, and generate the motion performance information of the (k+1)th frame motion image. The motion performance information of the (k+1)th frame motion image is used to indicate the difference between the actual motion performance and the standard motion performance in the (k+1)th frame motion image. The display module is used to display the (k+1)th frame motion image, the body health data, and the motion performance information corresponding to the (k+1)th frame motion image on the first user interface. The acquisition module is also used to acquire a pre-trained model and to acquire motion video samples of the user. The pre-trained model is obtained by pre-training a machine learning model using unlabeled motion videos. The display module is also used to display motion image samples and editable controls in the motion video samples on a second user interface. The editable controls are used to edit the user's calibration motion performance information in the motion image samples. The calibration motion performance information is used to indicate the difference between the actual motion performance in the motion image samples and the standard motion performance. The training module is used to determine the calibrated motion performance information as the label of the motion image sample; to train the pre-trained model based on the motion image sample and the label to obtain the motion feedback model used to provide feedback on the user's motion performance; and to call the pre-trained model to predict the motion features in the (i+1)th frame motion image sample of the motion video sample based on the motion feature vector of the i-th frame to obtain the (i+1)th frame predicted feature vector, wherein the i-th frame motion feature vector is extracted from the i-th frame motion image sample of the motion video sample. The training module is further configured to call the pre-trained model to extract motion features from the (i+1)th frame motion image sample, and obtain the (i+1)th frame motion feature vector; calculate the motion performance value based on the difference between the (i+1)th frame predicted feature vector and the (i+1)th frame motion feature vector, and generate reference motion performance information for the (i+1)th frame motion image sample; and display the reference motion performance information corresponding to the (i+1)th frame motion image sample in the editable control, where i is a positive integer; The training module is further configured to, in response to the editing operation on the editable control of the reference motion performance information, display the edited (i+1)th calibration motion performance information; and, in response to the editing completion operation on the terminal, determine the (i+1)th calibration motion performance information as the label of the (i+1)th frame motion image sample.

10. A terminal, characterized in that, The terminal includes a processor and a memory coupled to the processor, and program instructions stored in the memory, wherein the processor executes the program instructions to implement the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium storing program instructions, characterized in that, When the program instructions are executed by the processor, they implement the method as described in any one of claims 1 to 8.

12. A computer program product comprising computer instructions that are executed by a processor to implement the method as claimed in any one of claims 1 to 8.

Images