Exercise guidance method and device, equipment, storage medium and program product
By combining image matching and voice guidance, the similarity and guidance probability values between the user's movement posture and the correction text are calculated, and the target movement correction text is output. This solves the problem that users cannot concentrate on movement guidance in existing technologies, and achieves convenient and accurate movement correction.
Patent Information
- Application Number
- CN202411071611.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2026-02-06
AI Technical Summary
Existing exercise guidance methods require users to watch instructional videos while simultaneously looking at skeletal cues, which makes it difficult for users to concentrate, leads to frequent exercise errors, and makes the guidance inconvenient.
By acquiring real-time motion videos and image matching of instructional videos, the similarity and guidance probability values between motion postures and correction text are calculated. The target motion correction text is then output, and voice guidance is used to help users correct their movements, reducing reliance on displayed information.
It improves the convenience of exercise guidance, allowing users to make effective exercise corrections without constantly paying attention to the displayed information, thus enhancing the accuracy of exercise corrections and the user experience.
Smart Images
Figure CN121482854A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of data processing technology, and in particular relates to a motion guidance method, apparatus, device, storage medium and program product. Background Technology
[0002] With increasing work pressure and improved living standards, exercise has become a way for people to relieve stress and lose weight.
[0003] When users engage in a new sport, they may require exercise guidance. In an exemplary technique, based on a first real-time motion video of the user exercising, the motion skeleton image of the user in the first real-time motion video is analyzed and compared with the motion skeleton image of a standard movement to provide user guidance.
[0004] However, the exercise guidance mentioned above is displayed on the terminal device. Users need to watch the instructional video while looking at the skeletal prompts and other guidance information, which makes it difficult for users to concentrate on the exercise and causes users to make frequent exercise errors. In other words, the exercise guidance is not convenient enough. Summary of the Invention
[0005] This application provides a method, apparatus, device, storage medium, and program product for providing exercise guidance, in order to address the problem of inconvenient exercise guidance.
[0006] In a first aspect, embodiments of this application provide a method for guiding exercise, applied to an exercise guidance device, the method comprising:
[0007] Acquire the first exercise instruction video being played and the first real-time exercise video of the first user exercising based on the first exercise instruction video;
[0008] The first image corresponding to the current time is obtained from the first real-time motion video, and a second image matching the first image is obtained from the first motion teaching video.
[0009] Based on the first image and the second image, determine the similarity between the first user's motion posture at the current time and each target motion correction text, and determine the guidance probability value corresponding to each similarity.
[0010] In response to the maximum guidance probability value being greater than a preset threshold, target motion correction text is output, which is the target motion correction text used to determine the maximum guidance probability value.
[0011] In another embodiment, the step of determining the guidance probability value corresponding to each similarity includes:
[0012] Based on the first exercise instruction video, determine the probability value of when to provide guidance to the first user at the current time;
[0013] The product between the timing probability value and each similarity is determined to obtain the guidance probability value corresponding to each similarity.
[0014] In another embodiment, the step of determining the probability value of providing guidance to the first user at the current time based on the first exercise instruction video includes:
[0015] Obtain the background audio of the first exercise instruction video at the current time;
[0016] Based on the audio parameters of the human voice in the background audio, a probability value for providing guidance to the first user at the current time is determined, wherein the audio parameters include at least one of volume and pitch.
[0017] In another embodiment, the step of determining the similarity between the first user's motion posture at the current time and each target motion correction text based on the first image and the second image includes:
[0018] Image feature vectors are obtained based on the first image and the second image, and text feature vectors associated with each target motion-corrected text are obtained;
[0019] The similarity between the image feature vector and each of the text feature vectors is determined as the similarity between the first user's motion posture at the current time and each target motion correction text.
[0020] In another embodiment, prior to the step of determining the similarity between the first user's motion posture at the current time and each target motion correction text based on the first image and the second image, the method further includes:
[0021] The first image is compared with the second image to determine the motion error of the first user at the current time;
[0022] Obtain the target motion correction text for each of the motion error matches.
[0023] In another embodiment, the step of outputting the target motion-corrected text includes:
[0024] The target motion correction text is converted into guiding speech, and the guiding speech is played.
[0025] In another embodiment, the step of converting the target motion correction text into guiding speech includes:
[0026] Identify the target's voiceprint features;
[0027] The motion correction text is converted into guiding speech with the target voiceprint features.
[0028] In another embodiment, the motion guidance device is provided with a motion correction model, wherein,
[0029] The motion correction model is used to obtain a first image corresponding to the current time based on the first real-time motion video, and to obtain a second image matching the first image in the first motion teaching video.
[0030] The motion correction model is further configured to determine the similarity between the first user's motion posture at the current time and each target motion correction text based on the first image and the second image, and to determine the guidance probability value corresponding to each similarity.
[0031] In another embodiment, the training process of the motion correction model is as follows:
[0032] Acquire various training samples, including a second motion instruction video, a second real-time motion video obtained by a second user exercising based on the second motion instruction video, and various tags, the tags being used to indicate motion correction text for images with motion errors in the second real-time motion video;
[0033] Based on each of the training samples, the preset model is trained to obtain the motion correction model.
[0034] In another embodiment, the step of obtaining each training sample includes:
[0035] The second exercise instruction video and the second real-time exercise video are time-aligned, and a third image in the second real-time exercise video in which an exercise error occurs is determined based on the second exercise instruction video.
[0036] The motion-corrected text corresponding to the third image is determined, and the training text is constructed based on the time-aligned second motion video, the second real-time motion video, and each of the motion-corrected texts.
[0037] Secondly, embodiments of this application provide a motion guidance device, including:
[0038] The first acquisition module is used to acquire the first exercise instruction video being played and the first real-time exercise video of the first user exercising based on the first exercise instruction video.
[0039] The second acquisition module is used to acquire a first image corresponding to the current time based on the first real-time motion video, and to acquire a second image that matches the first image in the first motion teaching video.
[0040] The determination module is used to determine the similarity between the first user's motion posture at the current time and each target motion correction text based on the first image and the second image, and to determine the guidance probability value corresponding to each similarity.
[0041] The output module is used to output target motion correction text in response to the maximum guidance probability value being greater than a preset threshold. The target motion correction text is the target motion correction text used to determine the maximum guidance probability value.
[0042] Thirdly, embodiments of this application provide a motion guidance device, including: a processor and a memory storing computer program instructions;
[0043] When the processor executes the computer program instructions, it implements a motion guidance method as described in one aspect.
[0044] Fourthly, embodiments of this application provide a computer storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the motion guidance method as described in the first aspect is implemented.
[0045] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform the motion guidance method as described in the first aspect.
[0046] The exercise guidance method, apparatus, device, storage medium, and program product provided in this application acquire a playing exercise instruction video and a real-time exercise video of the user exercising based on the exercise instruction video. Based on the real-time exercise video, it acquires a first image corresponding to the current time and a second image matching the first image in the exercise instruction video. Using the first and second images, it determines the similarity between the user's exercise posture at the current time and various target exercise correction texts, and determines the guidance probability value corresponding to each similarity. If the maximum probability value is greater than a preset threshold, it outputs the target exercise correction text corresponding to the maximum guidance probability. In this application, when the guidance probability is high, guidance is provided for incorrect movements by the user, eliminating the need for the user to constantly monitor the display information on the device for exercise correction, thus improving the convenience of exercise guidance. Attached Figure Description
[0047] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0048] Figure 1 This is a schematic diagram of a scenario in which the exercise guidance method of this application can be implemented;
[0049] Figure 2 This is a flowchart illustrating the first embodiment of the exercise guidance method of this application;
[0050] Figure 3 This is a flowchart illustrating the second embodiment of the exercise guidance method of this application;
[0051] Figure 4 This is a flowchart illustrating the third embodiment of the exercise guidance method of this application;
[0052] Figure 5 This is a schematic diagram of the clock interval involved in this application;
[0053] Figure 6 This is a schematic diagram of the functional modules of the exercise guidance device involved in this application;
[0054] Figure 7 This is a schematic diagram of the structure of the exercise guidance device provided in the embodiments of this application. Detailed Implementation
[0055] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.
[0056] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.
[0057] With increasing work pressure and improved living standards, exercise has become a way for people to relieve stress and lose weight.
[0058] When users engage in a new sport, they may require exercise guidance. In an exemplary technique, based on a first real-time motion video of the user exercising, the motion skeleton image of the user in the first real-time motion video is analyzed and compared with the motion skeleton image of a standard movement to provide user guidance.
[0059] The inventors of this application have discovered that exercise guidance displays guidance information on a terminal device. Users need to watch instructional videos while simultaneously viewing skeletal prompts and other guidance information, which makes it difficult for users to concentrate on the exercise and leads to frequent exercise errors. In other words, exercise guidance is not convenient enough.
[0060] The inventors of this application therefore conceived of providing guidance to users on movements that are prone to errors when the probability of guidance is high, without requiring users to constantly monitor the display information on the device for movement correction, thus improving the convenience of movement guidance.
[0061] This application provides a method, apparatus, device, storage medium, and program product for guiding exercise. The following first illustrates an example of the framework for applying the exercise guidance method provided in this application.
[0062] like Figure 1As shown, user 100 exercises based on the exercise instruction video 210 played in the exercise guidance device 200, and the exercise guidance device 200 captures images of user 100's exercise posture in real time to obtain a real-time exercise video 220, which is then displayed. The exercise guidance device 200 acquires images of the exercise instruction video and the real-time exercise video at the current time, determines the similarity between the images and preset correction text, and determines the guidance probability value corresponding to each similarity. When the exercise guidance device 200 determines that the maximum guidance probability value is greater than a preset threshold, it outputs the target exercise correction text. For example, the exercise guidance device 200 displays exercise correction text 230, which may be "palm down".
[0063] Combination Figure 1 The exercise guidance method provided in the embodiments of this application will be introduced.
[0064] Figure 2 A flowchart illustrating a first embodiment of the exercise guidance method provided in this application is shown. This exercise guidance method can be applied to exercise guidance devices, such as… Figure 2 As shown, the methods for guiding exercise include:
[0065] Step S201: Obtain the first exercise instruction video being played and the first real-time exercise video of the first user exercising based on the first exercise instruction video.
[0066] In this embodiment, the executing entity is a motion guidance device. For ease of description, the term "device" will be used to refer to the motion guidance device below. The device can be a terminal device with display functionality, such as a television, tablet, computer, or mobile phone.
[0067] The device plays exercise instruction videos of users performing exercises; the user is defined as the first user, and the played exercise instruction video is defined as the first exercise instruction video. The device also has an image acquisition function, capturing real-time motion videos of the first user exercising based on the first exercise instruction video; these real-time motion videos are defined as the first real-time motion video. The first real-time motion video refers to the real-time video obtained when the first user imitates the postures in the first exercise instruction video. It should be noted that after acquiring the first exercise instruction video and the first real-time motion video, the device will display them in a split-screen format to allow the first user to correct their movements based on these videos.
[0068] Step S202: Obtain the first image corresponding to the current time from the first real-time motion video, and obtain the second image that matches the first image from the first motion teaching video.
[0069] The accuracy of the first user's correction based on the first exercise instruction video and the first real-time exercise video may not be high. Therefore, the device can output guidance information to guide the first user in making exercise corrections, so as to improve the accuracy of the user's exercise corrections.
[0070] To address this, the device acquires the image corresponding to the current time from the first real-time motion video, defining this image as the first image. The device then acquires a second image matching the first image from the first motion instruction video. Specifically, after playing the first motion instruction video, the device captures the first real-time motion instruction video and aligns the playback time of the first frame of the first real-time motion instruction video with the playback time of the first frame of the first motion instruction video. Therefore, each frame in the first real-time motion video has the same playback time point in the first motion instruction video. For example, the tenth frame in the first real-time motion video has the same playback time point as the tenth frame in the first motion instruction video. Thus, it can be concluded that the frame number of the second image in the first motion instruction video is the same as the frame number of the first image in the first real-time motion video.
[0071] Step S203: Based on the first image and the second image, determine the similarity between the first user's motion posture at the current time and each target motion correction text, and determine the guidance probability value corresponding to each similarity.
[0072] After acquiring the first and second images, the device determines the similarity between the first user's current motion posture and various target motion correction texts based on the first and second images. The target motion correction texts can be preset motion correction texts or motion correction texts generated in real-time based on the first image. For example, the first image can be input into a multimodal model, and the multimodal model outputs motion correction texts as the target correction texts. The multimodal model is trained using multiple training samples, including images and labels, where labels indicate the motion correction text corresponding to the image.
[0073] Specifically, the device compares the first image with the second image to determine whether the first user has made a movement error. If the movement posture of the instructor in the first image is different from the movement posture of the first user in the second image, it can be determined that the first user has made a movement error, and the device needs to provide movement guidance to the first user.
[0074] The device stores multiple motion correction texts, which are defined as target motion correction texts. For example, a target motion correction text might consist of one or more of the following: "face up," "turn head back," "chest out," "back straight," "palms down," "hands extended back," "feet together," and "faster." Furthermore, different motions are associated with different preset correction texts. For instance, the aforementioned preset correction text is associated with yoga. Other types of motions, such as sit-ups, might be associated with preset correction texts like "legs together and tense," or "hands behind head." The device acquires each preset correction text associated with the first user's motion and determines the similarity between the first image and each preset correction text, i.e., determining the image-text similarity. For example, the first image is encoded into an image feature vector, and the preset correction text is encoded into a text feature vector. The cosine similarity between the image feature vector and the text feature vector is then calculated as the similarity between the first user's current motion posture and the preset correction text.
[0075] After determining the similarity, the device determines a guidance probability value corresponding to each similarity. The guidance probability value indicates whether there is an opportunity to guide the first user at the current time. A higher guidance probability value indicates a better opportunity to guide the user to correct motion errors at the current time, and will result in better correction effects for the first user. In one example, the device converts each similarity into a corresponding guidance probability value, and the similarity and the corresponding guidance probability value are positively correlated.
[0076] Before determining similarity, the device compares the first image with the second image to identify the first user's motion errors at the current time. Motion errors include, for example, head errors, torso errors, arm errors, leg errors, and footwork errors. Different motion errors are associated with corresponding target motion correction text. The device obtains the motion correction text matching the type of motion performed by the first user, and then extracts the motion correction texts associated with the motion errors from these texts, using them as preset correction texts for motion error matching. In this way, the device obtains a small number of target motion correction texts related to motion errors, reducing the computational load and saving computational resources.
[0077] Step S204: In response to the maximum guidance probability value being greater than a preset threshold, output target motion correction text, which is the target motion correction text used to determine the maximum guidance probability value.
[0078] The device determines the largest guidance probability value among various guidance probability values and checks whether the largest guidance probability value is greater than a preset threshold. If the largest guidance probability value is greater than the preset threshold, it can be determined that the timing for motion correction at the current time is better, and the effect of motion correction performed by the first user is also better. In response, the device outputs the target motion correction text. The target correction text is the text used to determine the largest guidance probability value. For example, the preset correction texts are text 1, text 2, and text 3; the similarity corresponding to text 1 is similarity a, the similarity corresponding to text 2 is similarity b, and the similarity corresponding to text 3 is similarity c; the guidance probability value corresponding to similarity a is 0.5, the guidance probability value corresponding to similarity b is 0.7, and the guidance probability value corresponding to similarity c is 0.95. The largest guidance probability value is 0.95, and 0.95 is greater than the preset threshold of 0.9. Therefore, the similarity corresponding to the target correction text is similarity c, and similarity c corresponds to text 3, meaning the target correction text is text 3.
[0079] The device outputs target motion correction text, which can be displayed on the device or converted into speech for playback, i.e., converted into guiding speech and played back. Further, the device determines the target voiceprint features and converts the target correction text into guiding speech with those features. The target voiceprint features can be those of a person selected by the first user, such as a coach selected by the first user, or voiceprints of anime characters or authorized public figures. Playing the guiding speech corresponding to the target motion correction text eliminates the need for the user to view the device's display, thus avoiding errors caused by the user looking at the text and further improving the convenience of motion guidance.
[0080] In this embodiment, a playback exercise instruction video and a real-time exercise video of the user exercising based on the exercise instruction video are acquired. A first image corresponding to the current time and a second image matching the first image in the exercise instruction video are obtained based on the real-time exercise video. The similarity between the user's exercise posture at the current time and each target exercise correction text is determined using the first and second images, and a guidance probability value corresponding to each similarity is determined. If the maximum probability value is greater than a preset threshold, the target exercise correction text corresponding to the maximum guidance probability is output. In this embodiment, when the guidance probability is high, guidance is provided for incorrect movements by the user, eliminating the need for the user to constantly monitor the display information on the device for exercise correction, thus improving the convenience of exercise guidance.
[0081] Reference Figure 3 , Figure 3A flowchart illustrating a second embodiment of the exercise guidance method of this application is shown. Based on the first embodiment, step S203 includes:
[0082] Step S301: Based on the first exercise instruction video, determine the probability value of the timing to provide guidance to the first user at the current time.
[0083] In this embodiment, the device acquires a first exercise instruction video and determines the probability value P of providing guidance to the first user at the current time. chance .
[0084] In one example, the device determines the probability of a first exercise instruction video instructing a first user at the current time based on a classification model. This means the classification model obtains a temporal probability value from the first exercise instruction video and the current time input value. The device encodes the human voice and background audio in existing exercise instruction videos and then trains the classification model using this audio encoding. This classification model can obtain the probability value of exercise instruction videos in complex scenarios. The principle of training the classification model is: to perform motion correction on the user during playback times without human voice; that is, the labels of the feature vector obtained from the audio encoding include whether motion correction is possible and whether it is not. Existing exercise instruction videos are divided into multiple video segments, and the human voice and background audio in each video segment are encoded into feature vectors.
[0085] In another example, the device acquires the background audio of a first exercise instruction video at the current time and determines the probability value of providing guidance to the first user at the current time based on the audio parameters of the human voice in the background audio. The audio parameters include at least one of volume and pitch. For example, if the audio parameters include volume, and the volume of the human voice is below a preset threshold, it can be determined that the coach's voice in the first exercise instruction video is low and does not affect the device's playback of the speech corresponding to the exercise correction text; therefore, the probability value is set relatively high. If the audio parameters include pitch, and the frequency of the human voice's pitch is low, the penetration of the human voice is low and will not affect the device's playback of the speech corresponding to the exercise correction text; therefore, the probability value is set relatively low. It is understood that the smaller the audio parameters, the higher the probability value.
[0086] Step S302: Determine the product between the timing probability value and each similarity to obtain the guidance probability value corresponding to each similarity.
[0087] After determining the timing probability value, the device determines the product between the timing probability value and each similarity, thus obtaining the guidance probability value corresponding to each similarity.
[0088] For example, the similarity is P content (I teach , T), where I teachLet T represent the target motion correction text, and P represent the timing probability value. chance Then the guiding probability value P teach =P content (I teach ,T)*P chance .
[0089] In this embodiment, the device determines the probability value of providing guidance to the first user at the current time based on the first exercise instruction video, and determines the guidance probability value based on the probability value of the timing and similarity. That is, it provides exercise correction guidance based on the intervals of human voices in the exercise instruction video, thereby improving the user experience.
[0090] Reference Figure 4 , Figure 4 This application provides a schematic flowchart of a third embodiment of the exercise guidance method, based on... Figure 2 or Figure 3 In the embodiment shown, step S203 includes:
[0091] Step S401: Obtain image feature vectors based on the first image and the second image, and obtain text feature vectors associated with motion-corrected text for each target.
[0092] Step S402: Determine the similarity between the image feature vector and each text feature vector, as the similarity between the first user's motion pose at the current time and each target motion correction text.
[0093] In this embodiment, the device stitches the first image and the second image together. For example, the first image and the second image are stitched together vertically to obtain a stitched image. This stitching method is called alignment stitching. For example, the stitching can be completed by overlapping the lower boundary of the second image with the upper boundary of the first image.
[0094] After obtaining the stitched image, the device performs image encoding on the stitched image to obtain an image feature vector I, and performs text encoding on each target motion correction text to obtain a text feature vector T. The device determines the similarity between the image feature vector and each text feature vector as the similarity between the first user's motion pose at the current time and each target motion correction text. The similarity can be the cosine similarity between the text and image feature vectors. Furthermore, a model can be used to calculate the similarity between the image feature vector and the text feature vector; this model could be, for example, a multimodal model.
[0095] In this embodiment, the device obtains image feature vectors based on the first image and the second image, determines the text feature vector associated with the MiG target motion correction text, and then determines the similarity between the image feature vector and the text feature vector, thereby accurately determining the similarity between the first user's motion posture at the current time and each target motion correction text.
[0096] In one embodiment, the device includes a motion correction model. The motion correction model is used to acquire a first image corresponding to the current time from a first real-time motion video, and to acquire a second image matching the first image from a first motion instruction video. Furthermore, the motion correction model is also used to determine the similarity between the first user's motion posture at the current time and each target motion correction text based on the first and second images, and to determine a guidance probability value corresponding to each similarity. It is understood that by inputting the second image of the first motion instruction video and the first image of the first real-time motion video into the motion correction model, the guidance probability value for the current time can be output.
[0097] The training process of the motion correction model is as follows:
[0098] By acquiring various training samples and training the preset model based on these samples, a motion correction model can be obtained. The training samples include a second motion instruction video, a second real-time motion video obtained by a second user exercising based on the second motion instruction video, and various labels. The labels are used to indicate the motion correction text of images with motion errors in the second real-time motion video.
[0099] The specific method for obtaining training samples is as follows:
[0100] The second motion instruction video and the second real-time motion video are time-aligned, and a third image in the second real-time motion video in which a motion error occurs is determined based on the second motion instruction video.
[0101] Determine the motion-corrected text corresponding to the third image, and construct training text based on the time-aligned second motion video, the second real-time motion video, and each motion-corrected text.
[0102] Specifically, the device acquires a second exercise instruction video provided by the exercise instructor, and acquires a second real-time exercise video of a second user exercising according to the second exercise instruction video from the server. Acquiring the second real-time exercise video requires the user's authorization and consent, and acquiring the second exercise instruction video requires the authorization and consent of the exercise instructor.
[0103] Compared to the second motion instruction video, the second user's second real-time motion video may have a significant time lag. Therefore, alignment processing is required between the second motion instruction video and the second real-time motion video. This alignment can be achieved by stitching the first frame of the second motion instruction video with the first frame of the second real-time motion video, stitching the second frame of the second motion instruction video with the second frame of the second real-time motion video, and so on, to complete the alignment between the two videos. Furthermore, the device can preprocess the two videos, including but not limited to reducing resolution, reducing image grayscale, and stitching multiple frames. Preprocessing can reduce the complexity of model training and inference, and the multi-frame stitching method avoids the problem of motion misalignment caused by time discrepancies in the poses and movements between the motion instruction video and the real-time motion video. Multi-frame stitching refers to stitching the first frame of the image with the second frame, stitching the second frame with the third frame, stitching the first frame of the second motion instructional video with the first frame of the second real-time motion video to obtain the first frame of the image, stitching the second frame of the second motion instructional video with the second frame of the second real-time motion video to obtain the second frame of the image, and stitching the third frame of the second motion instructional video with the third frame of the second real-time motion video to obtain the third frame of the image.
[0104] It should be noted that the labels in the training samples can be configured based on the voice of the exercise instructor. Specifically, the device acquires real-time voice guidance information provided by the online exercise instructor in the second exercise instruction video. The voice guidance information is processed to obtain text information, which is then converted into labels. The labels are the target motion correction text in the model inference.
[0105] Additionally, considering the potential for lead-up or lag between instructional videos, user gestures, and guidance audio, data from appropriate clock intervals can be used for training and prediction. Clock intervals are defined based on the instructional video's voice, background music, and guidance audio; see the specific details below. Figure 5 , Figure 5 There are three different clock intervals, including one with voice but no guidance, one without voice and no guidance, and one without voice and guidance.
[0106] It should be noted that different motion associations have different motion correction models. The device acquires the first motion instruction video, determines the user's motion based on the first motion instruction video, and then guides the user's motion through the motion correction model of the motion association.
[0107] In addition, the target motion correction text includes, but is not limited to, correction of the user's motion posture, correction of motion rhythm, and correction of breathing.
[0108] In this embodiment, the motion correction model can quickly correct the user's motion, enabling the user to perform motion correction quickly and improving the user experience.
[0109] like Figure 6 As shown in the figure, this application embodiment also provides a motion guidance device 600, which includes:
[0110] The first acquisition module 610 is used to acquire the first exercise instruction video being played and the first real-time exercise video of the first user exercising based on the first exercise instruction video.
[0111] The second acquisition module 620 is used to acquire a first image corresponding to the current time based on the first real-time motion video, and to acquire a second image matching the first image in the first motion teaching video.
[0112] The determination module 630 is used to determine the similarity between the first user's motion posture at the current time and each target motion correction text based on the first image and the second image, and to determine the guidance probability value corresponding to each similarity.
[0113] The output module 640 is used to output target motion correction text in response to the maximum guidance probability value being greater than a preset threshold. The target motion correction text is the target motion correction text used to determine the maximum guidance probability value.
[0114] In one embodiment, the motion guidance device 600 is further used for:
[0115] Based on the first exercise instruction video, determine the probability value of the timing to provide guidance to the first user at the current time;
[0116] The product of the timing probability value and each similarity is determined to obtain the guidance probability value corresponding to each similarity.
[0117] In another embodiment, the motion guidance device 600 is also used for:
[0118] Get the background audio of the first sports instructional video at the current time;
[0119] Based on the audio parameters of human voices in the background audio, determine the probability value of providing guidance to the first user at the current time. The audio parameters include at least one of volume and pitch.
[0120] In another embodiment, the motion guidance device 600 is also used for:
[0121] Image feature vectors are obtained from the first and second images, and text feature vectors associated with motion-corrected text for each target are also obtained.
[0122] The similarity between the image feature vector and each text feature vector is determined as the similarity between the first user's motion pose at the current time and each target motion correction text.
[0123] In another embodiment, the motion guidance device 600 is also used for:
[0124] The first image is compared with the second image to determine the motion error of the first user at the current time;
[0125] Obtain the motion correction text for each target in motion error matching.
[0126] In another embodiment, the motion guidance device 600 is also used for:
[0127] Convert the target motion correction text into guiding speech and play the guiding speech.
[0128] In another embodiment, the motion guidance device 600 is also used for:
[0129] Identify the target's voiceprint features;
[0130] The motion correction text is converted into guiding speech with the target voiceprint features.
[0131] In another embodiment, the motion guidance device 600 is also used for:
[0132] A motion correction model is used to obtain a first image corresponding to the current time from a first real-time motion video, and to obtain a second image matching the first image from a first motion teaching video;
[0133] The motion correction model is also used to determine the similarity between the first user's motion posture at the current time and each target motion correction text based on the first image and the second image, and to determine the guidance probability value corresponding to each similarity.
[0134] In another embodiment, the motion guidance device 600 is also used for:
[0135] Acquire various training samples, including a second motion instruction video, a second real-time motion video obtained by a second user based on the second motion instruction video, and various labels. The labels are used to indicate motion correction text for images with motion errors in the second real-time motion video.
[0136] Based on each training sample, the preset model is trained to obtain the motion correction model.
[0137] In another embodiment, the motion guidance device 600 is also used for:
[0138] The second motion instruction video and the second real-time motion video are time-aligned, and a third image in the second real-time motion video in which a motion error occurs is determined based on the second motion instruction video.
[0139] Determine the motion-corrected text corresponding to the third image, and construct training text based on the time-aligned second motion video, the second real-time motion video, and each motion-corrected text.
[0140] Figure 7 A schematic diagram of the hardware structure of the motion guidance device provided in an embodiment of this application is shown.
[0141] The motion guidance device may include a processor 701 and a memory 702 storing computer program instructions.
[0142] Specifically, the processor 701 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.
[0143] Memory 702 may include mass storage for data or instructions. For example, and not limitingly, memory 702 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 702 may include removable or non-removable (or fixed) media. Where appropriate, memory 702 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 702 is non-volatile solid-state memory.
[0144] In a particular embodiment, memory 702 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Thus, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.
[0145] The processor 701 implements any of the motion guidance methods described in the above embodiments by reading and executing computer program instructions stored in the memory 702.
[0146] In one example, the motion guidance device may also include a communication interface 703 and a bus 77. Wherein, as... Figure 7 As shown, the processor 701, memory 702, and communication interface 703 are connected via bus 77 and communicate with each other.
[0147] The communication interface 703 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.
[0148] Bus 77 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 77 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.
[0149] Furthermore, in conjunction with the terminal device movement and movement guidance methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the movement guidance methods in the above embodiments.
[0150] This application also provides a computer program product, including a computer program, which, when executed, implements any of the motion guidance methods described in the above embodiments.
[0151] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.
[0152] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.
[0153] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.
[0154] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable terminal device motion device to produce a machine such that these instructions, executable via the processor of the computer or other programmable terminal device motion device, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.
[0155] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.
Claims
1. A method for guiding exercise, characterized in that, The exercise guidance method is applied to an exercise guidance device and includes: Acquire the first exercise instruction video being played and the first real-time exercise video of the first user exercising based on the first exercise instruction video; The first image corresponding to the current time is obtained from the first real-time motion video, and a second image matching the first image is obtained from the first motion teaching video. Based on the first image and the second image, determine the similarity between the first user's motion posture at the current time and each target motion correction text, and determine the guidance probability value corresponding to each similarity. In response to the maximum guidance probability value being greater than a preset threshold, target motion correction text is output, which is the target motion correction text used to determine the maximum guidance probability value.
2. The exercise guidance method according to claim 1, characterized in that, The step of determining the guidance probability value corresponding to each similarity includes: Based on the first exercise instruction video, determine the probability value of when to provide guidance to the first user at the current time; The product between the timing probability value and each similarity is determined to obtain the guidance probability value corresponding to each similarity.
3. The exercise guidance method according to claim 2, characterized in that, The step of determining the probability value of providing guidance to the first user at the current time based on the first exercise instruction video includes: Obtain the background audio of the first exercise instruction video at the current time; Based on the audio parameters of the human voice in the background audio, a probability value for when to provide guidance to the first user at the current time is determined, wherein the audio parameters include at least one of volume and pitch.
4. The exercise guidance method according to claim 1, characterized in that, The step of determining the similarity between the first user's motion posture at the current time and each target motion correction text based on the first image and the second image includes: Image feature vectors are obtained based on the first image and the second image, and text feature vectors associated with each target motion-corrected text are obtained; The similarity between the image feature vector and each of the text feature vectors is determined as the similarity between the first user's motion posture at the current time and each target motion correction text.
5. The exercise guidance method according to claim 1, characterized in that, Before the step of determining the similarity between the first user's motion posture at the current time and each target motion correction text based on the first image and the second image, the method further includes: The first image is compared with the second image to determine the motion error of the first user at the current time; Obtain the target motion correction text for each of the motion error matches.
6. The exercise guidance method according to claim 1, characterized in that, The steps for outputting the target motion-corrected text include: The target motion correction text is converted into guiding speech, and the guiding speech is played.
7. The exercise guidance method according to claim 6, characterized in that, The step of converting the target motion correction text into guiding speech includes: Determine the target's voiceprint characteristics; The motion correction text is converted into guiding speech with the target voiceprint features.
8. The exercise guidance method according to any one of claims 1-7, characterized in that, The motion guidance device is equipped with a motion correction model, wherein... The motion correction model is used to obtain a first image corresponding to the current time based on the first real-time motion video, and to obtain a second image matching the first image in the first motion teaching video. The motion correction model is further configured to determine the similarity between the first user's motion posture at the current time and each target motion correction text based on the first image and the second image, and to determine the guidance probability value corresponding to each similarity.
9. The exercise guidance method according to claim 8, characterized in that, The training process of the motion correction model is as follows: Acquire various training samples, including a second motion instruction video, a second real-time motion video obtained by a second user exercising based on the second motion instruction video, and various tags, the tags being used to indicate motion correction text for images with motion errors in the second real-time motion video; Based on each of the training samples, the preset model is trained to obtain the motion correction model.
10. The exercise guidance information according to claim 9, characterized in that, The steps for obtaining each training sample include: The second exercise instruction video and the second real-time exercise video are time-aligned, and a third image in the second real-time exercise video in which an exercise error occurs is determined based on the second exercise instruction video. The motion-corrected text corresponding to the third image is determined, and the training text is constructed based on the time-aligned second motion video, the second real-time motion video, and each of the motion-corrected texts.
11. A motion guidance device, characterized in that, include: The first acquisition module is used to acquire the first exercise instruction video being played and the first real-time exercise video of the first user exercising based on the first exercise instruction video. The second acquisition module is used to acquire a first image corresponding to the current time based on the first real-time motion video, and to acquire a second image that matches the first image in the first motion teaching video. The determination module is used to determine the similarity between the first user's motion posture at the current time and each target motion correction text based on the first image and the second image, and to determine the guidance probability value corresponding to each similarity. The output module is used to output target motion correction text in response to the maximum guidance probability value being greater than a preset threshold. The target motion correction text is the target motion correction text used to determine the maximum guidance probability value.
12. A sports guidance device, characterized in that, include: Processor and memory storing computer program instructions; When the processor executes the computer program instructions, it implements the motion guidance method as described in any one of claims 1-10.
13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the motion guidance method as described in any one of claims 1-10.
14. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device is able to perform the motion guidance method as described in any one of claims 1-10.