Video processing method and device based on large model, equipment and storage medium

Abstract

The invention provides a video processing method and device based on a large model, equipment and a storage medium, relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, large models and the like, and can be applied to scenes such as smart sports. The specific implementation scheme is as follows: collecting an imitation video made by a user based on a target video; based on the simulated video, extracting a three-dimensional attitude of the simulated video by adopting a pre-trained large model; and based on the three-dimensional attitude of the simulated video and a pre-acquired three-dimensional attitude of a target video, performing attitude evaluation on the simulated video by using the pre-trained large model to obtain an evaluation result.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technology, and especially to technical fields such as deep learning and large models, and can be applied to scenarios such as smart sports; in particular, it relates to a video processing method, device, equipment and storage medium based on large models. Background Art

[0002] With a high-intensity, high-pressure, fast-paced lifestyle, people are increasingly aware of the importance of the body and are paying more and more attention to physical training.

[0003] In the existing physical training, users can search for training videos and train at will. In order to improve the standardization of physical training movements and achieve training results, users can usually seek help from personal trainers, who will guide and evaluate the user's physical training, and assist the user to complete higher-standard physical training and achieve better training results. Summary of the invention

[0004] The present invention provides a video processing method, device, equipment and storage medium based on a large model.

[0005] According to one aspect of the present disclosure, a video processing method based on a large model is provided, the method comprising:

[0006] Collect imitation videos made by users based on the target video;

[0007] Based on the imitation video, using a pre-trained large model, extracting the three-dimensional posture of the imitation video;

[0008] Based on the three-dimensional posture of the imitation video and the three-dimensional posture of the pre-acquired target video, the pre-trained large model is used to perform posture evaluation on the imitation video to obtain an evaluation result.

[0009] According to another aspect of the present disclosure, a video processing device based on a large model is provided, the device comprising:

[0010] A video acquisition module is used to acquire imitation videos made by users based on target videos;

[0011] A posture extraction module, used to extract the three-dimensional posture of the imitation video based on the imitation video using a pre-trained large model;

[0012] The video evaluation module is used to perform posture evaluation on the imitation video based on the three-dimensional posture of the imitation video and the three-dimensional posture of the pre-acquired target video, using the pre-trained large model to obtain an evaluation result.

[0013] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any possible implementation manner and the aspects described above.

[0017] According to yet another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method of the above-mentioned aspect and any possible implementation manner.

[0018] According to yet another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements the above-mentioned aspects and any possible implementation method.

[0019] According to the technology disclosed in the present invention, it is possible to accurately and effectively evaluate the imitation video.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0024] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0025] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0026] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0027] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0028] Figure 7 is a schematic diagram according to a seventh embodiment of the present disclosure;

[0029] Figure 8 is a schematic diagram according to an eighth embodiment of the present disclosure;

[0030] Fig. 9 The block diagram is a block diagram of an electronic device for implementing the method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] Obviously, the described embodiments are only part of the embodiments of the present disclosure, but not all of them. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present disclosure.

[0033] It should be noted that the terminal devices involved in the embodiments of the present disclosure may include but are not limited to mobile phones, personal digital assistants (PDAs), wireless handheld devices, tablet computers and other smart devices; display devices may include but are not limited to personal computers, televisions and other devices with display functions.

[0034] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0035] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure; Figure 1 As shown, this embodiment provides a video processing method based on a large model, wherein the method includes:

[0036] S101, collecting imitation videos made by users based on target videos;

[0037] The executor of the large model-based video processing method of this embodiment is a large model-based video processing device, which can be an independent electronic entity or an application integrated with software. When in use, it runs on computers, mobile phones and other devices to realize physical training processing.

[0038] The scene of this embodiment can be a user training scene. The target video is a standard video that is acquired before the user training and is also a reference for the user training. In this embodiment, the training video made by the user based on the learning of the target video is called an imitation video.

[0039] S102, based on the imitation video, using a pre-trained large model to extract the three-dimensional posture of the imitation video;

[0040] During training, the designated part needs to complete the designated action, which will inevitably lead to changes in the three-dimensional posture of each point of the designated part. Therefore, in order to accurately evaluate the imitation video, it is necessary to extract the three-dimensional posture of the imitation video.

[0041] The three-dimensional posture of the imitating video in this embodiment may include imitating the three-dimensional posture of each key point of each part in the video.

[0042] When in use, the imitation video is input into a pre-trained large model, which can extract the three-dimensional posture of the imitation video.

[0043] S103, based on the 3D posture of the imitation video and the 3D posture of the target video acquired in advance, a pre-trained large model is used to perform posture evaluation on the imitation video to obtain an evaluation result.

[0044] Specifically, the large model can compare and analyze the three-dimensional posture of the imitation video and the three-dimensional posture of the target video to accurately evaluate the posture of the imitation video, obtain the evaluation results and output them.

[0045] The large model of this embodiment is mainly used to evaluate the user's training by performing posture evaluation on the imitation video.

[0046] The large model-based video processing method of this embodiment, by adopting the above technical solution, can accurately and effectively evaluate the imitation video based on the three-dimensional posture of the imitation video and the three-dimensional posture of the pre-acquired target video.

[0047] Further optionally, in one embodiment of the present disclosure, the above Figure 1 The specific evaluation of step S103 in the illustrated embodiment may include the following steps:

[0048] (I) Based on the 3D pose of the imitation video and the 3D pose of the target video, the difference of the 3D pose of the imitation video is calculated using a pre-trained large model;

[0049] For example, in a specific implementation, for the convenience of description, the video frame in the imitation video is referred to as the first video frame, and the video frame in the target video is referred to as the second video frame.

[0050] For each first video frame in the imitation video, the 3D posture difference of the key point of the specified part in the first video frame can be obtained based on the 3D posture of the key point of the specified part in the first video frame and the 3D posture of the corresponding key point of the corresponding specified part in the corresponding second video frame in the target video; in order to accurately evaluate, the imitation video and the target video can be aligned before the evaluation. Next, the first video frame of the imitation video at the beginning of training and the first video frame of the target video at the beginning of physical training can be aligned. Subsequently, the imitation video and the target video are compared and analyzed frame by frame in a similar manner, and then the 3D posture difference of the key point of the specified part in each first video frame of the imitation video can be obtained.

[0051] Then, the average value of the 3D posture difference of the key points of all designated parts in the first video frame is determined as the 3D posture difference of the first video frame; and the sum of the 3D posture difference of each first video frame in the imitation video is determined as the 3D posture difference of the imitation video.

[0052] By adopting the above method, the difference of the three-dimensional posture of the imitation video can be accurately calculated to more objectively and accurately represent the difference between the imitation video and the target video.

[0053] (II) Based on the difference of the 3D posture of the imitation video, a pre-trained large model is used to evaluate the posture of the imitation video and obtain the evaluation result.

[0054] Specifically, the difference of the three-dimensional posture of the imitation video is input into the pre-trained large model. The large model can perform posture evaluation on the imitation video based on the difference of the three-dimensional posture of the imitation video, obtain the evaluation result, and output it.

[0055] By adopting the above technical solution, the imitation video can be evaluated accurately and effectively.

[0056] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure; the video processing method based on the large model of this embodiment, in the above Figure 1 Based on the technical solution of the embodiment shown in the figure, the technical solution of the present disclosure is further described in more detail. Figure 1 Taking the application of the technical solution in a physical training scenario as an example, the corresponding target video may be a target physical training video, and the imitation video may be an imitation physical training video. Figure 2 As shown, the video processing method based on the large model of this embodiment may specifically include the following steps:

[0057] S201, collecting imitation physical training videos made by users based on recommended target physical training videos;

[0058] S202, based on the simulated physical training video, using a large model, extracting the three-dimensional posture of the simulated physical training video;

[0059] S203, acquiring the three-dimensional posture of the target physical training video from a pre-created multi-modal physical training database;

[0060] For example, the multimodal physical training database of this embodiment includes text, video and 3D posture physical training data. When this step is implemented, the 3D posture of the target physical training video can be directly obtained from the physical training database.

[0061] Optionally, in an embodiment of the present disclosure, before step S203, the following steps may also be included:

[0062] (1) Collect multiple physical training videos;

[0063] Specifically, it refers to collecting standard physical training videos of professionals.

[0064] (2) Configure corresponding text descriptions for each physical training video;

[0065] For example, the text description may be a description of the training objectives and main action introduction of the physical training video. For example, for a physical training video, the text description may include: the physical training video is a set of training actions for a specified part A and a specified part B, and the set of training actions includes the specified part A performing multiple actions such as step 1, step 2, and step 3; the specified part B performing multiple actions such as step 1', step 2', and step 3'.

[0066] (3) Annotate the three-dimensional posture of each physical training video;

[0067] In this embodiment, a physical training video can be used to train at least one designated part of the human body, including a video consisting of multiple video frames. For example, the designated part can refer to any part of the human body, including the shoulder, neck, elbow, knee, head, etc. For each physical training video, the three-dimensional posture of the physical training video can include the three-dimensional posture of each node of the action execution part in each physical training video frame. Based on step (2), the designated part corresponding to each physical training video can be known. Therefore, this step can only mark the three-dimensional posture of each node of each designated part. Further, since a designated part in the human skeleton can include many nodes, in this embodiment, for each designated part, only some representative key nodes can be selected, and each designated part can include at least one key node. Therefore, for each physical training video, the three-dimensional posture of the physical training video can include the three-dimensional posture of the key nodes of the designated part in each physical training video frame.

[0068] When this step is implemented specifically, it can be implemented based on hybrid inverse kinematics (hybrIK), mapping the physical training video collected in step (1) to the constructed three-dimensional human skeleton structure, and then converting the precise 3D joints into rotations of body parts through torsion and pendulum decomposition. Specifically, the swing rotation is solved by 3D joint analysis, while the twist rotation is derived from visual clues through a neural network. In this way, the three-dimensional posture of the key nodes of the specified parts in each physical training video frame included in each physical training video can be obtained. In order to improve the accuracy of the annotation, the three-dimensional posture of the key nodes of the specified parts in each physical training video frame obtained by the above method can also be reviewed by an expert reviewer. If there is any unreasonableness, it is adjusted in time to annotate the three-dimensional posture of each physical training video with higher accuracy.

[0069] (4) For each physical training video, multimodal physical training data is constructed based on the physical training video, the corresponding text description and the corresponding three-dimensional posture to obtain a multimodal physical training database.

[0070] For each physical training video, the text description and the corresponding three-dimensional posture are collected in the above manner, and added to the multimodal physical training database as a multimodal physical training data. In other words, the multimodal physical training database includes information of three modes: video, text, and three-dimensional posture of physical training, so it is called a multimodal physical training database. The multimodal physical training database data constructed by the above manner of this embodiment is very rich and comprehensive, and can provide effective data support for video evaluation of large models.

[0071] In the scenario of this embodiment, the multi-modal physical training database can be input into the large model in advance for use by the large model. Alternatively, an acquisition interface can be provided for the large model to be called at any time.

[0072] S204, based on the 3D posture of the simulated physical training video and the 3D posture of the target physical training video, using the pre-trained large model, calculating the difference of the 3D posture of the simulated physical training video;

[0073] S205, based on the difference of the three-dimensional posture of the simulated physical training video, using the pre-trained large model, evaluate the simulated physical training video to obtain an evaluation result;

[0074] Steps S204-S205 are all completed in the large model. The large model can calculate the difference between the three-dimensional posture of the simulated physical training video and the three-dimensional posture of the target physical training video based on the three-dimensional posture of the simulated physical training video and the three-dimensional posture of the target physical training video, as the difference of the three-dimensional posture of the simulated physical training video. Then, according to the difference of the three-dimensional posture of the simulated physical training video, the effect of the simulated physical training video can be evaluated. For example, the evaluation result can be a value between 0 and 1. The larger the value, the closer the simulated physical training video is to the target physical training video, and the better the training effect; conversely, the smaller the value, the greater the gap between the simulated physical training video and the target physical training video, and the worse the training effect.

[0075] For example, step S204 may include the following steps when implemented:

[0076] (a1), for each first video frame in the simulated physical training video, obtaining a difference in the three-dimensional posture of the key point of the specified part in the first video frame according to the three-dimensional posture of the key point of the specified part in the first video frame and the three-dimensional posture of the corresponding key point of the specified part in the corresponding second video frame in the target physical training video;

[0077] For the convenience of description, in this embodiment, the video frames in the simulated physical training video are referred to as first video frames, and the video frames in the target physical training video are referred to as second video frames.

[0078] In order to accurately evaluate, in this embodiment, the simulated physical training video and the target physical training video can be aligned before the evaluation. For example, the first frame of the simulated physical training video at the beginning of the physical training can be aligned with the first frame of the target physical training video at the beginning of the physical training. Then the two videos are compared and analyzed frame by frame. For example, the three-dimensional posture of the key points of the specified parts in each first video frame of the simulated physical training video is compared and analyzed with the three-dimensional posture of the corresponding key points of the corresponding specified parts in the corresponding second video frame of the target physical training video, and the three-dimensional posture difference of the key points of the specified parts in the first video frame is obtained.

[0079] If the designated part includes multiple key points, the 3D posture difference of each key point of the designated part in the first video frame can be obtained in the above manner. If the first video frame includes multiple designated parts, the 3D posture difference of each key point of each designated part needs to be obtained in the above manner.

[0080] (b1), obtaining an average value of the 3D posture difference values ​​of all key points of all designated parts in the first video frame as the 3D posture difference value of the first video frame;

[0081] For example, if the simulated physical training video and the target physical training video involve physical training of two designated parts, designated part A and designated part B, designated part A includes three key points 1, 2, and 3, and designated part B includes three key points 4, 5, and 6. The three-dimensional posture difference of the first video frame can be expressed by the following formula:

[0082] D=(|Z CA1 -Z TA1 |+|Z CA2 -Z TA2 |+|Z CA3 -Z TA3 |+|Z CB4 -Z TB4 |+

[0083] |Z CB5 -Z TB5 |+|Z CB6 -Z TB6 |) / 6

[0084] Among them, Z CA1 represents the 3D posture of key point 1 of the specified part A in the first video frame of the imitation physical training video, Z TA1 represents the 3D posture of the key point 1 of the specified part A of the second video frame aligned with the first video frame in the target physical training video, |Z CA1 -Z TA1 | represents the 3D posture difference of key point 1 of the specified part A in the first video frame of the simulated physical training video. Similarly, |Z CA2 -Z TA2 | represents the three-dimensional posture difference of the key point 2 of the specified part A in the first video frame of the simulated physical training video; | Z CA3 -Z TA3 | represents the three-dimensional posture difference of the key point 3 of the specified part A in the first video frame of the simulated physical training video; | Z CB4 -Z TB4 | represents the three-dimensional posture difference of the key point 4 of the designated part B in the first video frame of the imitation physical training video; | Z CB5 -Z TB5 | represents the three-dimensional posture difference of the key point 5 of the designated part B in the first video frame of the imitation physical training video; | Z CB6 -Z TB6 | represents the three-dimensional posture difference of the key point 6 of the specified part B of the first video frame in the simulated physical training video.

[0085] (c1) Obtaining the sum of the three-dimensional posture differences of each first video frame in the simulated physical training video as the three-dimensional posture difference of the simulated physical training video.

[0086] If the simulated physical training video includes multiple first video frames, the sum of the three-dimensional posture differences of the multiple first video frames is obtained as the three-dimensional posture difference of the simulated physical training video.

[0087] By adopting the above method, the difference of the three-dimensional posture of the simulated physical training video can be accurately calculated, so as to more objectively and accurately represent the difference between the simulated physical training video and the target physical training video.

[0088] S206, displaying the evaluation result to the user;

[0089] Specifically, the evaluation results can be directly displayed on the interface of the video processing device based on the large model, so that users can view them in time and understand the training effect.

[0090] S207, based on the evaluation results, using the large model to generate training improvement suggestions;

[0091] Specifically, the large model can know the quality of the imitation physical training video according to the evaluation results, and then give training modification suggestions. For example, if the evaluation result is greater than or equal to the first preset score, it means that the training effect is very good, and the training modification suggestions generated at this time can be: the action is completed very well, please continue to maintain; if the evaluation result is greater than or equal to the second preset score, but less than the first preset score, it means that the training effect is OK, and the training modification suggestions generated at this time can be: the action is completed OK, but please continue to practice and strive to complete it better. If the evaluation result is less than the second preset score, it means that the training effect is not ideal, and the training modification suggestions generated at this time can be: the action is not completed very well, but don't be discouraged, please refer to the target physical training video to relearn, and keep going. The above-mentioned first preset score and second preset score of this embodiment can be set according to actual conditions, and are not limited here. Moreover, in the scenario of this embodiment, the three-segment effect is divided into two preset scores as an example. In actual applications, the evaluation results can also be divided into two, four or other multiple segments, which are not limited here.

[0092] Further optionally, in one embodiment of the present disclosure, a large model can be used to generate training improvement suggestions based on the evaluation results and with reference to the three-dimensional posture of the imitation physical training video and the three-dimensional posture of the target physical training video. This allows the large model to refer to more information and obtain more accurate training modification suggestions. For example, the training improvement suggestions generated at this time can point out specific modification suggestions for designated parts, which can guide users more accurately.

[0093] S208. Display training improvement suggestions.

[0094] Step S207 and step S208 are supplementary opinions to provide further guidance to the user. Specifically, step S207 can be performed before step S206, and step S206 and step S208 can be combined and displayed together.

[0095] In this embodiment, the large model is mainly used to implement training evaluation. It can be considered that a training evaluation module is provided in the large model to implement the training evaluation of this embodiment.

[0096] Before use, the evaluation module of the large model needs to be trained. The specific training principle is similar to the above-mentioned application principle of this embodiment. By collecting multiple groups of target physical training videos and imitation physical training videos and marking the corresponding evaluation results, the evaluation module is trained so that the evaluation module learns the evaluation of physical training videos. For the specific training process, you can refer to the implementation principle of the relevant supervised training, which will not be repeated here.

[0097] The video processing method based on the large model of this embodiment can accurately and effectively evaluate the simulated physical training video by adopting the above technical solution and using the large model. Moreover, it can also give accurate and reasonable training improvement suggestions based on the evaluation results to provide users with further effective guidance.

[0098] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure; in this embodiment, in the above-mentioned FIG. Figure 2 Based on the technical solutions of the embodiments shown in the figure, the technical solutions of the present disclosure are further described in more detail. Figure 3 As shown, the video processing method based on the large model of this embodiment may specifically include the following steps:

[0099] S301, based on a pre-established physical training plan, using a pre-trained large model, obtaining a corresponding target physical training video from a multi-modal physical training database;

[0100] S302: Recommend target physical training videos to the user.

[0101] Above Figure 2 The target physical training video in the illustrated embodiment may be selected manually in advance or selected through other channels. In this embodiment, the function of the video processing device based on the large model may be further enriched, and the device may also provide a function of recommending training videos.

[0102] The physical training plan of this embodiment may also be formulated in advance manually, or formulated through other channels.

[0103] During specific use, a physical training plan is input into the large model. Based on the physical training plan, the model can filter and obtain target physical training videos from a multimodal physical training database and recommend them to users.

[0104] It should be noted that, in this embodiment, physical training is taken as an example, and in actual application, it can also be applied to other trainings, such as dance training, etc.

[0105] In the embodiment of the present invention, the target physical training video can be recommended to the user by directly displaying the target physical training video on the display interface of the video processing device based on the large model used by the user. Alternatively, the link of the target physical training video can be sent to the account of the user who logs into the video processing device based on the large model to recommend it to the user. In this way, when the user logs into the account in subsequent training, the recommended target physical training video can be viewed at any time.

[0106] The video processing method based on the big model of this embodiment mainly uses the big model to implement the recommendation function of the target physical training video to implement the recommendation of the training video of this embodiment. It can be considered that a training video recommendation module is set in the big model to implement the training video recommendation of this embodiment.

[0107] Before use, the training video recommendation module of the large model needs to be trained. The specific training principle is similar to the above-mentioned application principle of this embodiment. By collecting physical training plans and training videos to be recommended, the training video recommendation module is trained so that the training video recommendation module learns the recommendation of physical training videos. For the specific training process, reference can be made to the implementation principle of related supervised training, which will not be repeated here.

[0108] The large model-based video processing method of this embodiment can effectively utilize the physical training video recommendation function of the large model to accurately and effectively recommend target physical training videos to users.

[0109] Figure 4 is a schematic diagram according to the fourth embodiment of the present disclosure; Figure 2-Figure 3 Based on the technical solution of any of the illustrated embodiments, the technical solution of the present disclosure is further described in more detail. Figure 4 As shown, the video processing method based on the large model of this embodiment may specifically include the following steps:

[0110] S401, obtaining a user's physical training request;

[0111] S402, based on the physical training request, using the pre-trained large model to formulate a physical training plan for the user;

[0112] S403: Recommend a physical training plan to the user.

[0113] The physical training plan in the embodiment 3 above may also be manually formulated in advance, or formulated through other channels. In this embodiment, the function of the video processing device based on the large model can be further enriched, and the device can also provide the function of formulating a physical training plan.

[0114] During specific use, the user's physical training request is input into the large model, and the model can formulate a physical training plan for the user based on the user's physical training request and recommend it to the user.

[0115] Similarly, the method of recommending a physical training plan to a user in this embodiment can directly display the physical training plan on the display interface of the video processing device based on the large model used by the user to achieve the recommendation. Alternatively, the physical training plan can be sent to the account of the user who logs in to the video processing device based on the large model to recommend it to the user. In this way, when the user logs in to the account during subsequent training, he can view the recommended physical training plan at any time.

[0116] It should be noted that, in this embodiment, physical training is taken as an example. In actual applications, it can also be applied to other trainings, such as dance training, etc.

[0117] In this embodiment, the large model is mainly used to implement the physical training plan formulation function to implement the formulation of the physical training plan of this embodiment. It can be considered that a physical training plan formulation module is provided in the large model to implement the formulation of the physical training plan of this embodiment.

[0118] Specifically, before use, the physical training plan formulation module of the large model needs to be trained. Specifically, the training principle is similar to the above-mentioned application principle of this embodiment. By collecting physical training requests and marked formulated physical training plans, the physical training plan formulation module is trained so that the physical training plan formulation module learns the formulation of physical training plans. For the specific training process, reference can be made to the implementation principle of the relevant supervised training, which will not be repeated here.

[0119] Optionally, the physical training plan formulation module of this embodiment is the same as the above Figure 3 The training video recommendation modules in the illustrated embodiment may also be combined into one.

[0120] It should be noted that the big model of this embodiment can be based on the existing open source language big model. The model is first trained using physical fitness literature data, and then fine-tuned using a pre-created physical fitness training text database, so as to combine literature knowledge with training plan formulation tasks, thereby guiding practice with theory.

[0121] The physical training text database of this embodiment can be created in the following manner:

[0122] (a2) collecting multiple physical training files;

[0123] The physical training document of this embodiment may be an electronic book or document.

[0124] Using open source tools, optical character recognition (OCR) is used to digitize electronic books or documents, and then convert them into machine-readable text information in txt format. After that, the invalid content after digitization is deleted using the big model, such as invalid content such as table structure information during digitization. The big model is also used to optimize the sentence structure and expression to obtain the final physical training document to ensure the accuracy of subsequent processing.

[0125] (b2) Based on each physical training document, a pre-trained question-answer pair generation model is used to generate multiple pairs of physical training knowledge question-answer pairs, wherein the physical training knowledge question-answer pairs include physical training knowledge questions and physical training knowledge answers; the question-answer pair generation model is implemented using a large model;

[0126] (c2) Based on multiple pairs of physical training knowledge questions and answers, a physical training text database is constructed.

[0127] The question-answer pair generation model of the present embodiment can also be implemented based on a large model. By continuously adjusting the prompt words (prompt) of the large model, multiple pairs of physical training knowledge question-answer pairs for the current physical training document are generated, and each physical training knowledge question-answer pair includes a physical training knowledge question and a physical training knowledge answer. The physical training knowledge question-answer pairs generated by the present embodiment can be in Chinese or in English. Moreover, it can include a single round of professional physical knowledge question-answering, it can also include multiple rounds of professional physical knowledge question-answering, and it can also include long questions and answers.

[0128] The physical training knowledge question and answer pair of this embodiment may include: physical training request and physical training plan. Therefore, the physical training text database of this embodiment is used to train the large model, so that the large model can master the basic knowledge in the field of physical training and have a preliminary physical training plan formulation function. In order to further improve the formulation efficiency of the physical training plan formulation module of the large model, the physical training request and the marked formulated physical training plan can be further used for fine-tuning to enhance the formulation accuracy of the physical training plan formulation module.

[0129] The video processing method based on the big model of this embodiment can effectively utilize the physical training plan formulation function of the big model to accurately and effectively formulate a physical training plan for the user.

[0130] Based on the above embodiments, it can be known that in an application scenario of the present disclosure, a large model physical training plan formulation module can be first used, according to Figure 4 The technical solution of the embodiment shown in the figure formulates a physical training plan for the user. Further, based on the formulated physical training plan, a training video recommendation module of a large model is used to Figure 3 The technical solution of the illustrated embodiment recommends a target physical training video for the user. Furthermore, based on the target physical training video recommended for the user, the evaluation module of the large model is used to evaluate the imitation physical training video completed by the user. The above process can assist the user to complete the entire process of physical training. The text, video and three-dimensional posture data knowledge of physical training can be integrated during the processing process. Not only can training plans and teaching videos be recommended for users accurately and effectively, but also the training movements of users can be accurately and effectively evaluated to achieve instant feedback, thereby assisting users to improve training effects to the greatest extent.

[0131] Based on the above, it can be seen that the above Figure 2 , Figure 3 as well as Figure 4 The embodiments shown can exist independently as an application scenario of the present disclosure. They can also be combined together to form an application scenario of the present disclosure. In practical applications, they can also be combined to form an application scenario of the present embodiment. For example, Figure 2 and Figure 3 The embodiment shown is combined, Figure 2 and Figure 4 The embodiment shown is combined, Figure 3 and Figure 4 The combination of the shown embodiments can respectively constitute different application scenarios of the present disclosure.

[0132] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure; this embodiment provides a video processing device 500 based on a large model, including:

[0133] The video acquisition module 501 is used to acquire the imitation video made by the user based on the target video;

[0134] A posture extraction module 502 is used to extract the three-dimensional posture of the imitation video based on the imitation video using a pre-trained large model;

[0135] The video evaluation module 503 is used to perform posture evaluation on the imitation video based on the 3D posture of the imitation video and the 3D posture of the pre-acquired target video, using the pre-trained large model to obtain an evaluation result.

[0136] The large model-based video processing device 500 of this embodiment implements the implementation principle and technical effect of large model-based video processing by adopting the above-mentioned modules, which is the same as the implementation of the above-mentioned related method embodiments. For details, please refer to the records of the above-mentioned related method embodiments, which will not be repeated here.

[0137] Figure 6 is a schematic diagram according to the sixth embodiment of the present disclosure; the video processing device 600 based on the large model of this embodiment, in the above Figure 5 Based on the technical solutions of the embodiments shown in the figure, the technical solutions of the present disclosure are further described in more detail. Figure 6 As shown, the video processing device 600 based on the large model of this embodiment includes the above Figure 5 The modules with the same name and function are shown as follows: a video acquisition module 601 , a posture extraction module 602 and a video evaluation module 603 .

[0138] In this embodiment, the video evaluation module 603 includes:

[0139] A difference calculation unit 6031 is used to calculate the difference of the 3D posture of the imitation video based on the 3D posture of the imitation video and the 3D posture of the target video using the pre-trained large model;

[0140] The video evaluation unit 6032 is used to perform posture evaluation on the imitation video based on the difference of the three-dimensional posture of the imitation video and adopt the pre-trained large model to obtain the evaluation result.

[0141] Further optionally, in one embodiment of the present disclosure, the difference calculation unit 6031 is used to:

[0142] For each first video frame in the imitation video, obtaining a difference in the three-dimensional posture of the key point of the designated part in the first video frame according to the three-dimensional posture of the key point of the designated part in the first video frame and the three-dimensional posture of the corresponding key point of the designated part in the corresponding second video frame in the target video;

[0143] Determine an average value of the 3D posture differences of all key points of all designated parts in the first video frame as the 3D posture difference of the first video frame;

[0144] The sum of the three-dimensional posture differences of the first video frames in the imitation video is determined as the three-dimensional posture difference of the imitation video.

[0145] Further optionally, if Figure 6 As shown, in one embodiment of the present disclosure, the video processing device 600 based on the large model of this embodiment further includes:

[0146] A suggestion generation module 604 is used to generate training improvement suggestions based on the evaluation results and using the pre-trained large model;

[0147] The suggestion display module 605 is used to display the training improvement suggestions.

[0148] Further optionally, in one embodiment of the present disclosure, the suggestion generation module 604 is used to:

[0149] Based on the evaluation result, and with reference to the three-dimensional posture of the imitation video and the three-dimensional posture of the target video, the pre-trained large model is used to generate training improvement suggestions.

[0150] Further optionally, in one embodiment of the present disclosure, the device is applied to the processing of physical training videos, the target video includes a target physical training video, and the imitation video includes an imitation physical training video; the posture acquisition module 602 is also used to obtain the three-dimensional posture of the target physical training video from a multimodal physical training database; the multimodal physical training database includes text, video and three-dimensional posture physical training data.

[0151] Further optionally, if Figure 6 As shown, in one embodiment of the present disclosure, the video processing device 600 based on the large model of this embodiment further includes a construction module 606:

[0152] Collect multiple physical training videos;

[0153] Configuring a corresponding text description for each of the physical training videos;

[0154] annotating the three-dimensional posture of each of the physical training videos;

[0155] For each of the physical training videos, multimodal physical training data is constructed based on the physical training video, the corresponding text description and the corresponding three-dimensional posture to obtain the multimodal physical training database.

[0156] The large model-based video processing device 600 of this embodiment implements the implementation principle and technical effect of large model-based video processing by adopting the above-mentioned modules, which is the same as the implementation of the above-mentioned related method embodiments. For details, please refer to the records of the above-mentioned related method embodiments, which will not be repeated here.

[0157] Figure 7 is a schematic diagram according to the seventh embodiment of the present disclosure; the video processing device 700 based on the large model of this embodiment includes:

[0158] The video acquisition module 701 is used to acquire the corresponding target physical training video from the multimodal physical training database using the pre-trained large model based on the pre-established physical training plan;

[0159] The video recommendation module 702 is used to recommend the target physical training video to the user.

[0160] The large model-based video processing device 700 of this embodiment implements the implementation principle and technical effect of large model-based video processing by adopting the above-mentioned modules, which is the same as the implementation of the above-mentioned related method embodiments. For details, please refer to the records of the above-mentioned related method embodiments, which will not be repeated here.

[0161] Figure 8 is a schematic diagram according to an eighth embodiment of the present disclosure; the video processing device 800 based on a large model of this embodiment includes:

[0162] A request acquisition module 801 is used to acquire the user's physical training request;

[0163] A plan making module 802 is used to make the physical training plan for the user based on the physical training request and using the pre-trained large model;

[0164] The plan recommendation module 803 is used to recommend the physical training plan to the user.

[0165] The large model-based video processing device 800 of this embodiment implements the implementation principle and technical effect of large model-based video processing by adopting the above-mentioned modules, which is the same as the implementation of the above-mentioned related method embodiments. For details, please refer to the records of the above-mentioned related method embodiments, which will not be repeated here.

[0166] It should be noted that the above Figure 7 and Figure 8 The embodiments shown can be combined together to form an embodiment of the present disclosure; Figure 5 or Figure 6 The above embodiments are combined to form an embodiment of the present disclosure; they can also be combined with the above embodiments Figure 5 or Figure 6 The combination of the embodiments constitutes one embodiment of the present disclosure.

[0167] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0168] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0169] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0170] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0171] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0172] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the above-mentioned methods of the present disclosure. For example, in some embodiments, the above-mentioned methods of the present disclosure may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the above-mentioned methods of the present disclosure described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the above-mentioned methods of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0173] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0174] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0175] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0176] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0177] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0178] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0179] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0180] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A video processing method based on a large model, the method comprising: Collect imitation videos made by users based on the target video; Based on the imitation video, using a pre-trained large model, extracting the three-dimensional posture of the imitation video; Based on the three-dimensional posture of the imitation video and the three-dimensional posture of the pre-acquired target video, the pre-trained large model is used to perform posture evaluation on the imitation video to obtain an evaluation result.

2. The method according to claim 1, wherein: Based on the three-dimensional posture of the imitation video and the three-dimensional posture of the pre-acquired target video, the pre-trained large model is used to perform posture evaluation on the imitation video to obtain an evaluation result, including: Based on the three-dimensional posture of the imitation video and the three-dimensional posture of the target body, the pre-trained large model is used to calculate the difference of the three-dimensional posture of the imitation video; Based on the difference of the three-dimensional posture of the imitation video, the pre-trained large model is used to perform posture evaluation on the imitation video to obtain the evaluation result.

3. The method according to claim 2, wherein: Calculating a difference in the three-dimensional posture of the imitation video based on the three-dimensional posture of the imitation video and the three-dimensional posture of the target body includes: For each first video frame in the imitation video, obtaining a difference in the three-dimensional posture of the key point of the designated part in the first video frame according to the three-dimensional posture of the key point of the designated part in the first video frame and the three-dimensional posture of the corresponding key point of the designated part in the corresponding second video frame in the target video; Determine an average value of the three-dimensional posture difference values ​​of the key points of all designated parts in the first video frame as the three-dimensional posture difference value of the first video frame; The sum of the three-dimensional posture differences of the first video frames in the imitation video is determined as the three-dimensional posture difference of the imitation video.

4. The method according to claim 1, wherein: The method further comprises: Based on the evaluation results, using the large model, generating training improvement suggestions; Presenting said training improvement suggestion.

5. The method according to claim 4, wherein: Based on the evaluation results, the large model is used to generate training improvement suggestions, including: Based on the evaluation result, and with reference to the three-dimensional posture of the imitation video and the three-dimensional posture of the target video, the pre-trained large model is used to generate training improvement suggestions.

6. The method according to any one of claims 1 to 5, wherein: The method is applied to the processing of a physical training video, the target video includes a target physical training video, the imitation video includes an imitation physical training video, and the method further includes: The three-dimensional posture of the target physical training video is obtained from a multimodal physical training database; the multimodal physical training database includes physical training data of text, video and three-dimensional posture.

7. The method according to claim 6, further comprising: Collect multiple physical training videos; Configuring a corresponding text description for each of the physical training videos; annotating the three-dimensional posture of each of the physical training videos; For each of the physical training videos, multimodal physical training data is constructed based on the physical training video, the corresponding text description and the corresponding three-dimensional posture to obtain the multimodal physical training database.

8. The method according to claim 6, further comprising: Based on a pre-established physical training plan, the large model is used to obtain a corresponding target physical training video from the multimodal physical training database; The target physical training video is recommended to the user.

9. The method according to claim 8, further comprising: Obtaining a physical training request of the user; Based on the physical training request, using the large model, formulate the physical training plan for the user; The physical training plan is recommended to the user.

10. A video processing device based on a large model, the device comprising: A video acquisition module is used to acquire imitation videos made by users based on target videos; A posture extraction module, used to extract the three-dimensional posture of the imitation video based on the imitation video using a pre-trained large model; The video evaluation module is used to perform posture evaluation on the imitation video based on the three-dimensional posture of the imitation video and the three-dimensional posture of the pre-acquired target video, using the pre-trained large model to obtain an evaluation result.

11. The device according to claim 10, wherein: The video evaluation module comprises: A difference calculation unit, configured to calculate the difference of the 3D posture of the imitation video based on the 3D posture of the imitation video and the 3D posture of the target video using the pre-trained large model; The video evaluation unit is used to perform posture evaluation on the imitation video based on the difference of the three-dimensional posture of the imitation video and adopt the pre-trained large model to obtain the evaluation result.

12. The device according to claim 11, wherein The difference calculation unit is used for: For each first video frame in the imitation video, obtaining a difference in the three-dimensional posture of the key point of the designated part in the first video frame according to the three-dimensional posture of the key point of the designated part in the first video frame and the three-dimensional posture of the corresponding key point of the designated part in the corresponding second video frame in the target video; Determine an average value of the 3D posture differences of all key points of all designated parts in the first video frame as the 3D posture difference of the first video frame; The sum of the three-dimensional posture differences of the first video frames in the imitation video is determined as the three-dimensional posture difference of the imitation video.

13. The device according to claim 10, wherein: The device also includes: A suggestion generation module, for generating training improvement suggestions based on the evaluation results and using the large model; The suggestion display module is used to display the training improvement suggestions.

14. The device according to claim 13, wherein: The suggestion generation module is used to: Based on the evaluation result, and with reference to the three-dimensional posture of the imitation video and the three-dimensional posture of the target video, the pre-trained large model is used to generate training improvement suggestions.

15. The device according to any one of claims 10 to 14, wherein: The device is applied to the processing of physical training videos, wherein the target video includes a target physical training video, and the imitation video includes an imitation physical training video; the posture acquisition module is further used for: The three-dimensional posture of the target physical training video is obtained from a multimodal physical training database; the multimodal physical training database includes physical training data of text, video and three-dimensional posture.

16. The device according to claim 15, wherein: The device also includes a building block: Collect multiple physical training videos; Configuring a corresponding text description for each of the physical training videos; annotating the three-dimensional posture of each of the physical training videos; For each of the physical training videos, multimodal physical training data is constructed based on the physical training video, the corresponding text description and the corresponding three-dimensional posture to obtain the multimodal physical training database.

17. The device according to claim 16, wherein: The device also includes: A video acquisition module, configured to acquire a corresponding target physical training video from the multimodal physical training database using the large model based on a pre-established physical training plan; A video recommendation module is used to recommend the target physical training video to the user.

18. The device according to claim 17, wherein: The device also includes: A request acquisition module, used to acquire the user's physical training request; A plan making module, configured to make the physical training plan for the user based on the physical training request and using the large model; A plan recommendation module is used to recommend the physical training plan to the user.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

21. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images