Sports skill learning method and device based on style transfer and AR wearable facilities

Abstract

The present application relates to a method and device for learning sports skills based on style transfer and AR wearable facilities. The method includes: sampling the learner's action video stream to obtain the source video frame of the learner's own action; migrating the source video frame to the instructor's demonstration action video stream according to the style transfer algorithm to obtain the corrected learner action video stream; using AR wearable facilities to stack and compare the corrected learner action video stream and the learner action video stream in real time, and obtain a video superposition stream of the learner's real action and the learner's corrected action; migrating the single-frame picture of the learner at a specified time in the learner's action video stream to the video frame of the specified time in the corrected learner action video stream according to the joint step-by-step migration algorithm, and obtaining an action transfer slice; and learning sports skills according to the video superposition stream and the action transfer slice. The use of this method can improve the accuracy of sports skill learning.

Description

Technical Field

[0001] The present application relates to the technical fields of online education, sports training and augmented reality, and in particular to a method, apparatus, computer equipment and storage medium for learning sports skills based on style transfer and AR wearable facilities. Background Art

[0002] Due to the limitations of the online characteristics of the MOOC teaching model, practical courses that are closely coupled with the physical world, such as sports training, engineering practice, and military training, are currently difficult to achieve good teaching results on the MOOC platform.

[0003] However, the current MOOC education program has been able to build a digital twin of the learning environment, allowing learners to experience the virtual environment settings of the course (such as virtual stadiums, virtual machine tools, virtual laboratories, etc.) in an immersive way. However, it still lacks the empowerment of the course teaching content itself, making it impossible for learners to effectively practice and understand the key points of the movements demonstrated in the course, and it is even more difficult to correct their own wrong movements in a targeted manner. Taking the sports training MOOC course as an example, the core difficulties mainly lie in the following three aspects: 1) "Learning-Practice" time and space disconnection-when in motion, it is difficult for learners to take into account both the online teaching demonstration movements and their own real-time movements at the same time. In recent years, the emerging "smart full-body mirror" and other fixed display devices can display MOOC teaching videos on the display and allow users to see their own movements through the principle of mirror reflection, but they cannot solve the problem of learners leaving the display device due to movement needs, resulting in blind spots in the field of vision; such methods cannot support a larger range of motion and activity area. In addition, the completion time of the two types of action sequences, the demonstration action of the MOOC instructor and the learner's own real-time action, must match each other, so that accurate action comparison can be achieved and learners can efficiently discover and correct their own mistakes in action, rhythm, speed, etc. 2) The individual characteristics of the "teaching-learning" parties are very different - the learner's own actions (observed or filmed) and the MOOC video (filmed) instructor's demonstration actions, in addition to the possible differences in joint angles caused by the different completion of the action essentials, there are also many differences unrelated to the joint angles, making it difficult for learners to intuitively and effectively achieve rapid action correction through video comparison. These differences unrelated to joint angles include age, gender, body shape, clothing, sports equipment, camera shooting angle, focal length, etc. 3) The motion capture analysis system equipment is bulky and expensive - the existing motion capture and analysis system has serious dependence on the environment, equipment and supporting software. The supporting conditions such as full-body reflective balls, infrared motion capture cameras, and green screens are expensive, demanding, and poorly portable, and often require professional technicians to provide a high degree of manual intervention and intervention. Summary of the invention

[0004] Based on this, it is necessary to provide a sports skill learning method, device, computer equipment and storage medium based on style transfer and AR wearable facilities, which can improve the accuracy of sports skill learning, in response to the above technical problems.

[0005] A method for learning sports skills based on style transfer and AR wearable devices, the method comprising:

[0006] Acquire teaching action video; the teaching action video includes the instructor's demonstration action video stream and the learner's action video stream;

[0007] Sampling the learner's action video stream to obtain the source video frame of the learner's own action;

[0008] The source video frame is transferred to the instructor's demonstration action video stream according to the style transfer algorithm to obtain the corrected learner action video stream;

[0009] The corrected learner's action video stream and the learner's action video stream are stacked and compared online in real time using AR wearable equipment to obtain a video superposition stream of the learner's real action and the learner's corrected action;

[0010] According to the joint step-by-step transfer algorithm, a single frame image at a specified time of the learner in the learner's action video stream is transferred to a video frame at a specified time in the corrected learner's action video stream to obtain an action transfer slice;

[0011] Motor skill learning from video overlay streams and action transfer slices.

[0012] In one of the embodiments, the process of obtaining teaching action video includes: obtaining the instructor's demonstration action video from the MOOC platform, using a smart terminal to shoot the action video of the learner wearing the AR wearable device in real time on the spot, and based on the WIFI Direct protocol, realizing low-latency data transmission between the smart terminal and the AR wearable device.

[0013] In one embodiment, the source video frame is transferred to the instructor's demonstration action video stream according to the style transfer algorithm to obtain the corrected learner action video stream, including:

[0014] Obtain a target video frame of the instructor's demonstration action from the instructor's demonstration action video stream, calculate the source video frame and the target video frame using a human body mesh recovery toolkit, and obtain an association mapping corresponding to the source video frame and the target video frame; the association mapping includes weak perspective projection, body shape parameters, and 3D human body mesh;

[0015] According to the element-level matching of the association mappings corresponding to the source video frame and the target video frame, a transformation stream is obtained; the transformation stream refers to a transformation matrix from the association mapping of the source video frame to the association mapping of the target video frame;

[0016] Decompose the source video frame to obtain background projection components and foreground projection components;

[0017] Performing matrix transformation operation on the associated mapping of the source video frame using the transformation stream to obtain the transformation projection component from the target video frame to the source video frame;

[0018] The pre-constructed autoencoder neural network model is trained using the background projection component, the foreground projection component and the transformed projection component to obtain an autoencoder neural network model of the background projection component, an autoencoder neural network model of the foreground projection component and an autoencoder neural network model of the transformed projection component;

[0019] The source video frames and the target video frames of the instructor's demonstration action are input into the autoencoder neural network model of the background projection component, the autoencoder neural network model of the foreground projection component and the autoencoder neural network model of the transformation projection component. The three output pictures are integrated pixel by pixel to obtain the corrected learner action video stream.

[0020] In one embodiment, the corrected learner action video stream and the learner action video stream are stacked and compared online in real time by using an AR wearable device to obtain a video superposition stream of the learner's real action and the learner's corrected action, including:

[0021] Performing background transparency processing on the corrected learner action video stream and the learner action video stream to obtain the corrected learner action image and the actual image before correction;

[0022] The corrected learner action image is aligned and overlapped with the actual image before correction by using the axis in the center of gravity, and the actual image before correction is superimposed on the corrected learner action image in a semi-transparent manner to obtain a stacked video stream;

[0023] The stacked video stream is transmitted to the AR wearable device with low latency according to the WIFI Direct protocol, and refracted to the learner's retina through the visual imaging module of the AR wearable device for online comparison, obtaining a video superposition stream of the learner's real movement and the learner's corrected movement.

[0024] In one embodiment, a single frame image of a learner at a specified time in a learner action video stream is migrated to a video frame at a specified time in a corrected learner action video stream according to a joint step-by-step migration algorithm to obtain an action transfer slice, including:

[0025] Based on the 3D skeleton extraction tool K3M, a 3D skeleton image is extracted from a single-frame image of the learner at a specified moment in the learner's action video stream. According to the force chain principle, the 3D skeleton image is moved step by step along the core force chain sequence of "leg-hip-limbs". A frame in the action transfer slice is obtained in each movement until the action posture of the learner's action video frame after correction at the specified moment is fully adjusted to obtain a complete action transfer slice.

[0026] In one embodiment, motor skill learning is performed based on video overlay streams and action transfer slices, including:

[0027] The video overlay stream and motion transfer slices are displayed to users through AR wearable devices to assist and guide learners to correct their movements.

[0028] A sports skill learning device based on style transfer and AR wearable equipment, the device comprising:

[0029] The video stream sampling module is used to obtain teaching action video; the teaching action video includes the instructor's demonstration action video stream and the learner's action video stream; the learner's action video stream is sampled to obtain the source video frame of the learner's own action;

[0030] A style transfer algorithm module is used to transfer the source video frame to the instructor's demonstration action video stream according to the style transfer algorithm to obtain a corrected learner action video stream;

[0031] The stacking and comparison module is used to use AR wearable equipment to perform real-time stacking and online comparison on the corrected learner action video stream and the learner action video stream to obtain a video superposition stream of the learner's real action and the learner's corrected action;

[0032] The sports skill learning module is used to migrate the single-frame image of the learner at a specified time in the learner's action video stream to the video frame at a specified time in the corrected learner's action video stream according to the joint step-by-step migration algorithm, so as to obtain the action transfer slice; and perform sports skill learning based on the video superposition stream and the action transfer slice.

[0033] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0034] Acquire teaching action video; the teaching action video includes the instructor's demonstration action video stream and the learner's action video stream;

[0035] Sampling the learner's action video stream to obtain the source video frame of the learner's own action;

[0036] The source video frame is transferred to the instructor's demonstration action video stream according to the style transfer algorithm to obtain the corrected learner action video stream;

[0037] The corrected learner's action video stream and the learner's action video stream are stacked and compared online in real time using AR wearable equipment to obtain a video superposition stream of the learner's real action and the learner's corrected action;

[0038] According to the joint step-by-step transfer algorithm, a single frame image at a specified time of the learner in the learner's action video stream is transferred to a video frame at a specified time in the corrected learner's action video stream to obtain an action transfer slice;

[0039] Motor skill learning from video overlay streams and action transfer slices.

[0040] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0041] Acquire teaching action video; the teaching action video includes the instructor's demonstration action video stream and the learner's action video stream;

[0042] Sampling the learner's action video stream to obtain the source video frame of the learner's own action;

[0043] The source video frame is transferred to the instructor's demonstration action video stream according to the style transfer algorithm to obtain the corrected learner action video stream;

[0044] The corrected learner's action video stream and the learner's action video stream are stacked and compared online in real time using AR wearable equipment to obtain a video superposition stream of the learner's real action and the learner's corrected action;

[0045] According to the joint step-by-step transfer algorithm, a single frame image at a specified time of the learner in the learner's action video stream is transferred to a video frame at a specified time in the corrected learner's action video stream to obtain an action transfer slice;

[0046] Motor skill learning from video overlay streams and action transfer slices.

[0047] The above-mentioned sports skill learning method, device, computer equipment and storage medium based on style transfer and AR wearable facilities first transfer the source video frame to the instructor's demonstration action video stream according to the style transfer algorithm to obtain the corrected learner action video stream, and based on the style transfer algorithm, transfer the learner's own personalized characteristics unrelated to the joint angle to the MOOC video instructor, so that the generated video not only retains the instructor's standard actions, but also conforms to the learner's own body shape and appearance and other personality characteristics, and uses AR wearable facilities to stack and compare the learner's action video stream after correction and the learner's action video stream before correction in real time. Through the online comparison of the two actions, the learner can directly compare his own actions with the personalized demonstration actions in real time. The difference in the learner's action (i.e. the corrected learner's action) can be used to quickly and effectively correct the technical action, thereby realizing autonomous and real-time correction of the action. In addition, AR wearable facilities are cheaper and more universal than motion capture analysis system equipment. According to the joint step-by-step migration algorithm, the single-frame image of the learner at a specified moment in the learner's action video stream is migrated to the video frame at a specified moment in the corrected learner action video stream to obtain action transfer slices. The action transfer slices are orderly movement sequences that follow the core force chain of "leg-hip-limbs" in sequence and move the joint positions step by step until they coincide with the corrected learner action key frames. Using action slices to assist and guide learners in correcting their actions can specifically guide learners to correct their own actions and improve the accuracy of sports skill learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is an application scenario diagram of a method for learning sports skills based on style transfer and AR wearable equipment in an embodiment;

[0049] Figure 2 A schematic diagram of an application scenario of a method for learning sports skills in an embodiment;

[0050] Figure 3 A schematic diagram of a personalized style transfer model for a lecturer's demonstration action in one embodiment;

[0051] Figure 4 A schematic diagram of a joint step-by-step migration algorithm in another embodiment;

[0052] Figure 5 It is a structural block diagram of a sports skill learning device based on style transfer and AR wearable equipment in one embodiment;

[0053] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] In one embodiment, Figure 1 As shown, a method for learning sports skills based on style transfer and AR wearable facilities is provided, including the following steps:

[0056] Step 102, obtaining teaching action video; the teaching action video includes a video stream of the instructor's demonstration action and a video stream of the learner's action; sampling the learner's action video stream to obtain a source video frame of the learner's own action.

[0057] Step 104 , the source video frame is transferred to the instructor's demonstration action video stream according to the style transfer algorithm to obtain a corrected learner action video stream.

[0058] The source video frames are transferred to the video stream of the instructor's demonstration movements, and the learner's body shape, shooting angle and other features unrelated to the joint angles are transferred to the standard limb postures in the demonstration movement video. This allows the generated video to retain the instructor's standard movements and conform to the learner's own personality characteristics such as body shape and appearance. This solves the problem in the prior art that the instructor's movements cannot be accurately learned due to the large individual differences between the instructor and the learner.

[0059] Step 106, using AR wearable equipment to stack and compare the corrected learner action video stream and the learner action video stream in real time, to obtain a video superposition stream of the learner's real action and the learner's corrected action.

[0060] By using AR wearable equipment to stack and compare the learner's action video stream after correction and the learner's action video stream before correction in real time, learners can directly compare the difference between their own actions and personalized demonstration actions (i.e., learner's actions after correction) in real time, and quickly and effectively correct technical actions. In addition, AR wearable equipment is cheaper and more universal than motion capture analysis system equipment. Learners can realize autonomous and real-time correction of actions by comparing the two actions online.

[0061] Step 108, according to the joint step-by-step migration algorithm, migrate the single frame image at the specified time of the learner in the learner's action video stream to the video frame at the specified time in the corrected learner's action video stream to obtain action transfer slices; and perform sports skill learning based on the video superposition stream and the action transfer slices.

[0062] By using the joint step-by-step migration algorithm, the single-frame image (i.e., key action frame) at the specified moment of the learner in the learner's action video stream is migrated to the corrected learner action video frame at that moment to obtain the action transfer slice. The action transfer slice is an ordered movement sequence that moves the joint position step by step along the core force chain of "leg-hip-limb" until it coincides with the corrected learner action key frame. Assisting and guiding learners to correct their actions based on action slices can specifically guide learners to correct their own actions, provide learners with specific action correction guidance, and improve the accuracy of sports skill learning.

[0063] In the above-mentioned sports skill learning method based on style transfer and AR wearable facilities, the source video frame is first transferred to the instructor's demonstration action video stream according to the style transfer algorithm to obtain the corrected learner action video stream. Based on the style transfer algorithm, the learner's own personalized features that are not related to the joint angle are transferred to the MOOC video instructor, so that the generated video not only retains the instructor's standard movements, but also conforms to the learner's own personality characteristics such as body shape and appearance. The AR wearable facilities are used to stack and compare the learner's action video stream after correction and the learner's action video stream before correction in real time. The learner can directly compare his own movements with the personalized demonstration movements (i.e., the corrected learner's movement) in real time. The difference of learner's movements) can be used to correct technical movements quickly and effectively. In addition, AR wearable facilities are cheaper and more universal than motion capture analysis system equipment. According to the joint step-by-step migration algorithm, the single-frame picture of the learner at a specified time in the learner's movement video stream is migrated to the video frame at a specified time in the corrected learner movement video stream to obtain the movement transfer slice. The movement transfer slice is an ordered movement sequence along the core force chain of "leg-hip-limb" in sequence, and the joint position is moved step by step until it coincides with the corrected learner movement key frame. According to the action slice to assist and guide learners to correct their movements, it can specifically guide learners to correct their own movements and improve the accuracy of sports skill learning.

[0064] In one of the embodiments, the process of obtaining teaching action video includes: obtaining the instructor's demonstration action video from the MOOC platform, using a smart terminal to shoot the action video of the learner wearing the AR wearable device in real time on the spot, and based on the WIFI Direct protocol, realizing low-latency data transmission between the smart terminal and the AR wearable device.

[0065] like Figure 2As shown in the figure, based on the WIFI Direct protocol, low-latency data transmission is achieved between the smart terminal and the AR wearable facility. This protocol does not require additional data transmission through the Internet or LAN AP, and is very suitable for outdoor sports in a networkless environment. In addition, the WIFI Direct protocol is a wireless protocol, and the smart terminal does not need to be connected to the AR wearable facility worn by the user by wire, and the user does not need to carry the smart terminal with him, which makes it easier to flexibly complete large-scale and high-intensity body movements.

[0066] In one embodiment, the source video frame is transferred to the instructor's demonstration action video stream according to the style transfer algorithm to obtain the corrected learner action video stream, including:

[0067] Obtain a target video frame of the instructor's demonstration action from the instructor's demonstration action video stream, calculate the source video frame and the target video frame using a human body mesh recovery toolkit, and obtain an association mapping corresponding to the source video frame and the target video frame; the association mapping includes weak perspective projection, body shape parameters, and 3D human body mesh;

[0068] According to the element-level matching of the association mappings corresponding to the source video frame and the target video frame, a transformation stream is obtained; the transformation stream refers to a transformation matrix from the association mapping of the source video frame to the association mapping of the target video frame;

[0069] Decompose the source video frame to obtain background projection components and foreground projection components;

[0070] Performing matrix transformation operation on the associated mapping of the source video frame using the transformation stream to obtain the transformation projection component from the target video frame to the source video frame;

[0071] The pre-constructed autoencoder neural network model is trained using the background projection component, the foreground projection component and the transformed projection component to obtain an autoencoder neural network model of the background projection component, an autoencoder neural network model of the foreground projection component and an autoencoder neural network model of the transformed projection component;

[0072] The source video frames and the target video frames of the instructor's demonstration action are input into the autoencoder neural network model of the background projection component, the autoencoder neural network model of the foreground projection component and the autoencoder neural network model of the transformation projection component. The three output pictures are integrated pixel by pixel to obtain the corrected learner action video stream.

[0073] In a specific embodiment, a target video frame of the instructor's demonstration action is obtained from the instructor's demonstration action video stream, and the source video frame and the target video frame are calculated based on the currently mature human mesh recovery toolkit (such as the ResNet-50 model 3D posture estimator HMR), and the association map Map corresponding to the source video frame and the target video frame is obtained respectively. s and Mapt , and generates a transformation flow between the two by matching the two types of associated mappings at the element level; the associated mappings include weak perspective projection, body parameters and 3D human body mesh, and the transformation flow refers to the s To Map t The transformation matrix of

[0074] Using Map s Masking the source video frame, thereby decomposing the source video frame to obtain a background projection component and a foreground projection component;

[0075] Mapping the source video frames using transform streams s Perform matrix transformation operation to obtain the transformation projection component from the target video frame to the source video frame;

[0076] Using the three types of data sources, namely the background projection component, the foreground projection component and the transformed projection component, the pre-built autoencoder neural network model is trained respectively to obtain the autoencoder neural network model M of the background projection component and the transformed projection component. b , the autoencoder neural network model M of the foreground projection component f and the autoencoder neural network model M of the transformed projection component t ;

[0077] The source video frame and the target video frame of the instructor's demonstration action are input into the autoencoder neural network model of the background projection component, the autoencoder neural network model of the foreground projection component and the autoencoder neural network model of the transformation projection component, and the three output pictures are integrated pixel by pixel to obtain the corrected learner action video stream.

[0078] Among them, the pre-built self-encoding neural network model is preferably Autoencoder. Figure 3 As shown in the figure, based on transfer learning, the learner's body shape, shooting angle and other features unrelated to joint angles are transferred to the standard limb posture in the demonstration action video (VT), thereby obtaining a template video (VT') that conforms to the learner's actual situation and the standard action specification. Specifically, for each frame of the picture Pt in VT, a style transfer algorithm based on the autoencoder neural network model is used.

[0079] In one embodiment, the video stream of the learner's movements after correction and the video stream of the learner's movements before correction are stacked and compared online in real time by using an AR wearable device to obtain a video superposition stream of the learner's real movements and the learner's corrected movements, including:

[0080] Performing background transparency processing on the corrected learner action video stream and the learner action video stream to obtain the corrected learner action image and the actual image before correction;

[0081] The corrected learner action image is aligned and overlapped with the actual image before correction by using the axis in the center of gravity, and the actual image before correction is superimposed on the corrected learner action image in a semi-transparent manner to obtain a stacked video stream;

[0082] The stacked video stream is transmitted to the AR wearable device with low latency according to the WIFI Direct protocol, and refracted to the learner's retina through the visual imaging module of the AR wearable device for online comparison, obtaining a video superposition stream of the learner's real movement and the learner's corrected movement.

[0083] In a specific embodiment, after the intelligent terminal completes the above-mentioned character video personality feature migration and action generation, and makes the background in the video stream transparent, the personalized demonstration action of the instructor (that is, the corrected learner action image) is obtained. The learner's actual movement action is superimposed on it: the corrected learner image and the learner portrait are aligned based on the center of gravity axis, and the latter is superimposed on the upper layer in a semi-transparent manner. Through this superposition, the learner can simultaneously observe the completion of the personalized standard action and his or her actual action, and can directly compare and find the differences between his or her own action and the standard action, so that he or she can correct his or her own action posture online.

[0084] Based on the WIFI Direct protocol, the superimposed video stream is transmitted to the AR wearable device with low latency, and then refracted to the learner's retina through the visual imaging module of the AR wearable device, so that the learner can complete the movement correction online and independently through the visual comparison between his real movement and his corrected movement. Figure 2 , so that the video presented to the learner only retains the image of the person in the video (the portrait presented by the AR refraction system itself also has a certain degree of translucency and will not completely block the line of sight), while the background is still the real external environment, thus ensuring real and safe interaction with the on-site facilities during learning.

[0085] In one embodiment, a single frame image of a learner at a specified time in a learner action video stream is migrated to a video frame at a specified time in a corrected learner action video stream according to a joint step-by-step migration algorithm to obtain an action transfer slice, including:

[0086] Based on the currently mature 3D skeleton extraction tool K3M, the 3D skeleton image is extracted from the single-frame image (i.e., the key action frame) at the specified time in the learner's action video stream, and the joint position of the key action frame is moved step by step along the core force chain sequence of "leg-hip-limbs" according to the force chain principle. A frame in the action transfer slice is obtained in each movement until the action posture of the corrected learner action video frame at the specified time is fully adjusted to obtain a complete action transfer slice.

[0087] In one embodiment, motor skill learning is performed based on video overlay streams and action transfer slices, including:

[0088] The motion transfer slices are displayed to users through AR wearable devices to assist and guide learners to correct their movements.

[0089] In a specific embodiment, Figure 4 As shown in the figure, based on the currently mature 3D skeleton extraction tool K3M, the 3D skeleton posture graph is extracted from the single frame picture (i.e. key action frame) of the learner's specified time in the learner's action video stream. Then the following joint level-by-level migration algorithm is adopted: facing the key action frame, based on the principle of force chain, the key frame (f S ), along the core power chain of "legs-hips-limbs", move the joint positions step by step (get the intermediate frame f after each movement M i ) until it is fully adjusted to the corrected learner action key frame at the specified moment (f T’ ) to obtain a complete action transfer slice. The resulting action transfer slice (i.e., frame sequence {f S ,f M 1 ,f M 2 ,f M 3 ,...,f T’}) is displayed to the user through the AR device to assist and guide the learner to correct the action. The AR facility plays the video overlay stream in real time, which is convenient for the learner to correct the action in real time by comparing the actions before and after correction. The learner can also specify his own action at a specific moment in the AR facility, and generate the action transfer slice by querying to understand in detail how to gradually adjust from the action at the specified moment to the standard action.

[0090] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0091] In one embodiment, Figure 5 As shown, a sports skill learning device based on style transfer and AR wearable facilities is provided, including: a video stream sampling module 502, a style transfer algorithm module 504, a stacking comparison module 506 and a sports skill learning module 508, wherein:

[0092] The video stream sampling module 502 is used to obtain teaching action video images; the teaching action video images include the instructor's demonstration action video stream and the learner's action video stream; the learner's action video stream is sampled to obtain the source video frame of the learner's own action;

[0093] The style transfer algorithm module 504 is used to transfer the source video frame to the instructor's demonstration action video stream according to the style transfer algorithm to obtain a corrected learner action video stream;

[0094] The stacking and comparison module 506 is used to stack and compare the corrected learner action video stream and the learner action video stream in real time using the AR wearable device to obtain a video superposition stream of the learner's real action and the learner's corrected action;

[0095] The sports skill learning module 508 is used to migrate the single-frame image of the learner at a specified time in the learner's action video stream to the video frame at a specified time in the corrected learner's action video stream according to the joint step-by-step migration algorithm, so as to obtain action transfer slices; and perform sports skill learning based on the video superposition stream and the action transfer slices.

[0096] In one of the embodiments, the process of obtaining teaching action video includes: obtaining the instructor's demonstration action video from the MOOC platform, using a smart terminal to shoot the action video of the learner wearing the AR wearable device in real time on the spot, and based on the WIFI Direct protocol, realizing low-latency data transmission between the smart terminal and the AR wearable device.

[0097] In one embodiment, the style transfer algorithm module 504 is further used to transfer the source video frame to the instructor's demonstration action video stream according to the style transfer algorithm to obtain a corrected learner action video stream, including:

[0098] Obtain a target video frame of the instructor's demonstration action from the instructor's demonstration action video stream, calculate the source video frame and the target video frame using a human body mesh recovery toolkit, and obtain an association mapping corresponding to the source video frame and the target video frame; the association mapping includes weak perspective projection, body shape parameters, and 3D human body mesh;

[0099] According to the element-level matching of the association mappings corresponding to the source video frame and the target video frame, a transformation stream is obtained; the transformation stream refers to a transformation matrix from the association mapping of the source video frame to the association mapping of the target video frame;

[0100] Decompose the source video frame to obtain background projection components and foreground projection components;

[0101] Performing matrix transformation operation on the associated mapping of the source video frame using the transformation stream to obtain the transformation projection component from the target video frame to the source video frame;

[0102] The pre-constructed autoencoder neural network model is trained using the background projection component, the foreground projection component and the transformed projection component to obtain an autoencoder neural network model of the background projection component, an autoencoder neural network model of the foreground projection component and an autoencoder neural network model of the transformed projection component;

[0103] The source video frames and the target video frames of the instructor's demonstration action are input into the autoencoder neural network model of the background projection component, the autoencoder neural network model of the foreground projection component and the autoencoder neural network model of the transformation projection component. The three output pictures are integrated pixel by pixel to obtain the corrected learner action video stream.

[0104] In one embodiment, the stacking and comparing module 506 is further used to use the AR wearable device to perform real-time stacking and online comparison on the corrected learner action video stream and the learner action video stream to obtain a video superposition stream of the learner's real action and the learner's corrected action, including:

[0105] Performing background transparency processing on the corrected learner action video stream and the learner action video stream to obtain the corrected learner action image and the actual image before correction;

[0106] The corrected learner action image is aligned and overlapped with the actual image before correction by using the axis in the center of gravity, and the actual image before correction is superimposed on the corrected learner action image in a semi-transparent manner to obtain a stacked video stream;

[0107] According to the WIFI Direct protocol, the stacked video stream is transmitted to the AR wearable device with low latency, and refracted to the learner's retina through the visual imaging module of the AR wearable device for online comparison to obtain a video superposition stream of the learner's real movement and the learner's corrected movement.

[0108] In one embodiment, the sports skill learning module 508 is further used to migrate a single frame image at a specified time of the learner in the learner's action video stream to a video frame at a specified time in the corrected learner's action video stream according to a joint step-by-step migration algorithm to obtain an action transfer slice, including:

[0109] Based on the 3D skeleton extraction tool K3M, a 3D skeleton image is extracted from a single-frame image of the learner at a specified time in the learner's action video stream. According to the force chain principle, the 3D skeleton image is moved step by step along the core force chain sequence of "leg-hip-limbs". A frame in the action transfer slice is obtained in each movement until the action posture of the learner's action video frame after correction at the specified time is fully adjusted to obtain a complete action transfer slice.

[0110] In one embodiment, the sports skill learning module 508 is further configured to perform sports skill learning according to the video overlay stream and the action transfer slices, including:

[0111] The video overlay stream and motion transfer slices are displayed to users through AR wearable devices to assist and guide learners to correct their movements.

[0112] For the specific limitations of the sports skill learning device based on style transfer and AR wearable facilities, please refer to the limitations of the sports skill learning method based on style transfer and AR wearable facilities above, which will not be repeated here. Each module in the above-mentioned sports skill learning device based on style transfer and AR wearable facilities can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0113] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for learning sports skills based on style transfer and AR wearable facilities is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0114] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0115] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0116] In one embodiment, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0117] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0118] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0119] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for learning sports skills based on style transfer and AR wearable devices, characterized in that: The method comprises: Acquire teaching action video; the teaching action video includes a video stream of the instructor's demonstration action and a video stream of the learner's action; Sampling the learner's action video stream to obtain a source video frame of the learner's own action; Migrating the source video frame to the instructor's demonstration action video stream according to a style migration algorithm to obtain a corrected learner action video stream; Using AR wearable equipment to stack and compare the corrected learner action video stream and the learner action video stream in real time, so as to obtain a video superposition stream of the learner's real action and the learner's corrected action; Migrate a single frame image of the learner at a specified time in the learner's action video stream to a video frame at the specified time in the corrected learner's action video stream according to a joint step-by-step migration algorithm to obtain an action transfer slice; Motor skill learning is performed based on the video overlay stream and the action transfer slices.

2. The method according to claim 1, characterized in that: The process of obtaining teaching action video includes: The instructor's demonstration action video is obtained from the MOOC platform, and the action video of the learner wearing the AR wearable device is captured in real time on the spot through the smart terminal. Based on the WIFIDirect protocol, low-latency data transmission is achieved between the smart terminal and the AR wearable device.

3. The method according to claim 1, characterized in that The source video frame is transferred to the instructor's demonstration action video stream according to a style transfer algorithm to obtain a corrected learner action video stream, including: Obtaining a target video frame of the instructor's demonstration action from the instructor's demonstration action video stream, and calculating the source video frame and the target video frame using a human body mesh recovery toolkit to obtain an association map corresponding to the source video frame and the target video frame; the association map includes a weak perspective projection, body shape parameters, and a 3D human body mesh; According to element-wise matching of the association mappings corresponding to the source video frame and the target video frame, a transformation stream is obtained; the transformation stream refers to a transformation matrix from the association mapping of the source video frame to the association mapping of the target video frame; Decomposing the source video frame to obtain a background projection component and a foreground projection component; Using the transformation stream to perform a matrix transformation operation on the associated mapping of the source video frame to obtain a transformation projection component from the target video frame to the source video frame; The pre-constructed autoencoder neural network model is trained using the background projection component, the foreground projection component and the transformed projection component to obtain an autoencoder neural network model of the background projection component, an autoencoder neural network model of the foreground projection component and an autoencoder neural network model of the transformed projection component; The source video frame and the target video frame of the instructor's demonstration action are input into the autoencoder neural network model of the background projection component, the autoencoder neural network model of the foreground projection component and the autoencoder neural network model of the transformation projection component, and the three output pictures are integrated pixel by pixel to obtain a corrected learner action video stream.

4. The method according to any one of claims 1 to 3, characterized in that: The corrected learner action video stream and the learner action video stream are stacked and compared online in real time by using an AR wearable device to obtain a video superposition stream of the learner's real action and the learner's corrected action, including: Performing background transparency processing on the corrected learner action video stream and the learner action video stream to obtain a corrected learner action image and an actual image before correction; The corrected learner action image is aligned and overlapped with the actual image before correction by using the axis in the center of gravity, and the actual image before correction is superimposed on the corrected learner action image in a semi-transparent manner to obtain a stacked video stream; The stacked video stream is transmitted to the AR wearable device with low latency according to the WIFIDirect protocol, and is refracted to the learner's retina through the visual imaging module of the AR wearable device for online comparison, so as to obtain a video superposition stream of the learner's real action and the learner's corrected action.

5. The method according to claim 4, characterized in that The single frame image of the learner at a specified time in the learner action video stream is migrated to the video frame at the specified time in the corrected learner action video stream according to the joint step-by-step migration algorithm to obtain the action transfer slice, including: Based on the 3D skeleton extraction tool K3M, a 3D skeleton image is extracted from a single-frame image of the learner at a specified moment in the learner's action video stream. According to the force chain principle, the 3D skeleton image is moved step by step along the core force chain sequence of "leg-hip-limbs". A frame in the action transfer slice is obtained at each movement until the action posture of the corrected learner action video frame at the specified moment is fully adjusted to obtain a complete action transfer slice.

6. The method according to claim 5, characterized in that Motor skill learning is performed according to the video overlay stream and the action transfer slices, including: The video overlay stream and the motion transfer slices are displayed to the user through an AR wearable device to assist and guide the learner to correct the motion.

7. A sports skill learning device based on style transfer and AR wearable equipment, characterized in that: The device comprises: The video stream sampling module is used to obtain teaching action video; the teaching action video includes the instructor's demonstration action video stream and the learner's action video stream; the learner's action video stream is sampled to obtain the source video frame of the learner's own action; A style transfer algorithm module, used to transfer the source video frame to the instructor's demonstration action video stream according to the style transfer algorithm to obtain a corrected learner action video stream; A stacking and comparison module, used to use AR wearable equipment to perform real-time stacking and online comparison on the corrected learner action video stream and the learner action video stream to obtain a video superposition stream of the learner's real action and the learner's corrected action; A sports skill learning module is used to migrate a single-frame image of a learner at a specified moment in the learner's action video stream to a video frame at the specified moment in the corrected learner's action video stream according to a joint step-by-step migration algorithm to obtain a motion transfer slice; and to perform sports skill learning based on the video superposition stream and the motion transfer slice.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images