Method, system and device for generating high jump action video and medium

Abstract

The invention discloses a method, a system, equipment and a medium for generating a high jump action video, relates to the technical field of artificial intelligence, and solves the technical problems that a motion track cannot be accurately converted into a video form and personalized requirements cannot be met. And an efficient and accurate high jump video generation scheme is provided. In exercise training, seamless conversion from skeleton capture to high-quality video output can be achieved, and meanwhile, the requirements of different users are met through intelligent optimization and personalized customization. Through system optimization and detail processing, the method can effectively solve various challenges in motion video generation, such as occlusion, jitter, time discontinuity and the like, and provides powerful support for motion training and technical analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a method and system for generating high jump action videos, a device and a medium. BACKGROUND

[0002] The field of sports training is faced with a shortage of excellent coaches, making it difficult to meet the training needs of a large number of athletes. At the same time, the technical movements involved in sports training are complex and difficult to quantify, and coaches need to observe for a long time and accumulate experience to gradually identify the shortcomings in the movements of athletes.

[0003] Frontier research attempts to optimize the action guidance effect through multi-modal large models, typical schemes including the use of Transformer architecture to realize cross-modal motion feature fusion (visual / mechanical / bioelectric signals), but the above methods are still limited by high complexity of heterogeneous hardware deployment, difficulty in embedding kinematic prior knowledge, and other problems, making it difficult to realize end-to-end closed-loop application in real training scenarios. Multi-source data acquisition is the common basis for these schemes, and the coaching team often relies on multi-view cameras or wearable sensors to record the movement details of athletes at high frequency. However, the camera and sensor layout requirements are high, both to ensure proper lighting and angle, and to take into account the actual training experience of athletes, and to achieve precise alignment and synchronization of multi-channel data. For high-speed motion and complex scenarios, many systems use Kalman filtering and deep learning pose estimation models to perform temporal smoothing and spatial reconstruction of the joint nodes to reduce jitter and misjudgment. However, in the sudden change of actions such as take-off and landing, the model may still have missing or tracking breakpoints, especially in the absence of three-dimensional information, the restoration of action details is limited.

[0004] After completing the skeleton detection, these schemes usually present the training results to the coach and athlete through visualization or reporting, often marking the joint trajectory in the video or three-dimensional coordinates to allow the user to intuitively understand the action deviation. However, this static presentation can only provide a baseline reference, and the real personalized recommendations still need the coach to make a secondary interpretation based on experience. For professional athletes with large individual differences, simply relying on homogenized statistical results often makes it difficult to develop an effective action improvement plan. In addition, some systems lack real-time feedback capabilities, making it difficult to identify and correct problems in the athlete's training process in a timely manner, and it is also difficult for coaches to obtain first-hand action data comparisons.

[0005] To address the above problems, researchers are trying to integrate multi-modal large models into existing frameworks, using advanced technologies such as three-dimensional pose information, voice output, and text understanding to provide more flexible and dynamic training guidance for athletes and coaches. By combining automatic commentary and demonstration video generation, athletes can not only compare their movements to standard actions in a short period of time, but also use their individual body types and field parameters to conduct targeted posture analysis, thereby achieving higher levels of technical guidance and movement correction. However, the deployment of these new methods requires the cooperation of hardware environments and coaching teams, and requires continuous annotation of large-scale data and optimization of algorithms. If there is not enough time and resources to install multiple high-speed cameras or high-precision sensors, some advanced algorithms may only remain in the laboratory or small-scale pilot. In addition, professional athletes have more stringent requirements for training rhythm and intensity, and higher requirements for the real-time performance and accuracy of the system. Only when hardware cooperation, data governance, and algorithm optimization are all advancing together can existing technical solutions truly empower sports training and bring more efficient, accurate, and inclusive coaching services.

[0006] Therefore, the existing intelligent sports training auxiliary system has the following problems:

[0007] (1) It is difficult to balance motion capture accuracy and timing consistency. In the face of high-intensity or large-amplitude sports scenes, key frames are often lost or dynamic blurring occurs;

[0008] (2) Current methods lack personalized optimization methods when adapting to various body types and changing fields, requiring extensive annotation and targeted training for different populations and scenarios, which is costly and time-consuming;

[0009] (3) The system's real-time feedback capability is limited, with high computational overhead, especially when multiple perspectives are fused and video is understood, resulting in excessive delay or excessive hardware load.

[0010] To solve the above technical problems, the present application provides a method, system, device and medium for generating a high jump action video. SUMMARY

[0011] The present application provides a method, system, device and medium for generating a high jump action video, which aims to accurately convert motion trajectories into video format while maintaining efficiency and personalization needs. It not only addresses the limitations of traditional motion capture methods, but also effectively addresses the occlusion and joint jitter problems in high-speed motion.

[0012] The above technical purpose of the present application is achieved by the following technical solution:

[0013] A method for generating a high jump action video, comprising:

[0014] The joint coordinates of each frame of the real-time multi-view motion video are extracted by the YOLOv11-Pose model to generate skeleton data containing frame number, joint index and joint coordinates;

[0015] The skeleton data is preprocessed, and the preprocessed skeleton data is converted into a pose graph;

[0016] The image features of each frame of the real-time multi-view motion video are extracted by embedding the network;

[0017] The pose graph and the image features are fused to obtain a feature representation;

[0018] The pose graph and the feature representation are input into the stable video diffusion model SVD as control conditions, and the motion trajectory video is generated after decoding;

[0019] The motion trajectory video is processed by image enhancement technology to obtain a high jump demonstration video.

[0020] Further, the YOLOv11-Pose model comprises:

[0021] The expansion module is used to expand the annotation file, so that each frame feature map has an additional stage label, which is represented as: σ i ∈{Approach,Takeoff,Flight,Landing}, wherein Approach, Takeoff, Flight and Landing represent take-off label, flight label, over-bar label and landing label, respectively;

[0022] The temporary buffer is used to temporarily store the front frame feature map F i-1 and the rear frame feature map F i+1 .

[0023] The splicing module splices the front frame feature map F i-1 , the rear frame feature map F i+1 and the current frame feature map F i to obtain the spliced feature map (F i , F i-1 , F i+1 ).

[0024] The time sequence attention branch module performs shallow fusion on the spliced feature map, represented as: wherein, represents the fused feature map, and Ψ represents the time sequence attention branch module.

[0025] The detection head predicts the joint node coordinates in the fused feature map to generate skeleton data containing a frame number, a joint node index, and joint node coordinates; wherein the detection head comprises a C3k2 module and a C2PSA module, the C3k2 module strengthens the feature extraction capability by using a deep convolutional layer, and the C2PSA module captures global information through a self-attention mechanism;

[0026] The joint node coordinates predicted by the detection head are represented as:

[0027]

[0028] Wherein, represents the input fused feature map, and θ is a model parameter; k represents a joint node index, represents the kth joint node coordinate.

[0029] Further, the loss function of the YOLOv11-Pose model is represented as:

[0030] L=λ1L kp +λ2L stage ;

[0031] Wherein, L kp represents a joint node detection loss, L stage represents a stage recognition loss, λ1 and λ2 both represent weight coefficients; (x k , y k ) represents the kth joint node coordinate; σ i represents a discrete i-th class label, used to represent the action stage to which the sample belongs; represents the stage prediction value of the i'th sample by the YOLOv11-Pose model, that is, the probability distribution generated by the normalization function softmax.

[0032] Further, the YOLOv11-Pose model further comprises a multi-frame skeleton fusion module, the multi-frame skeleton fusion module is used to perform smoothing filtering on each joint node coordinate predicted by the detection head in the time axis, and the smoothed joint node coordinate is represented as:

[0033]

[0034] Wherein, Φ represents a smoothing function for eliminating inference noise and maintaining action coherence; represents the kth joint node coordinate predicted by the detection head at time t; represents the kth joint node coordinate predicted by the detection head at time t after smoothing.

[0035] Further, the pre-processing of the skeleton data comprises:

[0036] Smooth the time series of the joint coordinates in each frame by a one-dimensional Gaussian filter to obtain smoothed joint coordinates, denoted as:

[0037]

[0038] wherein p k,t+i denotes the kth joint coordinate at t+i; G(i;σ) denotes a standard Gaussian kernel, and σ denotes a filter width, denotes the kth joint coordinate at t after smoothing;

[0039] On the basis of the smoothed joint coordinates, the reasonable position of the missing point is inferred by a local motion model, denoted as:

[0040]

[0041] wherein Ψ(p) is a constraint term for limiting joint angles, motion ranges or physical consistency; λ denotes a balance coefficient; and p denotes the possible coordinate of the kth missing joint at t; denotes the kth joint coordinate at t-1 after smoothing; denotes the kth joint coordinate at t+1 after smoothing; denotes the inferred coordinate of the kth missing joint at t.

[0042] Further, the pre-processed skeleton data is converted into a pose graph, the pose graph comprising joint coordinates and motion information, and the pose graph should satisfy:

[0043] The drawing color of the current frame skeleton trajectory is brighter than that of the previous frame skeleton trajectory;

[0044] When generating multi-view graphs, the positions of the joints should be aligned and matched under all views.

[0045] Only key frames with significant changes are subjected to time trajectory drawing.

[0046] Further, the loss function of the stable video diffusion model SVD comprises a conditional consistency loss and a physical reasonableness loss.

[0047] The conditional consistency loss is denoted as:

[0048]

[0049] wherein p t,k denotes the input coordinate of the kth joint of the tth frame, represents the predicted position of the k th joint in the t th frame of the generated motion trajectory video;

[0050] The physical rationality loss is represented as:

[0051]

[0052] where θ t,k represents the angle of the k th joint in the t th frame, represents the target angle of the k th joint in the t th frame, v t,k represents the velocity of the k th joint in the t th frame, v t-1,k represents the velocity of the k th joint in the t-1 th frame.

[0053] A system for generating a high jump action video, the system for generating a high jump action video method comprising:

[0054] A coordinate extraction module extracts the joint coordinates of each frame of video in the multi-view motion video through a YOLOv11-Pose model, generating skeleton data containing frame numbers, joint indexes, and joint coordinates.

[0055] A preprocessing module pre-processes the skeleton data.

[0056] An attitude graph generation module generates an attitude graph from the pre-processed skeleton data.

[0057] A feature extraction module extracts image features of each frame of video in real-time multi-view motion video through an embedding network.

[0058] A fusion module performs feature fusion on the attitude graph and image features to obtain feature representation.

[0059] A trajectory generation module inputs the attitude graph and feature representation as control conditions into a stable video diffusion model SVD, and generates a motion trajectory video after decoding.

[0060] An enhancement module processes the motion trajectory video through image enhancement technology to obtain a high jump demonstration video.

[0061] An electronic device, the electronic device comprising:

[0062] At least one processor; and,

[0063] A memory communicatively connected to the at least one processor;

[0064] Wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the method for generating a high jump action video.

[0065] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a method of generating a high jump action video.

[0066] The application has the following beneficial effects:

[0067] (1) By introducing the fusion of multi-view motion video data and skeleton detection technology, combined with the conditional generation ability of controllable stable video diffusion model SVD, precise control of high jump action generation is achieved.

[0068] (2) Combined with multi-view motion video data and posture information, skeleton joint data is extracted, and through time smoothing and multi-frame interpolation completion technology, the problems of joint jitter and loss in high-speed motion are effectively solved.

[0069] (3) The generated standardized demonstration video provides clear action guidance for athletes, helping them master technical points faster. It realizes the transformation from static analysis to dynamic feedback, reduces the dependence on coaches' experience, provides real-time feedback and dynamic adjustment functions, and creates a flexible and efficient interactive platform for athletes and coaches. In addition, it can be used in professional sports venues, school training centers and family fitness scenes, and adapts to different user needs.

[0070] (4) By introducing the generation model (stable video diffusion model SVD), the application creates a new method of using generated demonstration videos in high jump and other complex movements. The model generation process supports high-precision reproduction of multi-view, time series and dynamic details, providing technical support for the expansion of other sports scenes.

[0071] (5) The efficient skeleton processing method ensures the data basis of the generation model, solving the common problems of jitter, loss and dynamic blur in traditional systems. The posture phase label and multi-modal fusion technology enhance the intelligence and personalization of the generated model, and improve the universality of the technical solution. BRIEF DESCRIPTION OF DRAWINGS

[0072] Figure 1 The flowchart of the method of generating a high jump action video in the embodiments of the application;

[0073] Figure 2 The flowchart of the model training in the embodiments of the application;

[0074] Figure 3 The system framework diagram of the method of generating a high jump action video in the embodiments of the application. DETAILED DESCRIPTION

[0075] The technical solutions of the application will be described in detail below with reference to the accompanying drawings.

[0076] As Figure 1As shown, the method for generating high jump action video described in the present application comprises:

[0077] 100: Extract the joint coordinates of each frame of the real-time multi-view motion video by the YOLOv11-Pose model, generate skeleton data containing frame number, joint index and joint coordinates.

[0078] In order to efficiently obtain human skeleton information in a multi-view motion training environment, the YOLOv11-Pose model is used in the skeleton detection stage, and its network structure and training strategy are optimized.

[0079] Specifically, the YOLOv11-Pose model combines the detection optimization strategies of multiple key points on the basis of the fast and accurate detection capability of the YOLO series. The C3k2 module and the C2PSA module are introduced in the detection head of the YOLOv11-Pose model, wherein the C3k2 module uses a deep convolution layer to strengthen the feature extraction capability, and the C2PSA module captures global information through a self-attention mechanism. In addition, the YOLOv11-Pose model effectively reduces the parameter amount through a depthwise separable convolution (DWConv), thereby balancing high precision and real-time inference performance. However, when aiming at professional sports scenes (such as high-speed action, special body position, and customized key point layout), the YOLOv11-Pose model still needs to be optimized and upgraded in terms of network structure, training strategy, and post-processing. The optimized YOLOv11-Pose model of the present application includes:

[0080] (1) Expansion module, used for expanding the annotation file, so that each frame has an additional phase label, which is denoted as: σ i ∈{Approach,Takeoff,Flight,Landing}, wherein Approach, Takeoff, Flight and Landing represent the take-off label, the take-off label, the over-bar label and the landing label, respectively.

[0081] Specifically, in terms of data strategy adjustment, since the present application focuses on the changes of the skeleton in the process of high-intensity sports, the method described in the present application increases the timing information of more joints in addition to the original data annotation, and adds “pose phase” annotation in the label. For example, in the process of high-level athlete high jump, the joint information of the phases such as take-off, take-off, over-bar and landing is not clearly distinguished in the original YOLO format. Therefore, the present application expands the annotation file, so that each frame has an additional phase label: σ i ∈{Approach,Takeoff,Flight,Landing}, these fine-grained supervision information plays a key role in network training.

[0082] (2) a temporary buffer at the input end for temporarily storing the previous frame feature map F i-1 and the next frame feature map F i+1 .

[0083] (3) a splicing module for splicing the previous frame feature map F i-1 , the next frame feature map F i+1 and the current frame feature map F i to obtain the spliced feature map (F i , F i-1 , F i+1 );

[0084] (4) a time sequence attention branch module for shallowly fusing the spliced feature map (F i , F i-1 , F i+1 ), represented as: wherein, represents the fused feature map, and Ψ represents the time sequence attention branch module.

[0085] Specifically, in terms of network structure, on the basis of the original C3k2 module and C2PSA module of the YOLOv11-Pose model, the application introduces a time sequence attention branch module Ψ to solve the problems of local blur and large-scale displacement caused by high-speed motion. The fused feature map then enters the original detection head for calculation of the joint node coordinates and the confidence.

[0086] (5) a detection head for predicting the joint node coordinates in the fused feature map to generate skeleton data containing the frame number, the joint node index and the joint node coordinates; wherein the detection head comprises a C3k2 module and a C2PSA module, the C3k2 module uses a deepened convolutional layer to strengthen the feature extraction capability, and the C2PSA module captures global information through a self-attention mechanism.

[0087] Preferably, in terms of joint node positioning, the body posture of a professional athlete changes greatly when he or she runs at high speed or turns in the air. During training, the application maps the 17 joint nodes to the output of the detection head of the YOLOv11-Pose model. The joint node coordinates predicted by the detection head are represented as:

[0088]

[0089] wherein, represents the input fused feature map, θ is a model parameter, and k represents the joint node index, represents the kth joint node coordinate. These joint nodes are of great help to improving the structural recognition of aerial actions.

[0090] (6) Loss function design

[0091] In the training process, the present application designs two branch loss functions for the joint point prediction and the phase recognition. The overall loss is represented as:

[0092] L = λ1L kp + λ2L stage (2)

[0093] The joint detection loss is represented as:

[0094]

[0095] The phase recognition loss is represented as L stage :

[0096]

[0097] wherein, L kp represents the joint detection loss; L stage represents the phase recognition loss; λ1, λ2 both represent weight coefficients; (x k , y k ) represents the kth joint coordinate; σ i represents the discrete ith class label (such as one-hot encoding form) for representing the action phase (such as “raising hands”, “squatting” and the like) to which the sample belongs; represents the phase prediction value of the i'th sample by the YOLOv11-Pose model, that is, the probability distribution generated by the normalization function softmax.

[0098] By adjusting λ1 and λ2, the present application balances between the joint positioning accuracy and the phase recognition ability to better support the subsequent action analysis and demonstration video generation.

[0099] (6) In the inference post-processing link, the present application adds a multi-frame skeleton fusion module to perform smoothing filtering (such as Kalman filtering) on each joint point on the time axis, compensates and interpolates for serious jitter or frame loss phenomenon, and the smoothed skeleton is represented as:

[0100]

[0101] wherein, Φ is a smoothing function for eliminating inference noise and maintaining action coherence; represents the kth joint coordinate predicted by the detection head at time t; represents the kth joint coordinate predicted by the detection head at time t after smoothing.

[0102] Finally, in order to make the detection head and the multi-frame skeleton fusion module into the multi-view training pipeline, the method described in the application is adapted to the data input end and the result output end respectively. At the input end, frames are extracted from the multi-view video, the resolution and frame rate are unified, and the input size of the YOLOv11-Pose model is ensured to be consistent with the YOLOv11-Pose model. At the output end, the smoothed joint sequence is synchronized with other multi-modal features on the time axis, providing high-quality skeleton data support for the subsequent differentiable rendering and action fusion steps.

[0103] 101: pre-process the skeleton data, and generate a pose graph through the pre-processed skeleton data.

[0104] Specifically, in the skeleton data preprocessing stage, the original skeleton data is often accompanied by significant noise and missing problems, which are mainly caused by motion blur, occlusion and limitations of the detection algorithm itself. In order to eliminate these influences, the application adopts a time smoothing technique to filter the skeleton sequence. Specifically, the time sequence of the joint coordinates in each frame is smoothed by a one-dimensional Gaussian filter to obtain smoothed joint coordinates, denoted as:

[0105]

[0106] where p k,t+i represents the kth joint coordinate at t+i; G(i;σ) represents a standard Gaussian kernel, and σ represents a filtering width, represents the kth joint coordinate at t after smoothing. Gaussian filtering can effectively eliminate high-frequency jitter caused by motion blur while maintaining the motion trend of the skeleton.

[0107] In addition, on the basis of the smoothed joint coordinates, the method described in the application introduces an interpolation method based on adjacent frames, which uses the least squares method and a local action model Φ(p i-1 ,p i+1 The local action model Φ(p i-1 ,p i+1 infers the reasonable position of the missing point according to the skeleton coordinates of the time neighborhood frames, denoted as:

[0108]

[0109] where Ψ(p) is a constraint term for limiting joint angles, motion ranges or physical consistency; λ represents a balance coefficient; p represents the possible coordinates of the missing kth joint at t; represents the kth joint coordinate at t-1 after smoothing; represents the kth joint coordinate at t+1 after smoothing; represents the inferred kth joint coordinate missing at t.

[0110] The above preprocessing method can effectively repair the problem of missing skeleton joint points caused by occlusion or detection errors. After preprocessing the skeleton data, the next step is to convert the optimized preprocessed skeleton data into a pose graph for input into the stable video diffusion model SVD (Stable Video Diffusion) for conditional video generation. The generation process of the pose graph requires projecting the two-dimensional or three-dimensional coordinates of the joints onto the image plane. For two-dimensional skeleton data, a skeleton connection graph is directly drawn according to the coordinates (x k,t ,y k,t ) of each joint point; (x k,t ,y k,t ) represents the coordinates of the kth joint point at time t.

[0111] Preferably, the pose graph includes joint coordinates and motion information, and the pose graph should satisfy:

[0112] (1) The drawing color of the current frame skeleton trajectory is brighter than that of the previous frame skeleton trajectory.

[0113] Specifically, when generating the pose graph, an important design point is how to express the time information. The simple skeleton coordinate graph cannot reflect the dynamic change trend of the joint points, so the present application encodes the time context by introducing color gradient or trajectory line in the pose graph. For example, the skeleton trajectory of the previous frame is drawn in a lighter color, while the current frame is drawn in a brighter color. In this way, in the pose graph, the model can perceive the motion direction and speed. In addition, the color change of the trajectory line is interpolated according to the timestamp, thereby constructing a continuous time sequence expression.

[0114] (2) When generating a multi-view graph, the positions of the joint points in all views should be aligned and matched.

[0115] Specifically, the introduction of multi-view fusion and time context encoding can significantly improve the understanding of dynamic motion by SVD, but also brings complexity. On the one hand, the pose graphs of different views may have joint occlusion or spatial inconsistency problems. To solve this problem, the present application introduces a multi-channel pose graph alignment method, i.e., when generating a multi-view graph, the positions of key joint points in all views are matched.

[0116] (3) Time trajectory drawing is performed only on key frames with significant changes through a sparse sampling strategy.

[0117] Since time encoding can cause information overload in the pose graph, especially in high-speed motion scenarios, the trajectory lines are too dense. To this end, the present application adopts a sparse sampling strategy, which only selects key frames with significant changes for time trajectory drawing, thereby ensuring information integrity while avoiding redundancy.

[0118] Ultimately, after processing by the preprocessing module and the pose graph generation module, this application obtains a series of high-quality, temporally context-rich pose graph sequences, which can serve as conditional inputs for the stable video diffusion model (SVD), providing an accurate and stable foundation for high jump video generation. The collaboration of these two modules not only resolves jitter and missing issues in motion data, but also enhances the expression of dynamic information through visualization, providing a strong foundation for subsequent generation.

[0119] 102:Extract image features of each frame in real-time multi-view motion video through embedding network.

[0120] In this application, the embedding network is the encoder part of the variational autoencoder, using OpenSora VAE 1.2. For details, please refer to the webpage: https: / / huggingface.co / hpcai-tech / OpenSora-VAE-v1.2.

[0121] 103: Perform feature fusion on the posture graph and image features to obtain feature representation.

[0122] 104: The posture graph and feature representation are input as control conditions into the stable video diffusion model SVD, and the motion trajectory video is generated after decoding.

[0123] Specifically, the Stable Video Diffusion (SVD) model is a core technology in the high jump video generation process. Its function is to generate high jump video sequences based on pose graphs. These pose graphs have been optimized through a skeleton data preprocessing module, containing sufficient temporal context and spatial information to effectively express the dynamic changes of the skeleton over time. The Stable Video Diffusion (SVD) model, a conditional generative network, controls the skeletal morphology of the generated images and videos based on these pose graphs, ensuring that the generated videos not only meet the timing requirements of the movement but also exhibit natural and smooth transitions.

[0124] The Stable Video Diffusion (SVD) model is a conditional generative model based on deep learning. It aims to control the generation process through explicit conditional inputs, ensuring that the output meets specific requirements. In the high jump video generation task, SVD's role is to generate precise action sequences based on a pre-processed pose graph. By receiving the pose graph as conditional information, it can accurately control the skeletal morphology and movement transitions in the generated video. The advantage of the Stable Video Diffusion (SVD) model lies in its ability to fine-tune the generated image by incorporating external conditions (such as the pose graph and joint positions), thereby ensuring that the generated video better conforms to the athlete's movement trajectory and physical laws.

[0125] The core principle of the stable video diffusion model SVD is to combine the input conditional information with the generation process through a conditional generation network, thereby affecting the specific details of the generated images or videos. In specific implementation, the stable video diffusion model SVD takes the input pose graph as the condition of the network, and maps the pose information to a high-dimensional feature space through the control of the conditional encoding layer in the network. These features are then input into the backbone generation network, and the generator generates the corresponding image or video frame according to these features. The design of the stable video diffusion model SVD allows the network to explicitly control the motion pose, joint angle and position, etc. in the generation process, thereby ensuring that the generated results meet the expectations.

[0126] The stable video diffusion model SVD generation process is represented by the following formula:

[0127] I i =G(P i ,C) (8)

[0128] Where I i represents the generated image of the i-th frame, P i represents the skeleton pose graph of the i-th frame (including joint position and action information), and C is additional conditional information (such as the body shape of the athlete, the training scene, etc.). The generator G is a conditional generation network that combines the input pose graph and other conditional information to generate the final output image through a series of convolutional layers and transformation layers.

[0129] The loss function includes two parts, conditional consistency loss and physical plausibility loss. The loss function not only constrains the consistency of the generated results with the input conditions, but also ensures that the generated skeleton is coherent in time and the skeleton action is physically reasonable. The conditional consistency loss function ensures that the generated image matches the input pose Figure 1 , i.e. the position and shape of the joint should match the input condition, which is represented as:

[0130]

[0131] Where p t,k represents the input coordinates of the k-th joint in the t-th frame, represents the predicted position of the k-th joint in the t-th frame in the generated motion trajectory video. By minimizing this loss, the model ensures that the generated results are consistent with the input conditions.

[0132] The physical plausibility function constrains the generated action to comply with the laws of physical kinematics, ensuring that the joint motion angle, velocity and acceleration are reasonable in the generation process. For example, the change of joint angle should comply with the physiological limit of human, and the change of velocity should be smooth, without sudden or abrupt changes. For this purpose, the present application uses joint angle constraint and velocity smoothing term, i.e. the physical plausibility loss is represented as:

[0133]

[0134] where θ t,k denotes the angle of the k th joint in the t th frame, denotes the target angle of the k th joint in the t th frame, v t,k denotes the velocity of the k th joint in the t th frame, v t-1,k denotes the velocity of the k th joint in the t-1 th frame. Through this physical constraint, SVD can generate more natural and smooth movements.

[0135] Each frame generated by the stable video diffusion model SVD not only accurately reflects the changes in the skeleton, but also preserves the details and physical rationality of the movement, ensuring the coordination of the video in terms of speed, angle, posture, etc. Through the constraint and optimization of each frame of movement, the generated high jump video reaches a high standard in terms of action fluency, detail presentation, and physical rationality, providing effective technical reference and training support for athletes and coaches.

[0136] 105: Process the movement trajectory video through image enhancement technology to obtain a high jump demonstration video.

[0137] Specifically, the present application optimizes the details in the generated high jump movement trajectory video, including adjusting the texture, color and background of the image, so that the final output video is more delicate and realistic. At the same time, users can fine-tune the skeleton coordinates according to actual needs to adapt to the characteristics of different body sizes, venues or athletes, thereby generating personalized high jump demonstration videos. This flexibility enables the system not only to meet standardized training needs, but also to provide customized action guidance for coaches and athletes.

[0138] The system for generating high jump action videos described in the present application, as shown in Figure 3 includes a coordinate extraction module, a preprocessing module, a pose graph generation module, a feature extraction module, a fusion module, a trajectory generation module and an enhancement module.

[0139] The coordinate extraction module extracts the joint coordinates of each frame of video in the multi-view movement video through the YOLOv11-Pose model, generating skeleton data containing frame number, joint index and joint coordinates.

[0140] The preprocessing module is used for preprocessing the skeleton data.

[0141] The pose graph generation module is used to generate a pose graph through the preprocessed skeleton data.

[0142] The feature extraction module is used to extract features through an embedding network Figure 3The encoder part and the decoder part in the image feature extraction model are used to extract image features of each frame of video in real-time multi-view motion video.

[0143] The fusion module is used for feature fusion of the pose graph and the image features to obtain a feature representation.

[0144] The trajectory generation module is used for inputting the pose graph and the feature representation as control conditions to a stable video diffusion model SVD, and generating a motion trajectory video after decoding.

[0145] The enhancement module is used for processing the motion trajectory video through an image enhancement technique to obtain a high jump demonstration video.

[0146] Figure 2 A model training flowchart in the embodiment of the present application is shown in FIG. 1. Figure 2 As shown in FIG. 1, the training flow mainly includes four stages of data preparation, model training, loss function design and optimization, and cross-modal feature fusion.

[0147] In the data preparation stage, multi-view high jump training videos are collected to ensure that various athlete poses, different site lighting conditions and motion stages (such as run-up, take-off, flight, over the bar, and landing) are covered. The joint coordinates in each frame of video are automatically extracted by the YOLOv11-Pose model, and are corrected by manual correction to ensure the accuracy of the annotation. At the same time, data enhancement techniques (such as random rotation, scaling and color change) are used to improve the generalization ability of the model.

[0148] In the model training stage, a phased training strategy is adopted. First, the skeleton detection module and the embedding network are independently trained to ensure their effectiveness in their respective tasks. Then, the skeleton detection and feature extraction modules are cooperatively optimized through joint training.

[0149] During the training process, a multi-task loss function is designed to balance the key point detection, feature alignment and image generation quality to improve the overall performance of the system.

[0150] Finally, the feature fusion part performs weighted fusion of the skeleton data and other modal features through an attention mechanism to improve the action accuracy and visual consistency.

[0151] The following evaluation index results are shown in Table 1 based on the method, system, device and medium for generating a high jump action video described in the present application, wherein the model represents a video generation model, and is mainly based on actually collected high jump videos of athletes. The data set contains high-precision video records of three professional high jump athletes during training. The video completely records the high jump process, including starting, running, jumping, passing the bar and landing. Three camera positions are used to shoot each high jump, located on the right side, left side and left front side of the athlete's starting position respectively. All videos are shot in 4K resolution and 60 frames per second to ensure the details and smoothness of the action capture. Fifteen key body joints of the athletes during the jumping process are labeled at 15 frames per second to support accurate jumping action analysis.

[0152] In comparison with mainstream video generation large model methods, the present application exceeds similar methods in video generation quality and timing consistency.

[0153] Table 1

[0154] Model FVD↓ PSNR↑ SSIM↑ MimicMotion 612.86 13.97 0.641 AnimateAnyone 376.54 16.73 0.724 The method of the present application 334.69 20.55 0.760

[0155] The above is an exemplary embodiment of the present application, and the protection scope of the present application is defined by the claims and their equivalents.

Claims

1. A method of generating a video of a high jump action, characterized by, The method comprises the following steps: extracting joint coordinates of each frame of video in real-time multi-view motion video through a YOLOv11-Pose model to generate skeleton data containing frame number, joint index and joint coordinates; preprocessing the skeleton data and converting the preprocessed skeleton data into a pose graph; extracting image features of each frame of video in real-time multi-view motion video through an embedding network; performing feature fusion on the pose graph and the image features to obtain feature representation; inputting the pose graph and the feature representation as control conditions into a stable video diffusion model SVD to generate a motion trajectory video after decoding; processing the motion trajectory video through an image enhancement technique to obtain a high jump demonstration video.

2. The method of claim 1, wherein, The YOLOv11-Pose model comprises: An extension module is configured to extend the annotation file, so that each frame feature map has an additional stage label, which is represented as: σ i ∈ {Approach, Takeoff, Flight, Landing}, wherein Approach, Takeoff, Flight and Landing represent a take-off label, a flight label, a pass-through label and a landing label, respectively. a temporary buffer for temporarily storing the preceding frame feature map F i-1 and the following frame feature map F i+1 ; A splicing module is configured to splice the preceding frame feature map F i-1 , the following frame feature map F i+1 and the current frame feature map F i to obtain spliced feature maps (F i , F i-1 , F i+1 ). The time sequence attention branch module performs shallow fusion on the spliced feature map, and is represented as: wherein, represents the fused feature map, and represents the time sequence attention branch module. a detection head for predicting joint coordinates in a fused feature map to generate skeleton data containing frame number, joint index and joint coordinates; wherein the detection head comprises a C3k2 module and a C2PSA module, the C3k2 module uses a deepened convolutional layer to strengthen feature extraction capability, and the C2PSA module captures global information through a self-attention mechanism; the joint coordinates predicted by the detection head are represented as: wherein, represents the input fused feature map, and θ is a model parameter; k represents a joint node index, represents the kth joint node coordinate.

3. The method of claim 1, wherein, a loss function of the YOLOv11-Pose model is represented as: L = λ1L kp + λ2L stage ; wherein L kp represents the joint detection loss, L stage represents the stage recognition loss, λ1, λ2 both represent weight coefficients; (x k , y k ) represents the kth joint coordinate; σ i represents the discrete ith class label, used to represent the action stage to which the sample belongs; represents the stage prediction value of the i'th sample by the YOLOv11-Pose model, i.e., the probability distribution generated by the normalization function softmax.

4. The method of claim 1, wherein, The YOLOv11-Pose model further comprises a multi-frame skeleton fusion module, which is used for performing smoothing filtering on each joint coordinate predicted by the detection head on a time axis, and the smoothed joint coordinates are represented as: wherein Φ represents a smoothing function for eliminating inference noise and maintaining action coherence; represents the kth joint coordinate predicted by the detection head at time t; represents the kth joint coordinate predicted by the detection head at time t after smoothing.

5. The method of claim 1, wherein, The preprocessing of the skeleton data comprises: performing smoothing on the time series of joint coordinates in each frame through a one-dimensional Gaussian filter to obtain smoothed joint coordinates, which are represented as: where p k,t+i G(i;σ) denotes a standard Gaussian kernel, and σ denotes a filter width, G(i;σ) denotes a standard Gaussian kernel, and σ denotes a filter width, on the basis of the smoothed joint coordinates, the reasonable positions of missing points are inferred through a local action model, which are represented as: Wherein, Ψ (p) is a constraint term, used to limit joint angle, range of motion or physical consistency; λ represents the balance coefficient; p represents the possible coordinates of the missing k th joint node at t time; represents the smoothed k th joint node coordinates at t-1 time; represents the smoothed k th joint node coordinates at t+1 time; represents the inferred missing k th joint node coordinates at t time.

6. The method of claim 1, wherein, the preprocessed skeleton data is converted into a pose graph, the pose graph comprises joint coordinates and action information, and the pose graph should satisfy: the drawing color of the current frame skeleton trajectory is brighter than the drawing color of the previous frame skeleton trajectory; when generating a multi-view graph, the positions of the joints in all views should be aligned and matched; only key frames with significant changes are subjected to time trajectory drawing.

7. The method of claim 1, wherein, The loss function of the stable video diffusion model SVD comprises conditional consistency loss and physical reasonableness loss; the conditional consistency loss is represented as: wherein p t,k represents the input coordinates of the kth joint node in the tth frame, represents the predicted position of the kth joint node in the tth frame in the generated motion trajectory video; the physical reasonableness loss is represented as: where θ t,k denotes the angle of the kth joint in the tth frame, denotes the target angle of the kth joint in the tth frame, v t,k denotes the velocity of the kth joint in the tth frame, v t-1,k denotes the velocity of the kth joint in the t-1th frame.

8. A system for generating a video of a high jump motion, the system being used in the method for generating a video of a high jump motion according to any one of claims 1 to 7, characterized in that The method comprises the following steps: a coordinate extraction module extracts joint coordinates of each frame of video in multi-view motion video through a YOLOv11-Pose model to generate skeleton data containing frame number, joint index and joint coordinates; a preprocessing module preprocesses the skeleton data; a pose graph generation module generates a pose graph through the preprocessed skeleton data; a feature extraction module extracts image features of each frame of video in real-time multi-view motion video through an embedding network; a fusion module performs feature fusion on the pose graph and the image features to obtain feature representation; The trajectory generation module inputs the pose graph and the feature representation as control conditions into a stable video diffusion model SVD, and generates a motion trajectory video after decoding; The enhancement module processes the motion trajectory video through image enhancement technology to obtain a high jump demonstration video.

9. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the method for generating a high jump action video according to any one of claims 1 to 7.

10. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the method for generating a high jump action video according to any one of claims 1 to 7. The computer program is executed by the processor to implement the method for generating a high jump action video according to any one of claims 1 to 7.

Images