A markerless dynamics analysis method and system for real-time tracking and photographing of a drone

Abstract

The application provides a kind of unmanned aerial vehicle real-time tracking shooting markless dynamics analysis method and system, comprising: by flight control to unmanned aerial vehicle, obtain the dynamic image data of runner in playground curve or curve path running in real-time tracking shooting of unmanned aerial vehicle;Synchronization and preprocessing are carried out to the dynamic image data recorded by different cameras;The 2D key points of human body are extracted from the preprocessed image, and the three-dimensional motion trajectory of human body under the local coordinate system of camera is reconstructed;The posture of three-dimensional motion trajectory is compensated to ensure the consistency of human body joint trajectory under the ground world reference system, and human body model scaling and muscle model scaling calculation are carried out, inverse kinematics analysis is carried out;According to inverse kinematics data, the centripetal acceleration of runner in curve motion is calculated;Based on centripetal acceleration, inverse dynamics calculation is carried out, and ground support force is calculated.The application can carry out unmanned aerial vehicle real-time tracking shooting markless dynamics analysis.

Description

Technical Field

[0001] This invention relates to the fields of sports biomechanics analysis, computer vision and intelligent mobile photography technology, and in particular to a label-free dynamics analysis method and system for real-time tracking and shooting by unmanned aerial vehicles (UAVs). Background Technology

[0002] In recent years, with the rapid development of sports science, rehabilitation medicine and intelligent health monitoring, markerless motion capture technology, as a non-invasive method of human motion capture, has been widely studied and applied in many fields due to its advantages such as convenience, efficiency and strong adaptability to the environment.

[0003] Typical label-free kinematics / dynamics analysis schemes are mostly based on fixed multi-camera arrays or depth cameras, combined with human pose estimation algorithms (such as OpenPose, MediaPipe, VIBE, OpenCap, etc.) to estimate motion trajectories and biomechanical parameters through 2D / 3D keypoint reconstruction. These systems are mostly deployed in laboratory environments and require fixed viewing angles, stable lighting, and accurate calibration, thus their adaptability to outdoor or large open-field environments is relatively weak.

[0004] Meanwhile, existing systems are mostly statically deployed and cannot move freely with the athlete. They are only suitable for motion analysis in limited spaces within a controlled area, which is insufficient to meet the needs of practical scenarios such as running training, track and field testing, and gait analysis in the wild. Summary of the Invention

[0005] To address the technical problems existing in the prior art, the present invention provides a markerless dynamics analysis method and system for real-time tracking and imaging by unmanned aerial vehicles (UAVs), the technical solution of which is as follows:

[0006] On the one hand, a label-free dynamics analysis method for real-time tracking and filming by unmanned aerial vehicles is provided, the method including:

[0007] S1. By controlling the flight of the drone, obtain dynamic image data of athletes running on the bends or curved paths of the playground in real time by tracking and capturing the drone.

[0008] S2. Synchronize and preprocess dynamic image data recorded by different cameras;

[0009] S3. Use human pose recognition algorithm to extract 2D key points of human body from preprocessed image, and combine binocular or multi-view visual triangulation and UAV flight altitude data to reconstruct the three-dimensional motion trajectory of human body in camera local coordinate system.

[0010] S4. Based on the heading angle data recorded by the IMU on the UAV, the attitude of the three-dimensional motion trajectory is compensated to ensure the consistency of the human joint trajectory in the ground world reference frame. The human body model scaling and muscle model scaling calculations are performed, and inverse kinematics analysis is carried out to obtain inverse kinematics data.

[0011] S5. Calculate the centripetal acceleration of the athlete during the curve motion based on the inverse kinematics data.

[0012] S6. Based on the centripetal acceleration, perform inverse dynamics calculations to estimate the ground support force.

[0013] Optionally, the flight control of the UAV in S1 specifically includes three parts:

[0014] Part 1: Speed ​​Control;

[0015] The control objectives for speed control during cornering are: to maintain a constant relative distance d(t) between the drone and the operator, and to adjust the drone's linear velocity v(t) to counteract the centripetal acceleration during turning. The control equation is as follows:

[0016] v d (t)=v h (t)+K d ·e d (t)+K c ·(a c，traget -a c (t))

[0017] Where: v h (t) represents the velocity of the athlete; K d For distance control gain, it is adjusted empirically or controlled by PID; e d (t) represents the distance error, e d (t)=d target -d(t), K target d(t) represents the set distance between the drone and the athlete, and d(t) represents the actual distance between the drone and the athlete at the current moment; K c Empirical parameter tuning for centripetal acceleration compensation gain in UAVs; a c，traget : Centripetal acceleration of the UAV target; a c (t): Actual centripetal acceleration of the UAV;

[0018] Part Two: Trajectory Prediction Layer;

[0019] Predicting the position, velocity, acceleration, and turning radius of the mover within the next N frames, as well as the drone's heading angle, provides fundamental data for drone path optimization, including:

[0020] Trajectory data acquisition: Collect trajectory points (x, y, z) of the mover over the past M frames.h y h );

[0021] Fitting historical trajectory points using the least squares method:

[0022] y h (t)=a·x h (t) 2 +b·x h (t)+c

[0023] Where a, b, and c are the quadratic curve fitting coefficients;

[0024] The predicted position at time t+k is:

[0025]

[0026] y h (t+k)=a·x h (t+k) 2 +b·x h (t+k)+c

[0027] Where x h (t): The x-coordinate of the mover at the current moment; The velocity of the mover in the x-direction at the current moment; The current acceleration of the moving person in the x-direction; (x h (t+k), y h (t+k)): Predicted position of the mover after k frames; k: Number of frames to be predicted; Δt: Time interval between each frame;

[0028] Predicted speed:

[0029]

[0030] in: Predict the velocity in the x-direction; Predict the velocity in the y-direction;

[0031] Predicted acceleration:

[0032]

[0033] in The acceleration is in the x-direction; The acceleration is in the y-direction; a h (t+k) is the total acceleration;

[0034] Turning radius: Predicting the turning radius of a mover through circular arc fitting.

[0035]

[0036] Heading angle estimation:

[0037]

[0038] Where: (x h (t), y h (t) represents the current location of the drone; (x) represents the current location of the drone. c y c ) represents the position of the previous frame; θ h (t) represents the estimated current heading angle of the UAV;

[0039] Part Three: Path Optimization Layer;

[0040] Taking into account path error, energy consumption, and centripetal acceleration control, the optimal control path for the UAV in a curve is generated. The objectives are: to minimize path error while maintaining a constant relative distance between the UAV and the operator; to avoid coordinating paths between the two UAVs and prevent mutual obstruction; and to reduce energy consumption by optimizing propulsion and steering energy consumption and controlling battery wear. Therefore, the optimization objective function is defined as follows:

[0041] J = α·J path +β·J collision +γ·J energy

[0042] The optimal path control strategy is obtained by solving the problem through gradient descent, iterative optimization, or Nash equilibrium. The minimum value of J and its corresponding parameters are the optimal control path. Here, α, β, and γ are weight coefficients that are adjusted according to the actual task.

[0043] J path For path error terms,

[0044] Q d Q R , Weighting coefficients are used to adjust the impact of path error, turning radius error, and centripetal acceleration error on the control strategy; e d，k e represents the distance error. d，k =d target -d(t+k), d target d(t+k) is the preset distance between the drone and the athlete; d(t+k) is the predicted actual distance between the drone and the athlete at time k; e R,k e represents the turning radius error. R,k =R target -R(t+k), R target The expected turning radius of the player is the predicted value, R(t+k): the actual turning radius of the player at time k in frame k; Centripetal acceleration error actarget : The expected centripetal acceleration of the athlete, which is the predicted value, a c (t+k): The actual centripetal acceleration of the mover at time k;

[0045] J collision For obstacle avoidance,

[0046] Where d ij，k : The distance between two UAVs i and j at time k;

[0047] J energy This is an energy consumption item.

[0048] Where α and β are energy consumption weights, α is related to air resistance and propulsion efficiency, and β is related to moment of inertia and frictional torque; v d (t) represents the linear propulsion speed of the UAV; ω d (t) Rotation speed of the UAV.

[0049] Optionally, S3 specifically includes:

[0050] A deep learning model is used to extract 2D key points from each camera, and 3D reconstruction is performed using triangulation. The least squares method is then used to optimize the reconstruction of the 3D points. The calculation method is as follows:

[0051]

[0052] Where: X = [X, Y, Z] T : The coordinates of the three-dimensional point to be determined; X i : 2D projection point in the i-th camera; K i : This is the camera intrinsic parameter matrix, representing the camera's internal parameters; [R] i |T i ]: Camera extrinsic parameter matrix; R i : Rotation matrix; T i The translation vector is calculated based on the camera calibration process;

[0053] Adjust the 3D trajectory based on the drone's flight altitude data:

[0054] Assuming the drone's altitude at time t is H(t), the three-dimensional point coordinates become:

[0055]

[0056] Where H ref : The set altitude of the drone;

[0057] The solution yields the three-dimensional motion trajectory data of the camera in the local coordinate system.

[0058] Optionally, S4 specifically includes:

[0059] The attitude compensation for the three-dimensional motion trajectory is performed by inputting the three-dimensional motion trajectory and IMU heading angle data. The attitude compensation method is as follows: IMU data fusion is performed to construct a rotation matrix.

[0060] R IMU =R z (ψ)·R y (θ)·R x (φ)

[0061] Where: R z (ψ) is the heading angle rotation matrix of the UAV, which adjusts the heading angle of the UAV for left and right rotation:

[0062]

[0063] R y (θ): The drone pitch angle rotation matrix, which adjusts the drone's pitch angle up and down;

[0064]

[0065] R x (φ): The roll angle rotation matrix of the drone, which adjusts the roll angle of the drone when it rolls left and right;

[0066]

[0067] The set of 3D points obtained in the local coordinate system is as follows:

[0068] P local =[X local ,Y local Z local ] T

[0069] The formula for converting to a 3D point in the world coordinate system is:

[0070] P world =R IMU ·P locat

[0071] Then, the human body model and muscle model are scaled and calculated, and inverse kinematics analysis is performed to obtain inverse kinematics data. The inverse kinematics data includes: the trajectory of the human body's center of mass, joint motion angles, skeletal motion trajectories, and the lengths of some muscles and ligaments.

[0072] Optionally, S5 specifically includes:

[0073] Combining the camera parameters, drone attitude changes, and data sampling errors in the UAV system, the formula for centripetal acceleration is expressed as:

[0074]

[0075] Where Δv represents the motion velocity estimation error introduced by camera distortion correction and frame rate fluctuations, calculated based on optical flow, frame difference, and fused IMU data; ΔR represents the radius of curvature estimation error, the distance error of the radius of curvature caused by insufficient camera distortion correction and lens angle changes, corrected based on the radial distortion model; k1, k2, k3 are weighting coefficients used to adjust the contribution of attitude angle and viewpoint errors to centripetal acceleration; φ, θ are the roll and pitch angles of the UAV, which affect the camera's viewpoint during rapid turns; δ view Viewpoint error, caused by changes in the drone's yaw angle and resulting camera field of view shift, reflects the impact of these changes on displacement estimation. The calculation formula is as follows: Δx is the viewpoint offset, and f is the focal length. This represents the change in the field of view angle.

[0076] Optionally, S6 specifically includes:

[0077] Ground support force (GRF) is a result of the acceleration of the human body's center of mass, and is expressed as:

[0078] F GRF =[F GRF,x F GRF，y F GRF,z ]

[0079] Wherein: F GRF,z F represents the force perpendicular to the direction supporting the body weight. GRF,x F represents the horizontal force in the direction of the curve / centripetal force; GRF,y Force in the forward / backward direction, indicating the direction of acceleration / deceleration;

[0080] F in the vertical direction GRF,z :

[0081] The vertical GRF is used to support body weight, F GRF,z =m·(a z +g), where m is the mass of the human body; g is the acceleration due to gravity; a z The vertical acceleration is calculated by taking the second derivative of the data obtained from the trajectory of the human body's center of mass. Z x (t) represents the trajectory data of the human body's center of mass in the vertical direction;

[0082] F in the horizontal direction GRF，x:

[0083] The horizontal GRF provides the centripetal force during motion, F GRF，x =m·(a c +a x ), where a c Centripetal acceleration; a x The horizontal translational acceleration is calculated by taking the second derivative of the data on the trajectory of the human body's center of mass. Z x (t) represents the trajectory data of the human body's center of mass in the horizontal direction;

[0084] F in the forward and backward directions GRF，y :

[0085] The GRF in the forward and backward directions is used for propulsion or braking, F GRF，y =m·a y , where a y Acceleration in the forward and backward directions is calculated by taking the second derivative of the data on the trajectory of the human body's center of mass. Z y (t) represents the trajectory data of the human body's center of mass in the forward and backward directions;

[0086] Ground support force (GRF):

[0087]

[0088] On the other hand, a label-free dynamics analysis system for real-time tracking and filming by a drone is provided, the system comprising:

[0089] The acquisition module is used to acquire dynamic image data of athletes running on the bends or curved paths of the playground in real time by controlling the flight of the drone.

[0090] The synchronous preprocessing module is used to synchronize and preprocess dynamic image data recorded by different cameras;

[0091] The extraction and reconstruction module is used to extract 2D key points of the human body from the preprocessed image using a human pose recognition algorithm, and reconstruct the three-dimensional motion trajectory of the human body in the local coordinate system of the camera by combining binocular or multi-view visual triangulation and UAV flight altitude data.

[0092] The compensation module is used to compensate the attitude of the three-dimensional motion trajectory based on the heading angle data recorded by the IMU on the UAV, to ensure the consistency of the human joint trajectory in the ground world reference frame, and to perform human model scaling and muscle model scaling calculations, perform inverse kinematics analysis, and obtain inverse kinematics data.

[0093] The calculation module is used to calculate the centripetal acceleration of the athlete during cornering motion based on the inverse kinematics data.

[0094] The calculation module is used to perform inverse dynamics calculations based on the centripetal acceleration to calculate the ground support force.

[0095] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the aforementioned labelless dynamics analysis method for real-time tracking and filming by a drone.

[0096] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement the above-described markerless dynamics analysis method for real-time tracking and filming by a drone.

[0097] The beneficial effects of the technical solution provided by this invention include at least the following:

[0098] 1) Wide range of outdoor applicability: The testing site is no longer limited to a specific location, and dynamic analysis can be carried out without the need to pre-deploy multiple fixed camera devices, making it suitable for a wide range of real-world motion scenarios. This invention breaks through the limitations of existing systems on site and space, and is applicable to motion data collection and analysis in outdoor open environments (especially playground curves or curved paths).

[0099] 2) Flexible deployment and easy expansion: One or more drones can be used to collect data from different angles as needed, resulting in a more comprehensive collection perspective. It is also easy to extend to applications such as outdoor sports training, monitoring of athletes' daily practice, and rehabilitation follow-up, enabling the collection and analysis of motion dynamics data anytime, anywhere. The calibration process is simpler than fixed-location data collection, reducing the complexity of hardware deployment and calibration, and offering greater flexibility.

[0100] 3) Analysis can be performed without labeling the athlete: Artificial intelligence algorithms are integrated into the label-free dynamic parameter estimation process, which improves the accuracy and robustness of parameter estimation.

[0101] 4) Strong adaptability to dynamic motion: By introducing the flight speed of the UAV as an auxiliary modeling parameter, the adaptability of dynamic parameter estimation to dynamic situations such as changes in motion speed is improved. Even if the tester's speed suddenly increases or decreases, the system can adaptively collect data based on the tester's speed.

[0102] 5) Centripetal acceleration modeling: It can adapt to motion in different venues, paths and scenarios, accurately capture and compensate for the influence of inertial forces received by the tester, improve the calculation accuracy of various results output, and enhance the versatility and robustness of the sports biomechanical analysis system. Attached Figure Description

[0103] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0104] Figure 1 This is a flowchart of a markerless dynamics analysis method for real-time tracking and photography by a drone, provided by an embodiment of the present invention.

[0105] Figure 2 This is a schematic diagram of the relative flight between the drone and the athlete provided in an embodiment of the present invention;

[0106] Figure 3 This is an acceleration analysis diagram of an athlete running on a curved or curved path on a playground, provided by an embodiment of the present invention.

[0107] Figure 4 This is a block diagram of a markerless dynamics analysis system for real-time tracking and photography by a drone, provided in an embodiment of the present invention.

[0108] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0109] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0110] This invention provides a markerless dynamics analysis method for real-time tracking and filming by a drone. This method can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The diagram shown is a flowchart of the method. The processing flow may include the following steps:

[0111] S1. By controlling the flight of the drone, obtain dynamic image data of athletes running on the bends or curved paths of the playground in real time by tracking and capturing the drone.

[0112] The practical embodiment of this invention mainly includes at least one multi-rotor drone with stable flight capability, and each drone includes three cameras, such as... Figure 2As shown, one camera is located on the underside of the drone, while the other two are located at either end of a dedicated mounting bracket. When there is only one drone, the dedicated mounting bracket is longer than 1.5m, with two high-definition cameras mounted at both ends. The cameras are horizontally arranged with a distance of more than 2.5m between them, forming a wide-baseline binocular vision system for real-time acquisition of image sequences of human movement. When there are multiple drones, the dedicated mounting bracket is longer than 0.5m, with two high-definition cameras mounted at both ends. The cameras are horizontally arranged with a distance of more than 0.8m between them. The drone is equipped with a GPS module, an inertial measurement unit (IMU), and a visual navigation system to acquire real-time location information and determine the flight trajectory. Visual / fusion target detection and tracking technology, flight path control algorithms, and drone attitude and speed control algorithms are used to achieve human body following and filming functions, maintaining a fixed relative distance with the test subject. The camera on the underside of the drone is connected to the drone's flight control system to follow the human body's movement trajectory, maintaining a fixed relative distance with the test subject.

[0113] Optionally, the flight control of the UAV in S1 specifically includes three parts:

[0114] Part 1: Speed ​​Control;

[0115] The control objectives for speed control during cornering are: to maintain a constant relative distance d(t) between the drone and the operator, and to adjust the drone's linear velocity v. d (t) To counteract the centripetal acceleration during turning, the control equation is:

[0116] v d (t)=v h (t)+K d ·e d (t)+K c ·(a c，traget -a c (t))

[0117] Where: v h (t) represents the velocity of the athlete; K d For distance control gain, it is adjusted empirically or controlled by PID; e d (t) represents the distance error, e d (t)=d target -d(t), d target d(t) represents the set distance between the drone and the athlete, and d(t) represents the actual distance between the drone and the athlete at the current moment; K c Empirical parameter tuning for centripetal acceleration compensation gain in UAVs; a c，traget : Centripetal acceleration of the UAV target; a c (t): Actual centripetal acceleration of the UAV;

[0118] Part Two: Trajectory Prediction Layer;

[0119] Predicting the position, velocity, acceleration, and turning radius of the mover within the next N frames, as well as the drone's heading angle, provides fundamental data for drone path optimization, including:

[0120] Trajectory data acquisition: Collect trajectory points (x, y, z) of the mover over the past M frames. h y h );

[0121] Fitting historical trajectory points using the least squares method:

[0122] y h (t)=a·x h (t) 2 +b·x h (t)+c

[0123] Where a, b, and c are the quadratic curve fitting coefficients;

[0124] The predicted position at time t+k is:

[0125]

[0126] y h (t+k)=a·x h (t+k) 2 +b·x h (t+k)+c

[0127] Where x h (t): The x-coordinate of the mover at the current moment; The velocity of the mover in the x-direction at the current moment; The current acceleration of the moving person in the x-direction; (x h (t+k), y h (t+k)): Predicted position of the mover after k frames; k: Number of frames to be predicted; Δt: Time interval between each frame;

[0128] Predicted speed:

[0129]

[0130] in: Predict the velocity in the x-direction; Predict the velocity in the y-direction;

[0131] Predicted acceleration:

[0132]

[0133] in The acceleration is in the x-direction; The acceleration is in the y-direction; a h (t+k) is the total acceleration;

[0134] Turning radius: Predicting the turning radius of a mover through circular arc fitting.

[0135]

[0136] Heading angle estimation:

[0137]

[0138] Where: (x h (t), y h (t) represents the current location of the drone; (x) represents the current location of the drone. c y c ) represents the position of the previous frame; θ h (t) represents the estimated current heading angle of the UAV;

[0139] Part Three: Path Optimization Layer;

[0140] Taking into account path error, energy consumption, and centripetal acceleration control, the optimal control path for the UAV in a curve is generated. The objectives are: to minimize path error while maintaining a constant relative distance between the UAV and the operator; to avoid coordinating paths between the two UAVs and prevent mutual obstruction; and to reduce energy consumption by optimizing propulsion and steering energy consumption and controlling battery wear. Therefore, the optimization objective function is defined as follows:

[0141] J = α·J path +β·J collision +γ·J energy

[0142] The optimal path control strategy is obtained by solving the problem through gradient descent, iterative optimization, or Nash equilibrium. The minimum value of J and its corresponding parameters are the optimal control path. Here, α, β, and γ are weight coefficients that are adjusted according to the actual task.

[0143] J path For path error terms,

[0144] Q d Q R , Weighting coefficients, adjusting the impact of path error, turning radius error, and centripetal acceleration error on the control strategy; e d,k e represents the distance error. d，k =d target -d(t+k), d target d(t+k) is the preset distance between the drone and the athlete; d(t+k) is the predicted actual distance between the drone and the athlete at time k; e R，ke represents the turning radius error. R,k =R target -R(t+k), R target The expected turning radius of the player is the predicted value, R(t+k): the actual turning radius of the player at time k in frame k; Centripetal acceleration error a ctarget : The expected centripetal acceleration of the athlete, which is the predicted value, a c (t+k): The actual centripetal acceleration of the mover at time k;

[0145] J collision For obstacle avoidance,

[0146] Where d ij,k : The distance between two UAVs i and j at time k;

[0147] J energy This is an energy consumption item.

[0148] Where α and β are energy consumption weights, α is related to air resistance and propulsion efficiency, and β is related to moment of inertia and frictional torque; v d (t) represents the linear propulsion speed of the UAV; ω d (t) Rotation speed of the UAV.

[0149] S2. Synchronize and preprocess dynamic image data recorded by different cameras;

[0150] The embodiments of the present invention require the synchronization of dynamic image data recorded by different cameras. The specific synchronization method is existing technology and will not be described in detail here.

[0151] The data preprocessing in this embodiment includes distortion correction of data recorded by different cameras; smoothing of images using Gaussian filtering to reduce noise, etc.

[0152] S3. Use human pose recognition algorithm to extract 2D key points of human body from preprocessed image, and combine binocular or multi-view visual triangulation and UAV flight altitude data to reconstruct the three-dimensional motion trajectory of human body in camera local coordinate system.

[0153] Optionally, S3 specifically includes:

[0154] A deep learning model is used to extract 2D key points from each camera, and 3D reconstruction is performed using triangulation. The least squares method is then used to optimize the reconstruction of the 3D points. The calculation method is as follows:

[0155]

[0156] Where: X = [X, Y, Z] T : The coordinates of the three-dimensional point to be determined; X i : 2D projection point in the i-th camera; K i : This is the camera intrinsic parameter matrix, representing the camera's internal parameters; [R] i |T i ]: Camera extrinsic parameter matrix; R i : Rotation matrix; T i The translation vector is calculated based on the camera calibration process;

[0157] Adjust the 3D trajectory based on the drone's flight altitude data:

[0158] Assuming the drone's altitude at time t is H(t), the three-dimensional point coordinates become:

[0159]

[0160] Where H ref : The set altitude of the drone;

[0161] The solution yields the three-dimensional motion trajectory data of the camera in the local coordinate system.

[0162] S4. Based on the heading angle data recorded by the IMU on the UAV, the attitude of the three-dimensional motion trajectory is compensated to ensure the consistency of the human joint trajectory in the ground world reference frame. The human body model scaling and muscle model scaling calculations are performed, and inverse kinematics analysis is carried out to obtain inverse kinematics data.

[0163] Optionally, S4 specifically includes:

[0164] The attitude compensation for the three-dimensional motion trajectory is performed by inputting the three-dimensional motion trajectory and IMU heading angle data. The attitude compensation method is as follows: IMU data fusion is performed to construct a rotation matrix.

[0165] R IMU =R z (ψ)·R y (θ)·R x (φ)

[0166] Where: R z (ψ) is the heading angle rotation matrix of the UAV, which adjusts the heading angle of the UAV for left and right rotation:

[0167]

[0168] R y (θ): The drone pitch angle rotation matrix, which adjusts the drone's pitch angle up and down;

[0169]

[0170] R x (φ): The roll angle rotation matrix of the drone, which adjusts the roll angle of the drone when it rolls left and right;

[0171]

[0172] The set of 3D points obtained in the local coordinate system is as follows:

[0173] P locot =[X rocot ,Y locat Z tocal ] T

[0174] The formula for converting to a 3D point in the world coordinate system is:

[0175] P world =R IMU ·P local

[0176] Then, the human body model and muscle model are scaled and calculated, and inverse kinematics analysis is performed to obtain inverse kinematics data (the specific method is existing technology and will not be described in detail here). The inverse kinematics data includes: the trajectory of the human body's center of mass, joint motion angles, skeletal motion trajectories, and the lengths of some muscles and ligaments.

[0177] S5. Based on the aforementioned inverse kinematics data, calculate the centripetal acceleration of the athlete during cornering motion, such as... Figure 3 As shown;

[0178] Optionally, S5 specifically includes:

[0179] Combining the camera parameters, drone attitude changes, and data sampling errors in the UAV system, the formula for centripetal acceleration is expressed as:

[0180]

[0181] Where Δv represents the motion velocity estimation error introduced by camera distortion correction and frame rate fluctuations, calculated based on optical flow, frame difference, and fused IMU data; ΔR represents the radius of curvature estimation error, the distance error of the radius of curvature caused by insufficient camera distortion correction and lens angle changes, corrected based on the radial distortion model; k1, k2, k3 are weighting coefficients used to adjust the contribution of attitude angle and viewpoint errors to centripetal acceleration; φ, θ are the roll and pitch angles of the UAV, which affect the camera's viewpoint during rapid turns; δ viewViewpoint error, caused by changes in the drone's yaw angle and resulting camera field of view shift, reflects the impact of these changes on displacement estimation. The calculation formula is as follows: Δx is the viewpoint offset, and f is the focal length. This represents the change in the field of view angle.

[0182] S6. Based on the centripetal acceleration, perform inverse dynamics calculations to estimate the ground support force.

[0183] Optionally, S6 specifically includes:

[0184] Ground support force (GRF) is a result of the acceleration of the human body's center of mass, and is expressed as:

[0185] F GRF =[F GRF，x F GRF,y ,F GRF,z ]

[0186] Wherein: F GRF，z F represents the force perpendicular to the direction supporting the body weight. GRF，x F represents the horizontal force in the direction of the curve / centripetal force; CRF，y Force in the forward / backward direction, indicating the direction of acceleration / deceleration;

[0187] F in the vertical direction CRF，z :

[0188] The vertical GRF is used to support body weight, F GRF，z =m·(a z +g), where m is the mass of the human body; g is the acceleration due to gravity; a z The vertical acceleration is calculated by taking the second derivative of the data on the trajectory of the human body's center of mass (including data in three directions): Z z (t) represents the trajectory data of the human body's center of mass in the vertical direction;

[0189] F in the horizontal direction GRF，x :

[0190] The horizontal GRF provides the centripetal force during motion, F GRF，x =m·(a c +a x ), where a c Centripetal acceleration; a x The horizontal translational acceleration is calculated by taking the second derivative of the data on the trajectory of the human body's center of mass. Z x (t) represents the trajectory data of the human body's center of mass in the horizontal direction;

[0191] F in the forward and backward directions GRF，y :

[0192] The GRF in the forward and backward directions is used for propulsion or braking, F GRF，y =m·a y , where a y Acceleration in the forward and backward directions is calculated by taking the second derivative of the data on the trajectory of the human body's center of mass. z y (t) represents the trajectory data of the human body's center of mass in the forward and backward directions;

[0193] Ground support force (GRF):

[0194]

[0195] like Figure 4 As shown, this embodiment of the invention also provides a markerless dynamics analysis system for real-time tracking and filming by a drone, the system comprising:

[0196] The acquisition module 410 is used to acquire dynamic image data of athletes running on the bends or curved paths of the playground in real time by controlling the flight of the drone.

[0197] The synchronous preprocessing module 420 is used to synchronize and preprocess dynamic image data recorded by different cameras;

[0198] The extraction and reconstruction module 430 is used to extract the 2D key points of the human body from the preprocessed image using a human pose recognition algorithm, and reconstruct the three-dimensional motion trajectory of the human body in the local coordinate system of the camera by combining binocular or multi-view visual triangulation and UAV flight altitude data.

[0199] The compensation module 440 is used to compensate the attitude of the three-dimensional motion trajectory based on the heading angle data recorded by the IMU on the UAV, to ensure the consistency of the human joint trajectory in the ground world reference frame, and to perform human model scaling and muscle model scaling calculations, perform inverse kinematics analysis, and obtain inverse kinematics data.

[0200] The calculation module 450 is used to calculate the centripetal acceleration of the athlete in the curve motion based on the inverse kinematics data.

[0201] The calculation module 460 is used to perform inverse dynamics calculations based on the centripetal acceleration to calculate the ground support force.

[0202] The labelless dynamics analysis system for real-time tracking and shooting by UAVs provided in this embodiment of the invention has a functional structure that corresponds to the labelless dynamics analysis method for real-time tracking and shooting by UAVs provided in this embodiment of the invention, and will not be described again here.

[0203] Figure 5 This is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present invention. The electronic device 500 can vary considerably due to differences in configuration or performance, and may include one or more processors (central processing units, CPUs) 501 and one or more memories 502. The memories 502 store at least one instruction, which is generated by the processors 501.

[0204] 01 Load and execute the steps to implement the above-described method for real-time tracking and image capture of unmarked dynamics analysis by UAV.

[0205] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the aforementioned markerless dynamics analysis method for real-time tracking and filming by a drone. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0206] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0207] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A markerless dynamics analysis method for real-time tracking and filming by unmanned aerial vehicles (UAVs), characterized in that, The method includes: S1. By controlling the flight of the drone, obtain dynamic image data of the athlete running on the curved path in real time by tracking and capturing the athlete's movements. S2. Synchronize and preprocess dynamic image data recorded by different cameras; S3. Use human pose recognition algorithm to extract 2D key points of human body from preprocessed image, and combine binocular or multi-view visual triangulation and UAV flight altitude data to reconstruct the three-dimensional motion trajectory of human body in camera local coordinate system. S4. Based on the heading angle data recorded by the IMU on the UAV, the attitude of the three-dimensional motion trajectory is compensated to ensure the consistency of the human joint trajectory in the ground world reference frame. The human body model scaling and muscle model scaling calculations are performed, and inverse kinematics analysis is carried out to obtain inverse kinematics data. S5. Calculate the centripetal acceleration of the mover in the curved path motion based on the inverse kinematics data; S6. Based on the centripetal acceleration, perform inverse dynamics calculations to estimate the ground support force; The flight control of the UAV in S1 specifically includes three parts: Part 1: Speed ​​Control; The control objectives for speed control on a curved path are: to maintain a constant relative distance d(t) between the drone and the operator, and to adjust the drone's linear velocity. To counteract centripetal acceleration during turning, the control equation is: ； in: For the speed of the athlete; The distance control gain is determined by empirical parameter tuning or PID control. For distance error, , The set distance between the drone and the athlete. The actual distance between the drone and the athlete at the current moment; Empirical parameter tuning for centripetal acceleration compensation gain of UAVs; Centripetal acceleration of the drone target; Actual centripetal acceleration of the drone; Part Two: Trajectory Prediction Layer; Predicting the future of athletes Intra-frame position, velocity, acceleration, and turning radius, as well as the drone's heading angle, provide fundamental data for drone path optimization, including: Trajectory data acquisition: Collect trajectory points of the athlete over the past M frames. ; Fitting historical trajectory points using the least squares method: ； Where a, b, and c are the quadratic curve fitting coefficients; The predicted position at time t+k is: ； ； in The x-coordinate of the moving person at the current moment; The velocity of the mover in the x-direction at the current moment; : The x-axis acceleration of the moving person at the current moment; : Predicted mover position after k frames; : The predicted number of frames; The time interval between each frame; Predicted speed: ； ； ； in: Predict the velocity in the x-direction; Predict the velocity in the y-direction; Predicted acceleration: ； ； ； in The acceleration is in the x-direction; The acceleration is in the y-direction; This is the total acceleration; Turning radius: Predicting the turning radius of a mover through circular arc fitting. ； Heading angle estimation: ； in: This indicates the current location of the drone. The position of the previous frame; This is an estimate of the current heading angle of the UAV; Part Three: Path Optimization Layer; Taking into account path error, energy consumption, and centripetal acceleration control, the optimal control path for the UAV in a curved path is generated. The objectives are: to minimize path error while maintaining a constant relative distance between the UAV and the operator; to avoid path coordination between the two UAVs and prevent mutual occlusion; and to reduce energy consumption by optimizing propulsion and turning energy consumption and controlling battery wear. Therefore, the optimization objective function is defined as follows: ； The optimal path control strategy is obtained by solving the problem using gradient descent, iterative optimization, or Nash equilibrium. The minimum value of and its corresponding parameters constitute the optimal control path; among them, These are weighting coefficients, which are adjusted according to the actual task. For path error terms, ; in , , Weighting coefficients are used to adjust the impact of path error, turning radius error, and centripetal acceleration error on the control strategy. For distance error, , The set distance between the drone and the athlete is a preset value; Predicted drone and athlete frames k The actual distance at any given moment; For turning radius error, , The expected turning radius of the athlete is a predicted value. : The actual turning radius of the player at time k in frame k; Centripetal acceleration error ; The expected centripetal acceleration of the athlete is the predicted value. : The actual centripetal acceleration of the mover at time k; For obstacle avoidance, ; in At time k, the two drones The distance between; This is an energy consumption item. ; in As a weight for energy consumption, Related to air resistance and propulsion efficiency, Related to moment of inertia and frictional torque; This refers to the linear propulsion speed of the drone; Drone turning rotation speed.

2. The method according to claim 1, characterized in that, S3 specifically includes: A deep learning model is used to extract 2D key points from each camera, and 3D reconstruction is performed using triangulation. The least squares method is then used to optimize the reconstruction of the 3D points. The calculation method is as follows: ； in: The coordinates of the three-dimensional point to be determined; : 2D projection point in the i-th camera; : This is the camera intrinsic parameter matrix, representing the camera's internal parameters; Camera extrinsic matrix; Rotation matrix; The translation vector is calculated based on the camera calibration process; Adjust the 3D trajectory based on the drone's flight altitude data: Assume the drone's altitude at time t is Then the coordinates of the three-dimensional point become: ； ； ； in : The set altitude of the drone; The solution yields the three-dimensional motion trajectory data of the camera in the local coordinate system.

3. The method according to claim 1, characterized in that, S4 specifically includes: The attitude compensation for the three-dimensional motion trajectory is performed by inputting the three-dimensional motion trajectory and IMU heading angle data. The attitude compensation method is as follows: IMU data fusion is performed to construct a rotation matrix. ； in: The heading angle rotation matrix is ​​used to adjust the heading angle of the drone for left and right rotation: ； : UAV pitch angle rotation matrix, adjusts the pitch angle of the UAV tilting up and down; ； : The roll angle rotation matrix of the drone, which adjusts the roll angle of the drone when it rolls left and right; ； The set of 3D points obtained in the local coordinate system is as follows: ； The formula for converting to a 3D point in the world coordinate system is: ； Then, the scaling calculations for the human body model and muscle model are performed, and inverse kinematics analysis is conducted to obtain inverse kinematics data, which includes the trajectory of the human body's center of mass.

4. The method according to claim 1, characterized in that, S5 specifically includes: Combining the camera parameters, drone attitude changes, and data sampling errors in the UAV system, the formula for centripetal acceleration is expressed as: ； in The motion speed estimation error caused by camera distortion correction and frame rate fluctuation is calculated based on optical flow method, frame difference method and fused IMU data. The radius of curvature estimation error is due to the distance error of the radius of curvature caused by insufficient camera distortion correction and lens angle change factors, and is corrected based on the radial distortion model; : Weighting coefficients, used to adjust the contribution of attitude angle and viewpoint error to centripetal acceleration; The roll and pitch angles of the drone, during rapid turns, the changes in the drone's attitude will affect the camera's field of view; Viewpoint error, caused by changes in the drone's yaw angle and resulting camera field of view shift, reflects the impact of these changes on displacement estimation. The calculation formula is as follows: , This is the viewpoint offset. Focal length This represents the change in the field of view angle.

5. The method according to claim 1, characterized in that, S6 specifically includes: Ground support force (GRF) is a result of the acceleration of the human body's center of mass, and is expressed as: ； in: This refers to the force perpendicular to the direction supporting the body weight. This indicates the horizontal force in the direction of the curved path / centripetal force. Force in the forward / backward direction, indicating the direction of acceleration / deceleration; Vertical direction : The vertical GRF is used to support body weight. ,in, m For human body mass; g It is the acceleration due to gravity; The vertical acceleration is calculated by taking the second derivative of the data obtained from the trajectory of the human body's center of mass. , This refers to the trajectory data of the human body's center of mass in the vertical direction. Horizontal direction : The horizontal GRF provides the centripetal force during motion. ,in Centripetal acceleration; The horizontal translational acceleration is calculated by taking the second derivative of the data on the trajectory of the human body's center of mass. , This refers to the trajectory data of the human body's center of mass in the horizontal direction. Front and back directions : The GRF in the forward and backward directions is used for propulsion or braking. ,in Acceleration in the forward and backward directions is calculated by taking the second derivative of the data on the trajectory of the human body's center of mass. , This refers to the trajectory data of the human body's center of mass in the forward and backward directions. Ground support force (GRF): 。 6. A markerless dynamics analysis system for real-time tracking and filming by unmanned aerial vehicles (UAVs), characterized in that, The system includes: The acquisition module is used to acquire dynamic image data of athletes running on a curved path or curved path on the playground in real time by controlling the flight of the drone. The synchronous preprocessing module is used to synchronize and preprocess dynamic image data recorded by different cameras; The extraction and reconstruction module is used to extract 2D key points of the human body from the preprocessed image using a human pose recognition algorithm, and reconstruct the three-dimensional motion trajectory of the human body in the local coordinate system of the camera by combining binocular or multi-view visual triangulation and UAV flight altitude data. The compensation module is used to compensate the attitude of the three-dimensional motion trajectory based on the heading angle data recorded by the IMU on the UAV, to ensure the consistency of the human joint trajectory in the ground world reference frame, and to perform human model scaling and muscle model scaling calculations, perform inverse kinematics analysis, and obtain inverse kinematics data. The calculation module is used to calculate the centripetal acceleration of the mover in the curved path motion based on the inverse kinematics data; The calculation module is used to perform inverse dynamics calculations based on the centripetal acceleration to calculate the ground support force; The acquisition module is specifically used for: Part 1: Speed ​​Control; The control objectives for speed control on a curved path are: to maintain a constant relative distance d(t) between the drone and the operator, and to adjust the drone's linear velocity. To counteract centripetal acceleration during turning, the control equation is: ； in: For the speed of the athlete; The distance control gain is determined by empirical parameter tuning or PID control. For distance error, , The set distance between the drone and the athlete. The actual distance between the drone and the athlete at the current moment; Empirical parameter tuning for centripetal acceleration compensation gain of UAVs; Centripetal acceleration of the drone target; Actual centripetal acceleration of the drone; Part Two: Trajectory Prediction Layer; Predicting the future of athletes Intra-frame position, velocity, acceleration, and turning radius, as well as the drone's heading angle, provide fundamental data for drone path optimization, including: Trajectory data acquisition: Collect trajectory points of the athlete over the past M frames. ; Fitting historical trajectory points using the least squares method: ； Where a, b, and c are the quadratic curve fitting coefficients; The predicted position at time t+k is: ； ； in The x-coordinate of the moving person at the current moment; The velocity of the mover in the x-direction at the current moment; : The x-axis acceleration of the moving person at the current moment; : Predicted mover position after k frames; : The predicted number of frames; The time interval between each frame; Predicted speed: ； ； ； in: Predict the velocity in the x-direction; Predict the velocity in the y-direction; Predicted acceleration: ； ； ； in The acceleration is in the x-direction; The acceleration is in the y-direction; This is the total acceleration; Turning radius: Predicting the turning radius of a mover through circular arc fitting. ； Heading angle estimation: ； in: This indicates the current location of the drone. The position of the previous frame; This is an estimate of the current heading angle of the UAV; Part Three: Path Optimization Layer; Taking into account path error, energy consumption, and centripetal acceleration control, the optimal control path for the UAV in a curved path is generated. The objectives are: to minimize path error while maintaining a constant relative distance between the UAV and the operator; to avoid path coordination between the two UAVs and prevent mutual occlusion; and to reduce energy consumption by optimizing propulsion and turning energy consumption and controlling battery wear. Therefore, the optimization objective function is defined as follows: The optimal path control strategy is obtained by solving the problem using gradient descent, iterative optimization, or Nash equilibrium. The minimum value of and its corresponding parameters constitute the optimal control path; among them, These are weighting coefficients, which are adjusted according to the actual task. For path error terms, ; in , , Weighting coefficients are used to adjust the impact of path error, turning radius error, and centripetal acceleration error on the control strategy. For distance error, , The set distance between the drone and the athlete is a preset value; Predicted drone and athlete frames k The actual distance at any given moment; For turning radius error, , The expected turning radius of the athlete is a predicted value. : The actual turning radius of the player at time k in frame k; Centripetal acceleration error ; The expected centripetal acceleration of the athlete is the predicted value. : The actual centripetal acceleration of the mover at time k; For obstacle avoidance, ; in At time k, the two drones The distance between; This is an energy consumption item. ; in As a weight for energy consumption, Related to air resistance and propulsion efficiency, Related to moment of inertia and frictional torque; This refers to the linear propulsion speed of the drone; Drone turning rotation speed.

7. An electronic device comprising a processor and a memory, wherein the memory stores at least one instruction, characterized in that, The processor loads and executes at least one instruction to implement the markerless dynamics analysis method for real-time tracking and filming by a drone as described in any one of claims 1-5.

8. A computer-readable storage medium storing at least one instruction, characterized in that, The at least one instruction is loaded and executed by the processor to implement the markerless dynamics analysis method for real-time tracking and filming by the UAV as described in any one of claims 1-5.

Images