Intelligent hopping test analysis method and system based on computer vision and multi-modal data fusion

Abstract

The application discloses an intelligent jumping test analysis method and system based on computer vision and multi-modal data fusion, relates to the technical field of human motion analysis, and comprises the following steps: individual modeling is performed on a subject before testing, individualized limb segment parameters, visual quality baseline and dynamic characteristic benchmark are obtained, multi-view visual image streams and inertial measurement data are collected in formal testing, after three-dimensional reconstruction and inertial calculation, the jumping phase boundary is determined based on the individual dynamic characteristic benchmark, and three-dimensional joint trajectory is output, kinematic parameters and ground reaction force are calculated, the phase boundary is iteratively corrected through dynamic reverse checking closed loop, and the jumping ability evaluation result is output. Through the synergistic effect of individual modeling driven full-process self-adaptation, phase perception multi-modal fusion and dynamic reverse checking closed loop, the precision and robustness of the jumping test analysis are improved, and the applicability to subjects with different body types and motion abilities is enhanced.

Description

Technical Field

[0001] This invention relates to the field of human motion analysis technology, and in particular to an intelligent jump test analysis method and system based on computer vision and multimodal data fusion. Background Technology

[0002] Jump testing is an important tool in sports medicine, competitive sports, and rehabilitation assessment for evaluating lower limb explosive power and neuromuscular control. Existing methods include force table solutions, which rely on laboratory environments and cannot provide kinematic parameters; inertial measurement unit (IMU) solutions, while portable, lack sufficient long-term accuracy; and vision-based solutions, which can non-contactly acquire whole-body motion information, suffer from limited 3D reconstruction accuracy in high-dynamic scenarios. None of these single-modal solutions can simultaneously meet the requirement of obtaining complete kinematic and dynamic parameters.

[0003] While existing multimodal fusion schemes have made up for the limitations of single-modal fusion to some extent, they still have shortcomings in terms of fusion adaptability and reliability of analysis results in complex dynamic scenarios, making it difficult to meet the actual needs of refined jump test analysis. Summary of the Invention

[0004] The main objective of this invention is to provide an intelligent jump test analysis method and system based on computer vision and multimodal data fusion, aiming to solve the technical problems of insufficient multimodal fusion adaptability, lack of individual difference compensation, and difficulty in ensuring consistency between kinematic and dynamic results in existing jump test analysis methods.

[0005] In a first aspect, the present invention provides an intelligent jump test analysis method based on computer vision and multimodal data fusion, comprising: Before the jump test, static body shape scanning and calibrated jump data were performed on the subjects to obtain individualized limb segment parameters, individual visual quality baseline and individual dynamic characteristic benchmark. Collect multi-view visual image streams and inertial measurement data during the subject's actual jumping process; The three-dimensional coordinate sequence of each joint is obtained by performing three-dimensional reconstruction on the multi-view vision image stream, and the inertial attitude sequence is obtained by solving the inertial measurement data; Using individual dynamic characteristics as a reference, the three-dimensional coordinate sequence and inertial attitude sequence are fused to define the jump phase boundary and output the phase label; Based on individual visual quality baselines and phase labels, the fusion weights of the 3D coordinate sequence and the inertial attitude sequence are dynamically adjusted to output the 3D joint trajectory. Kinematic parameters are calculated based on three-dimensional joint trajectories, and ground reaction force is calculated based on individualized limb segment parameters; The characteristic moments of the ground reaction force are compared with the phase boundary for consistency, and the phase boundary is iteratively corrected until convergence is achieved when there is a deviation, and the jump capability assessment result is output.

[0006] Optionally, static body shape scanning and calibration jump acquisition include: Multi-view images of subjects standing still are acquired using multi-view cameras. A parametric body model of the subjects is reconstructed based on the multi-view images to obtain individualized limb segment parameters. Based on multi-view images, a statistical distribution model is performed on the detection confidence of key points of each joint to obtain a static visual quality baseline. Collect multi-view visual calibration jump image streams and joint acceleration data during the submaximal effort calibration jump process of the subjects; Based on the dynamic confidence distribution of multi-view visual calibration skip image stream statistics, combined with static visual quality baseline correction, the individual visual quality baseline is obtained. The peak vertical acceleration and impact characteristic values ​​of the landing were extracted from the joint acceleration data to obtain the individual dynamic characteristic benchmark.

[0007] Optionally, the 3D reconstruction includes: Human keypoint detection is performed frame by frame on the multi-view image stream to obtain the two-dimensional coordinates of keypoints from multiple perspectives and their corresponding confidence scores in each frame. Based on the extrinsic matrix of multi-view cameras, the two-dimensional coordinates of key points from multiple perspectives are triangulated and reconstructed. The triangulation contribution of each perspective is weighted by the confidence level to obtain the three-dimensional coordinate sequence of each joint. By using individualized limb segment parameters as constraints, the three-dimensional coordinate sequence is optimized with invariant bone length constraints to obtain a smoothed three-dimensional coordinate sequence. The inertial measurement data is denoised and the gravity component is separated. The joint acceleration sequence is obtained based on the processed acceleration data, and the attitude sequence is obtained by quaternion integration based on the processed angular velocity data.

[0008] Optionally, defining the jump phase boundary and outputting the phase label includes: Visual modal features are extracted based on 3D coordinate sequences, and inertial modal features are extracted based on inertial attitude sequences to obtain dual-modal phase features; Using individual dynamic characteristics as a reference, an individualized phase criterion threshold is set for the dual-modal phase characteristics; Based on individualized phase criterion thresholds and dual-modal phase characteristics, the take-off start time, take-off time, peak air time, and landing time are determined sequentially to obtain the jump phase boundary. Based on the jump phase boundary, the jump action is divided into the approach preparation phase, the take-off and extension phase, the flight phase, and the landing and cushioning phase, and the phase labels are output.

[0009] Optionally, the output process of the three-dimensional joint trajectory includes: The visual confidence level of the current frame is evaluated based on the individual visual quality baseline, and the visual confidence level bias is obtained. The visual modal weights and inertial modal weights of the current phase are determined based on the deviation between the phase label and the visual confidence level. The visual observation noise covariance is adjusted using visual modality weights, and the inertial prediction noise covariance is adjusted using inertial modality weights. Kalman filter framework is adaptively selected based on phase labels; Based on the filtering framework, the sampling rate of the inertial measurement unit is used as the time reference. Fusion updates are performed at the corresponding time of the visual frame, and predictions are performed at other time points to output the three-dimensional joint trajectory.

[0010] Optionally, the calculation process for kinematic parameters and ground reactions includes: Based on the three-dimensional joint trajectory and jump phase boundary, the trajectory of the center of mass, the height of the airborne, the time series of the angles of each joint, and the limb symmetry index are calculated to obtain kinematic parameters; The second-order numerical differential of the joint coordinates is performed based on the three-dimensional joint trajectory to obtain the acceleration sequence of each joint. A rigid body model of the human body is established using individualized limb segment parameters, and the torque of each joint and the ground reaction force are calculated based on the acceleration sequence of each joint.

[0011] Optionally, consistency comparison includes: The zero-crossing point and the peak impact point of the ground reaction force are extracted and used as the lift-off characteristic moment and the landing characteristic moment, respectively. The takeoff and landing characteristic moments are compared with the corresponding moments in the jump phase boundary, and the deviation is calculated. When the deviation exceeds the preset convergence threshold, the jump phase boundary is corrected and the phase label is updated using the takeoff feature time and landing feature time. The phase-aware adaptive multimodal fusion and individualized parameter calculation are re-executed using the corrected phase label. The above comparison and correction are performed iteratively until the deviation converges to within the preset convergence threshold, and the jump capability evaluation result is output.

[0012] Secondly, this invention provides an intelligent jump test analysis system based on computer vision and multimodal data fusion, comprising: The individual modeling module is used to perform static body shape scanning and calibration jump acquisition on subjects before the jump test to obtain individualized limb segment parameters, individual visual quality baseline and individual dynamic characteristic benchmark; The acquisition module is used to acquire multi-view visual image streams and inertial measurement data during the subject's actual jumping process; The calculation module is used to perform 3D reconstruction on the multi-view vision image stream to obtain the 3D coordinate sequence of each joint, and to calculate the inertial attitude sequence from the inertial measurement data. The phase segmentation module is used to define the jump phase boundary and output the phase label by fusing the three-dimensional coordinate sequence and the inertial attitude sequence with reference to the individual dynamic characteristics benchmark. The fusion module is used to dynamically adjust the fusion weights of the 3D coordinate sequence and the inertial attitude sequence based on the individual visual quality baseline and phase label, and output the 3D joint trajectory. The parameter calculation module is used to calculate kinematic parameters based on three-dimensional joint trajectories and to calculate ground reaction forces based on individualized limb segment parameters. The verification module is used to compare the characteristic moments of the ground reaction force with the phase boundary for consistency, and to iteratively correct the phase boundary until convergence when there is a deviation, and output the jump capability evaluation result.

[0013] Thirdly, the present invention provides an electronic device, comprising: Memory, used to store computer programs; The processor is used to execute computer programs to implement the aforementioned intelligent jump test analysis method based on computer vision and multimodal data fusion.

[0014] Fourthly, the present invention provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned intelligent jump test analysis method based on computer vision and multimodal data fusion.

[0015] The beneficial effects of this invention are as follows: Individual modeling with three-way parameter output is integrated throughout the entire process of kinematic reconstruction, phase segmentation, and dynamics calculation, achieving full-process adaptive compensation for individual differences; the phase-aware fusion mechanism fully leverages the complementary advantages of visual and inertial dual-modality approaches at different motion stages, improving the accuracy and temporal resolution of the fused trajectory; the dynamics reverse verification closed loop incorporates kinematic and dynamics estimations into a unified consistency constraint framework, ensuring global self-consistency of the analysis results. The synergy of these three elements significantly improves the accuracy, robustness, and applicability of jump test analysis. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the intelligent jump test analysis method based on computer vision and multimodal data fusion provided in an embodiment of the present invention.

[0018] Figure 2The flowchart illustrates the jump phase boundary delineation process of the intelligent jump test analysis method based on computer vision and multimodal data fusion, as provided in this embodiment of the invention.

[0019] Figure 3 A flowchart illustrating the three-dimensional joint trajectory generation process of the intelligent jump test analysis method based on computer vision and multimodal data fusion provided in this embodiment of the invention.

[0020] Figure 4 This is a structural block diagram of an intelligent jump test and analysis system based on computer vision and multimodal data fusion provided in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Reference Figure 1 This invention discloses an intelligent jump test analysis method based on computer vision and multimodal data fusion, which may include: S1: Before the jump test, static body shape scanning and calibrated jump data were performed on the subjects to obtain individualized limb segment parameters, individual visual quality baseline and individual dynamic characteristic benchmark.

[0023] Specifically, step S1 includes: S1.1: Multi-view images of the subject in a static standing state are acquired using a multi-view camera. Based on the multi-view images, a parametric body shape model of the subject is reconstructed to obtain individualized limb segment parameters.

[0024] Specifically, human key point detection is performed on multi-view images, and the key points are triangulated and reconstructed based on the extrinsic matrix of the multi-view camera to obtain the three-dimensional coordinates of the key points of the whole body. A parameterized body shape model is fitted based on the three-dimensional coordinates of the key points of the whole body, and the geometric shape and volume distribution of each limb segment are extracted based on the parameterized body shape model. Based on the geometric shape and volume distribution of each limb segment, the mass and moment of inertia of each limb segment are calculated in combination with human tissue density parameters to obtain individualized limb segment parameters.

[0025] S1.2: Based on multi-view images, statistical distribution modeling is performed on the detection confidence of key points of each joint to obtain a static visual quality baseline.

[0026] Furthermore, human keypoint detection is performed frame by frame on the multi-view images to obtain the detection confidence of keypoints of each joint from each viewpoint; statistical distribution modeling is performed on the detection confidence of keypoints of each joint from each viewpoint to obtain the mean and variance of the confidence of each joint under each viewpoint; based on the mean and variance of the confidence, the static visual quality baseline is obtained.

[0027] S1.3: Collect multi-view visual calibration jump image stream and joint acceleration data during the subject's submaximal effort calibration jump process.

[0028] It should be noted that the joint acceleration data was collected by inertial measurement units worn on both hip, knee and ankle joints of the subject, and included triaxial acceleration signals.

[0029] S1.4: Based on the statistical dynamic confidence distribution of multi-view visual calibration skip image stream, combined with static visual quality baseline correction, the individual visual quality baseline is obtained.

[0030] In one embodiment, human keypoint detection is performed frame by frame on the multi-view calibration jump image stream to obtain the dynamic confidence of keypoints of each joint from each viewpoint during the calibration jump; the confidence distribution of each joint throughout the calibration jump is statistically analyzed based on the dynamic confidence to obtain the dynamic visual quality distribution; the dynamic visual quality distribution is compared with the static visual quality baseline to calculate the attenuation coefficient of the confidence of each joint relative to the static baseline in the dynamic scene; the static visual quality baseline is corrected based on the attenuation coefficient to obtain the individual visual quality baseline.

[0031] S1.5: Extract the peak vertical acceleration and impact characteristic value from the joint acceleration data to obtain the individual dynamic characteristic benchmark.

[0032] Among them, the peak vertical acceleration corresponds to the maximum value of the vertical acceleration component during the take-off and extension phase, and the landing impact characteristic value corresponds to the impact peak value of the vertical acceleration component during the landing cushioning phase.

[0033] Through a unified calibration phase before testing, three individualized parameters are established simultaneously: individualized limb segment parameters, individual visual quality baseline, and individual dynamic characteristic benchmark. These parameters drive subsequent inverse dynamics calculations, adaptive adjustment of fusion weights, and individualized setting of phase criteria, respectively. This enables adaptive compensation of individual differences throughout the entire process of kinematic and dynamic estimation, improving applicability to subjects with different body types and motor abilities.

[0034] S2: Simultaneously acquire multi-view visual image streams and inertial measurement data of the subject during the actual jumping process.

[0035] In one specific implementation, multiple cameras are deployed around the test area, with no fewer than two cameras, and the frame rate is no less than 60fps; inertial measurement units are worn on the subject's bilateral hip joints, knee joints and ankle joints, with a sampling frequency of no less than 100Hz, and the three-axis acceleration data and three-axis angular velocity data of each wearing node are collected simultaneously, and the inertial measurement data are output.

[0036] Time synchronization between multiple cameras and the inertial measurement unit (IMU) is achieved through one of two methods: First, based on a hardware trigger signal, the acquisition clocks of each camera and IMU are synchronously started by the same trigger source; second, based on a network time protocol, the timestamps of each device are software-aligned to control the timestamp deviation within a preset synchronization threshold. The preset synchronization threshold is no greater than half the frame interval of the camera acquisition to ensure that the time alignment error between visual and inertial data does not exceed the order of magnitude of a single frame reconstruction error.

[0037] After synchronous acquisition is completed, time-aligned multi-view visual image stream and inertial measurement data are output.

[0038] S3: Perform 3D reconstruction on the multi-view vision image stream to obtain the 3D coordinate sequence of each joint, and calculate the inertial attitude sequence from the inertial measurement data.

[0039] Specifically, step S3 includes: S3.1: Perform human keypoint detection frame by frame on the multi-view image stream to obtain the two-dimensional coordinates of keypoints from multiple perspectives and their corresponding confidence scores in each frame.

[0040] S3.2: Based on the extrinsic matrix of the multi-view camera, the two-dimensional coordinates of the key points from multiple perspectives are triangulated and reconstructed. The triangulation contribution of each perspective is weighted by the confidence level to obtain the three-dimensional coordinate sequence of each joint.

[0041] Specifically, a triangulated equation set is constructed based on the extrinsic parameter matrix of the multi-view key points in two dimensions. The triangulated equation set of each view is weighted by confidence level, and the weights of the equation set corresponding to the view with confidence level lower than the preset confidence level threshold are reset to zero. The triangulated equation set is solved by weighted least squares method to obtain the three-dimensional coordinates of each joint, thus forming a three-dimensional coordinate sequence of each joint.

[0042] The preset confidence threshold is determined based on the mean and variance of the confidence distribution of each joint in the static visual quality baseline, and the mean minus twice the standard deviation is used as the confidence validity criterion for the corresponding viewpoint of each joint.

[0043] S3.3: The three-dimensional coordinate sequence is optimized by using individualized limb segment parameters as constraints to obtain a smoothed three-dimensional coordinate sequence with invariant bone length.

[0044] Furthermore, based on individualized limb segment parameters, the length of each limb segment is extracted to construct a skeleton length constraint set; using the skeleton length constraint set as a hard constraint and minimizing the change in joint coordinates between adjacent frames as a soft constraint, the three-dimensional coordinate sequence is jointly optimized to obtain a smoothed three-dimensional coordinate sequence.

[0045] S3.4: Denoise and separate the gravity component from the inertial measurement data, obtain the joint acceleration sequence based on the processed acceleration data, and obtain the attitude sequence by quaternion integration based on the processed angular velocity data.

[0046] Furthermore, the triaxial acceleration and triaxial angular velocity signals in the inertial measurement data are denoised to obtain denoised acceleration and angular velocity data. Based on the calibration extrinsic parameters of the multi-view camera and the inertial measurement unit, the denoised acceleration data is transformed from the inertial measurement unit coordinate system to the world coordinate system, and the gravity component is separated to obtain the joint acceleration sequence. Starting from the initial attitude obtained in the static body shape scanning stage, quaternion integration is performed based on the denoised angular velocity data to obtain the attitude sequence.

[0047] It should be noted that the joint acceleration sequence and the attitude sequence together constitute the inertial attitude sequence, which serves as the inertial mode input for the subsequent multimodal fusion step.

[0048] Based on the above scheme, confidence-weighted triangulation reconstruction automatically reduces the weight of low-quality viewpoints in the fusion calculation, effectively suppressing the impact of occlusion and motion blur on the accuracy of 3D reconstruction; bone length invariant constraint optimization eliminates inter-frame joint coordinate jitter while maintaining the prior knowledge of individual anatomical structures, improving the temporal consistency of the 3D coordinate sequence; IMU denoising and gravity separation ensure the reliability of joint acceleration and pose sequences, providing high-quality inertial modal input for subsequent multimodal fusion. The three work together to ensure the quality of the original visual and inertial dual-modal sequences, laying the foundation for subsequent phase delineation and adaptive fusion.

[0049] S4: Using individual dynamic characteristics as a reference, the three-dimensional coordinate sequence and inertial attitude sequence are fused to define the jump phase boundary and output the phase label.

[0050] Specifically, the flowchart for defining the jump phase boundary is as follows: Figure 2 As shown, it includes: S4.1: Visual modal features are extracted based on the three-dimensional coordinate sequence, and inertial modal features are extracted based on the inertial attitude sequence to obtain dual-modal phase features.

[0051] Among them, the visual modal features include the vertical displacement features of the center of mass, the rate of change of the vertical coordinates of the ankle joint, and the rate of change of the angle of the knee joint, while the inertial modal features include the vertical acceleration features.

[0052] Specifically, the position of the center of mass of each limb segment is estimated based on the midpoint of adjacent joints in the three-dimensional coordinate sequence. The weighted summation is performed with the mass proportion of each limb segment as the weight to obtain the three-dimensional coordinate sequence of the center of mass. The vertical component of the three-dimensional coordinate sequence of the center of mass is extracted to obtain the vertical displacement feature of the center of mass. The vertical coordinate component of the ankle joint is extracted from the three-dimensional coordinate sequence, and the vertical coordinate component of the ankle joint is subjected to time difference to obtain the vertical coordinate change rate of the ankle joint. The knee joint angle time series is calculated based on the spatial vector angle between the thigh segment and the lower leg segment in the three-dimensional coordinate sequence. The knee joint angle time series is subjected to time difference to obtain the knee joint angle change rate. The vertical component of the joint acceleration sequence in the inertial attitude sequence is extracted to obtain the vertical acceleration feature.

[0053] S4.2: Using the individual dynamic characteristics benchmark as a reference, set an individualized phase criterion threshold for the dual-modal phase characteristics.

[0054] Furthermore, based on the peak vertical acceleration in the individual dynamic characteristic benchmark, a preset proportional coefficient is used to determine the vertical acceleration criterion threshold at the start of the take-off extension; based on the landing impact characteristic value in the individual dynamic characteristic benchmark, a preset proportional coefficient is used to determine the vertical acceleration criterion threshold at the landing moment; based on the statistical distribution of the ankle joint vertical coordinate change rate and knee joint angle change rate extracted from the calibration jump acquisition data, corresponding auxiliary criterion thresholds are determined respectively.

[0055] The preset proportional coefficient is determined based on the statistical law of acceleration characteristics of jumping action in sports biomechanics research. The vertical acceleration criterion threshold at the start of take-off extension can be 0.2 to 0.4 times the peak value of individual vertical acceleration, and the vertical acceleration criterion threshold at the moment of landing can be 0.4 to 0.6 times the impact characteristic value of individual landing. The specific values ​​can be adjusted according to the actual test scenario.

[0056] S4.3: Based on the individualized phase criterion threshold and dual-modal phase characteristics, the take-off extension start time, take-off time, peak air time and landing time are determined sequentially to obtain the jump phase boundary.

[0057] In one specific embodiment, the dual-modal phase features are detected time-by-time based on individualized phase criterion thresholds. The vertical displacement feature of the center of mass and the vertical acceleration feature are the main criteria, and the vertical coordinate change rate of the ankle joint and the angle change rate of the knee joint are the auxiliary criteria. The take-off extension start time, the time of departure from the ground, the peak time of the flight, and the landing time are determined in sequence. The take-off extension start time, the time of departure from the ground, the peak time of the flight, and the landing time constitute the jump phase boundary.

[0058] S4.4: Based on the jump phase boundary, divide the jump action into the approach preparation phase, take-off and extension phase, flight phase, and landing and cushioning phase, and output phase labels.

[0059] Based on the above scheme, the dual-modal phase feature fusion integrates visual and inertial signals. The visual modality provides joint spatial position information, while the inertial modality provides high-frequency dynamic response information, complementarily covering the blind spots of the single modality. The individualized phase criterion threshold is dynamically set with reference to the individual dynamic feature benchmark, avoiding misjudgment problems between different subjects with a fixed threshold. The main and auxiliary criterion joint detection mechanism can still maintain the accuracy of phase boundary detection when the quality of the single modality signal deteriorates, improving the robustness of phase segmentation in complex motion scenarios. The output phase label provides a temporal basis for subsequent adaptive fusion weight adjustment and inverse dynamics calculation.

[0060] S5: Based on individual visual quality baseline and phase label, dynamically adjust the fusion weight of 3D coordinate sequence and inertial attitude sequence to output 3D joint trajectory.

[0061] Specifically, the flowchart for generating 3D joint trajectories is as follows: Figure 3 As shown, it includes: S5.1: Evaluate the visual confidence of the current frame based on the individual visual quality baseline to obtain the visual confidence deviation.

[0062] It should be noted that the visual confidence of the current frame comes from the joint keypoint detection confidence obtained by performing human keypoint detection frame by frame on the multi-view visual image stream.

[0063] S5.2: Determine the visual modal weight and inertial modal weight of the current phase based on the deviation between the phase label and the visual confidence level.

[0064] In one embodiment, the visual modal reference weight and inertial modal reference weight corresponding to the current phase are queried from a preset phase weight table based on the phase label; the visual modal reference weight is corrected based on the visual confidence deviation, and when the visual confidence deviation is negative, the visual modal weight is reduced and the inertial modal weight is increased accordingly, and when the visual confidence deviation is positive, the reference weight remains unchanged; the corrected visual modal weight and inertial modal weight are normalized to obtain the visual modal weight and inertial modal weight of the current phase.

[0065] The preset phase weight table is determined based on the signal quality characteristics of visual and inertial modes within each phase: visual reconstruction is reliable in the takeoff phase, and the visual mode reference weight is the highest; the dynamics are strong in the takeoff and landing buffer phases, and the inertial mode reference weight is relatively high; the movement is stable in the approach run preparation phase, and the visual and inertial mode reference weights are evenly distributed.

[0066] S5.3: Adjust the visual observation noise covariance with visual modality weights and adjust the inertial prediction noise covariance with inertial modality weights.

[0067] Specifically, a higher visual modality weight results in a smaller visual observation noise covariance, while a higher inertial modality weight results in a smaller inertial prediction noise covariance. In the Kalman filter framework, a smaller observation noise covariance indicates a higher level of confidence in the observations of that modality, and the fusion result converges more towards that modality. This achieves an adaptive fusion effect where high-quality visual phase converges towards visual observations, and high-quality inertial signal phase converges towards inertial predictions.

[0068] S5.4: Adaptively select the Kalman filter framework based on the phase label.

[0069] Specifically, the current phase is determined based on the phase label. An extended Kalman filter framework is selected in the non-landing buffer phase, and an unscented Kalman filter framework is selected in the landing buffer phase, which will be used as the filtering framework for subsequent fusion calculations.

[0070] S5.5: Based on the filtering framework, the sampling rate of the inertial measurement unit is used as the time reference. The fusion update is performed at the corresponding time of the visual frame, and the prediction is performed at other time points to output the three-dimensional joint trajectory.

[0071] Furthermore, an output time axis is established based on the sampling rate of the inertial measurement unit. At each sampling moment, state prediction is performed based on the inertial attitude sequence and the filter framework determined in S5.4. When the current sampling moment corresponds to the visual frame moment, the joint coordinates of the corresponding frame in the three-dimensional coordinate sequence are used as the observation. A fusion update is performed based on the adjusted visual observation noise covariance to correct the predicted state and the prediction covariance. The state estimates at each moment are continuously output according to the sampling rate of the inertial measurement unit to form a three-dimensional joint trajectory.

[0072] Using the above scheme, the confidence bias assessment based on the individual visual quality baseline enables individualized dynamic adjustment of the fusion weights. The adaptive switching between phase-driven extended Kalman filtering and unscented Kalman filtering effectively addresses the strong nonlinearity of the landing buffer phase. The two work together to improve the estimation accuracy and continuity of the three-dimensional joint trajectory in the entire phase of the jump.

[0073] S6: Calculates kinematic parameters based on three-dimensional joint trajectories and calculates ground reaction force based on individualized limb segment parameters.

[0074] Specifically, step S6 includes: S6.1: Based on the mass proportion of each limb segment in the three-dimensional joint trajectory, the position of the center of mass of each limb segment is weighted and summed to obtain the three-dimensional coordinate sequence of the center of mass, i.e., the center of mass trajectory.

[0075] In one embodiment, based on the mass proportion of each limb segment in the three-dimensional joint trajectory, the weighted summation of the centroid positions of each limb segment is performed to obtain the three-dimensional coordinate sequence of the centroid, i.e., the centroid trajectory; based on the takeoff time and the peak takeoff time in the jump phase boundary, the difference between the vertical coordinates of the centroid at the peak takeoff time and the takeoff time is calculated to obtain the takeoff height; based on the angle between the spatial vectors of adjacent limb segments in the three-dimensional joint trajectory, the time series of the angles of each joint of the hip, knee, and ankle are calculated; the time series of the angles of the corresponding joints on the left and right sides are extracted respectively, and the amplitude difference and correlation coefficient of the time series of the angles of the left and right sides are calculated to obtain the limb symmetry index.

[0076] Among them, the amplitude difference reflects the degree of asymmetry in the amplitude of the left and right sides, and the correlation coefficient reflects the synchronicity of the movement sequence of the left and right sides. The weighted sum of the two results in the limb symmetry index, and the weight can be adjusted according to the actual assessment needs.

[0077] S6.2: Based on the three-dimensional joint trajectory, perform second-order numerical differentiation on the coordinates of each joint to obtain the acceleration sequence of each joint.

[0078] S6.3: Establish a rigid body model of the human body using individualized limb segment parameters, and calculate the torque of each joint and the ground reaction force based on the acceleration sequence of each joint.

[0079] Specifically, each limb segment is modeled using the rigid body assumption, and a series of kinematic chains from the feet to the torso are constructed with joints as connection points. The rigid body segment model of the human body is completed by combining the length, mass, and moment of inertia of each limb segment in the individualized limb segment parameters. On this basis, based on the acceleration sequence of each joint and the rigid body segment model of the human body, the torque of each joint is calculated from bottom to top using the inverse Newton-Euler dynamics equations, and the ground reaction force is calculated using the acceleration of the whole body center of mass and the total mass.

[0080] By constructing a rigid body segment model of the human body based on individualized limb segment parameters, the inverse dynamics calculation can fully reflect the body shape characteristics of the subject, improve the accuracy of ground reaction force estimation, and provide a reliable constraint basis for dynamic inverse verification.

[0081] S7: Compare the characteristic moments of the ground reaction force with the phase boundary for consistency, and iteratively correct the phase boundary until convergence when there is a deviation, and output the jump capability assessment result.

[0082] Specifically, the zero-crossing moment and the peak impact moment of the ground reaction force are extracted and used as the takeoff feature moment and landing feature moment, respectively. The takeoff feature moment and landing feature moment are compared with the corresponding moments in the jump phase boundary to calculate the deviation. When the deviation exceeds the preset convergence threshold, the jump phase boundary is corrected and the phase label is updated using the takeoff feature moment and landing feature moment. The phase-aware adaptive multimodal fusion and individualized parameter calculation are re-executed with the corrected phase label. The above comparison and correction are iteratively executed until the deviation converges to within the preset convergence threshold, and the jump capability assessment result is output.

[0083] It should be noted that the jump phase boundary correction method is as follows: the takeoff time in the jump phase boundary is replaced with the takeoff characteristic time, and the landing time in the jump phase boundary is replaced with the landing characteristic time. The takeoff extension start time and the peak flight time remain unchanged. Based on the corrected jump phase boundary, the time periods of each phase are redefined and the phase labels are updated. The preset convergence threshold is set to no more than half of the time difference between adjacent visual frames to ensure that the phase boundary correction accuracy matches the visual temporal resolution.

[0084] The dynamic back-end verification closed loop eliminates the systematic error of inconsistency between kinematic and dynamic results in a single forward estimation by constraining the iterative consistency between the two; the iterative convergence mechanism ensures the global self-consistency of the final output results at the kinematic and dynamic levels, significantly improving the reliability of jump test analysis results.

[0085] Accordingly, refer to Figure 4 The present invention also provides an intelligent jump test analysis system based on computer vision and multimodal data fusion, which may include: The individual modeling module is used to perform static body shape scanning and calibration jump acquisition on subjects before the jump test to obtain individualized limb segment parameters, individual visual quality baseline and individual dynamic characteristic benchmark; The acquisition module is used to acquire multi-view visual image streams and inertial measurement data during the subject's actual jumping process; The calculation module is used to perform 3D reconstruction on the multi-view vision image stream to obtain the 3D coordinate sequence of each joint, and to calculate the inertial attitude sequence from the inertial measurement data. The phase segmentation module is used to define the jump phase boundary and output the phase label by fusing the three-dimensional coordinate sequence and the inertial attitude sequence with reference to the individual dynamic characteristics benchmark. The fusion module is used to dynamically adjust the fusion weights of the 3D coordinate sequence and the inertial attitude sequence based on the individual visual quality baseline and phase label, and output the 3D joint trajectory. The parameter calculation module is used to calculate kinematic parameters based on three-dimensional joint trajectories and to calculate ground reaction forces based on individualized limb segment parameters. The verification module is used to compare the characteristic moments of the ground reaction force with the phase boundary for consistency, and to iteratively correct the phase boundary until convergence when there is a deviation, and output the jump capability evaluation result.

[0086] This invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program; when the computer program is executed by the processor, the processor performs the steps of the intelligent jump test analysis method based on computer vision and multimodal data fusion as described in any of the above embodiments.

[0087] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the intelligent jump test analysis method based on computer vision and multimodal data fusion as described in any of the above embodiments.

[0088] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0089] For a description of the computer-readable storage medium provided by the present invention, please refer to the above method embodiments; the present invention will not be described in detail here.

[0090] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0091] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0092] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A smart jump test analysis method based on computer vision and multimodal data fusion, characterized in that, include: Before the jump test, static body shape scanning and calibrated jump data were performed on the subjects to obtain individualized limb segment parameters, individual visual quality baseline and individual dynamic characteristic benchmark. Collect multi-view visual image streams and inertial measurement data during the subject's actual jumping process; The multi-view vision image stream is reconstructed in three dimensions to obtain the three-dimensional coordinate sequence of each joint, and the inertial measurement data is solved to obtain the inertial attitude sequence; Using the individual dynamic characteristic benchmark as a reference, the three-dimensional coordinate sequence and the inertial attitude sequence are fused to define the jump phase boundary and output the phase label; Based on the individual visual quality baseline and the phase label, the fusion weights of the three-dimensional coordinate sequence and the inertial attitude sequence are dynamically adjusted to output the three-dimensional joint trajectory; Kinematic parameters are calculated based on the three-dimensional joint trajectory, and ground reaction force is calculated based on the individualized limb segment parameters. The characteristic moments of the ground reaction force are compared with the phase boundary for consistency, and the phase boundary is iteratively corrected until convergence is achieved when there is a deviation, and the jump capability evaluation result is output.

2. The intelligent jump test analysis method based on computer vision and multimodal data fusion according to claim 1, characterized in that, The static body shape scanning and calibration jump acquisition includes: Multi-view images of subjects standing still are acquired using multi-view cameras. A parametric body model of the subjects is reconstructed based on the multi-view images to obtain individualized limb segment parameters. Based on the multi-view images, a statistical distribution model is performed on the detection confidence of each joint key point to obtain a static visual quality baseline. Collect multi-view visual calibration jump image streams and joint acceleration data during the submaximal effort calibration jump process of the subjects; Based on the statistical dynamic confidence distribution of the multi-view visual calibration skip image stream, and combined with the static visual quality baseline correction, the individual visual quality baseline is obtained. The peak vertical acceleration and the impact characteristic value of the landing are extracted from the joint acceleration data to obtain the individual dynamic characteristic benchmark.

3. The intelligent jump test analysis method based on computer vision and multimodal data fusion according to claim 1, characterized in that, The three-dimensional reconstruction includes: Human keypoint detection is performed frame by frame on the multi-view image stream to obtain the two-dimensional coordinates of keypoints from multiple perspectives and their corresponding confidence scores in each frame. Based on the extrinsic matrix of the multi-view camera, the two-dimensional coordinates of the key points of the multi-view are triangulated and reconstructed. The triangulation contribution of each view is weighted by the confidence level to obtain the three-dimensional coordinate sequence of each joint. The three-dimensional coordinate sequence is optimized by bone length invariance constraint using individualized limb segment parameters to obtain a smoothed three-dimensional coordinate sequence. The inertial measurement data is denoised and the gravity component is separated. The joint acceleration sequence is obtained based on the processed acceleration data, and the attitude sequence is obtained by quaternion integration based on the processed angular velocity data.

4. The intelligent jump test analysis method based on computer vision and multimodal data fusion according to claim 1, characterized in that, The process of defining the jump phase boundary and outputting the phase label includes: Visual modal features are extracted based on 3D coordinate sequences, and inertial modal features are extracted based on inertial attitude sequences to obtain dual-modal phase features; Using individual dynamic characteristics as a reference, an individualized phase criterion threshold is set for the dual-modal phase characteristics; Based on the individualized phase criterion threshold and the dual-modal phase characteristics, the take-off extension start time, take-off time, peak air time and landing time are determined sequentially to obtain the jump phase boundary; Based on the jump phase boundary, the jump action is divided into the approach preparation phase, the take-off and extension phase, the flight phase, and the landing and cushioning phase, and a phase label is output.

5. The intelligent jump test analysis method based on computer vision and multimodal data fusion according to claim 1, characterized in that, The output process of the three-dimensional joint trajectory includes: The visual confidence level of the current frame is evaluated based on the individual visual quality baseline, and the visual confidence level bias is obtained. Based on the deviation between the phase label and the visual confidence level, the visual modal weight and inertial modal weight of the current phase are determined. The visual observation noise covariance is adjusted using the visual modality weights, and the inertial prediction noise covariance is adjusted using the inertial modality weights. Kalman filter framework is adaptively selected based on phase labels; Based on the filtering framework, the sampling rate of the inertial measurement unit is used as the time reference. Fusion updates are performed at the corresponding time of the visual frame, and predictions are performed at other time points to output the three-dimensional joint trajectory.

6. The intelligent jump test analysis method based on computer vision and multimodal data fusion according to claim 1, characterized in that, The calculation process for the kinematic parameters and ground reaction forces includes: Based on the three-dimensional joint trajectory and jump phase boundary, the trajectory of the center of mass, the height of the airborne, the time series of the angles of each joint, and the limb symmetry index are calculated to obtain kinematic parameters; The second-order numerical differential of the joint coordinates is performed based on the three-dimensional joint trajectory to obtain the acceleration sequence of each joint. A rigid body model of the human body is established using individualized limb segment parameters, and the torque of each joint and the ground reaction force are calculated based on the acceleration sequence of each joint.

7. The intelligent jump test analysis method based on computer vision and multimodal data fusion according to claim 1, characterized in that, The consistency comparison includes: The zero-crossing point and the peak impact point of the ground reaction force are extracted and used as the lift-off characteristic moment and the landing characteristic moment, respectively. The takeoff and landing characteristic moments are compared with the corresponding moments in the jump phase boundary, and the deviation is calculated. When the deviation exceeds the preset convergence threshold, the jump phase boundary is corrected and the phase label is updated using the takeoff feature time and landing feature time. The phase-aware adaptive multimodal fusion and individualized parameter calculation are re-executed using the corrected phase label. The above comparison and correction are performed iteratively until the deviation converges to within the preset convergence threshold, and the jump capability evaluation result is output.

8. An intelligent jump test and analysis system based on computer vision and multimodal data fusion, characterized in that, include: The individual modeling module is used to perform static body shape scanning and calibration jump acquisition on subjects before the jump test to obtain individualized limb segment parameters, individual visual quality baseline and individual dynamic characteristic benchmark; The acquisition module is used to acquire multi-view visual image streams and inertial measurement data during the subject's actual jumping process; The calculation module is used to perform three-dimensional reconstruction on the multi-view vision image stream to obtain the three-dimensional coordinate sequence of each joint, and to calculate the inertial attitude sequence from the inertial measurement data; The phase segmentation module is used to define the jump phase boundary and output the phase label by fusing the three-dimensional coordinate sequence and the inertial attitude sequence with the individual dynamic characteristic reference as a reference. The fusion module is used to dynamically adjust the fusion weights of the three-dimensional coordinate sequence and the inertial attitude sequence based on the individual visual quality baseline and the phase label, and output the three-dimensional joint trajectory. The parameter calculation module is used to calculate kinematic parameters based on the three-dimensional joint trajectory and to calculate ground reaction force based on the individualized limb segment parameters. The verification module is used to compare the characteristic moments of the ground reaction force with the phase boundary for consistency, and to iteratively correct the phase boundary until convergence when there is a deviation, and output the jump capability evaluation result.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the intelligent jump test analysis method based on computer vision and multimodal data fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the intelligent jump test analysis method based on computer vision and multimodal data fusion as described in any one of claims 1 to 7.

Images