Joint point detection dynamic model algorithm for racing boat

Abstract

The invention relates to the technical field of motion detection, and discloses a joint point detection dynamic model algorithm oriented to boat racing motion. According to the algorithm, firstly, a racing boat motion video is collected to generate a multi-view motion frame, and joint point candidate areas of a key motion stage are determined through processing of a spatial-temporal feature fusion algorithm; performing feature enhancement on the candidate region to generate a high-dimensional motion feature map, and marking a potential joint point dynamic trajectory in combination with an attention mechanism and dynamic time warping; detecting a region of interest based on the generated joint points; analyzing the gradient change of the motion vector of the region, and evaluating whether the spatial displacement fluctuation of the articulation point accords with the motion law or not; analyzing the relevance between skeleton and muscle movement in the region, and evaluating whether the collaborative coupling strength is in a normal range; and judging whether model parameter calibration processing needs to be carried out in real time or not according to the evaluation result. The algorithm can accurately capture the dynamic change of the joint, identify the abnormal motion, and improve the adaptability and accuracy of detection.

Description

Technical Field

[0001] This invention relates to the field of motion detection technology, specifically to a dynamic model algorithm for joint detection in rowing. Background Technology

[0002] Rowing, a sport demanding extremely high levels of physical coordination and precision, relies heavily on standardized techniques to directly impact athletic performance. Accurate detection of the movement status of athletes' joints is crucial for understanding movement patterns and optimizing techniques during rowing training and competition analysis. Traditional joint detection methods often rely on single-view image acquisition and static feature extraction to determine joint positions; however, this approach struggles to capture the dynamic changes in the coordinated movements of multiple joints in rowing.

[0003] In rowing, the athletes' limb movements exhibit clear periodicity and continuity. From the initial stroke, pull, exit, and return of the oar, the motion trajectories of key joints such as the shoulder, elbow, hip, and knee are interconnected, forming a complex dynamic coupling relationship. Single-view image sequences often have limitations in perspective and are prone to errors in joint detection due to occlusion, motion blur, and other issues. For example, when the boat is rocking or the athlete's limbs are crossing, single-view images may lose motion information of some joints, leading to breaks or misjudgments in subsequent trajectory analysis.

[0004] In existing technologies, some dynamic joint detection models attempt to introduce spatiotemporal feature fusion, but they often face problems of feature redundancy or conflict when processing multi-view data. The lack of an effective collaborative mechanism for integrating information from multi-view motion frames leads to inaccurate determination of candidate joint regions during key motion phases, thus affecting the accuracy of subsequent trajectory tracking. Furthermore, these models often employ fixed feature weight allocation methods when labeling dynamic trajectories of joints, failing to consider the dynamic differences in importance of different joints during motion, making the trajectory labeling of potential joints susceptible to interference from non-critical motion features.

[0005] Traditional models rarely perform real-time motion conformity assessments on the detection results during joint detection. In rowing, the spatial displacement of joints fluctuates within a specific range, and the coordination of joint motion states exhibits inherent patterns. When the joint displacement detected by the model exceeds the normal range of motion, or when abnormal coupling occurs in the coordinated motion between joints, existing models cannot promptly identify such deviations and lack corresponding dynamic calibration mechanisms. Consequently, the reliability of the detection results gradually decreases as the motion progresses.

[0006] In practical applications, environmental factors in rowing also affect joint detection. External interferences such as water surface reflection, lighting changes, and boat vibrations can introduce noise into motion features in image sequences, further increasing the difficulty of joint detection. Traditional model parameter settings are mostly based on static scene adjustments, making it difficult to adapt to feature changes in dynamic environments. This results in significant fluctuations in detection accuracy in complex scenes, failing to provide stable and accurate joint motion data for rowing technical analysis.

[0007] Currently, joint detection methods for dynamic motion scenarios still have significant shortcomings in handling multi-view data fusion, dynamic trajectory tracking, and real-time deviation correction, making it difficult to meet the high precision and dynamic requirements of rowing for joint detection. These problems prevent coaches and researchers from obtaining comprehensive and accurate joint motion information when analyzing athletes' movements, thus affecting the accurate evaluation and optimization of technical movements. Summary of the Invention

[0008] The purpose of this invention is to provide a dynamic model algorithm for joint detection in rowing to solve the problems mentioned in the background art.

[0009] To achieve the above objectives, this invention provides a dynamic model algorithm for joint detection in rowing, the algorithm comprising: Dynamic image sequence acquisition is performed on rowing videos to generate multi-view motion frames. The multi-view motion frames are then processed based on a spatiotemporal feature fusion algorithm to determine candidate regions for key motion phases. Feature enhancement is performed on candidate joint regions to generate high-dimensional motion feature maps, and dynamic trajectories of potential joints are marked based on attention mechanisms and dynamic time warping. Based on the candidate regions of joints and the dynamic trajectories of potential joints, a region of interest for joint detection is generated. By analyzing the gradient changes of motion vectors in the region of interest of joint detection, we can assess whether the fluctuation amplitude of the spatial displacement of the joint conforms to the law of motion; by analyzing the spatiotemporal correlation between skeletal connections and muscle movements in the region of interest of joint detection, we can assess whether the synergistic coupling strength of the joint motion state is within the normal range. Based on the evaluation results of the fluctuation amplitude of joint spatial displacement and the cooperative coupling strength of joint motion state, it is determined whether the region of interest for joint detection needs real-time model parameter calibration.

[0010] Preferably, the rowing video is dynamically imaged to generate multi-view motion frames, and the multi-view motion frames are processed based on a spatiotemporal feature fusion algorithm to determine candidate regions for key motion phases, specifically including: Data is collected by using a multi-camera synchronous acquisition device to capture dynamic images of the rowing process, obtaining multi-view motion sequences including the boat's posture and the athlete's movements. Dynamic feature extraction is performed on the spatial distribution characteristics of athlete joints based on multi-view motion sequences to generate initial joint positioning information. The initial localization information of the joints is fused in multiple dimensions using a spatiotemporal feature fusion algorithm. Based on the feature weight distribution obtained by fusion, candidate regions with joint detection requirements in key motion stages are identified and marked as joint candidate regions.

[0011] Preferably, feature enhancement is performed on candidate joint regions to generate high-dimensional motion feature maps, and the dynamic trajectories of potential joints are marked based on attention mechanisms and dynamic time warping, specifically including: A convolutional neural network is used to enhance the features of dynamic images of candidate regions of joints, and a feature pyramid network is used to perform multi-scale fusion processing on the enhanced features. Temporal correlation of motion features in candidate joint regions is extracted using a bidirectional long short-term memory network. Attention mechanisms and dynamic time warping algorithms are used to dynamically match temporal correlation features, and the dynamic trajectories of potential joints in the joint candidate region are marked according to the matching results.

[0012] Preferably, a region of interest for joint detection is generated based on candidate joint regions and dynamic trajectories of potential joints, specifically including: Extract spatiotemporal coordinate data of candidate regions for joints and dynamic trajectories of potential joints; The spatiotemporal coordinate data of candidate joint regions and dynamic trajectories of potential joints are intersected to determine the smallest spatial bounding box covering the core range of the two regions, and the region of interest for joint detection is generated.

[0013] Preferably, by analyzing the gradient changes of motion vectors in the region of interest of the joint detection, the fluctuation amplitude of the spatial displacement of the joint is evaluated to determine whether it conforms to the law of motion. Specifically, this includes: Extract joint motion vector data within the region of interest for joint detection, and generate a three-dimensional dynamic distribution map of joint spatial displacement based on the motion vector data; Gradient analysis is performed on the three-dimensional dynamic distribution map of joint spatial displacement to calculate the gradient change of joint spatial displacement at different time nodes; The fluctuation amplitude of the spatial displacement of joints within the region of interest is calculated based on the gradient change of the spatial displacement of joints at different time nodes. The fluctuation amplitude includes the average gradient change value and the degree of dispersion of the fluctuation amplitude. The fluctuation amplitude of the joint spatial displacement is compared with the preset fluctuation amplitude benchmark range to evaluate whether the fluctuation amplitude of the joint spatial displacement in the region of interest of joint detection conforms to the law of motion.

[0014] The preferred method for obtaining the dispersion of the average gradient change value and the fluctuation amplitude is as follows: Average gradient change value: obtained by calculating the arithmetic mean of the gradient change values ​​of all time nodes in the region of interest of the key detection; The degree of dispersion of fluctuation amplitude is obtained by calculating the arithmetic mean of the sum of squared deviations between the gradient change values ​​of all time nodes in the region of interest of the key point detection and the average gradient change value.

[0015] Preferably, by analyzing the spatiotemporal correlation between skeletal connections and muscle movement within the region of interest in joint detection, the synergistic coupling strength of joint motion states is assessed to determine whether it is within the normal range. Specifically, this includes: Extract time-series data of bone connection angles and muscle movement amplitude within the region of interest for joint detection, and combine them with a biomechanical model to generate associated feature information of joint motion state; Standardize the associated feature information and extract spatiotemporal correlation indicators by combining mutual information entropy calculation method; A multidimensional feature matrix of spatiotemporal correlation indicators is constructed, and the multidimensional feature matrix is ​​decomposed by canonical correlation analysis to obtain the coupling coefficient matrix of spatiotemporal correlation between skeletal connection and muscle movement. The coupling coefficient matrix includes the correlation weights of time dimension and space dimension as well as the correlation weights of space dimension and correlation indicators. Based on the difference between positive and negative correlations in the coupling coefficient matrix, the cooperative coupling strength of the joint motion state is calculated. By comparing the cooperative coupling strength with the standard range of cooperative coupling strength, it is determined whether the cooperative coupling strength of the joint motion state within the region of interest in joint detection is within the normal range.

[0016] Preferably, based on the evaluation results of the fluctuation amplitude of the joint spatial displacement and the cooperative coupling strength of the joint motion state, it is determined whether the region of interest for joint detection needs real-time model parameter calibration, specifically including: If the fluctuation amplitude of the spatial displacement of the joints in the region of interest detected by the joint point detection conforms to the law of motion, and the cooperative coupling strength of the joint motion state is within the normal range, then it is determined that the region of interest detected by the joint point detection does not need to undergo real-time model parameter calibration; otherwise, it is determined that the region of interest detected by the joint point detection needs to undergo real-time model parameter calibration.

[0017] Preferably, the dynamic image sequence acquisition further includes multimodal data fusion processing, specifically including: By synchronously collecting inertial measurement data of athletes using motion sensors, kinematic parameters including acceleration and angular velocity can be obtained. Spatiotemporal registration of visual features and kinematic parameters of multi-view motion frames is performed to generate a multimodal fusion feature matrix; Weights are assigned to the multimodal fusion feature matrix based on an attention mechanism to enhance the correlation between the motion state of joints and physical motion parameters.

[0018] Preferably, specific methods for weight allocation of the multimodal fusion feature matrix include: Construct a correlation evaluation function between visual features and kinematic parameters, and calculate the contribution of different modal features to the joint detection results; The weighting coefficients of the visual feature channel and the kinematic parameter channel are dynamically adjusted based on the contribution, and the weighting coefficients of the kinematic parameter channel are adaptively updated with the phase change of the paddling cycle. The fusion feature matrix is ​​generated by weighted summation.

[0019] Compared with the prior art, the beneficial effects of the present invention are: By acquiring motion frames from multiple perspectives and fusing them with spatiotemporal features, the limitations of single-view images can be overcome, allowing for a more comprehensive capture of the dynamic changes of each joint during rowing motion and avoiding detection biases caused by viewpoint occlusion or missing information. This multi-dimensional information acquisition and processing method enables the determination of candidate regions for joints in key motion phases to better reflect the actual motion state, laying a more solid foundation for subsequent trajectory tracking.

[0020] Introducing attention mechanisms and dynamic time warping techniques to label the dynamic trajectories of potential joints effectively focuses on key joint features during movement, filtering out irrelevant and interfering information. In the cyclical movements of rowing, the importance of different joints changes with the phase of the movement. The attention mechanism can dynamically adjust feature weights to ensure that the labeling of potential trajectories more closely matches the actual laws of joint movement, reducing the impact of non-critical features on trajectory continuity. Simultaneously, dynamic time warping techniques can adapt to the differences in movement rhythms among different athletes, maintaining consistency in trajectory labeling over time and improving the robustness of trajectory tracking.

[0021] Generating regions of interest (ROIs) for joint detection further narrows the detection range and reduces interference from irrelevant regions. By analyzing the changes in motion vector gradients and the spatiotemporal correlation of skeletal muscles within this region, the motion state of joints can be evaluated from both spatial displacement and co-movement perspectives. This multi-dimensional evaluation method can more accurately identify abnormal joint movements that do not conform to the laws of motion, capturing both abnormal fluctuations in spatial displacement and abnormal changes in the strength of inter-joint co-coupling in a timely manner.

[0022] Real-time model parameter calibration based on evaluation results enables the detection model to dynamically adapt to various changes during motion. In rowing, the rhythm of the athlete's movements and external lighting conditions can change over time, making it difficult for traditional fixed-parameter models to maintain high-precision detection. This model, however, continuously optimizes the accuracy of feature extraction and trajectory tracking during detection through real-time parameter calibration, ensuring that the joint detection results remain consistent with the actual motion state under various dynamic changes. This dynamic adaptive capability not only improves the reliability of the detection results but also enhances the model's applicability in different rowing scenarios, providing stable and accurate joint motion data for both training analysis and competition technique evaluation. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating the working principle of the dynamic model algorithm for joint detection in rowing sports as described in this invention. Figure 2 A flowchart for enhancing candidate region features and dynamic trajectory labeling for key points; Figure 3 A flowchart for assessing the spatial displacement fluctuation amplitude of joint points; Figure 4 A flowchart for assessing the strength of synergistic coupling; Figure 5 This is a flowchart for multimodal data fusion processing. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Please see Figure 1 This invention provides a dynamic model algorithm for joint detection in rowing, the algorithm comprising: Dynamic image sequence acquisition is performed on rowing videos to generate multi-view motion frames. The multi-view motion frames are then processed based on a spatiotemporal feature fusion algorithm to determine candidate regions for key motion phases. Feature enhancement is performed on candidate joint regions to generate high-dimensional motion feature maps, and dynamic trajectories of potential joints are marked based on attention mechanisms and dynamic time warping. Based on the candidate regions of joints and the dynamic trajectories of potential joints, a region of interest for joint detection is generated. By analyzing the gradient changes of motion vectors in the region of interest of joint detection, we can assess whether the fluctuation amplitude of the spatial displacement of the joint conforms to the law of motion; by analyzing the spatiotemporal correlation between skeletal connections and muscle movements in the region of interest of joint detection, we can assess whether the synergistic coupling strength of the joint motion state is within the normal range. Based on the evaluation results of the fluctuation amplitude of joint spatial displacement and the cooperative coupling strength of joint motion state, it is determined whether the region of interest for joint detection needs real-time model parameter calibration.

[0026] Example 1: See Figure 2 The system uses a multi-camera synchronous acquisition device to collect dynamic images of rowing movements. This device consists of multiple high-definition cameras distributed across different locations within the rowing training or competition venue, such as the sides of the boathouse, the finish line, and even mounted on aerial drones, to achieve comprehensive coverage of the rowing scene. The camera parameters are set to meet the high-speed and detail-capturing requirements of rowing, with a resolution of at least 1920×1080 and a frame rate exceeding 60fps, ensuring clear recording of every detail of the athlete's paddling movements, including changes in hand grip and arm muscle contraction. The acquired multi-view motion sequences not only include the athlete's motion data, such as the stroke amplitude of the upper limbs, the rotation angle of the torso, and the force of the lower limb push-off, but also cover relevant boat attitude information, such as the boat's tilt angle on the water, the amplitude of lateral rolling, and the deviation in forward direction.

[0027] Dynamic feature extraction of the spatial distribution characteristics of athlete joints is performed based on multi-view motion sequences. First, key joints requiring focus are identified, including but not limited to the wrist, elbow, shoulder, collarbone, hip, waist, knee, and ankle, as these joints directly participate in the force generation and coordination of rowing movements in rowing. For each joint, its corresponding pixel position is identified in images from different viewpoints. Combined with pre-calibrated intrinsic and extrinsic parameters of each camera, such as focal length, principal point coordinates, and distortion coefficients, the pixel coordinates are converted into three-dimensional spatial coordinates, thereby generating initial joint location information. This information is presented in the form of a data list, with each data point containing the joint identifier, three-dimensional spatial coordinates (X, Y, Z), and a corresponding timestamp accurate to the millisecond level to ensure the accuracy of the time series.

[0028] A spatiotemporal feature fusion algorithm is used to perform multi-dimensional fusion processing on the initial localization information of joints. The algorithm first fuses the spatial dimension by unifying the 3D coordinates of joints from different viewpoints into the same world coordinate system through coordinate transformation, eliminating spatial deviations caused by different camera positions. During the fusion process, a confidence evaluation mechanism is introduced. For the localization results of each joint from different viewpoints, different confidence weights are assigned based on factors such as image clarity and occlusion. Localization results with higher confidence scores have a greater weight in the fusion process. Subsequently, temporal fusion is performed using a sliding window method to analyze the joint localization information of consecutive time frames, extracting the temporal continuity features of joint motion, such as displacement changes between adjacent frames and the continuity of motion speed. Based on the feature weight distribution obtained after fusion, and combined with the biomechanics characteristics of rowing, regions with complex joint motion trajectories, significant displacement changes, and important value for motion technology analysis in key motion stages such as entry into the water, pull-out, exit from the water, and return to the surface are marked as candidate joint regions. Each candidate region is represented by a cubic bounding box in 3D space, containing the vertex coordinates of the bounding box and the corresponding time interval.

[0029] A convolutional neural network (CNN) is used to enhance the features of dynamic images of joint candidate regions. This CNN employs a deep architecture, including an input layer, multiple convolutional layers, pooling layers, and activation function layers. The input layer receives dynamic image data of joint candidate regions, with the image size uniformly adjusted to 256×256 pixels. The convolutional layers use kernels of different sizes (e.g., 3×3, 5×5) for feature extraction, capturing local features in the image through a sliding window operation, such as muscle contours around the joints and clothing wrinkles. The pooling layers use max pooling to reduce the feature map size while retaining key feature information and reducing computational cost. The ReLU activation function is used to enhance the network's non-linear expressive power. After processing by the CNN, the enhanced features are fused at multiple scales using a feature pyramid network. The feature pyramid network consists of multiple layers, with different feature scales corresponding to different levels. Lower-level features contain rich detailed information (e.g., the edge contours of the joints), while higher-level features reflect more abstract semantic information (e.g., the motion state of the joints). By fusing features from different levels through lateral concatenation and upsampling operations, a high-dimensional motion feature map containing multi-scale information is generated. This map is stored in tensor form with dimensions of (number of time frames, feature map height, feature map width, and number of feature channels).

[0030] The temporal correlation of motion features in candidate joint regions is extracted using a Bidirectional Long Short-Term Memory (Bi-LSTM) network. The Bi-LSTM network consists of a forward LSTM and a backward LSTM. The forward LSTM processes the feature sequence chronologically from the first time frame to the last, while the backward LSTM processes the feature sequence in reverse order from the last time frame to the first. The network input is a sequence of feature vectors expanded temporally from a high-dimensional motion feature map. The input feature vector at each time step contains multi-scale feature information for that moment. Through calculations in the network's hidden layers, the temporal dependencies of joint motion are captured, such as the relationship between the current joint position and its positions at the previous and next moments. The network output feature vector not only contains the motion features of the current moment but also incorporates relevant information from historical and future moments, thus fully expressing the temporal correlation of motion features.

[0031] An attention mechanism and a dynamic time warping algorithm are used to dynamically match temporally correlated features. The attention mechanism calculates the attention weight of each feature at each time step using a fully connected neural network; a larger weight indicates a more significant impact of that feature on the overall motion analysis. The dynamic time warping algorithm addresses the inconsistency of time axes across different motion cycles. It constructs a distance matrix, where each element represents the similarity of feature vectors from two different time series, and then finds an optimal path that minimizes the sum of distances along the path, thus aligning the time series. Based on the matching results of the attention weights and dynamic time warping, the position coordinates of key points in consecutive time frames are connected to form a smooth curved trajectory, which is then marked as the dynamic trajectory of potential key points. The trajectory data, containing the 3D coordinates, velocity, and acceleration information at each sampling moment, is stored as a sequence for subsequent region of interest generation.

[0032] Example 2: See Figure 3 The system extracts spatiotemporal coordinate data of candidate joint regions and dynamic trajectories of potential joints. The spatiotemporal coordinate data of candidate joint regions includes their boundary information in three-dimensional space, including the maximum and minimum values ​​of each boundary on the X, Y, and Z axes, as well as the corresponding time intervals, i.e., start and end times. The spatiotemporal coordinate data of potential joint dynamic trajectories is more detailed, containing the three-dimensional coordinates of each sampling point on the trajectory, i.e., X-axis, Y-axis, and Z-axis coordinates, and a precise timestamp corresponding to each sampling point. The timestamps are accurate to a time unit matching the frame rate of the acquisition device, ensuring accurate reflection of the joint's position at every instant.

[0033] The spatiotemporal coordinate data of candidate key regions and potential key dynamic trajectories are intersected. Spatially, the boundary coordinates of the candidate key regions are compared with the coordinates of each point on the potential key dynamic trajectory to determine which trajectory points fall within the spatial range of the candidate regions; these points constitute the spatial intersection. Temporally, the time intervals of the candidate regions and the timestamps of the trajectory points are compared, and trajectory points whose timestamps fall within the time intervals of the candidate regions are selected, forming the temporal intersection. Combining the spatial and temporal intersection results, an intersection region that simultaneously satisfies both spatial and temporal conditions is obtained. Based on this intersection region, the smallest spatial bounding box covering the core range of both regions is determined. Spatially, this bounding box is the smallest cube that can completely contain all points in the intersection region; temporally, it is the shortest time interval that contains all timestamps in the intersection region. This generates the region of interest for key detection.

[0034] The process involves extracting motion vector data of joints within a region of interest (ROI). By calculating the positional changes of joints within the ROI across consecutive time frames, motion vectors are obtained for each time interval. These motion vectors contain both direction and magnitude information; the direction is determined by the coordinate difference between the positions of two consecutive frames, and the magnitude is the modulus of that coordinate difference. Based on this motion vector data, a three-dimensional dynamic distribution map of joint spatial displacement is generated. This map, with time as the horizontal axis and three-dimensional spatial coordinates as the vertical axis, visually displays the spatial positions of joints at different time points and the changes in the direction and magnitude of motion vectors through a set of continuous line segments or points, clearly presenting the motion path and trend of the joints.

[0035] Gradient analysis is performed on the three-dimensional dynamic distribution map of joint spatial displacement. Multiple consecutive time nodes are selected on the distribution map, and the change in joint spatial displacement between adjacent time nodes is calculated. Combined with the time interval between two time nodes, the gradient change at each time node is obtained. The gradient change reflects the rate of change of the joint's motion velocity. Based on these gradient changes at different time nodes, the fluctuation amplitude of joint spatial displacement within the region of interest is calculated. The fluctuation amplitude includes the average gradient change value and the dispersion of the fluctuation amplitude. The average gradient change value is obtained by summing the gradient change values ​​of all time nodes and dividing by the total number of time nodes. The dispersion of the fluctuation amplitude is obtained by calculating the difference between the gradient change value and the average gradient change value at each time node, squaring these differences, summing them, and then dividing by the total number of time nodes.

[0036] The fluctuation amplitude of joint spatial displacement is compared with a preset fluctuation amplitude benchmark range. This benchmark range is determined through statistical analysis based on a large amount of historical rowing data and encompasses common intervals of joint spatial displacement fluctuation in normal rowing. By comparing the calculated average gradient change value and the dispersion of the fluctuation amplitude with the benchmark range, if both are within the benchmark range, the fluctuation amplitude of the joint spatial displacement in the region of interest for joint detection is considered to conform to the motion laws; if either exceeds the benchmark range, it is considered not to conform to the motion laws.

[0037] Example 3: See Figure 4 The average gradient change value is obtained by calculating the arithmetic mean of the gradient change values ​​of all time nodes within the region of interest in joint detection. In rowing, for example, the gradient change values ​​of the elbow joint when an athlete is rowing are collected at multiple consecutive time nodes within the region of interest. These values ​​are added together and divided by the total number of time nodes to obtain the average gradient change value.

[0038] The dispersion of the fluctuation amplitude is obtained by calculating the arithmetic mean of the sum of squared deviations between the gradient change values ​​and the average gradient change values ​​at all time points within the region of interest in the joint detection. Taking the shoulder joint of a rower as an example, after obtaining the gradient change value and the average gradient change value at each time point, the difference between each gradient change value and the average gradient change value is first calculated. Then, these differences are squared respectively. Finally, all the squared results are added together, and the sum is divided by the total number of time points to obtain the dispersion of the fluctuation amplitude.

[0039] This study extracts temporal data on skeletal connection angles and muscle amplitude within regions of interest for joint detection. Taking the upper limb of a rower as an example, it collects values ​​of elbow skeletal connection angles at different time points, while simultaneously recording amplitude data for related muscles such as the biceps and triceps. These data are arranged chronologically to form temporal data. Combined with a biomechanical model that includes information on skeletal structure and muscle attachment points, the data on skeletal connection angles and muscle amplitude are integrated to analyze their interaction during movement, generating associated feature information on joint motion states.

[0040] Standardization of related feature information is performed to transform different types of data, such as skeletal connection angles and muscle movement amplitude, into the same numerical range, eliminating the influence of different data units and magnitudes. Combining mutual information entropy calculation methods, the dependency between skeletal connection angles and muscle movement amplitude is analyzed, extracting indicators that reflect the degree of correlation between the two in time and space, namely, spatiotemporal correlation indicators. These indicators include the correlation strength between the two at different time points and their correlation characteristics in spatial distribution.

[0041] A multidimensional feature matrix of spatiotemporal correlation indicators is constructed. Rows represent different time points, and columns represent various spatiotemporal correlation indicators. Each matrix element corresponds to the value of a specific correlation indicator at a particular time point. This multidimensional feature matrix is ​​decomposed using canonical correlation analysis, a method that identifies linear combinations of indicators related to skeletal connectivity and muscle movement, resulting in the strongest correlation among the combined indicators. The decomposition yields a coupling coefficient matrix of the spatiotemporal correlation between skeletal connectivity and muscle movement. Elements in this matrix represent the correlation strength between different dimensions, including the correlation weights of the time and space dimensions (i.e., the influence of different time and space factors on the correlation) and the correlation weights of the spatial dimension and the correlation indicators (i.e., the correlation between spatial distribution characteristics and the correlation indicators). Positive values ​​indicate positive correlation, and negative values ​​indicate negative correlation.

[0042] Based on the difference between positive and negative correlations in the coupling coefficient matrix, the synergistic coupling strength of the joint motion state is calculated. For example, in the leg movement of a rower, if the sum of the positively correlated coupling coefficients in the coupling coefficient matrix between the knee joint bone connection and the leg muscle movement is greater than the sum of the negatively correlated coupling coefficients, it indicates that the two have good synergy. Subtracting the sum of the negatively correlated coupling coefficients from the sum of the positively correlated coupling coefficients yields the synergistic coupling strength.

[0043] The synergistic coupling strength is compared with a standard range determined by analyzing a large amount of data from rowers during normal movement. When the calculated synergistic coupling strength falls within this standard range, the synergistic coupling strength of the joint movement state within the region of interest in joint point detection is considered to be within the normal range; when it exceeds this range, it is considered to be outside the normal range. Taking the lumbar joint in rowing as an example, if its synergistic coupling strength is within the standard range, it indicates that the lumbar skeleton and surrounding muscles move in coordination; if it exceeds the range, it indicates that there may be a lack of coordination.

[0044] Example 4: In the joint detection of rowing, after the evaluation of the fluctuation amplitude of the joint spatial displacement and the synergistic coupling strength of the joint motion state is completed, the stage of determining whether real-time model parameter calibration is required is entered.

[0045] Taking the rowing motion of a rower as an example, when analyzing the detection data of their elbow joint, the first step is to examine the evaluation results of the fluctuation amplitude of the joint's spatial displacement. If this fluctuation amplitude is within the preset benchmark range, it means that the spatial movement of the elbow during the rowing process conforms to the expected motion law of the elbow in rowing. For example, during the pull phase, the displacement change amplitude of the elbow from extension to flexion is within the normal range, without any sudden excessive or insufficient fluctuations. Simultaneously, the evaluation results of the synergistic coupling strength of the elbow joint's motion state are examined. If it is within the normal range, it indicates that the elbow skeletal connection and the movement of surrounding muscles are coordinated. For example, when the elbow is flexed, the contraction of the biceps brachii and the change in bone angle are correlated, conforming to a normal motion coordination relationship. When both evaluation results meet the conditions, it is determined that the region of interest where the elbow joint is located does not require real-time model parameter calibration, and the model can continue to perform subsequent joint detection in that region according to the current parameters.

[0046] Taking the shoulder joint of a rower as an example, if the evaluation reveals that the fluctuation range of its spatial displacement exceeds the preset benchmark range—for instance, during the return stroke, if the forward and backward movement of the shoulder suddenly increases, exceeding the normal fluctuation range of the shoulder in rowing, this indicates that there may be an abnormality in the shoulder movement, and the model's detection of this area may be biased. In this case, regardless of whether the synergistic coupling strength of the shoulder joint movement state is within the normal range, it is determined that the region of interest where the shoulder joint is located needs to undergo real-time model parameter calibration.

[0047] Another scenario is that, during the detection of a rower's hip joint, the fluctuation range of its spatial displacement conforms to the laws of motion, but the strength of the synergistic coupling exceeds the normal range. For example, the correlation between the rotation angle of the hip bone and the movement amplitude of the surrounding muscles weakens, and the muscle movement cannot coordinate well with the bone rotation. This abnormality in synergy also indicates that there may be a problem with the model's detection of the hip joint. In this case, it is also determined that the region of interest to be detected needs to undergo real-time model parameter calibration.

[0048] When model parameter calibration is deemed necessary, the calibration process adjusts the model parameters based on the specific evaluation results. If there are abnormal fluctuations in spatial displacement, parameters related to motion vector gradient calculation may be adjusted to more accurately capture the spatial movement of joints. If there are abnormalities in the strength of synergistic coupling, parameters related to the correlation analysis of skeletal and muscular motion may be adjusted to enhance the model's ability to identify the synergistic relationship between the two. Through such real-time parameter calibration, the model can better adapt to different motion states, improving the accuracy of joint detection.

[0049] Throughout the judgment and calibration process, the model continuously monitors the evaluation results of the regions of interest (ROIs) at each joint and dynamically adjusts whether parameter calibration is required based on real-time data to ensure that the detection of joints in rowing motion maintains good performance. Whether an abnormal evaluation result occurs at a single joint or multiple joints, the model will promptly determine whether calibration is needed for the corresponding ROI and perform appropriate processing.

[0050] Example 5: See Figure 5 The system synchronously collects inertial measurement data from athletes using motion sensors. These lightweight sensors are fixed to the athlete's wrist, forearm, upper arm, shoulder, hip, thigh, calf, and ankle. The sampling frequency of the sensors is consistent with that of the multi-camera synchronous acquisition device, ensuring data synchronization in the time dimension. The collected kinematic parameters include the components of acceleration on the three-dimensional coordinate axes (X-axis, Y-axis, Z-axis), reflecting the speed of the athlete's limb movements; the components of angular velocity on the three-dimensional coordinate axes, reflecting the rate of limb rotation; and the attitude angle of the motion sensor itself, used to help determine the orientation of the limb in space. This data is transmitted wirelessly to a data processing terminal in real time, forming a continuous sequence of kinematic parameters.

[0051] Spatiotemporal registration of visual features and kinematic parameters from multi-view motion frames is performed. First, visual features and kinematic parameters at the same moment are associated based on timestamps to ensure that each set of data corresponds to the same instant of motion. Visual features include joint contours, limb trajectories, and the relative positions of the boat and athlete extracted from the multi-view motion frames; kinematic parameters are data such as acceleration and angular velocity collected by sensors. For spatial registration, a preset coordinate transformation rule is used to convert the local coordinate system data acquired by the motion sensors into the same global coordinate system as the visual features, eliminating spatial coordinate differences caused by different sensor installation locations. After registration, the visual features and kinematic parameters are arranged chronologically and combined into a multimodal fusion feature matrix. Each row in the matrix represents the fused data of one time frame, and each column corresponds to a specific feature or parameter.

[0052] A correlation evaluation function between visual features and kinematic parameters is constructed. This function determines the degree of correlation between the two types of data by analyzing their changing trends during movement. For example, when an athlete is paddling, does the positional change in the visual features of the wrist show a similar pattern to the wrist acceleration change collected by the motion sensor? If the changing trends are consistent, the correlation is high. Based on this correlation, the contribution of different modal features to the joint detection results is calculated. In paddling motion, kinematic parameters related to force exertion (such as arm acceleration) and limb deformation features in visual features may contribute more to joint detection, and their contribution values ​​are relatively high.

[0053] The weighting coefficients of the visual feature channels and kinematic parameter channels are dynamically adjusted based on their contribution. Channels with higher contribution are given greater weight, allowing their features to play a more significant role in the fusion process. Specifically, the weighting coefficients of the kinematic parameter channels are adaptively updated according to the phase changes of the paddling cycle, which includes four phases: entry, pull, exit, and recovery. During the pull phase, the athlete primarily relies on limb force to propel the boat forward, and the weighting coefficients of force-related kinematic parameters (such as the vertical component of acceleration) increase. During the recovery phase, the limbs primarily perform recovery movements, and the weighting coefficients of velocity-related kinematic parameters (such as angular velocity) increase accordingly. By weighted summing of the adjusted visual features and kinematic parameters, the feature values ​​of different channels are added according to their weight ratios to generate a fused feature matrix. This matrix combines the advantages of both modalities, including both intuitive visual motion information and fine-grained motion parameters captured by sensors, providing more comprehensive feature support for subsequent joint detection.

[0054] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0055] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A dynamic model algorithm for joint detection in rowing, characterized in that, Includes the following steps: Dynamic image sequence acquisition is performed on rowing videos to generate multi-view motion frames. The multi-view motion frames are then processed based on a spatiotemporal feature fusion algorithm to determine candidate regions for key motion phases. Feature enhancement is performed on candidate joint regions to generate high-dimensional motion feature maps, and dynamic trajectories of potential joints are marked based on attention mechanisms and dynamic time warping. Based on the candidate regions of joints and the dynamic trajectories of potential joints, a region of interest for joint detection is generated. By analyzing the gradient changes of motion vectors in the region of interest of joint detection, we can assess whether the fluctuation amplitude of the spatial displacement of the joint conforms to the law of motion; by analyzing the spatiotemporal correlation between skeletal connections and muscle movements in the region of interest of joint detection, we can assess whether the synergistic coupling strength of the joint motion state is within the normal range. Based on the evaluation results of the fluctuation amplitude of joint spatial displacement and the cooperative coupling strength of joint motion state, it is determined whether the region of interest for joint detection needs real-time model parameter calibration.

2. The joint detection dynamic model algorithm for rowing sports according to claim 1, characterized in that, The process involves acquiring dynamic image sequences from rowing videos to generate multi-view motion frames. These frames are then processed using a spatiotemporal feature fusion algorithm to determine candidate regions for key motion phases. Specifically, this includes: Data is collected by using a multi-camera synchronous acquisition device to capture dynamic images of the rowing process, obtaining multi-view motion sequences including the boat's posture and the athlete's movements. Dynamic feature extraction is performed on the spatial distribution characteristics of athlete joints based on multi-view motion sequences to generate initial joint positioning information. The initial localization information of the joints is fused in multiple dimensions using a spatiotemporal feature fusion algorithm. Based on the feature weight distribution obtained by fusion, candidate regions with joint detection requirements in key motion stages are identified and marked as joint candidate regions.

3. The joint detection dynamic model algorithm for rowing sports according to claim 2, characterized in that, Feature enhancement is performed on candidate joint regions to generate high-dimensional motion feature maps. Potential joint dynamic trajectories are then labeled based on attention mechanisms and dynamic time warping, specifically including: A convolutional neural network is used to enhance the features of dynamic images of candidate regions of joints, and a feature pyramid network is used to perform multi-scale fusion processing on the enhanced features. Temporal correlation of motion features in candidate joint regions is extracted using a bidirectional long short-term memory network. Attention mechanisms and dynamic time warping algorithms are used to dynamically match temporal correlation features, and the dynamic trajectories of potential joints in the joint candidate region are marked according to the matching results.

4. The joint detection dynamic model algorithm for rowing sports according to claim 3, characterized in that, Based on candidate joint regions and dynamic trajectories of potential joints, regions of interest for joint detection are generated, specifically including: Extract spatiotemporal coordinate data of candidate regions for joints and dynamic trajectories of potential joints; The spatiotemporal coordinate data of candidate joint regions and dynamic trajectories of potential joints are intersected to determine the smallest spatial bounding box covering the core range of the two regions, and the region of interest for joint detection is generated.

5. The joint detection dynamic model algorithm for rowing sports according to claim 4, characterized in that, By analyzing the gradient changes of motion vectors in the region of interest at the joints, the evaluation assesses whether the fluctuation amplitude of the spatial displacement of the joints conforms to the laws of motion. Specifically, this includes: Extract joint motion vector data within the region of interest for joint detection, and generate a three-dimensional dynamic distribution map of joint spatial displacement based on the motion vector data; Gradient analysis is performed on the three-dimensional dynamic distribution map of joint spatial displacement to calculate the gradient change of joint spatial displacement at different time nodes; The fluctuation amplitude of the spatial displacement of joints within the region of interest is calculated based on the gradient change of the spatial displacement of joints at different time nodes. The fluctuation amplitude includes the average gradient change value and the degree of dispersion of the fluctuation amplitude. The fluctuation amplitude of the joint spatial displacement is compared with the preset fluctuation amplitude benchmark range to evaluate whether the fluctuation amplitude of the joint spatial displacement in the region of interest of joint detection conforms to the law of motion.

6. The joint detection dynamic model algorithm for rowing sports according to claim 5, characterized in that, The specific methods for obtaining the dispersion of the average gradient change value and the fluctuation amplitude are as follows: Average gradient change value: obtained by calculating the arithmetic mean of the gradient change values ​​of all time nodes in the region of interest of the key detection; The degree of dispersion of fluctuation amplitude is obtained by calculating the arithmetic mean of the sum of squared deviations between the gradient change values ​​of all time nodes in the region of interest of the key point detection and the average gradient change value.

7. The joint detection dynamic model algorithm for rowing sports according to claim 6, characterized in that, By analyzing the spatiotemporal correlation between skeletal connections and muscle movement within the region of interest at joint points, the synergistic coupling strength of joint motion states is assessed to determine whether it falls within the normal range. Specifically, this includes: Extract time-series data of bone connection angles and muscle movement amplitude within the region of interest for joint detection, and combine them with a biomechanical model to generate associated feature information of joint motion state; Standardize the associated feature information and extract spatiotemporal correlation indicators by combining mutual information entropy calculation method; A multidimensional feature matrix of spatiotemporal correlation indicators is constructed, and the multidimensional feature matrix is ​​decomposed by canonical correlation analysis to obtain the coupling coefficient matrix of spatiotemporal correlation between skeletal connection and muscle movement. The coupling coefficient matrix includes the correlation weights of time dimension and space dimension as well as the correlation weights of space dimension and correlation indicators. Based on the difference between positive and negative correlations in the coupling coefficient matrix, the cooperative coupling strength of the joint motion state is calculated. By comparing the cooperative coupling strength with the standard range of cooperative coupling strength, it is determined whether the cooperative coupling strength of the joint motion state within the region of interest in joint detection is within the normal range.

8. The joint detection dynamic model algorithm for rowing sports according to claim 7, characterized in that, Based on the evaluation results of the fluctuation amplitude of joint spatial displacement and the cooperative coupling strength of joint motion state, it is determined whether the region of interest for joint detection needs real-time model parameter calibration, specifically including: If the fluctuation amplitude of the spatial displacement of the joints in the region of interest detected by the joint point detection conforms to the law of motion, and the cooperative coupling strength of the joint motion state is within the normal range, then it is determined that the region of interest detected by the joint point detection does not need to undergo real-time model parameter calibration; otherwise, it is determined that the region of interest detected by the joint point detection needs to undergo real-time model parameter calibration.

9. The joint detection dynamic model algorithm for rowing as described in claim 1, characterized in that, The dynamic image sequence acquisition also includes multimodal data fusion processing, specifically including: By synchronously collecting inertial measurement data of athletes using motion sensors, kinematic parameters including acceleration and angular velocity can be obtained. Spatiotemporal registration of visual features and kinematic parameters of multi-view motion frames is performed to generate a multimodal fusion feature matrix; Weights are assigned to the multimodal fusion feature matrix based on an attention mechanism to enhance the correlation between the motion state of joints and physical motion parameters.

10. The joint detection dynamic model algorithm for rowing as described in claim 11, characterized in that, Specific methods for weight allocation of the multimodal fusion feature matrix include: Construct a correlation evaluation function between visual features and kinematic parameters, and calculate the contribution of different modal features to the joint detection results; The weighting coefficients of the visual feature channel and the kinematic parameter channel are dynamically adjusted based on the contribution, and the weighting coefficients of the kinematic parameter channel are adaptively updated with the phase change of the paddling cycle. The fusion feature matrix is ​​generated by weighted summation.

Images