Standardized introduction of training videos for rock climbing, single-person screening and structured drop-in methods

Abstract

The present application relates to the technical field of rock climbing training video processing, and particularly relates to a rock climbing training video standardized import single-person screening and structured disk dumping method. By acquiring a rock climbing training video and synchronously collecting a depth sensing data stream and performing timestamp alignment, encoding format and frame rate normalization is performed on the video to generate a standardized color frame sequence, the depth data stream is resampled according to the normalized frame rate to generate a standardized depth frame sequence, two-dimensional coordinates of human body skeleton key points are extracted frame by frame and mapped to three-dimensional coordinates according to the depth frame sequence, a three-dimensional motion trajectory is constructed and single-person continuity and spatial constraint verification is performed to obtain a single-person effective sequence, action boundary detection is performed on the single-person effective sequence to extract action segments, spatiotemporal feature coding is performed on three-dimensional coordinates of skeleton key points of each action segment to generate action feature descriptors and input a classification network to determine an action category, and a hierarchical classification storage path is constructed to store the action segments and category labels in association.

Description

Technical Field

[0001] This invention relates to the field of rock climbing training video processing technology, and in particular to a standardized method for importing rock climbing training videos into individual screening and structured landing methods. Background Technology

[0002] In the field of rock climbing training, current technology primarily relies on recording training videos using ordinary cameras or action cameras. Coaches or athletes then manually review the footage for technical movement analysis and evaluation. While some venues are equipped with depth sensors or inertial motion capture systems, cost and usage barriers limit the use of single-camera video in most training scenarios. Video data is typically stored locally or in the cloud in its raw format, lacking a unified preprocessing standard, resulting in significant differences in encoding format, frame rate, and resolution between videos captured by different devices. Furthermore, for climbing wall scenarios where multiple climbers train simultaneously, existing methods struggle to automatically distinguish the movement trajectories of different climbers, usually requiring manual marking or the use of color markers for identification.

[0003] Conventional practices in this field have significant shortcomings in the individual screening stage. When multiple climbers are simultaneously climbing or resting on a climbing wall, multiple dynamic human targets appear in the video footage. Traditional target tracking algorithms lose their identities or produce trajectory intersection errors due to frequent target occlusion and cross-movement, making it impossible to reliably and continuously track the complete climbing process of a single trainee. This forces analysts to manually trim individual video clips for each trainee, which is time-consuming and error-prone. Especially in long training videos, manual screening is extremely inefficient and may miss key movement segments.

[0004] Another common deficiency lies in data organization and storage. In existing technologies, processed climbing videos are often stored as continuous long clips or files named by timestamps, lacking structured classification based on action semantics. Action analysis results are usually stored separately from the video as text annotations or separate labeled files. This means that when specific technical actions (such as dynamic force exertion or static balance) need to be retrieved or statistically analyzed, the original video must be re-examined and manual judgment must be relied upon, hindering efficient data reuse and cross-trainer comparative analysis. This loose data management approach impedes the accumulation of climbing training knowledge and systematic modeling. Summary of the Invention

[0005] This invention provides a standardized method and system for importing rock climbing training videos into individual screening and structured landing patterns, which can solve the problems in the prior art.

[0006] A first aspect of this invention provides a method for standardized import of rock climbing training videos into single-person screening and structured landing patterns, comprising:

[0007] The system acquires the climbing training video to be processed and the synchronously acquired depth sensing data stream. It performs timestamp alignment between the depth sensing data stream and the climbing training video, performs encoding format and frame rate normalization processing on the climbing training video to generate a standardized color frame sequence, and resamples the depth sensing data stream according to the normalized frame rate to generate a standardized depth frame sequence.

[0008] Two-dimensional coordinates of key points of the human target skeleton are extracted frame by frame from the standardized color frame sequence. The two-dimensional coordinates of the key points of the skeleton are mapped to three-dimensional spatial coordinates according to the standardized depth frame sequence. A three-dimensional motion trajectory vector is constructed based on the inter-frame displacement of the three-dimensional coordinates of the same target skeleton key points between consecutive frames. The single-person continuity and climbing wall spatial constraint verification are performed on the three-dimensional motion trajectory vector. The frame segment that tracks the trajectory to a single climber is marked as a single-person valid sequence.

[0009] For single-person valid sequences, the climbing action boundary is detected based on the temporal change of 3D displacement energy of skeletal key points, and the start and end frame indices of each action segment are extracted. Spatiotemporal feature encoding is performed on the 3D coordinates of skeletal key points of each action segment to generate action feature descriptors. The action feature descriptors are input into the action classification network for action category determination. Based on the determination results, a hierarchical classification storage path is constructed. After the single-person valid sequence is segmented by action segment, it is associated with the action category label and stored in the target storage node according to the classification storage path.

[0010] Perform timestamp alignment between the depth sensing data stream and the climbing training video, including:

[0011] The hardware trigger timestamps of each depth frame in the depth sensing data stream and the hardware trigger timestamps of each color frame in the rock climbing training video are collected to construct the depth frame timestamp sequence and the color frame timestamp sequence.

[0012] Using the hardware synchronization trigger pulse signal shared by the depth sensing module and the color camera module as the alignment reference, the timestamp deviation function between the depth frame timestamp sequence and the color frame timestamp sequence is extracted. The timestamp deviation function describes the offset and offset direction of the depth frame sampling time relative to the color frame sampling time.

[0013] Interpolation correction compensation is performed on the original timestamps of each depth frame based on the timestamp deviation function. The timestamps of each depth frame are corrected in reverse by using the deviation value of the timestamp deviation function at the corresponding sampling time, so that the deviation between the corrected depth frame timestamp and the corresponding color frame timestamp converges to the preset alignment tolerance.

[0014] The depth frames with corrected timestamps are assigned to the sampling times of each color frame according to the nearest neighbor matching principle with the timestamps of the color frames, thus establishing a one-to-one correspondence between depth frames and color frames.

[0015] Based on the correspondence between depth frames and color frames, bilinear interpolation resampling is performed on the spatial resolution of each depth frame to make the spatial resolution of the depth frame completely consistent with the spatial resolution of the standardized color frame sequence.

[0016] Based on the standardized depth frame sequence, the two-dimensional coordinates of skeletal keypoints are mapped to three-dimensional spatial coordinates, including:

[0017] Obtain the two-dimensional pixel coordinates of each skeletal keypoint detected in the normalized color frame sequence, which are composed of pixel column coordinates and pixel row coordinates in the corresponding normalized color frame;

[0018] Using the pixel 2D coordinates of the skeletal keypoint as the index position, in the standardized depth frame whose timestamp matches the standardized color frame, perform a depth value query operation at the same pixel column coordinate and pixel row coordinate position to extract the depth measurement value corresponding to the pixel position of the skeletal keypoint.

[0019] Numerical validity verification is performed on the extracted depth measurement values. The verification conditions include whether the depth measurement value is empty, whether the depth measurement value exceeds the effective range of the depth sensing module, and whether the variance of the depth value in the neighborhood of the pixel where the depth value is located exceeds the preset smoothness tolerance threshold. The depth measurement value that passes all verification conditions is taken as the effective depth value of the skeletal key point.

[0020] The pixel coordinates of the skeletal key points are combined with the effective depth values ​​to form homogeneous pixel coordinates. Combined with the camera intrinsic parameter matrix constructed from the focal length parameters and principal point coordinate parameters of the color camera module, the homogeneous pixel coordinates are back-projected to the camera coordinate system through inverse perspective projection transformation, and the three-dimensional spatial coordinates of the skeletal key points in the camera coordinate system are calculated and output.

[0021] After back-projecting the pixel 2D coordinates and effective depth values ​​of the skeletal keypoints to the camera coordinate system to obtain the 3D spatial coordinates, the following are included:

[0022] Based on the predefined human skeleton topology connection relationship, two adjacent skeleton key points with skeleton connection relationship in the same frame are formed into a skeleton segment, and the Euclidean space length of the skeleton segment in the camera coordinate system is calculated.

[0023] The Euclidean space length of each bone segment is compared with the standard reference length of the corresponding bone segment in the preset human skeleton proportion model segment by segment, and the length deviation rate of each bone segment is calculated. The length deviation rate is defined as the quotient obtained by dividing the absolute value of the difference between the Euclidean space length of the bone segment and the standard reference length by the standard reference length.

[0024] Two bone keypoints associated with bone segments whose length deviation rate exceeds the preset limit of rigid body deformation are marked as abnormal mapping points and removed from the set of bone keypoints of that frame.

[0025] Temporal adaptive filtering is performed on the three-dimensional spatial coordinate sequence of the same skeletal key point in the camera coordinate system between consecutive frames. Abrupt jump coordinates that exceed the physiological limit of human joint displacement between frames are detected in the three-dimensional spatial coordinate sequence. The abrupt jump coordinates are replaced and corrected with the weighted interpolation results of the coordinates of each neighboring coordinate in the temporal neighborhood window of the coordinate, so that the change of the three-dimensional coordinates of the skeletal key point between consecutive frames meets the constraints of human motion smoothness and consistency.

[0026] A 3D motion trajectory vector is constructed based on the inter-frame displacement of the 3D spatial coordinates of key points of the same human skeleton in consecutive frames. The 3D motion trajectory vector undergoes dual verification of single-person continuity and climbing wall spatial constraints, including:

[0027] The displacement of the three-dimensional spatial coordinates of each skeletal key point of the same human target in consecutive frames is weighted and summed according to the preset motion contribution weight corresponding to the type of skeletal key point, so as to generate a sequence of three-dimensional motion trajectory vectors of the human centroid of the human target in consecutive frames.

[0028] Single-person continuous verification: Scan the three-dimensional motion trajectory vector sequence of the human body's center of mass sequentially along the time axis, and detect whether there are trajectory vector breaks caused by missing detection frames, and whether there are trajectory vector abrupt jumps caused by the human body's center of mass spatial displacement exceeding the rock climbing motion limit displacement threshold between two consecutive frames. Frame segments without trajectory vector breaks and without trajectory vector abrupt jumps are judged as passing the single-person continuous verification.

[0029] Climbing wall spatial constraint verification: Obtain the three-dimensional spatial bounding box parameters of the climbing wall in the camera coordinate system, and check frame by frame whether the three-dimensional spatial coordinates of the human centroid corresponding to each frame time in the three-dimensional motion trajectory vector sequence of the human centroid are all located inside the bounding box of the climbing wall spatial boundary. Frame segments in which the three-dimensional spatial coordinates of the centroid are entirely located inside the bounding box are judged to pass the climbing wall spatial constraint verification.

[0030] Frames that pass both the single-person persistence check and the climbing wall spatial constraint check are marked as single-person valid sequences.

[0031] In single-person continuous verification, trajectory vector breakpoints detected are determined to be retained or removed after skeletal topology matching, including:

[0032] When the single-person continuous verification detects a change in the number of human targets between frames, the first skeletal topology descriptor of the human target in the last frame before the change in number is extracted, and the second skeletal topology descriptor of each candidate human target in the first frame after the change in number is extracted. The skeletal topology descriptor contains a skeletal proportion feature vector composed of the length ratio of each skeletal segment and a joint angle distribution feature vector composed of the angle values ​​of each joint.

[0033] Calculate the topological matching similarity value between the first bone topological structure descriptor and each of the second bone topological structure descriptors. The topological matching similarity value is obtained by weighted fusion operation of the cosine similarity of the bone proportion feature vector and the Bach distance of the joint angle distribution feature vector.

[0034] If the calculated topology matching similarity value exceeds the preset topology matching judgment threshold, it is determined that the human targets before and after the change in number belong to the same rock climber. The frame segment is retained and a linear interpolation trajectory completion operation based on the trajectory vector of the neighboring frame is performed at the trajectory vector breakpoint.

[0035] If the calculated topology matching similarity value does not exceed the preset topology matching judgment threshold, and the human targets before and after the change in the number of judgments have crossed targets, the frame segment is removed.

[0036] A hierarchical classification storage path is constructed based on the action category determination results. Valid sequences for a single person are segmented into action fragments and then associated with action category tags, stored in the target storage node pointed to by the hierarchical classification storage path. This includes:

[0037] A three-level hierarchical tag tree structure is constructed, with action category tags as the root node of the category tree, rock climbing difficulty level tags as intermediate nodes, and action segment time interval tags as leaf nodes.

[0038] The tag values ​​of each level from the root node to the leaf node in the three-level hierarchical tag tree structure are concatenated in order according to the directory separator to generate a complete category storage path string from the root directory to the storage file;

[0039] Get the remaining available storage capacity and write throughput of each candidate storage node in the available candidate storage node set;

[0040] A load balancing selection function is constructed based on the remaining available storage capacity of each candidate storage node and the current write operation throughput per unit time. The write priority weight of each candidate storage node under the load balancing selection function is calculated, and the candidate storage node with the highest write priority weight is selected as the target storage node.

[0041] The video data blocks corresponding to each action segment, along with the action category label, climbing difficulty level label, and time interval label metadata associated with that action segment, are persistently written to the file system of the target storage node according to the complete classification storage path.

[0042] A second aspect of this invention provides a standardized import system for rock climbing training videos, enabling individual screening and structured landing, comprising:

[0043] The data preprocessing unit is used to acquire the climbing training video to be processed and the synchronously acquired depth sensing data stream, perform timestamp alignment on the depth sensing data stream and the climbing training video, perform encoding format and frame rate normalization processing on the climbing training video to generate a standardized color frame sequence, and resample the depth sensing data stream according to the normalized frame rate to generate a standardized depth frame sequence.

[0044] The skeleton tracking unit is used to extract the two-dimensional coordinates of key points of the human target skeleton frame by frame from the standardized color frame sequence, map the two-dimensional coordinates of the key points of the skeleton to three-dimensional spatial coordinates according to the standardized depth frame sequence, construct a three-dimensional motion trajectory vector based on the inter-frame displacement of the three-dimensional coordinates of the same target skeleton key points between consecutive frames, perform single-person continuity and climbing wall spatial constraint verification on the three-dimensional motion trajectory vector, and mark the frame segment that tracks the trajectory to a single climber as a single-person valid sequence.

[0045] The action recognition unit is used to detect the boundary of rock climbing action based on the temporal change of the three-dimensional displacement energy of the skeletal key points of a single person's valid sequence and extract the start and end frame indexes of each action segment. It performs spatiotemporal feature encoding on the three-dimensional coordinates of the skeletal key points of each action segment to generate action feature descriptors. The action feature descriptors are input into the action classification network to determine the action category. Based on the determination results, a hierarchical classification storage path is constructed. The single person's valid sequence is segmented by action segments and associated with the action category label according to the classification storage path and stored in the target storage node.

[0046] A third aspect of the present invention provides an electronic device, comprising:

[0047] processor;

[0048] Memory used to store processor-executable instructions;

[0049] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0050] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0051] The process acquires climbing training videos and depth sensing data streams for strict timeline alignment. After standardized encoding and frame rate normalization, standardized color and depth frame sequences are generated, eliminating spatiotemporal discrepancies between multimodal data at the source. This ensures high consistency and repeatability in subsequent skeletal keypoint extraction and 3D mapping. Depth data resampling and frame rate matching further guarantee the accuracy of 3D coordinate calculations, laying a reliable data foundation for long-term continuous climbing training video processing and significantly reducing data preprocessing errors and manual correction costs caused by inconsistent formats and frame rates.

[0052] By extracting the two-dimensional coordinates of key points of the human skeleton frame by frame from standardized color and depth frame sequences and accurately mapping them to three-dimensional space, a three-dimensional motion trajectory vector is constructed based on the displacement of the same key points between adjacent frames. Simultaneously, by combining single-person continuity verification with the spatial constraints of the climbing wall, irrelevant interference and overlapping trajectories are eliminated from both the dimensions of multi-frame motion continuity and spatial physical constraints, ensuring that the entire process is uniquely tracked to a single climber and obtaining a valid single-person sequence. This mechanism effectively avoids mistracking and target loss caused by multi-target occlusion, temporary stillness, or rapid movement segments, significantly improving the accuracy and robustness of single-person screening and providing a clean and reliable data source for subsequent motion analysis.

[0053] Based on the temporal changes in the three-dimensional displacement energy of skeletal key points, the system intelligently detects action boundaries and accurately extracts the start and end frame indices of each action segment for effective sequences of a single person. Action feature descriptors are generated through spatiotemporal feature encoding and input into a classification network to determine the action category. Based on the determination results, a hierarchical classification storage path is automatically constructed, and the segmented action segments are associated with their corresponding category labels in a structured manner and stored in designated nodes. This achieves fully automated processing of rock climbing training videos, from raw acquisition to refined action segmentation, category recognition, and standardized landing, eliminating the tedious steps of manual annotation, classification, and storage. It significantly improves the efficiency of training data management and the convenience of retrieval, providing standardized, high-definition structured data support for quantitative analysis of sports data, evaluation of training action quality, and personalized guidance. Attached Figure Description

[0054] Figure 1 A flowchart illustrating the process of standardizing rock climbing training videos by incorporating individual screening and structured landing methods. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0057] Figure 1 This is a flowchart illustrating the standardized import method for single-person screening and structured landing of rock climbing training videos according to an embodiment of the present invention. The standardized import method for single-person screening and structured landing of rock climbing training videos includes:

[0058] The system acquires the climbing training video to be processed and the synchronously acquired depth sensing data stream. It performs timestamp alignment between the depth sensing data stream and the climbing training video, performs encoding format and frame rate normalization processing on the climbing training video to generate a standardized color frame sequence, and resamples the depth sensing data stream according to the normalized frame rate to generate a standardized depth frame sequence.

[0059] Two-dimensional coordinates of key points of the human target skeleton are extracted frame by frame from the standardized color frame sequence. The two-dimensional coordinates of the key points of the skeleton are mapped to three-dimensional spatial coordinates according to the standardized depth frame sequence. A three-dimensional motion trajectory vector is constructed based on the inter-frame displacement of the three-dimensional coordinates of the same target skeleton key points between consecutive frames. The single-person continuity and climbing wall spatial constraint verification are performed on the three-dimensional motion trajectory vector. The frame segment that tracks the trajectory to a single climber is marked as a single-person valid sequence.

[0060] For single-person valid sequences, the climbing action boundary is detected based on the temporal change of 3D displacement energy of skeletal key points, and the start and end frame indices of each action segment are extracted. Spatiotemporal feature encoding is performed on the 3D coordinates of skeletal key points of each action segment to generate action feature descriptors. The action feature descriptors are input into the action classification network for action category determination. Based on the determination results, a hierarchical classification storage path is constructed. After the single-person valid sequence is segmented by action segment, it is associated with the action category label and stored in the target storage node according to the classification storage path.

[0061] Perform timestamp alignment between the depth sensing data stream and the climbing training video, including:

[0062] The hardware trigger timestamps of each depth frame in the depth sensing data stream and the hardware trigger timestamps of each color frame in the rock climbing training video are collected to construct the depth frame timestamp sequence and the color frame timestamp sequence.

[0063] Using the hardware synchronization trigger pulse signal shared by the depth sensing module and the color camera module as the alignment reference, the timestamp deviation function between the depth frame timestamp sequence and the color frame timestamp sequence is extracted. The timestamp deviation function describes the offset and offset direction of the depth frame sampling time relative to the color frame sampling time.

[0064] Interpolation correction compensation is performed on the original timestamps of each depth frame based on the timestamp deviation function. The timestamps of each depth frame are corrected in reverse by using the deviation value of the timestamp deviation function at the corresponding sampling time, so that the deviation between the corrected depth frame timestamp and the corresponding color frame timestamp converges to the preset alignment tolerance.

[0065] The depth frames with corrected timestamps are assigned to the sampling times of each color frame according to the nearest neighbor matching principle with the timestamps of the color frames, thus establishing a one-to-one correspondence between depth frames and color frames.

[0066] Based on the correspondence between depth frames and color frames, bilinear interpolation resampling is performed on the spatial resolution of each depth frame to make the spatial resolution of the depth frame completely consistent with the spatial resolution of the standardized color frame sequence.

[0067] In real-world rock climbing training video capture scenarios, depth sensing modules and color camera modules are often driven by different hardware, each maintaining its own independent internal clock. Even if the two devices share the same trigger pulse at the physical level, due to crystal oscillator frequency deviations, driver layer delays, and data buffer queue queuing times, there will still be a slight but not negligible systematic deviation in the actual sampling times of depth frames and color frames on the time axis. If this deviation is not accurately compensated for, subsequent 3D coordinate mapping of skeletal key points will result in spatial misalignment, leading to distorted motion trajectory reconstruction.

[0068] When acquiring the hardware trigger timestamps for each depth frame in the depth sensing data stream, the hardware counter of the depth sensing module latches the current count value at the moment each frame's exposure begins and records it as the frame's original timestamp with nanosecond precision. The color camera module uses the same mechanism to add a hardware trigger timestamp to each color image frame. Arranging the timestamps of all depth frames in the acquisition order yields the depth frame timestamp sequence; arranging the timestamps of all color frames in the acquisition order yields the color frame timestamp sequence. The two sequences may have unequal numbers of entries because the nominal frame rates of the depth sensing module and the color camera module may differ. For example, if the depth sensing module operates at 30fps while the color camera module operates at 60fps, the number of entries in the depth frame timestamp sequence will be approximately half that of the color frame timestamp sequence.

[0069] Using a shared hardware synchronization trigger pulse signal between the depth sensing module and the color camera module as the alignment reference is a crucial prerequisite for eliminating systematic deviations between the two timestamp sequences. The hardware synchronization trigger pulse is sent simultaneously to both devices by a dedicated synchronization control board within a fixed period. Upon receiving the pulse, each device initiates a one-frame acquisition process. Due to the different drive response delays of the two devices, the actual exposure start time has a fixed delay relative to the pulse arrival time. Let the... The depth frame timestamp corresponding to each synchronization pulse is: The corresponding color frame timestamp is Then the timestamp deviation function Defined as This describes the offset and direction of the depth frame sampling time relative to the color frame sampling time. When When, it indicates that the depth frame sampling time is later than the color frame; when When, it indicates that the depth frame sampling time is earlier than the color frame. Under ideal rigid clock conditions, It should be a constant; in practical engineering environments, due to crystal oscillator frequency drift, It changes slowly over time, so piecewise linear fitting or low-order polynomial fitting is required to obtain an estimate of the bias in the continuous time domain.

[0070] When performing interpolation correction compensation on the original timestamps of each depth frame based on the timestamp deviation function, for the Frame depth frame, its original timestamp is ,exist The estimated deviation value is obtained by querying the timestamp deviation function at the corresponding continuous time position. Then the corrected timestamp according to The calculation involves mapping the time reference of the depth frame from the internal clock of the depth sensing module to the internal clock of the color camera module, allowing both data streams to share the same time reference. After calibration, the absolute value of the residual between the corrected timestamp and the corresponding color frame timestamp is verified for each depth frame to ensure it converges within a preset alignment tolerance. This preset alignment tolerance is typically determined based on the human movement speed and depth measurement accuracy in the application scenario. For example, in a standard rock climbing training scenario, it can be set to the time length corresponding to 0.5 color frame periods, which is approximately 8.3 milliseconds at 60fps. If the residual of a frame exceeds the tolerance, the local re-evaluation of the deviation function is re-executed for that frame, and iterative correction is performed until the residual meets the convergence condition.

[0071] The timestamp-corrected depth frames are assigned to the sampling times of each color frame according to the nearest neighbor matching principle with the color frame timestamps. Specifically, for each color frame, the depth frame with the smallest absolute difference from the color frame timestamp is searched in the corrected depth frame timestamp sequence, and a one-to-one correspondence is established between the depth frame and the color frame. When the frame rate of the depth sensing module is lower than that of the color camera module, some consecutive color frames may be assigned to the same depth frame. In this case, the depth data of the depth frame will be reused for the corresponding multiple color frames. When the frame rate of the depth sensing module is higher than that of the color camera module, some depth frames will not correspond to any color frame and will be discarded directly. The time distance threshold for nearest neighbor matching is also constrained by a preset alignment tolerance. If a color frame cannot find a depth frame that meets the tolerance requirements, the color frame is marked as a depth missing frame and filled in by linear interpolation of adjacent valid depth frames in subsequent processing.

[0072] After establishing a one-to-one correspondence between depth frames and color frames, the inconsistency in spatial resolution between the two images needs to be addressed. Due to hardware limitations imposed by time-of-flight or structured light technologies, the output resolution of depth sensing modules is typically lower than that of color camera modules. For example, the spatial resolution of a color frame is 1920×1080 pixels, while the corresponding spatial resolution of a depth frame is only 640×480 pixels. When performing bilinear interpolation resampling on the spatial resolution of each depth frame, the target resolution (i.e., the spatial resolution of the normalized color frame sequence) is used as the output grid. The depth value at each target pixel location in the depth frame is calculated using weighted interpolation based on the depth values ​​of its four neighboring pixels in the original depth frame coordinate system. Bilinear interpolation performs linear interpolation in both the horizontal and vertical directions, which can suppress aliasing artifacts while maintaining depth continuity. It is important to note that depth images contain invalid depth pixels due to occlusion, reflection, or exceeding the measurement range. These invalid pixels are typically represented by 0 or a specific invalid flag. During bilinear interpolation, invalid pixels are not included in the weighted calculation; instead, they are filled with the weighted average of their valid neighboring pixels to prevent invalid depth values ​​from contaminating the valid area. After resampling, the spatial resolution of each depth frame is completely consistent with the normalized color frame sequence, and the pixel coordinate system is aligned, providing a reliable spatial correspondence for the subsequent accurate mapping of 2D coordinates to 3D spatial coordinates of skeletal keypoints.

[0073] The entire timestamp alignment process forms a complete closed loop from hardware trigger pulse acquisition, deviation function modeling, interpolation correction compensation, nearest neighbor frame matching to spatial resolution alignment, ensuring that depth sensing data and color video meet consistency requirements in both the temporal and spatial domains, thus providing a stable and reliable data foundation for subsequent 3D motion trajectory reconstruction.

[0074] Based on the standardized depth frame sequence, the two-dimensional coordinates of skeletal keypoints are mapped to three-dimensional spatial coordinates, including:

[0075] Obtain the two-dimensional pixel coordinates of each skeletal keypoint detected in the normalized color frame sequence, which are composed of pixel column coordinates and pixel row coordinates in the corresponding normalized color frame;

[0076] Using the pixel 2D coordinates of the skeletal keypoint as the index position, in the standardized depth frame whose timestamp matches the standardized color frame, perform a depth value query operation at the same pixel column coordinate and pixel row coordinate position to extract the depth measurement value corresponding to the pixel position of the skeletal keypoint.

[0077] Numerical validity verification is performed on the extracted depth measurement values. The verification conditions include whether the depth measurement value is empty, whether the depth measurement value exceeds the effective range of the depth sensing module, and whether the variance of the depth value in the neighborhood of the pixel where the depth value is located exceeds the preset smoothness tolerance threshold. The depth measurement value that passes all verification conditions is taken as the effective depth value of the skeletal key point.

[0078] The pixel coordinates of the skeletal key points are combined with the effective depth values ​​to form homogeneous pixel coordinates. Combined with the camera intrinsic parameter matrix constructed from the focal length parameters and principal point coordinate parameters of the color camera module, the homogeneous pixel coordinates are back-projected to the camera coordinate system through inverse perspective projection transformation, and the three-dimensional spatial coordinates of the skeletal key points in the camera coordinate system are calculated and output.

[0079] When acquiring the pixel 2D coordinates of each skeletal keypoint, for the human pose detection output of each frame in the standardized color frame sequence, the pixel column coordinates and pixel row coordinates of each skeletal keypoint in the image coordinate system of that frame are recorded to constitute the pixel 2D coordinates of that keypoint. ,in Represents the column coordinates of pixels. This represents the pixel row coordinates. Since the color frames have undergone encoding format and frame rate normalization, their image resolution and coordinate system definition remain consistent. Therefore, the pixel coordinates of the detected skeletal keypoints in each frame have a consistent physical meaning in the same resolution space and can be directly used as the index for subsequent depth value queries.

[0080] Using the pixel 2D coordinates of skeletal keypoints As an index location, the same pixel column coordinates are used in the normalized depth frame that matches the normalized color frame timestamp. with pixel row coordinates A depth value lookup operation is performed at the location. The depth frame and color frame have already undergone synchronization correction during the timestamp alignment phase, and they have also undergone alignment mapping in terms of spatial resolution. Therefore, the pixel coordinates in the color frame can be directly used as the lookup index for the depth frame to read the stored depth measurement value at that pixel location. This value represents the distance from the scene point at the corresponding pixel location to the optical center of the sensor, as measured by the depth sensing module at the corresponding sampling time. In actual rock climbing scenarios, due to factors such as the surface material of the rock slab, lighting conditions, and sensor noise, the depth values ​​of some pixel locations in the depth frame may have issues such as missing values, abnormalities, or local discontinuities. Therefore, it is necessary to perform strict numerical validity verification before using the depth measurement values.

[0081] Extracted depth measurements When performing numerical validity checks, three conditions are checked sequentially. The first condition is to determine... The first condition checks if the value is null, meaning whether a valid depth reading exists at that pixel location. If the depth frame does not return any value at that location, the depth of that skeletal keypoint is directly marked as invalid, and subsequent processing uses interpolation compensation or a skip strategy for that keypoint. The second condition is to determine... Does it exceed the effective range calibrated by the depth sensing module? Let the lower limit of the effective range be... The upper limit of the effective range is ,like or If the depth value is not within the sensor's reliable operating range, it is considered invalid. In rock climbing training scenarios, the distance between the climber and the camera is typically between 1.5 meters and 6 meters. and The specific values ​​are calibrated and written to the configuration file during system initialization, based on the hardware specifications of the depth sensing module used. The third condition is to calculate the variance of the depth values ​​within the pixel neighborhood where the depth value is located, using the pixel coordinates of the skeletal keypoints. Centered on, take a side length of Calculate the variance of all valid depth pixel values ​​within a square neighborhood window. ,like Exceeding the preset smoothness tolerance threshold If the pixel location is not found in a depth discontinuity edge or noise region, the depth value is considered unreliable and is marked as invalid. All three verification conditions must be passed before the pixel can be considered valid. The effective depth value of this skeletal key point This is used for subsequent 3D coordinate calculation.

[0082] In the 3D coordinate back projection calculation stage, the pixel 2D coordinates of the skeletal key points are... With effective depth value Combined into homogeneous pixel coordinates The camera intrinsic parameter matrix is ​​constructed by combining the focal length parameters of the color camera module with the principal point coordinate parameters. Perform the inverse perspective projection transformation. Camera intrinsic parameter matrix. Defined as:

[0083]

[0084] in and These are the equivalent focal lengths of the color camera module in the pixel column direction and pixel row direction, respectively. and These are the coordinates of the principal point in the pixel column direction and pixel row direction, respectively. All four parameters are acquired and stored during the system deployment phase through the camera module calibration process. The inverse perspective projection transform projects the homogeneous pixel coordinates back to the camera coordinate system. The calculation process is as follows:

[0085]

[0086]

[0087] Output three-dimensional space coordinates This refers to the position of the keypoint of the skeleton in the camera coordinate system, where This represents the coordinate components of the key point along the horizontal direction of the camera. This represents the coordinate components along the vertical direction of the camera. This represents the coordinate components along the camera's optical axis, i.e., the depth direction. The above-described pixel coordinate reading, depth value query, validity verification, and back-projection calculation process is repeatedly executed for all skeletal keypoints detected in each frame of the standardized color frame sequence. Frame by frame, the 3D spatial coordinates of all keypoints in the camera coordinate system are output, providing spatial position input for the construction of 3D motion trajectory vectors between subsequent consecutive frames.

[0088] In practical engineering implementation, the side length of the neighborhood window The value of this parameter needs to be determined by a trade-off between noise suppression capability and key point positioning accuracy. If... If the value is too small, the number of neighborhood samples is insufficient, and the variance estimation is unstable; if If the value is too large, it may include the depth values ​​of adjacent rock points or background areas in the calculation, leading to misclassification of skeletal keypoints within a continuous depth region as invalid. In rock climbing training video processing scenarios, it is recommended to use a smaller value. Set to an odd value within the range of 5 to 9 pixels, and determine the optimal value experimentally based on the resolution and noise characteristics of the depth sensing module used. Smoothness tolerance threshold Similarly, calibration is required based on the specific sensor model and the environmental conditions of the climbing site to ensure that no large number of invalid markers are generated in areas with rich variations in the texture of the climbing surface due to normal depth fluctuations. It also effectively filters depth jump noise caused by sensor blind spots or occluded areas. For skeletal keypoints that fail calibration, linear interpolation of the effective depth values ​​of the same keypoint in temporally adjacent frames can be used for compensation to maintain the temporal continuity of the skeletal sequence and ensure the stability of subsequent action boundary detection and feature encoding.

[0089] After back-projecting the pixel 2D coordinates and effective depth values ​​of the skeletal keypoints to the camera coordinate system to obtain the 3D spatial coordinates, the following are included:

[0090] Based on the predefined human skeleton topology connection relationship, two adjacent skeleton key points with skeleton connection relationship in the same frame are formed into a skeleton segment, and the Euclidean space length of the skeleton segment in the camera coordinate system is calculated.

[0091] The Euclidean space length of each bone segment is compared with the standard reference length of the corresponding bone segment in the preset human skeleton proportion model segment by segment, and the length deviation rate of each bone segment is calculated. The length deviation rate is defined as the quotient obtained by dividing the absolute value of the difference between the Euclidean space length of the bone segment and the standard reference length by the standard reference length.

[0092] Two bone keypoints associated with bone segments whose length deviation rate exceeds the preset limit of rigid body deformation are marked as abnormal mapping points and removed from the set of bone keypoints of that frame.

[0093] Temporal adaptive filtering is performed on the three-dimensional spatial coordinate sequence of the same skeletal key point in the camera coordinate system between consecutive frames. Abrupt jump coordinates that exceed the physiological limit of human joint displacement between frames are detected in the three-dimensional spatial coordinate sequence. The abrupt jump coordinates are replaced and corrected with the weighted interpolation results of the coordinates of each neighboring coordinate in the temporal neighborhood window of the coordinate, so that the change of the three-dimensional coordinates of the skeletal key point between consecutive frames meets the constraints of human motion smoothness and consistency.

[0094] After back-projecting the skeletal keypoints from pixel 2D coordinates to camera coordinate system 3D coordinates, the resulting 3D coordinates need to be physically validated to eliminate skeletal structure distortion caused by depth measurement noise or pose estimation errors. The human skeleton exhibits near-rigid body characteristics under normal motion, meaning the length of bone segments between adjacent keypoints remains physiologically relatively constant and does not change significantly with changes in viewing angle or amplitude of movement. Utilizing this constraint, abnormal keypoints generated during the 3D mapping process can be effectively identified and eliminated.

[0095] Based on the predefined topological connections of the human skeleton, two adjacent skeletal keypoints with skeletal connections within the same frame are considered as a single skeletal segment. The Euclidean length of this skeletal segment in the camera coordinate system is then calculated. Let the three-dimensional coordinates of the two endpoint keypoints of the skeletal segment in the camera coordinate system be... and Then the Euclidean length of that bone segment Defined as ,in , , Key points Coordinate components along the three axes of the camera coordinate system , , Key point The corresponding coordinate components. The topological connections of the human skeleton typically include major bone segments such as the neck, upper arm, forearm, torso, thigh, and lower leg. Each bone segment corresponds to a standard reference length in a preset human skeleton scale model. The reference length is determined based on anthropometry statistics and can be linearly scaled to fit the height parameters of the trainee.

[0096] Calculate the length deviation rate for each bone segment. It is defined as the Euclidean space length of a bone segment. Corresponding standard reference length The quotient obtained by dividing the absolute value of the difference by the standard reference length is, i.e. .when Exceeding the preset rigid body deformation limit At that time, it was assumed that at least one of the two endpoint keypoints of the bone segment had a 3D mapping anomaly. In practice, The value is typically set between 0.15 and 0.25, and the specific value can be adjusted based on the measurement accuracy of the depth sensor and the range of climbing movements. For bone segments exceeding the tolerance, both associated bone keypoints are marked as abnormal mapping points and removed from the current frame's bone keypoint set to avoid introducing incorrect 3D coordinate information in subsequent motion analysis. It is important to note that when the same keypoint is associated with multiple bone segments, if the length deviation rate of any one of the bone segments exceeds the tolerance, the keypoint is marked as abnormal and removed, thus ensuring the overall physical consistency of the skeletal structure.

[0097] After completing the rigid body constraint verification of the bone segments and removing abnormal mapping points, it is also necessary to smooth the temporal sequence of the three-dimensional coordinates of the same bone key points between consecutive frames to eliminate the interference of inter-frame coordinate jumps on motion trajectory analysis. In rock climbing, the movement speed of the athlete's limbs is limited by the physiological structure of the human joints, and the three-dimensional displacement of a single bone key point between adjacent frames should not exceed a certain physiological limit threshold. . Based on the frame rate and the maximum angular velocity of human joints, it is estimated that, for example, at a capture frame rate of 30 frames per second, the physiological limit of inter-frame displacement of the wrist key point is about 0.3 to 0.5 meters, and that of the torso key point is even smaller.

[0098] When performing temporal adaptive filtering on 3D coordinates, the Euclidean distance between the 3D coordinates of the skeletal keypoints in each frame and the corresponding coordinates in the previous frame is first calculated. If this distance exceeds... If the current frame coordinates are detected as a sudden jump coordinate, then the weighted interpolation result of the coordinates in each neighboring region within the temporal neighborhood window is used for replacement and correction. Let the frame number of the sudden jump frame be... ,by Take a temporal neighborhood window centered at the center, with a window radius of . The coordinates of each neighboring frame within the window are: ( (excluding the current mutation frame itself), corresponding weight is Then the corrected coordinates It is obtained from weighted interpolation, that is ,in Window radius is assigned based on the temporal distance between neighboring frames and the current frame, using a Gaussian decay method; the closer the frame, the greater the weight, and the farther the frame, the smaller the weight. Generally, 2 to 4 frames are used, which can be adaptively adjusted according to the frame rate and motion speed. When the motion speed is fast, the frame rate can be reduced appropriately. To preserve movement details, increase the speed appropriately when the movement speed is slow. To enhance the smoothing effect.

[0099] It is worth noting that if multiple consecutive abrupt change frames exist within the temporal neighborhood window (e.g., keypoints are lost in several consecutive frames due to occlusion), the number of valid frames that can participate in interpolation in the neighborhood coordinates decreases. In this case, the coordinates of the nearest valid non-abrupt change frame should be used for interpolation first, and the weight ratio of far-neighboring frames should be appropriately reduced to ensure the physical rationality of the correction result. For consecutive abrupt change frames exceeding the window radius... In extreme cases, the entire frame segment is marked as an unreliable segment and excluded during subsequent motion segment extraction to avoid introducing coordinate data that cannot be reliably corrected.

[0100] After rigid body constraint verification and temporal adaptive filtering, the resulting 3D coordinate sequence of key points on the skeleton spatially satisfies the human skeleton proportion constraints and temporally satisfies the smoothness and consistency constraints of joint movements. This provides high-quality, physically reasonable input data for subsequent construction of 3D motion trajectory vectors and detection of climbing action boundaries. The entire verification and filtering process is executed online in a sliding window manner based on frame-by-frame processing, without waiting for all frames to be processed, demonstrating good real-time processing capabilities and making it suitable for batch structured processing scenarios of large-scale climbing training videos.

[0101] A 3D motion trajectory vector is constructed based on the inter-frame displacement of the 3D spatial coordinates of key points of the same human skeleton in consecutive frames. The 3D motion trajectory vector undergoes dual verification of single-person continuity and climbing wall spatial constraints, including:

[0102] The displacement of the three-dimensional spatial coordinates of each skeletal key point of the same human target in consecutive frames is weighted and summed according to the preset motion contribution weight corresponding to the type of skeletal key point, so as to generate a sequence of three-dimensional motion trajectory vectors of the human centroid of the human target in consecutive frames.

[0103] Single-person continuous verification:

[0104] The sequence of three-dimensional motion trajectory vectors of the human body's center of mass is scanned sequentially along the time axis. The sequence is checked for trajectory vector breaks caused by missing detection frames, and for abrupt jumps in trajectory vectors caused by the human body's center of mass spatial displacement exceeding the rock climbing motion limit threshold between two consecutive frames. Frames without trajectory vector breaks and without abrupt jumps are judged as passing the single-person continuous verification.

[0105] Climbing wall spatial constraint verification:

[0106] Obtain the bounding box parameters of the three-dimensional space boundary of the climbing wall in the camera coordinate system, and detect frame by frame whether the three-dimensional space coordinates of the human centroid corresponding to each frame time in the three-dimensional motion trajectory vector sequence of the human centroid are all located in the inner region of the bounding box of the climbing wall space boundary. Frame segments in which the three-dimensional space coordinates of the centroid are entirely located inside the bounding box are judged to pass the climbing wall space constraint verification.

[0107] Frames that pass both the single-person persistence check and the climbing wall spatial constraint check are marked as single-person valid sequences.

[0108] After obtaining the three-dimensional spatial coordinates of each skeletal keypoint, it is necessary to integrate the motion information of the same human target into a trajectory vector sequence that can represent the overall motion trend. Since the contribution of skeletal keypoints in different parts of the human body to the overall center of mass motion varies significantly—for example, keypoints at the center of the torso (such as the midpoint of the hip joint) are far more representative of the center of mass position than keypoints at the extremities (such as fingertips or toes)—it is necessary to pre-set corresponding motion contribution weights for different types of skeletal keypoints when calculating the three-dimensional motion trajectory of the human center of mass. Specifically, for the same human target in the [missing information - likely a specific motion vector sequence], ... Frame and the Between frames, the 3D spatial coordinate displacement of each skeletal keypoint is calculated one by one. Then, the displacements are weighted and summed according to the preset motion contribution weights corresponding to each keypoint type, thereby generating the 3D motion trajectory vector of the human target's center of mass across consecutive frames. Arranging the 3D motion trajectory vectors of the center of mass in chronological order across all frames constitutes a complete sequence of 3D motion trajectory vectors of the human center of mass. The preset motion contribution weights can be determined offline based on a human biomechanical model, with higher weights assigned to core keypoints of the torso and lower weights assigned to keypoints of the extremities. The sum of the weights of all keypoints is normalized to 1 to ensure that the center of mass trajectory vectors are physically comparable.

[0109] After constructing the three-dimensional motion trajectory vector sequence of the human center of mass, a single-person continuous verification is required to eliminate target loss or target confusion during detection. The trajectory vector sequence is scanned sequentially along the time axis. First, it checks for trajectory vector breakpoints caused by missing detection frames. A trajectory vector breakpoint refers to a missing detection result of skeletal keypoints in one or more consecutive frames on the time axis, making it impossible to calculate the three-dimensional motion trajectory vector of the center of mass at the corresponding moment, thus creating a gap in the sequence. Such breakpoints may occur when the human target in the video frame is occluded, there are sudden changes in lighting, or the confidence of the pose estimation network is too low. Second, it checks for abrupt trajectory vector jumps caused by the spatial displacement of the human center of mass exceeding the rock climbing limit displacement threshold between two consecutive frames. The rock climbing limit displacement threshold is set based on the biomechanical characteristics of rock climbing, reflecting the maximum reasonable displacement upper limit that the human center of mass can undergo within a single frame time interval. If the displacement of the center of mass between two frames exceeds this threshold, it is determined that a trajectory jump has occurred, usually corresponding to a mismatch in target detection or a switch in target identity in a multi-person scene. Only when the trajectory vector sequence has neither breaks nor sudden jumps within the scanning range is the corresponding frame segment judged to have passed the single-person continuity check.

[0110] The purpose of climbing wall spatial constraint verification is to further exclude frames where the human target is continuously tracked but is not actually in a climbing state. The 3D bounding box parameters of the climbing wall in the camera coordinate system are obtained through a pre-defined site calibration process. During calibration, a calibration board of known size or a laser rangefinder is used to measure the 3D coordinates of the four corner points of the climbing wall, thereby determining the six boundary surface parameters of the bounding box in the camera coordinate system, corresponding to the left and right boundaries in the horizontal direction, the top and bottom boundaries in the vertical direction, and the front and back boundaries in the depth direction. The bounding box parameters are stored in the form of a 3D axis-aligned bounding box (AABB), described by the minimum and maximum values ​​in three dimensions. During climbing wall spatial constraint verification, the 3D spatial coordinates of the human centroid are extracted frame by frame, and it is determined whether these coordinates satisfy the condition of being simultaneously within the six boundary constraints of the bounding box, i.e., the horizontal, vertical, and depth components of the centroid coordinates fall between the minimum and maximum values ​​of their respective dimensions. If the 3D spatial coordinates of the human centroid in a frame exceed the bounding box boundaries, that frame is determined to not meet the climbing wall spatial constraints. A frame segment is considered to have passed the climbing wall spatial constraint check only when the three-dimensional spatial coordinates of the human centroid in all frames within the segment are located inside the bounding box.

[0111] In practical applications, the parameters of the climbing wall bounding box can be set with a certain tolerance extension on top of strict boundaries to cope with measurement deviations caused by slight camera shake or calibration errors. The size of the tolerance extension is determined based on the calibration accuracy evaluation results, and is usually extended outward by several centimeters from the boundaries in each dimension. Furthermore, for situations where the climber's center of gravity coordinates may temporarily deviate from the front area of ​​the climbing wall during the take-off preparation phase or the landing buffer phase, false judgments can be avoided by setting a minimum consecutive frame threshold. That is, only when the number of frames in which the center of gravity coordinates continuously exceeds the bounding box boundary exceeds a preset threshold is the corresponding frame segment marked as failing, thereby enhancing the robustness of the verification process to temporary boundary deviations.

[0112] Frames that pass both the single-person persistence check and the climbing wall spatial constraint check are marked as valid single-person sequences. These two checks are complementary: the single-person persistence check focuses on ensuring the temporal continuity of tracking the same human target, eliminating target loss and identity confusion; the climbing wall spatial constraint check focuses on ensuring the human target is indeed within the valid area of ​​the climbing wall, excluding frames where irrelevant personnel enter the shot or the climber leaves the wall. Only frames that simultaneously meet both checks can be used as valid climbing training data in subsequent processes for action boundary detection and feature encoding, thus ensuring the quality and purity of the final landing data. Frames that fail either check are skipped to avoid introducing noisy data that could interfere with the training effect of the subsequent action classification network.

[0113] In single-person continuous verification, trajectory vector breakpoints detected are determined to be retained or removed after skeletal topology matching, including:

[0114] When the single-person continuous verification detects a change in the number of human targets between frames, the first skeletal topology descriptor of the human target in the last frame before the change in number is extracted, and the second skeletal topology descriptor of each candidate human target in the first frame after the change in number is extracted. The skeletal topology descriptor contains a skeletal proportion feature vector composed of the length ratio of each skeletal segment and a joint angle distribution feature vector composed of the angle values ​​of each joint.

[0115] Calculate the topological matching similarity value between the first bone topological structure descriptor and each of the second bone topological structure descriptors. The topological matching similarity value is obtained by weighted fusion operation of the cosine similarity of the bone proportion feature vector and the Bach distance of the joint angle distribution feature vector.

[0116] If the calculated topology matching similarity value exceeds the preset topology matching judgment threshold, it is determined that the human targets before and after the change in number belong to the same rock climber. The frame segment is retained and a linear interpolation trajectory completion operation based on the trajectory vector of the neighboring frame is performed at the trajectory vector breakpoint.

[0117] If the calculated topology matching similarity value does not exceed the preset topology matching judgment threshold, and the human targets before and after the change in the number of judgments have crossed targets, the frame segment is removed.

[0118] In rock climbing training scenarios, it is common for multiple people to appear in the camera's field of view simultaneously, such as when an instructor is demonstrating or other trainees are warming up on an adjacent route. When a change in the number of human targets is detected between frames during single-person continuous verification, it cannot be directly determined that the tracked target has switched or been occluded. Fine-grained matching of the skeletal topology is needed to distinguish between the two fundamentally different situations: "the same person is briefly occluded and then reappears" and "target overlap and confusion occurs between different people".

[0119] The skeletal topology descriptor of the tracked human target in the last frame before the change in number is extracted as the first skeletal topology descriptor. This descriptor consists of two parts: a skeletal proportion feature vector and a joint angle distribution feature vector. The skeletal proportion feature vector is constructed by calculating the length ratio of each skeletal segment relative to a reference skeletal segment (usually the main trunk segment) based on several predefined skeletal segments in the human skeletal topology map, and arranging all ratios in a fixed order to form a vector. This normalized ratio expression method has natural scale invariance to changes in shooting distance, and can effectively eliminate the difference in absolute length of skeletal segments caused by different distances between the target and the camera. The joint angle distribution feature vector is constructed by calculating the bending angle values ​​of each major joint (including shoulder, elbow, wrist, hip, knee, ankle, etc.) in three-dimensional space, arranging these angle values ​​in the order of joint numbers to form a vector, reflecting the overall posture configuration of the human body in the current frame.

[0120] After the quantity changes, there may be multiple candidate body targets in the first frame. For each candidate target, a second skeleton topology descriptor is extracted. The construction method is completely consistent with the first skeleton topology descriptor to ensure that the two are comparable in the feature space.

[0121] When calculating the topological matching similarity between the first skeletal topological descriptor and each of the second skeletal topological descriptors, different similarity measurement methods are used for the skeletal proportion feature vector and the joint angle distribution feature vector. Let the skeletal proportion feature vector in the first skeletal topological descriptor be... The second skeleton proportion feature vector corresponding to a certain candidate target is The cosine similarity between the two is The calculation method is as follows The range of cosine similarity values ​​is within... arrive Between, when the directions of the two bone proportion vectors are completely consistent, the value is taken as... This indicates that the skeletal proportions of the two targets are highly similar, suggesting that they are likely from the same person.

[0122] For the joint angle distribution feature vector, the Bhattacharyya distance is used to measure the similarity between the two distributions. Let the joint angle distribution feature vector in the first skeleton topological descriptor be... The feature vector of the second joint angle distribution corresponding to the candidate target is After normalizing the two vectors, they are treated as probability distributions, and the Bach distance is used. The calculation method is as follows ,in and Representing vectors respectively and The Middle The normalized values ​​of the joint angle components. The smaller the Bach distance, the closer the distributions of the two joint angles, and the more similar the target poses. To convert the Bach distance into a similarity value with the same dimension as cosine similarity, it is transformed by a negative exponential function to obtain the joint angle similarity. The range of values ​​is The higher the value, the higher the similarity.

[0123] Topological matching similarity value The similarity is obtained by weighted fusion of the cosine similarity of the skeletal proportion feature vector and the similarity of the joint angle distribution feature vector, specifically as follows: ,in The fusion weight coefficient for skeletal proportion features has a value range of [value range missing]. In practical applications, because skeletal proportions are stable physiological characteristics of the human body and have a strong ability to distinguish the identity of the same person, while joint angle distribution is greatly affected by the current state of movement, therefore, it is usually considered that... Set to slightly greater than The value (e.g.) to This gives higher matching weight to bone proportion features.

[0124] when Exceeding the preset topology matching threshold When the number of human targets before and after the change in quantity is determined to belong to the same climber, that frame segment is retained. At this point, trajectory completion is required at the breakpoint in the trajectory vector. The completion method uses linear interpolation based on the trajectory vectors of adjacent frames: taking the trajectory vectors of several frames before the breakpoint as a reference, the linear interpolation between the 3D coordinates of corresponding skeletal keypoints in the frames before and after the breakpoint is calculated. The 3D coordinates of the skeletal keypoints in the missing frames are then evenly filled according to the number of frames the breakpoint lasts, ensuring a smooth transition of the trajectory vector at the breakpoint. Linear interpolation can provide reasonable trajectory estimation when the number of breakpoint frames is small (usually no more than a few frames), avoiding the impact of trajectory interruptions caused by brief occlusion on the integrity of subsequent action segment analysis.

[0125] when The preset topology matching threshold was not exceeded. When the number of human targets before and after the change is determined to be different, a target overlap has occurred. Target overlap usually occurs when the routes of two climbers intersect, become obscured, and then separate again. The tracking algorithm may mistakenly identify the other climber appearing after the obscuration as the original tracking target. In this case, the frame segment is discarded to avoid mixing the movement trajectories of different people into the single valid sequence, thereby ensuring the purity of human identity in subsequent action analysis.

[0126] Preset topology matching threshold The threshold setting needs to be adjusted according to the specific application scenario. In professional rock climbing training scenarios, the body shape differences among athletes are significant, so the threshold can be appropriately increased to enhance the discrimination ability; in youth training scenarios, the body shapes of subjects are relatively similar, so the threshold can be appropriately decreased to reduce misjudgments. This threshold can be determined by statistically analyzing the optimal classification boundary on a known labeled dataset, or it can be manually configured by the user according to actual business needs. The entire skeletal topology matching and discrimination process is executed one by one at the breakpoints of the single-person continuous verification, forming a refined processing mechanism for trajectory breakpoints, effectively improving the accuracy and completeness of single-person effective sequence extraction.

[0127] A hierarchical classification storage path is constructed based on the action category determination results. Valid sequences for a single person are segmented into action fragments and then associated with action category tags, stored in the target storage node pointed to by the hierarchical classification storage path. This includes:

[0128] A three-level hierarchical tag tree structure is constructed, with action category tags as the root node of the category tree, rock climbing difficulty level tags as intermediate nodes, and action segment time interval tags as leaf nodes.

[0129] The tag values ​​of each level from the root node to the leaf node in the three-level hierarchical tag tree structure are concatenated in order according to the directory separator to generate a complete category storage path string from the root directory to the storage file;

[0130] Get the remaining available storage capacity and write throughput of each candidate storage node in the available candidate storage node set;

[0131] A load balancing selection function is constructed based on the remaining available storage capacity of each candidate storage node and the current write operation throughput per unit time. The write priority weight of each candidate storage node under the load balancing selection function is calculated, and the candidate storage node with the highest write priority weight is selected as the target storage node.

[0132] The video data blocks corresponding to each action segment, along with the action category label, climbing difficulty level label, and time interval label metadata associated with that action segment, are persistently written to the file system of the target storage node according to the complete classification storage path.

[0133] After identifying and classifying the action segments of a single person's valid sequence, the segmented video data of each action segment and its associated tags need to be persistently stored on disk in a structured manner. To this end, a three-level hierarchical classification storage path mechanism is introduced, which encodes the semantic classification information, difficulty level information, and time positioning information of rock climbing actions into the file system path in the form of a directory tree, so that the storage structure itself carries complete semantic indexing capabilities.

[0134] The tag tree structure is constructed with action category tags as the root node of the classification directory tree. Action category tags are derived from the action classification network's judgment results for each action segment. For example, climbing-specific action types such as "side pull," "step," and "high center of gravity shift" exist as independent root-level directories. Below the root-level nodes, climbing difficulty level tags serve as intermediate-level nodes. Difficulty levels can be assigned according to the International Federation of Sport Climbing (IFSC) general difficulty rating system or the training venue's custom grading system, such as "V3," "V5," and "5.10c." Each action category can have multiple difficulty level subdirectories, forming the second-level branches of a multi-branch tree. Below the intermediate-level nodes, action segment time interval tags serve as leaf-level nodes. Time interval tags are generated by concatenating the start and end frame indices of the action segment in the original video, with a format such as "frame_0120_0245," accurately identifying the position of the action segment on the original timeline. The three-level hierarchical relationship is thus formed: action category → difficulty level → time interval. The entire label tree covers the complete classification and localization information of all identified action segments in a training video.

[0135] When generating the complete categorized storage path string, the tag values ​​of each level from the root node to the leaf node in the tag tree are extracted sequentially and concatenated using the operating system's common directory separator " / ". This results in a complete path prefix in the form of " / side pull / V5 / frame\_0120\_0245 / ". The video data block filename and metadata filename are then appended to this path prefix, resulting in the complete categorized storage path string from the root directory to the specific storage file. This path string also serves as the target addressing basis for subsequent write operations, ensuring that the video data of each action segment and its semantic tag remain physically aligned at the file system level.

[0136] When determining the target storage node, the optimal node needs to be dynamically selected from the candidate storage node set based on a load balancing strategy. For each candidate node in the candidate storage node set, its current remaining available storage capacity and current write operation throughput per unit time are collected. The remaining available storage capacity reflects the node's space capacity margin, and the write operation throughput reflects the node's current I / O load pressure. These two indicators together determine whether the node is suitable for handling new write tasks.

[0137] Based on the above two metrics, a load balancing selection function is constructed to calculate the write priority weight of each candidate storage node. Let the first... The normalized value of the remaining available storage capacity of each candidate storage node is: Its current normalized write operation throughput per unit time is Then the write priority weight of this node Calculated in the following way:

[0138]

[0139] in This is the balancing weighting coefficient between the capacity factor and the throughput factor, with a value range of [value range missing]. This is used to adjust the relative contribution of storage capacity margin and current write pressure in priority evaluation. The larger the value, the more space the node has remaining. A larger value indicates a heavier write load on the node. Candidate nodes with higher numerical values ​​have the advantages of both ample space and lighter load. The candidate storage node set is traversed to select... The candidate node with the highest value will be selected as the target storage node for this write task. When there are multiple candidate nodes... When the values ​​are equal, the node with the smaller number is given priority to ensure the certainty of the selection result.

[0140] The normalization method is as follows: Let the maximum remaining capacity of all nodes in the candidate set be _____. ,but equal to the The original remaining capacity of each node divided by Similarly, let the maximum current throughput of all nodes be [value]. ,but equal to the The original throughput of each node divided by Normalization ensures that the two indicators are within the same dimension, avoiding bias in weight calculation caused by differences in dimensions. The default value can be set to 0.6, which means that the storage capacity margin is moderately biased towards the write load pressure to prevent write failures due to node space exhaustion. In actual deployment, It can be optimized based on the hardware configuration and business access patterns of the storage cluster.

[0141] Once the target storage node is determined, the video data blocks corresponding to each action segment, along with the associated action category label, climbing difficulty level label, and time interval label metadata, are persistently written to the target storage node's file system according to the complete classification storage path. During writing, the video data blocks are written to disk as independent binary files, with the filename named using the unique identifier of the action segment; the metadata is stored in the same directory as the video data blocks in a structured text format (such as JSON or XML). The metadata file records fields such as the action category label string, difficulty level label string, start and end frame index values, the original video file path, and the write timestamp, ensuring that the context information of the action segment can be completely restored during subsequent retrieval and training data loading.

[0142] After persistent writing is complete, a write integrity check is performed on the target storage node. The file size before and after writing is compared with the checksum to confirm that the data has been written to disk without loss. If the check fails, a rewrite process is triggered, re-executing the target storage node selection and write operation. After a maximum of a preset number of retries, if it still fails, the action segment write task is marked as abnormal and recorded in the error log for subsequent manual review. This hierarchical classification storage process directly maps the semantic structure of the climbing training videos to the directory structure of the file system, ensuring that the organization of the training dataset is consistent with the action classification system, significantly reducing data loading and indexing overhead in the subsequent model training phase.

[0143] A second aspect of this invention provides a standardized import system for rock climbing training videos, enabling individual screening and structured landing, comprising:

[0144] The data preprocessing unit is used to acquire the climbing training video to be processed and the synchronously acquired depth sensing data stream, perform timestamp alignment on the depth sensing data stream and the climbing training video, perform encoding format and frame rate normalization processing on the climbing training video to generate a standardized color frame sequence, and resample the depth sensing data stream according to the normalized frame rate to generate a standardized depth frame sequence.

[0145] The skeleton tracking unit is used to extract the two-dimensional coordinates of key points of the human target skeleton frame by frame from the standardized color frame sequence, map the two-dimensional coordinates of the key points of the skeleton to three-dimensional spatial coordinates according to the standardized depth frame sequence, construct a three-dimensional motion trajectory vector based on the inter-frame displacement of the three-dimensional coordinates of the same target skeleton key points between consecutive frames, perform single-person continuity and climbing wall spatial constraint verification on the three-dimensional motion trajectory vector, and mark the frame segment that tracks the trajectory to a single climber as a single-person valid sequence.

[0146] The action recognition unit is used to detect the boundary of rock climbing action based on the temporal change of the three-dimensional displacement energy of the skeletal key points of a single person's valid sequence and extract the start and end frame indexes of each action segment. It performs spatiotemporal feature encoding on the three-dimensional coordinates of the skeletal key points of each action segment to generate action feature descriptors. The action feature descriptors are input into the action classification network to determine the action category. Based on the determination results, a hierarchical classification storage path is constructed. The single person's valid sequence is segmented by action segments and associated with the action category label according to the classification storage path and stored in the target storage node.

[0147] A third aspect of the present invention provides an electronic device, comprising:

[0148] processor;

[0149] Memory used to store processor-executable instructions;

[0150] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0151] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0152] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A standardized method for importing rock climbing training videos into a single-person screening and structured landing method, characterized by: include: The system acquires the climbing training video to be processed and the synchronously acquired depth sensing data stream. It performs timestamp alignment between the depth sensing data stream and the climbing training video, performs encoding format and frame rate normalization processing on the climbing training video to generate a standardized color frame sequence, and resamples the depth sensing data stream according to the normalized frame rate to generate a standardized depth frame sequence. Two-dimensional coordinates of key points of the human target skeleton are extracted frame by frame from the standardized color frame sequence. The two-dimensional coordinates of the key points of the skeleton are mapped to three-dimensional spatial coordinates according to the standardized depth frame sequence. A three-dimensional motion trajectory vector is constructed based on the inter-frame displacement of the three-dimensional coordinates of the same target skeleton key points between consecutive frames. The single-person continuity and climbing wall spatial constraint verification are performed on the three-dimensional motion trajectory vector. The frame segment that tracks the trajectory to a single climber is marked as a single-person valid sequence. For single-person valid sequences, the climbing action boundary is detected based on the temporal change of 3D displacement energy of skeletal key points, and the start and end frame indices of each action segment are extracted. Spatiotemporal feature encoding is performed on the 3D coordinates of skeletal key points of each action segment to generate action feature descriptors. The action feature descriptors are input into the action classification network for action category determination. Based on the determination results, a hierarchical classification storage path is constructed. After the single-person valid sequence is segmented by action segment, it is associated with the action category label and stored in the target storage node according to the classification storage path.

2. The method according to claim 1, characterized in that, Perform timestamp alignment between the depth sensing data stream and the climbing training video, including: The hardware trigger timestamps of each depth frame in the depth sensing data stream and the hardware trigger timestamps of each color frame in the rock climbing training video are collected to construct the depth frame timestamp sequence and the color frame timestamp sequence. Using the hardware synchronization trigger pulse signal shared by the depth sensing module and the color camera module as the alignment reference, the timestamp deviation function between the depth frame timestamp sequence and the color frame timestamp sequence is extracted. The timestamp deviation function describes the offset and offset direction of the depth frame sampling time relative to the color frame sampling time. Interpolation correction compensation is performed on the original timestamps of each depth frame based on the timestamp deviation function. The timestamps of each depth frame are corrected in reverse by using the deviation value of the timestamp deviation function at the corresponding sampling time, so that the deviation between the corrected depth frame timestamp and the corresponding color frame timestamp converges to the preset alignment tolerance. The depth frames with corrected timestamps are assigned to the sampling times of each color frame according to the nearest neighbor matching principle with the timestamps of the color frames, thus establishing a one-to-one correspondence between depth frames and color frames. Based on the correspondence between depth frames and color frames, bilinear interpolation resampling is performed on the spatial resolution of each depth frame to make the spatial resolution of the depth frame completely consistent with the spatial resolution of the standardized color frame sequence.

3. The method according to claim 1, characterized in that, Based on a standardized depth frame sequence, the two-dimensional coordinates of skeletal keypoints are mapped to three-dimensional spatial coordinates, including: Obtain the two-dimensional pixel coordinates of each skeletal keypoint detected in the normalized color frame sequence, which are composed of pixel column coordinates and pixel row coordinates in the corresponding normalized color frame; Using the pixel 2D coordinates of the skeletal keypoint as the index position, in the standardized depth frame whose timestamp matches the standardized color frame, perform a depth value query operation at the same pixel column coordinate and pixel row coordinate position to extract the depth measurement value corresponding to the pixel position of the skeletal keypoint. Numerical validity verification is performed on the extracted depth measurement values. The verification conditions include whether the depth measurement value is empty, whether the depth measurement value exceeds the effective range of the depth sensing module, and whether the variance of the depth value in the neighborhood of the pixel where the depth value is located exceeds the preset smoothness tolerance threshold. The depth measurement value that passes all verification conditions is taken as the effective depth value of the skeletal key point. The pixel coordinates of the skeletal key points are combined with the effective depth values ​​to form homogeneous pixel coordinates. Combined with the camera intrinsic parameter matrix constructed from the focal length parameters and principal point coordinate parameters of the color camera module, the homogeneous pixel coordinates are back-projected to the camera coordinate system through inverse perspective projection transformation, and the three-dimensional spatial coordinates of the skeletal key points in the camera coordinate system are calculated and output.

4. The method according to claim 3, characterized in that, After back-projecting the pixel 2D coordinates and effective depth values ​​of the skeletal keypoints to the camera coordinate system to obtain the 3D spatial coordinates, the following is included: Based on the predefined human skeleton topology connection relationship, two adjacent skeleton key points with skeleton connection relationship in the same frame are formed into a skeleton segment, and the Euclidean space length of the skeleton segment in the camera coordinate system is calculated. The Euclidean space length of each bone segment is compared with the standard reference length of the corresponding bone segment in the preset human skeleton proportion model segment by segment, and the length deviation rate of each bone segment is calculated. The length deviation rate is defined as the quotient obtained by dividing the absolute value of the difference between the Euclidean space length of the bone segment and the standard reference length by the standard reference length. Two bone keypoints associated with bone segments whose length deviation rate exceeds the preset limit of rigid body deformation are marked as abnormal mapping points and removed from the set of bone keypoints of that frame. Temporal adaptive filtering is performed on the three-dimensional spatial coordinate sequence of the same skeletal key point in the camera coordinate system between consecutive frames. Abrupt jump coordinates that exceed the physiological limit of human joint displacement between frames are detected in the three-dimensional spatial coordinate sequence. The abrupt jump coordinates are replaced and corrected with the weighted interpolation results of the coordinates of each neighboring coordinate in the temporal neighborhood window of the coordinate, so that the change of the three-dimensional coordinates of the skeletal key point between consecutive frames meets the constraints of human motion smoothness and consistency.

5. The method according to claim 1, characterized in that, A 3D motion trajectory vector is constructed based on the inter-frame displacement of the 3D spatial coordinates of key points of the same human skeleton in consecutive frames. The 3D motion trajectory vector undergoes dual verification of single-person continuity and climbing wall spatial constraints, including: The displacement of the three-dimensional spatial coordinates of each skeletal key point of the same human target in consecutive frames is weighted and summed according to the preset motion contribution weight corresponding to the type of skeletal key point, so as to generate a sequence of three-dimensional motion trajectory vectors of the human centroid of the human target in consecutive frames. Single-person continuous verification: Scan the three-dimensional motion trajectory vector sequence of the human body's center of mass sequentially along the time axis, and detect whether there are trajectory vector breaks caused by missing detection frames, and whether there are trajectory vector abrupt jumps caused by the human body's center of mass spatial displacement exceeding the rock climbing motion limit displacement threshold between two consecutive frames. Frame segments without trajectory vector breaks and without trajectory vector abrupt jumps are judged as passing the single-person continuous verification. Climbing wall spatial constraint verification: Obtain the three-dimensional spatial bounding box parameters of the climbing wall in the camera coordinate system, and check frame by frame whether the three-dimensional spatial coordinates of the human centroid corresponding to each frame time in the three-dimensional motion trajectory vector sequence of the human centroid are all located inside the bounding box of the climbing wall spatial boundary. Frame segments in which the three-dimensional spatial coordinates of the centroid are entirely located inside the bounding box are judged to pass the climbing wall spatial constraint verification. Frames that pass both the single-person persistence check and the climbing wall spatial constraint check are marked as single-person valid sequences.

6. The method according to claim 5, characterized in that, In single-person continuous verification, trajectory vector breakpoints detected are determined to be retained or removed after skeletal topology matching, including: When the single-person continuous verification detects a change in the number of human targets between frames, the first skeletal topology descriptor of the human target in the last frame before the change in number is extracted, and the second skeletal topology descriptor of each candidate human target in the first frame after the change in number is extracted. The skeletal topology descriptor contains a skeletal proportion feature vector composed of the length ratio of each skeletal segment and a joint angle distribution feature vector composed of the angle values ​​of each joint. Calculate the topological matching similarity value between the first bone topological structure descriptor and each of the second bone topological structure descriptors. The topological matching similarity value is obtained by weighted fusion operation of the cosine similarity of the bone proportion feature vector and the Bach distance of the joint angle distribution feature vector. If the calculated topology matching similarity value exceeds the preset topology matching judgment threshold, it is determined that the human targets before and after the change in number belong to the same rock climber. The frame segment is retained and a linear interpolation trajectory completion operation based on the trajectory vector of the neighboring frame is performed at the trajectory vector breakpoint. If the calculated topology matching similarity value does not exceed the preset topology matching judgment threshold, and the human targets before and after the change in the number of judgments have crossed targets, the frame segment is removed.

7. The method according to claim 1, characterized in that, A hierarchical classification storage path is constructed based on the action category determination results. Valid sequences for a single person are segmented into action fragments and then associated with action category tags, stored in the target storage node pointed to by the hierarchical classification storage path. This includes: A three-level hierarchical tag tree structure is constructed, with action category tags as the root node of the category tree, rock climbing difficulty level tags as intermediate nodes, and action segment time interval tags as leaf nodes. The tag values ​​of each level from the root node to the leaf node in the three-level hierarchical tag tree structure are concatenated in order according to the directory separator to generate a complete category storage path string from the root directory to the storage file; Get the remaining available storage capacity and write throughput of each candidate storage node in the available candidate storage node set; A load balancing selection function is constructed based on the remaining available storage capacity of each candidate storage node and the current write operation throughput per unit time. The write priority weight of each candidate storage node under the load balancing selection function is calculated, and the candidate storage node with the highest write priority weight is selected as the target storage node. The video data blocks corresponding to each action segment, along with the action category label, climbing difficulty level label, and time interval label metadata associated with that action segment, are persistently written to the file system of the target storage node according to the complete classification storage path.

8. A standardized import system for rock climbing training videos, used for individual screening and structured landing, to implement the method as described in any one of claims 1-7, characterized in that... include: The data preprocessing unit is used to acquire the climbing training video to be processed and the synchronously acquired depth sensing data stream, perform timestamp alignment on the depth sensing data stream and the climbing training video, perform encoding format and frame rate normalization processing on the climbing training video to generate a standardized color frame sequence, and resample the depth sensing data stream according to the normalized frame rate to generate a standardized depth frame sequence. The skeleton tracking unit is used to extract the two-dimensional coordinates of key points of the human target skeleton frame by frame from the standardized color frame sequence, map the two-dimensional coordinates of the key points of the skeleton to three-dimensional spatial coordinates according to the standardized depth frame sequence, construct a three-dimensional motion trajectory vector based on the inter-frame displacement of the three-dimensional coordinates of the same target skeleton key points between consecutive frames, perform single-person continuity and climbing wall spatial constraint verification on the three-dimensional motion trajectory vector, and mark the frame segment that tracks the trajectory to a single climber as a single-person valid sequence. The action recognition unit is used to detect the boundary of rock climbing action based on the temporal change of the three-dimensional displacement energy of the skeletal key points of a single person's valid sequence and extract the start and end frame indexes of each action segment. It performs spatiotemporal feature encoding on the three-dimensional coordinates of the skeletal key points of each action segment to generate action feature descriptors. The action feature descriptors are input into the action classification network to determine the action category. Based on the determination results, a hierarchical classification storage path is constructed. The single person's valid sequence is segmented by action segments and associated with the action category label according to the classification storage path and stored in the target storage node.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images