A method, device and medium for predicting motion trajectory based on deep learning

Abstract

This invention discloses a motion trajectory prediction method, device, and medium based on deep learning, relating to the field of computer vision technology. The method includes: acquiring a set of collected videos; generating a timestamp sequence and a set of labeled frames; and creating a list of bone segment relationships. It also includes: training a keypoint regressor; inferring and generating a sequence of keypoint pixel coordinates; calculating bone length and scale references; obtaining motion start and end thresholds and segmenting motion segments; generating a joint optical flow vector sequence; obtaining an occlusion gating threshold; generating gated joint trajectories; training and embedding a temporal inferrer; inferring and generating unconstrained future trajectories; generating constrained future trajectories; acquiring standard motion videos; generating a set of standard gated joint trajectories and standard constrained future trajectories; obtaining a correction trigger threshold and performing phase alignment; calculating a correction risk index; generating correction prompts and visual overlay content; reducing trajectory jumps while maintaining the continuity of motion temporal sequence; and enhancing the interpretability and practicality of motion guidance.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, device and medium for predicting motion trajectories based on deep learning. Background Technology

[0002] In the field of computer vision, motion trajectory prediction typically uses color video or image sequences as input. It extracts keypoint pixel coordinate sequences through human pose estimation algorithms and combines techniques such as optical flow analysis, temporal modeling, and prediction window segmentation to infer joint motion trends for several future frames. To adapt to scenarios such as real-time interaction, training guidance, and motion evaluation, convolutional neural networks are usually used to extract features. Combined with deep learning models such as recurrent neural networks and temporal convolutional networks, historical motion information is modeled within the review period to output future trajectories, thereby achieving motion trend prediction, real-time feedback, and visualization.

[0003] However, conventional technical approaches still have room for improvement. On the one hand, in videos, occlusion, image blurring, and local false detections cause key point coordinate jumps, reducing the temporal consistency of historical trajectories and increasing the risk of drift in future predicted trajectories. On the other hand, relying solely on data-driven temporal inference cannot guarantee the geometric consistency of human bone segments, and future trajectories may exhibit structural deviations such as non-conservation of bone length, affecting the accuracy of cross-individual scale alignment and reducing the credibility of correction prompts. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a deep learning-based motion trajectory prediction method to solve the problems of unstable key point trajectories caused by occlusion and false detection, and reduced consistency of future trajectory structures due to lack of bone segment geometric constraints in existing technologies.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a motion trajectory prediction method based on deep learning, comprising: acquiring a color video stream, obtaining a set of acquired videos, generating a timestamp sequence and a set of labeled frames, and establishing a list of bone segment relationships; training a keypoint regressor based on the set of labeled frames, inferring a sequence of keypoint pixel coordinates, calculating bone length benchmarks and scale benchmarks, obtaining motion start and end thresholds and segmenting motion segments, generating a sequence of joint optical flow vectors, and obtaining an occlusion gating threshold; establishing a frame-by-frame processing sequence for motion segments, combining the occlusion gating threshold to generate gated joint trajectories, constructing a training sample set for a temporal inferrer, training and solidifying the temporal inferrer, inferring unconstrained future trajectories, performing bone length constraint processing based on the list of bone segment relationships and bone length benchmarks, and generating constrained future trajectories; acquiring standard motion videos, generating a set of standard gated joint trajectories and a set of standard constrained future trajectories, obtaining a correction trigger threshold and performing phase alignment, calculating a correction risk index, and generating correction prompts and visual overlay content.

[0008] As a preferred embodiment of the deep learning-based motion trajectory prediction method of the present invention, the step of formulating the bone segment relationship list includes: fixing the acquisition conditions, setting camera parameters, acquiring acquisition parameter records, acquiring color video streams according to the script, forming an acquired video set, generating a timestamp sequence and verifying it; extracting candidate labeled frames from the acquired video set, labeling key point pixel coordinates and verifying consistency, generating a labeled frame set; and, based on the labeled frame set, formulating and archiving the bone segment relationship list according to the human joint connection relationship, and fixing the playback duration and prediction duration.

[0009] As a preferred embodiment of the deep learning-based motion trajectory prediction method of the present invention, the step of obtaining the action start and end thresholds and segmenting the action segments includes: generating a training sample set based on the labeled frame set, training a keypoint regressor, unifying the inference caliber, and obtaining the trained keypoint regressor and inference caliber file; calling the trained keypoint regressor and inference caliber file to infer the acquired video set frame by frame, inversely transforming the coordinates to the original resolution, and generating a keypoint pixel coordinate sequence; based on the keypoint pixel coordinate sequence and the bone segment relationship list, calculating the bone length benchmark and scale benchmark, statistically analyzing the frame velocity of the stationary segment, generating the action start and end thresholds, and segmenting the action segments.

[0010] As a preferred embodiment of the deep learning-based motion trajectory prediction method of the present invention, the step of obtaining the occlusion gating threshold includes: calculating a dense optical flow field within the action segment, sampling to generate a joint optical flow vector sequence, obtaining a key point displacement vector based on the key point pixel coordinate sequence, calculating a consistency angle, and generating an occlusion gating threshold based on the consistency angle.

[0011] As a preferred embodiment of the deep learning-based motion trajectory prediction method of the present invention, the training sample pair set for constructing the temporal inferrer includes: reading the start and end frame numbers of the action segment, extracting the keypoint pixel coordinate sequence, timestamp sequence, and joint optical flow vector sequence to generate a frame-by-frame processing sequence for the action segment; calculating the consistency angle between the keypoint displacement and optical flow direction based on the frame-by-frame processing sequence of the action segment, combining the occlusion gating threshold, performing a reliable update or propagation update, writing an update marker, and generating a gated joint trajectory; according to the gated joint trajectory, extracting historical windows and future windows according to the review duration and prediction duration to form training sample pairs, performing relativization processing, filtering samples through the action start and end threshold, and generating a training sample pair set for the temporal inferrer.

[0012] As a preferred embodiment of the deep learning-based motion trajectory prediction method of the present invention, the generation of constrained future trajectories includes: dividing the training sample pair set of the temporal inferrer into a training set and a validation set, training the temporal inferrer, and generating a trained temporal inferrer; using the trained temporal inferrer, inferring and generating an unconstrained future trajectory, and combining the bone segment relationship list and bone length benchmark, performing bone segment-by-bone constraint processing to generate a constrained future trajectory.

[0013] As a preferred embodiment of the deep learning-based motion trajectory prediction method of the present invention, the generation of standard gated joint trajectories and standard constrained future trajectory sets includes: acquiring standard motion videos, calling a trained keypoint regressor, inferring and generating standard keypoint pixel coordinate sequences, extracting standard motion segments, generating standard joint optical flow vector sequences, performing gating updates, and obtaining standard gated joint trajectories; filtering the predicted starting point frame sequence number within the standard motion segment, extracting future windows from the standard gated joint trajectories, obtaining standard unconstrained future trajectories, performing bone length constraint processing, and generating a set of standard constrained future trajectories.

[0014] As a preferred embodiment of the deep learning-based motion trajectory prediction method of the present invention, the generation of correction prompts and visual overlay content includes: determining the allowable pixel deviation of the action; calculating the correction trigger threshold based on a scale benchmark; reading the user's gating joint trajectory to generate the user's current posture; matching the user's current posture with a standard candidate posture sequence for phase alignment; indexing the standard constrained future trajectory set; generating a standard future window; reading the user's constrained future trajectory and the standard future window; calculating the pixel coordinate difference of key points frame by frame; generating a correction risk index; comparing it with the correction trigger threshold; generating a correction judgment conclusion; and based on the correction judgment conclusion and the correction risk index, locating the moment of maximum deviation and the prompt key point, generating a correction prompt, drawing an arrow, and generating visual overlay content.

[0015] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the deep learning-based motion trajectory prediction method described in the first aspect of the present invention.

[0016] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the deep learning-based motion trajectory prediction method described in the first aspect of the present invention.

[0017] The beneficial effects of this invention are as follows: by acquiring the occlusion gating threshold, gating joint trajectories are generated, enabling reliable control over key point updates, reducing trajectory jumps and maintaining the continuity of motion sequence, and stabilizing the output motion trend; by processing bone length constraints, constrained future trajectories are obtained, ensuring that the geometric relationship of bone segments remains consistent during the prediction process, thus improving the usability and accuracy of assessment and correction prompts; by using phase alignment and correction risk index, key points requiring correction are located, and correction prompts and visual overlay content are generated, enhancing the interpretability and practicality of motion guidance. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a deep learning-based motion trajectory prediction method.

[0020] Figure 2 This is a flowchart of data acquisition and preprocessing.

[0021] Figure 3 A flowchart for obtaining the occlusion gating threshold.

[0022] Figure 4 A flowchart for generating constrained future trajectories.

[0023] Figure 5 This is a heatmap of bone segment deviations without constrained future trajectories.

[0024] Figure 6 Heatmap of bone segment deviation to constrain future trajectories.

[0025] Figure 7 This is a box plot of bone segment deviation. Detailed Implementation

[0026] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0027] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0028] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0029] Reference Figures 1-7 This is one embodiment of the present invention, which provides a motion trajectory prediction method based on deep learning, including the following steps:

[0030] S1. Acquire color video stream, obtain the acquired video set, generate timestamp sequence and labeled frame set, and formulate bone segment relationship list.

[0031] Set fixed acquisition conditions, set camera parameters, obtain acquisition parameter records, acquire color video streams according to the script, form an acquired video set, generate a timestamp sequence and verify it; extract candidate annotation frames from the acquired video set, annotate the pixel coordinates of key points and verify consistency, and generate an annotation frame set.

[0032] Furthermore, to fix the acquisition conditions, specifically, the 2D camera is mounted on a bracket and locked so that the lens optical axis points to the center of the shooting area; standing lines are affixed to the ground so that participants stand in the same position each time; and indoor lighting is kept stable and background interference is minimized.

[0033] Information such as camera resolution, frame rate, file encoding method, camera installation height, and station position line is summarized and written into the acquisition parameter record. The upper limit of pixel speed is calculated based on the camera resolution and frame rate, and the upper limit of pixel speed is written into the acquisition parameter record. The acquisition parameter record is saved as the basis for this acquisition.

[0034] Specifically, the upper limit of pixel speed is expressed as:

[0035] ;

[0036] in, Indicates the upper limit of pixel speed. This indicates the number of pixels in the image width corresponding to the camera resolution. This indicates the number of pixels in the image height corresponding to the camera resolution. Indicates frame rate.

[0037] Furthermore, the two-dimensional camera is activated to capture color video. Color video streams are captured according to the scripts for static segments, action segments, and static segments. Specifically, the user first stands still for a period of time, then performs the target action, and then stands still for a period of time again. The capture personnel use a unified command to indicate the start and end, and save the video files captured each time in sequence by number, forming a video collection.

[0038] For each video segment in the acquired video set, the timestamp of each frame is recorded synchronously during the acquisition of the color video stream, and written to a file in order of frame number to form a timestamp sequence. When saving, the timestamp sequence file is stored with the same name as the corresponding video file. A consistency check is performed, specifically, checking whether the number of frames in each video file is consistent with the number of timestamps. If the number of frames in a video file is inconsistent with the number of timestamps, the video file is marked as invalid and re-acquired.

[0039] Furthermore, annotation frames are extracted from the collected video sets one by one. Specifically, the extraction strictly covers three types of video segments, and several frames are extracted at equal intervals from the first static segment, the action segment, and the second static segment to form a candidate annotation frame set. The key point pixel coordinates are annotated frame by frame in the candidate annotation frame set, and the annotation file is bound to the frame sequence number for saving. The annotation frames and the annotation key point pixel coordinates are summarized to form an annotation frame set. An annotation consistency check is performed to verify that the same key point name and order are completely consistent in all annotation frames. Any missing annotations are immediately corrected.

[0040] Based on the annotated frame set, a list of bone segment relationships is compiled and archived according to the human joint connection relationship, and the playback duration and prediction duration are fixed.

[0041] Furthermore, based on the keypoint set of the annotated frame set, a bone segment relationship list is formulated according to the connection relationship between adjacent joints in the human body. Specifically, each bone segment is defined as a pair of adjacent keypoint names. The bone segment relationship list is generated by using the keypoint name order and breadth-first traversal rules. All bone segments are arranged and saved according to the bone segment relationship list. The bone segment relationship list and the annotated frame set are archived together.

[0042] The order of key point names comes from the annotation frame set. Specifically, the order of key point names in any annotation frame within the annotation frame set is recorded as the key point name order. An annotation consistency check is performed to ensure that the key point name order remains consistent across all annotation frames.

[0043] The connection relationship between adjacent joints in the human body is represented as a set of adjacent key point name pairs. Each adjacent key point name pair corresponds to a bone segment. The first key point name in the key point name sequence is selected as the root key point name.

[0044] Furthermore, a breadth-first traversal is performed starting from the root keypoint name. Specifically, the queue initially contains only the root keypoint name. After retrieving the parent keypoint name from the queue, a set of candidate child keypoint names connected to the parent keypoint name is collected from the set of adjacent keypoint name pairs. The set of candidate child keypoint names is sorted from front to back according to the keypoint name order. The sorted candidate child keypoint names are enqueued in sequence. The parent keypoint name and each candidate child keypoint name form a parent-child bone segment record in sequence and are written into the bone segment relationship list. Keypoint names that have already been written into the bone segment relationship list during the breadth-first traversal are not enqueued again.

[0045] It should be noted that the order in which bone segment relationships are written is the same as the order in which bone segment relationships are arranged, and the order in which bone segment relationships are arranged remains unique under the same annotation frame set and the same human body adjacent joint connection relationship.

[0046] Furthermore, the playback duration is set to one second and the prediction duration is set to two seconds, and these values ​​are written into the same acquisition caliber file as the acquisition parameter records and saved together. Fixing the playback duration stabilizes the input size of the timing inferr and reduces online inference latency.

[0047] S2. Based on the labeled frame set, train the keypoint regressor, infer and generate the keypoint pixel coordinate sequence, calculate the bone length reference and scale reference, obtain the action start and end thresholds and segment the action segments, generate the joint optical flow vector sequence, and obtain the occlusion gating threshold.

[0048] Based on the labeled frame set, a training sample set is generated, a keypoint regressor is trained, the inference criteria are standardized, and the trained keypoint regressor and inference criteria file are obtained. The trained keypoint regressor and inference criteria file are called to infer the keypoint regressor frame by frame on the acquired video set, the coordinates are inversely transformed to the original resolution, and a keypoint pixel coordinate sequence is generated.

[0049] Furthermore, the image files in the labeled frame set are read frame by frame, and the pixel coordinate label files of key points with the same name or index as the image are read. Consistency checks are performed, specifically checking whether each frame contains all key points, whether the key point names are consistent, and whether the order is consistent. If missing labels or inconsistent order are found, the process returns to the labeling stage to complete or correct them. The image path, corresponding label, and frame number of each frame are written into the training sample list file to form a training sample set.

[0050] Furthermore, the keypoint regressor is trained by standardizing the image input size, scaling each frame of the image according to the inference resolution and padding as necessary, and synchronously transforming the keypoint pixel coordinates of each frame of the image according to the same scaling ratio and padding offset.

[0051] It should be noted that the inference resolution includes the number of pixels in the inference width and the number of pixels in the inference height. The value of the inference width in pixels ranges from 256 pixels to the original resolution width in pixels, and the value of the inference height in pixels ranges from 192 pixels to the original resolution height in pixels.

[0052] A supervision heatmap is generated for each key point. A key point heatmap is formed at the position of the transformed coordinates. The peak of the heatmap is located in the neighborhood of the labeled point, which serves as the training supervision signal.

[0053] The training data includes a training set and a validation set. The training set and validation set are divided from the training sample set by sample number. The training set sample numbers are the first 80% of the sample numbers, and the validation set sample numbers are the last 20% of the sample numbers. The sample numbers are obtained by sorting them in ascending order according to the collection timestamp. The total number of training set samples and the total number of validation set samples are written into the training caliber file.

[0054] A keypoint regressor is constructed, specifically employing a cascaded encoder-decoder convolutional layer structure. The encoder side progressively downsamples to extract features, while the decoder side progressively upsamples to restore spatial resolution and outputs a heatmap for each keypoint. The image is used as input, and the heatmap for each keypoint is used as output. Training is performed by minimizing the mean squared error between the predicted and supervised heatmaps. During training, batch size and the number of training epochs are used as training hyperparameters, written into a training calibrator file. The batch size is an integer value between 4 and 64, and the number of training epochs is an integer value between 20 and 100. During training, a convergence stopping condition is executed based on the mean squared error of the validation set to validate the regression. Training is stopped when the mean squared error of the training set decreases by less than 1% over 10 consecutive training rounds. The number of rounds corresponding to the stop training is written as the training round number to the training calibrator file. After training is completed, the keypoint regressor parameter file is exported, and the inference resolution, scaling ratio, and padding offset are written to the inference calibrator file. The inference calibrator file contains the inference resolution, scaling ratio, and padding offset. The inference calibrator file is bound to the keypoint regressor parameter file and saved together. The training calibrator file is bound to the keypoint regressor parameter file and includes, but is not limited to, batch size, number of training rounds, inference resolution, scaling ratio, padding offset, total number of training set samples, and total number of validation set samples.

[0055] The acquired video set is processed frame by frame. Specifically, a video segment and a sequence of timestamps with the same name are read, and the images are extracted frame by frame. Each frame of the image is scaled and filled according to the inference caliber file, and then input into the keypoint regressor to obtain heatmaps of each keypoint. The peak position of each heatmap is used as the position of the keypoint in the inference resolution coordinate system. According to the inference caliber file, the position is transformed back to the original resolution coordinate system to obtain the original pixel coordinates of the keypoints in the image frame. The frame number, timestamp, and original pixel coordinates of each keypoint are written into a structured record file, and the process is repeated for each frame of the video to obtain the sequence of keypoint pixel coordinates.

[0056] Based on the keypoint pixel coordinate sequence and the bone segment relationship list, calculate the bone length benchmark and scale benchmark, count the frame velocity of the static segment, generate the motion start and end thresholds and segment the motion segment; calculate the dense optical flow field within the motion segment, sample and generate the joint optical flow vector sequence, obtain the keypoint displacement vector based on the keypoint pixel coordinate sequence, calculate the consistency angle, and generate the occlusion gating threshold based on the consistency angle.

[0057] Furthermore, the bone segment relationship list is read to determine the two key points constituting each bone segment. Statistical analysis is performed within the static segment frame range. Specifically, the coordinates of the two ends of the key points in the corresponding frame of the static segment are extracted from the key point pixel coordinate sequence. The pixel distance between the two ends of the bone segment is calculated frame by frame to obtain the bone segment distance sequence. The median of the bone segment distance sequence is used as the bone length benchmark, representing the pixel scale of the human bone segment under the acquisition aperture. The median of the bone length benchmarks of all bone segments is used as the scale benchmark to normalize the deviation to a uniform scale and reduce the impact of changes in standing distance on the consistency of prompt triggering.

[0058] Furthermore, the displacement of each keypoint in adjacent frames is taken frame by frame, and the pixel velocity of the keypoint between adjacent frames is calculated in combination with the time interval between adjacent frames; the median of the pixel velocities of all keypoints in the same frame is calculated to obtain the frame velocity and form a static segment frame velocity sequence.

[0059] Specifically, the pixel velocity of key points between adjacent frames is represented as:

[0060] ;

[0061] in, Indicates the first The pixel velocity of a key point between adjacent frames Indicates the first Key moments Key pixel coordinates, Indicates the first A key point in the previous moment Key pixel coordinates, Indicates the time interval between two adjacent frames, subscript Indicates the key point index, subscript Indicates the frame time index.

[0062] The upper quartile of the frame velocity sequence in the still segment is used as the action start and end threshold, representing the upper limit of still jitter. The frame velocity is calculated frame by frame in the entire video segment. When the frame velocity first exceeds the action start and end threshold, it is marked as the start of the action segment. When the frame velocity falls back and meets the stability condition, it is marked as the end of the action segment. The start and end frame numbers of the action segment are obtained and written to the action segment record file.

[0063] It should be noted that, specifically, the stability condition is as follows: the frame number in which the frame rate first falls below the action start-end threshold is taken as the candidate action segment end frame number. Starting from the candidate action segment end frame number, the timestamp difference is accumulated frame by frame until the accumulated timestamp difference covers the playback duration, forming a stability window. When the frame rate corresponding to each frame in the stability window is lower than the action start-end threshold, the stability condition is determined to be met, and the candidate action segment end frame number is marked as the action segment end.

[0064] The lower limit of the action start and stop threshold is zero, and the upper limit is the pixel speed limit. The lower limit of zero satisfies the non-negative characteristic of frame speed, and the upper limit is the pixel speed limit that covers the maximum observable displacement of the human body in the picture. This suppresses outlier frame speed values ​​caused by timestamp anomalies and key point jumps, ensuring stable action start and stop determination.

[0065] The dense optical flow field of adjacent frames within the frame range of the action segment is calculated. For each frame and each keypoint, a neighborhood window centered on the pixel coordinates of the keypoint is taken. The optical flow vector of each pixel within the neighborhood window is sampled from the dense optical flow field. The size of the neighborhood window is determined by the number of pixels of the neighborhood side length, which is written into the optical flow sampling aperture file. The number of pixels of the neighborhood side length is an odd number between 3 and 15 pixels. The median of the horizontal and vertical components within the neighborhood window is taken to obtain the current frame joint optical flow vector of the keypoint. The joint optical flow vector is in units of pixel displacement and represents the pixel displacement vector from the previous frame to the current frame. The neighborhood sampling and median taking process is repeated for each frame and each keypoint of the action segment, and the results are saved with the corresponding timestamp to form a joint optical flow vector sequence. The optical flow sampling aperture file is bound and saved with the joint optical flow vector sequence. The optical flow sampling aperture file includes, but is not limited to, the number of pixels of the neighborhood side length, the frame range of the action segment, and the identification information of the dense optical flow field calculation method.

[0066] It should be noted that when the neighborhood window exceeds the image boundary, the neighborhood window undergoes boundary truncation processing, after which the neighborhood window completely falls within the image boundary range; when the number of pixels in the neighborhood window is less than 1 pixel, the joint optical flow vector is set to zero and an anomaly marker is written.

[0067] Furthermore, the consistency angle is calculated frame by frame. Specifically, the displacement vector of each key point in the adjacent frames and the joint optical flow vector in the current frame are taken, and the angle between them is calculated. The larger the angle, the more inconsistent the displacement direction of the key point is with the direction of image motion, and the more likely there will be occlusion false detection or skipping points.

[0068] Specifically, from a consistency perspective, this is expressed as:

[0069] ;

[0070] in, Indicates the first A key point at a moment From the perspective of consistency, Indicates the first The key point is the joint optical flow vector at time t. This represents the inverse cosine function.

[0071] It should be noted that, in order to avoid instability of the included angle due to small displacements, only frames with frame rates not lower than the action start and end thresholds are included in the angle statistics; after collecting angle samples of all included frames in the action segment, the upper quartile of the angle samples is used as the occlusion gating threshold.

[0072] The occlusion gating threshold has a lower limit of 0 and an upper limit of π. A lower limit of 0 indicates no directional deviation, while an upper limit of π covers the entire range of the inverse cosine output angle and avoids numerical overflow caused by the vector norm being close to 0, thus ensuring stable occlusion gating judgment.

[0073] S3. Establish a frame-by-frame processing sequence for action segments, combine it with occlusion gating thresholds to generate gating joint trajectories, construct a training sample set for the temporal inferrer, train and solidify the temporal inferrer, infer and generate unconstrained future trajectories, perform bone length constraint processing based on the bone segment relationship list and bone length benchmark, and generate constrained future trajectories.

[0074] Read the start and end frame numbers of the action segment, extract the key point pixel coordinate sequence, timestamp sequence, and joint optical flow vector sequence to generate the frame-by-frame processing sequence of the action segment; based on the frame-by-frame processing sequence of the action segment, calculate the consistency angle of key point displacement and optical flow direction, combine with the occlusion gating threshold, perform reliable update or propagation update, write the update flag, and generate the gating joint trajectory.

[0075] Furthermore, the action segment is read, and the start frame number and end frame number corresponding to the action segment are obtained. Using the frame number range of the action segment, the key point pixel coordinates of the corresponding frame are extracted frame by frame from the key point pixel coordinate sequence to form key point data fragments within the action segment, and the frame number of each frame is retained.

[0076] Using the same frame number range, extract the timestamp of the corresponding frame from the timestamp sequence frame by frame, and check whether the timestamp increases monotonically with the frame number; if timestamps are found to be backtracking, duplicated, or missing, mark the video segment as invalid and return to the acquisition stage for re-acquisition.

[0077] Using the same frame number range, extract the joint optical flow vector of the corresponding frame from the joint optical flow vector sequence frame by frame, and check whether the corresponding joint optical flow vector exists for each key point of the corresponding frame; if the corresponding frame is missing a joint optical flow vector, mark the corresponding frame as unusable, fall back to the optical flow generation stage, and recalculate the frame pair corresponding to the frame with missing joint optical flow vector.

[0078] For each frame within the action segment, keypoint pixel coordinates, joint optical flow vectors, and timestamps are generated and stored in ascending order of frame number, forming a frame-by-frame processing sequence for the action segment.

[0079] Create a gated joint trajectory file. Write the key point coordinates into the gated joint trajectory file in frame sequence. Write the key point pixel coordinates of the first frame of the action segment into the gated joint trajectory file. Use the gate coordinates of the first frame of the action segment as the initial coordinates for propagation and update.

[0080] Starting from the second frame of the action segment, for each frame and each key point, gating judgment and gating update are performed. Specifically, the pixel coordinates of the key point in the current frame and the pixel coordinates of the key point in the previous frame are read, the displacement direction is calculated, the joint optical flow vector in the current frame is read, the optical flow direction is extracted, and the consistency angle between the displacement direction and the optical flow direction is calculated. The consistency angle calculation aperture is consistent with the occlusion gating threshold generation aperture.

[0081] Furthermore, the consistency angle is compared with the occlusion gating threshold. If the consistency angle does not exceed the occlusion gating threshold, the pixel coordinates of the key point in the current frame are written as the gating coordinates of the current frame.

[0082] If the consistency angle exceeds the occlusion gating threshold, a propagation update is performed. Specifically, the gating coordinates of the previous frame and the joint optical flow vector of the current frame are read. The propagation update uses the joint optical flow vector as the pixel displacement, superimposes the joint optical flow vector onto the gating coordinates of the previous frame, and obtains the gating coordinates of the current frame.

[0083] Write an update tag for each frame. The update tag value can be either a reliable update or a propagation update. The update tag is used for training sample selection and quality backtracking.

[0084] Based on the gating joint trajectory, historical and future windows are extracted according to the review duration and prediction duration to form training sample pairs. Relativization processing is performed, and samples are filtered through action start and end thresholds to generate a training sample pair set for the timing inferrer. The training sample pair set of the timing inferrer is divided into a training set and a validation set to train the timing inferrer and generate the trained timing inferrer.

[0085] Furthermore, the playback duration and prediction duration are read, and the timestamp records within the action segment frame range are read. The historical window and future window are determined by accumulating timestamps. Specifically, for the candidate starting frame number, the frame number is traced backward from the candidate starting frame number, and the timestamp difference is accumulated until it covers the playback duration to obtain the historical window frame set; for the candidate starting frame number, the frame number is advanced backward from the candidate starting frame number, and the timestamp difference is accumulated until it covers the prediction duration to obtain the future window frame set.

[0086] The gating trajectory segments corresponding to the historical window frame set of the gating joint trajectory are extracted as input samples, and the gating trajectory segments corresponding to the future window frame set of the gating joint trajectory are extracted as output samples to form training sample pairs.

[0087] The training sample pairs are subjected to relativization processing. Specifically, the gating coordinates of the last frame of the historical window are used as the reference coordinates, and the gating trajectory segments of the historical window and the gating trajectory segments of the future window are converted into relative displacement sequences.

[0088] The candidate starting frame number is filtered using an action start and end threshold. Only the training sample pairs corresponding to the candidate starting frame number that is in an action state are retained, reducing the proportion of static micro-motion samples. Specifically, the action start and end threshold is used to filter the training sample pairs. The frame velocity near the candidate starting frame is calculated. When the frame velocity continuously exceeds the action start and end threshold, the training sample pair corresponding to the candidate starting frame number is retained. When the frame velocity does not continuously exceed the action start and end threshold, the training sample pair is discarded, generating a training sample pair set for the timing inferrer.

[0089] The timing inferrer is trained and solidified. The input of the timing inferrer is the relative displacement sequence of the historical window, and the output is the relative displacement sequence of the future window. The pixel coordinate recovery rule is the reference coordinate back-addition rule, and the reference coordinate is the gated coordinate of the last frame of the historical window.

[0090] The temporal inferr structure is a causal one-dimensional convolutional layer concatenation structure. The causal one-dimensional convolutional layer concatenation structure only uses past frame information to satisfy real-time inference. The number of causal one-dimensional convolutional layers and the length of the convolutional kernel are written as structural hyperparameters into the inference solidification configuration file. The inference solidification configuration file is used to ensure that the training caliber is consistent with the deployment caliber.

[0091] During the training phase, the training sample set is divided into a training set and a validation set. The training set is used for parameter updates, and the validation set is used for version selection. The training process uses the mean square criterion of frame-by-frame error for optimization. The optimization process uses batch size and number of training epochs as training hyperparameters. The batch size and number of training epochs are written into the inference solidification configuration file. The batch size is an integer value between 4 and 64, and the number of training epochs is an integer value between 20 and 100. During training, a stopping condition is executed based on the average error of the validation set. Training stops when the average error of the validation set decreases by less than 1% in 10 consecutive training epochs. The number of epochs corresponding to the stop training is written into the inference solidification configuration file as the number of training epochs. After the training epochs are completed, the average error is calculated on the validation set, and the version with the smallest average error is saved as the solidified version.

[0092] Export the time series inferr parameter file and the inference solidification configuration file. The inference solidification configuration file includes, but is not limited to, the historical window determination rules, the future window determination rules, the relativization processing rules, the pixel coordinate recovery rules, the number of causal one-dimensional convolutional layers, the convolutional kernel length, the batch size, the number of training rounds, the stopping condition parameters, the number of training set sample pairs, and the number of validation set sample pairs. The time series inferr parameter file and the inference solidification configuration file are bound and saved together.

[0093] The trained temporal inferrer infers and generates unconstrained future trajectories. By combining the list of bone segment relationships and bone length benchmarks, it performs segment-by-segment constraint processing to generate constrained future trajectories.

[0094] Furthermore, during real-time operation, the most recent historical window segment that meets the review duration is extracted from the latest gating joint trajectory, and converted into a relative displacement sequence according to the inferred solidified configuration file. The relative displacement sequence is then sent to the time series inferrer to obtain the future relative displacement sequence that meets the prediction duration.

[0095] Using the gating coordinates of the last frame of the historical window as a reference, the future relative displacement is added back to restore the key point pixel coordinate sequence of the future window, and written to the file in chronological order to form an unconstrained future trajectory.

[0096] Read the bone segment relationship list and bone length reference. Read the parent and child keypoint names from the bone segment relationship list one by one, using the order of the bone segment relationship list as the order of bone segment constraint processing. Perform bone segment constraint in each future frame. Specifically, read the pixel coordinates of all keypoints in the future frame as initial values ​​from the unconstrained future trajectory. Process each bone segment one by one according to the order of the bone segment relationship list. For each bone segment, first obtain the parent keypoint coordinates, then the child keypoint coordinates, and calculate the direction vector from the parent keypoint to the child keypoint. Keep the direction vector from the parent keypoint to the child keypoint unchanged, ensuring that the pixel distance from the parent keypoint to the child keypoint is equal to the pixel distance of the corresponding bone segment on the bone length reference. Obtain the constrained child keypoint coordinates, write the adjusted child keypoint pixel coordinates into the constrained future trajectory frame record, and continue processing the next bone segment connected to the child keypoint using the constrained child keypoint coordinates.

[0097] It should be noted that the order of the bone segment relationship list comes from the output order of the breadth-first traversal. The parent key point name enters the bone segment relationship list before the child key point name. The coordinate propagation direction of the bone segment constraint processing process is consistent with the parent-child relationship.

[0098] Specifically, the coordinates of the sub-keypoints after bone length constraint are represented as follows:

[0099] ;

[0100] in, Indicates time Coordinates of subkeypoints after bone length constraint Indicates time Sub-keypoint pixel coordinates, Indicates time The parent keypoint pixel coordinates, Indicates the bone length reference of the corresponding bone segments of the parent and child keypoints, subscript Index of the parent keypoint, subscript Indicates the index of the subkeypoint.

[0101] Repeat the bone-segment constraint for each frame within the future window, obtain the sequence of key point pixel coordinates after constraint for each frame, and save them in chronological order as the constrained future trajectory.

[0102] S4. Collect standard motion videos, generate standard gating joint trajectories and standard constrained future trajectory sets, obtain correction trigger thresholds and perform phase alignment, calculate correction risk index, and generate correction prompts and visual overlay content.

[0103] Acquire standard action videos, call the trained keypoint regressor, infer and generate standard keypoint pixel coordinate sequences, extract standard action segments, generate standard joint optical flow vector sequences, perform gating updates, and obtain standard gated joint trajectories; filter the predicted starting frame sequence number within the standard action segment, extract future windows from the standard gated joint trajectories, obtain standard unconstrained future trajectories, perform bone length constraint processing, and generate a set of standard constrained future trajectories.

[0104] Furthermore, standard action videos are collected. Specifically, the camera installation method, shooting area, position line position, resolution, and frame rate recorded in the acquisition parameters are reused. The instructor stands at the position line and completes the target action according to the script of static segment, action segment, and static segment. The color video stream is collected and saved as a standard action video. During the acquisition process, a monotonically increasing timestamp is written to each frame to form a timestamp sequence that corresponds one-to-one with the frame number. The timestamp sequence is saved with the same name as the standard action video. A consistency check is performed to verify whether the number of frames in the standard action video matches the number of entries in the timestamp sequence. If they do not match, the standard action video is deemed invalid and re-acquired. Thirty standard action video segments are repeatedly collected, numbered sequentially, and saved to form a standard action video set. Standard action video set number 1 is written into the baseline standard action video marker, and standard action video sets numbered 2 to 30 are written into the statistical standard action video marker.

[0105] The trained keypoint regressor is invoked to infer frame-by-frame from the standard motion video. Specifically, the standard motion video images are read frame by frame, scaled and padded according to the inference calibrator file, and then input into the keypoint regressor to output a keypoint heatmap, with the peak position used as the inference coordinate. The inference coordinates are then transformed back to the original resolution coordinate system according to the inference calibrator file to obtain the keypoint pixel coordinates of the standard motion video image frames. The keypoint pixel coordinates of each frame are bound to the timestamp of the same frame and written into a structured record file to form a sequence of standard keypoint pixel coordinates.

[0106] Obtain the standard action segment. Specifically, reuse the action start and end thresholds, calculate the frame rate in the static segment area of ​​the standard video according to the same caliber, calculate the frame rate frame by frame in the entire video range, mark the entry into the action segment when the frame rate exceeds the action start and end thresholds, and mark the exit from the action segment when the frame rate drops and meets the stability condition, and obtain the start and end frame numbers of the standard action segment.

[0107] It should be noted that, specifically, the stability condition is determined by using the frame number where the frame rate first falls below the action start-end threshold as the candidate standard action segment endpoint frame number. Starting from the candidate standard action segment endpoint frame number, the timestamp difference is accumulated frame by frame until the accumulated timestamp difference covers the playback duration, forming a standard stability window. When the frame rate corresponding to each frame within the standard stability window is lower than the action start-end threshold, the stability condition is determined to be met, and the candidate standard action segment endpoint frame number is marked as the standard action segment endpoint.

[0108] For adjacent frames within the standard action segment, a dense optical flow field is calculated. For each frame and each keypoint, a neighborhood window is taken centered on the keypoint's pixel coordinates. The optical flow vector within the neighborhood window is sampled from the dense optical flow field. The median of the horizontal and vertical components is taken to obtain the standard joint optical flow vector of the keypoint in the image frame. The size of the neighborhood window is determined by the number of pixels of the neighborhood side length, which is written into the inference caliber file. The number of pixels of the neighborhood side length is an odd number between 3 and 15 pixels. This process is repeated for all frames and all keypoints and saved to form a standard joint optical flow vector sequence.

[0109] It should be noted that when the neighboring window exceeds the image boundary, the neighboring window undergoes boundary truncation processing, after which the neighboring window falls within the image boundary range.

[0110] Generate a standard gating joint trajectory. Specifically, write the key point pixel coordinates of the first frame of the standard action segment into the standard gating joint trajectory as the initial value. For each subsequent frame and each key point, calculate the consistency angle between the adjacent frame displacement direction of the key point and the optical flow direction of the same frame, and compare it with the occlusion gating threshold.

[0111] If the consistency angle does not exceed the occlusion gating threshold, the key point detection is considered reliable, and the standard gating joint trajectory is updated using the key point pixel coordinates. If the consistency angle exceeds the occlusion gating threshold, there is a risk of false detection due to occlusion. The key point coordinates in the standard gating joint trajectory of the previous frame and the standard joint optical flow vector of the key point in the current frame are read. The short-time propagation update uses the standard joint optical flow vector as the pixel displacement, superimposes the standard joint optical flow vector onto the gating coordinates of the previous frame, obtains the gating coordinates of the current frame, and writes them into the standard gating joint trajectory to achieve short-time propagation update.

[0112] Within the standard action segment, frame numbers that can serve as prediction start points are selected frame by frame. The selection criteria are to determine the future window frame set by accumulating timestamps from the prediction start frame number onwards. The cumulative timestamp difference corresponding to the future window frame set covers the prediction duration. For each prediction start point, the gated trajectory segment corresponding to the future window frame set after the prediction start frame number is extracted from the standard gated joint trajectory to obtain the standard unconstrained future trajectory.

[0113] It should be noted that the timestamp accumulation method specifically involves advancing the frame number backward from the prediction start frame number, calculating the timestamp difference between adjacent frames frame by frame, and accumulating the timestamp difference. When the accumulated timestamp difference first covers the prediction duration, the frame number corresponding to the predicted duration is recorded as the future window termination frame number. The future window frame set is the set of frame numbers from the prediction start frame number to the future window termination frame number. The number of frames in the future window frame set is the future window termination frame number minus the prediction start frame number. If the accumulated timestamp difference cannot be obtained to cover the prediction duration even after advancing the prediction start frame number to the standard motion video termination frame number, the prediction start frame number is not included in the prediction start frame number set.

[0114] For the standard unconstrained future trajectory, bone length constraint processing is performed frame by frame. Specifically, the bone segment relationship list is read to determine the bone segment connection relationship and traversal order. For each bone segment, the child keypoint is projected along the direction from the parent keypoint to the child keypoint to the distance corresponding to the bone length reference, so that the distance between the keypoints at both ends of the bone segment fits the bone length reference, and the standard constrained future trajectory is obtained. All predicted starting points are repeatedly truncated and constrained, and the predicted starting point frame number is used as an index to save the standard constrained future trajectory set.

[0115] The permissible pixel deviation for the action is determined, and a correction trigger threshold is calculated based on a scale benchmark. The user's current pose is generated by reading the user's gating joint trajectory. The user's current pose is matched with a standard candidate pose sequence for phase alignment. The standard constrained future trajectory set is indexed to generate a standard future window. The user's constrained future trajectory and standard future window are read, and the pixel coordinate difference of key points is calculated frame by frame to generate a correction risk index. This index is compared with the correction trigger threshold to generate a correction judgment conclusion. Based on the correction judgment conclusion and the correction risk index, the moment of maximum deviation and the key point for prompting are located, a correction prompt is generated, an arrow is drawn, and a visual overlay content is generated.

[0116] Furthermore, the allowable pixel deviation corresponding to the allowable deviation range of the action is determined, the allowable pixel deviation is normalized using the scale reference, and the ratio of the allowable pixel deviation to the scale reference is calculated as the correction trigger threshold.

[0117] Specifically, determining the allowable pixel deviation for an action involves reading a set of standard action videos, reading the benchmark standard action video and marking the corresponding standard action video, calling the trained keypoint regressor to infer frame by frame, generating a benchmark keypoint pixel coordinate sequence; extracting standard action segments based on the benchmark keypoint pixel coordinate sequence to generate benchmark gated joint trajectories; and extracting future windows based on the benchmark gated joint trajectories, performing bone length constraint processing, and generating a set of benchmark constrained future trajectories.

[0118] Read the standard action video set, read the statistical standard action video tag corresponding to the standard action video one by one, and perform frame-by-frame inference through the trained keypoint regressor to generate a statistical keypoint pixel coordinate sequence; extract the standard action segment from the statistical keypoint pixel coordinate sequence to generate a statistical gated joint trajectory; extract the future window through the statistical gated joint trajectory, perform bone length constraint processing, and generate a statistically constrained future trajectory.

[0119] Read the key point pixel coordinates of the statistical gating joint trajectory at the current moment to generate the statistical current pose; read the key point pixel coordinates of the baseline gating joint trajectory in each frame of the standard action segment to generate the baseline candidate pose sequence; calculate the pose distance frame by frame for the baseline candidate pose sequence, and take the standard frame with the smallest pose distance as the alignment frame.

[0120] Read the frame number corresponding to the alignment frame, index the standard constraint future trajectory in the baseline constraint future trajectory set, and generate a standard future window; read the statistical constraint future trajectory and generate a user future window; the length of the standard future window is the same as the length of the user future window.

[0121] Calculate the difference in keypoint pixel coordinates between the user's future window and the standard future window frame by frame to obtain the future window difference sequence. Write the pixel value with the largest difference in the future window difference sequence into the maximum difference record table. For each statistical standard action video segment, generate the user's future window and the standard future window in sequence. Repeat the actions of generating the future window difference sequence and writing the maximum difference record table until the maximum difference record table covers all statistical standard action videos in the standard action video set.

[0122] Sort the maximum difference pixel values ​​in the maximum difference record table from smallest to largest, and take the 75th percentile of the maximum difference pixel values ​​as the statistical allowable pixel deviation; if the statistical allowable pixel deviation is greater than the scale reference, then the scale reference is used as the motion allowable pixel deviation; if the statistical allowable pixel deviation is less than the scale reference, then the statistical allowable pixel deviation is used as the motion allowable pixel deviation.

[0123] It should be noted that the lower limit of the correction trigger threshold is 0 and the upper limit is 1. The lower limit of 0 satisfies the non-negative characteristic of the ratio of the allowable pixel deviation of the action to the scale reference. The upper limit of 1 limits the allowable pixel deviation of the action to not exceed the scale reference, so as to avoid the correction trigger threshold being abnormally amplified due to the scale reference being too small or the key point jump, thus ensuring the stability of the correction trigger judgment and reducing the frequency of false triggers.

[0124] Furthermore, phase alignment is performed. Specifically, the pixel coordinates of key points of the gating joint trajectory at the current moment on the user side are read as the user's current pose; the pixel coordinates of key points of the standard gating joint trajectory in each frame within the standard action segment are read as the standard candidate pose sequence.

[0125] The pose distance is calculated frame by frame for the standard candidate pose sequence. Specifically, the Euclidean distance between each key point and the key point corresponding to the user's current pose is calculated, and the median of the Euclidean distance is taken as the pose distance. The standard frame with the smallest pose distance is selected as the alignment frame.

[0126] Furthermore, the corresponding standard constrained future trajectory is indexed in the standard constrained future trajectory set using the frame number of the aligned frame, and used as the standard future window for this comparison.

[0127] The user-side constrained future trajectory is read as the user's future window, and the standard constrained future trajectory is read as the standard future window. Both have the same length and are determined by the prediction duration, corresponding one-to-one frame by frame.

[0128] The difference between the keypoint pixel coordinates of the user's future window and the standard future window is calculated frame by frame. The difference is calculated for each future time, and the maximum difference is taken within the entire future window. The maximum difference is normalized using a scale benchmark to obtain the correction risk index.

[0129] Specifically, the corrective risk index is expressed as:

[0130] ;

[0131] in, This indicates a corrected risk index. Represents the set of moments in a future window. Represents the set of keypoint indices. This indicates that the user is constraining the future trajectory at time [time]. The Key pixel coordinates, This indicates that the standard constraint on the future trajectory is at time [time]. The Key pixel coordinates, Indicates the scale reference.

[0132] It should be noted that a higher corrected risk index indicates a more significant structural bias in the future window.

[0133] Furthermore, the correction risk index is compared with the correction trigger threshold. If the correction risk index is greater than or equal to the correction trigger threshold, it is determined that a correction prompt is needed; if the correction risk index is less than the correction trigger threshold, it is determined that a prompt or positive incentive is needed.

[0134] Furthermore, when it is determined that correction is needed, the system locates the object to be prompted, finds the moment when the difference reaches its maximum within the future window, and uses the key point with the largest time difference as the prompt key point; it generates direction information, specifically, it reads the coordinate difference between the prompt key point in the user's future window and the standard future window, determines the left, right, up, and down directions according to the positive and negative values ​​of the horizontal and vertical components, and maps the prompt key point name to a short name of a human body part to form a complete prompt text.

[0135] Furthermore, visual overlay content is generated. Specifically, the pixel coordinates of the prompt key points of the user's gating joint trajectory at the current moment are used as the starting point of the arrow, and the pixel coordinates of the prompt key points of the standard constrained future trajectory at the moment of maximum deviation are used as the target point of the arrow. An arrow line segment pointing from the starting point to the target point is drawn on the screen, and prompt text is displayed next to the arrow. At the same time, the key point trajectory lines of the standard future trajectory and the key point trajectory lines of the user's future trajectory are overlaid, distinguished by different line types, to help the user understand the difference between the present and the target.

[0136] In this embodiment, simulation experiments were conducted to verify the synergistic effect of occlusion gating threshold and bone length constraint in improving future trajectory stability and enhancing the usability and accuracy of correction prompts. Specifically, a monocular video acquisition and offline calculation method was used for the simulation experiment. The detection environment included a monocular camera device, a fixed bracket, a continuous illumination source, a non-reflective background, and a flat ground. The monocular camera device was mounted on the fixed bracket with the lens optical axis facing the front of the subject. Key points of the subject's entire body were kept within the frame range. The acquisition frame rate was set to 30 frames per second, and the image resolution was set to 1280. 720; To cover objective disturbances such as occlusion and noise, four occlusion levels (no, light, medium, and heavy) are set. By combining the occlusion occurrence probability, occlusion duration in frames, coordinate noise intensity, and jump point probability, a repeatable, controllable, and comparable detection environment is formed. Multiple samples are automatically generated under each occlusion level, and comparisons are made under four configurations: full enable, occlusion gating threshold disabled, bone length constraint processing disabled, and phase alignment disabled. Quantifiable data such as relative bone segment deviation and 95th percentile of bone segment deviation are output.

[0137] Figure 5 This is a heatmap of relative deviations of bone segments without constrained future trajectories. The relative deviation of a bone segment is the normalized difference between the pixel distance between keypoints at both ends of a bone segment and the bone length baseline within the future window. Specifically, in each frame of the future window, the pixel distance between keypoints at both ends of a bone segment is calculated line by line according to the bone segment relationship list. The absolute value of the difference between the pixel distance and the bone length baseline is calculated, and the relative deviation of the bone segment is obtained by calculating the ratio of the absolute value of the difference to the bone length baseline. Figure 5 The shade of color indicates the relative deviation of bone segments. Lighter areas correspond to areas where the geometric consistency of bone segments decreases. Unconstrained future trajectories are prone to bone segment length drift under the influence of occlusion and jump points. Bone segment length drift amplifies the uncertainty of the source of difference in the calculation of the correction risk index.

[0138] Figure 6 To constrain the future trajectory, a heatmap of relative deviations of bone segments is generated, wherein the calculation process for relative deviations of bone segments is similar to... Figure 5 To maintain consistency, the computational object is switched to constraining future trajectories. Figure 6 The overall lighter color indicates that the relative deviation of bone segments has converged. Bone length constraint processing ensures that the geometric relationship of bone segments remains consistent during the prediction process. Improved consistency of bone segment geometric relationship reduces the artificial increase in difference caused by skeletal distortion, and simultaneously improves the stability of correction prompt pointing and correction prompt position.

[0139] Figure 7The box plot of bone segment deviation distribution is used. The bone segment deviation comes from the 95th percentile position value of the bone segment relative deviation sample set. Specifically, to obtain the bone segment relative deviation sample set, the occlusion level and video sample number are fixed. Future window trajectories are generated under four experimental configurations: full enable, occlusion gating threshold disabled, bone length constraint processing disabled, and phase alignment disabled. According to the bone segment relationship list, the pixel distance between the key points at both ends of the bone segment is calculated for each frame of the future window. The relative deviation of the bone segment is calculated. The relative deviation of the bone segment for all frames and all bone segments in the future window is collected to form the bone segment relative deviation sample set. The bone segment relative deviation sample set is sorted, and the value at the 95th percentile position after sorting is taken as the bone segment deviation. Figure 7 Each box plot corresponds to a test configuration. The upper and lower boundaries of the box reflect the interquartile range, the median line reflects the median level, and outliers reflect extreme deviation samples. Figure 7 The distribution corresponding to the discontinuation of bone length constraint treatment usually shifts upward and spreads, indicating that the consistency of bone segment geometric relationships is difficult to maintain within the future window; the distribution corresponding to the full activation usually shifts downward and converges, indicating that bone length constraint treatment has a stabilizing and suppressive effect on high-level deviations.

[0140] This embodiment also provides a computer device applicable to motion trajectory prediction methods based on deep learning, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the motion trajectory prediction method based on deep learning as proposed in the above embodiment.

[0141] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0142] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the deep learning-based motion trajectory prediction method proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0143] In summary, this invention achieves reliable control over key point updates by acquiring occlusion gating thresholds and generating gated joint trajectories, reducing trajectory jumps and maintaining the continuity of motion sequence, thus stabilizing the output of movement trends. Through bone length constraint processing, it obtains constrained future trajectories, ensuring that the geometric relationships of bone segments remain consistent during the prediction process, improving the usability and accuracy of assessment and correction prompts. Furthermore, through phase alignment and correction risk indices, it locates key points requiring correction, generates correction prompts and visual overlay content, enhancing the interpretability and practicality of motion guidance.

[0144] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A motion trajectory prediction method based on deep learning, characterized in that: include, Acquire color video streams, obtain the acquired video set, generate timestamp sequences and labeled frame sets, and formulate a list of bone segment relationships; Based on the labeled frame set, a keypoint regressor is trained, a sequence of keypoint pixel coordinates is generated through inference, bone length reference and scale reference are calculated, action start and end thresholds are obtained and action segments are segmented, a sequence of joint optical flow vectors is generated, and occlusion gating thresholds are obtained. Establish a frame-by-frame processing sequence for action segments, combine it with occlusion gating thresholds to generate gating joint trajectories, construct a training sample set for the temporal inferrer, train and solidify the temporal inferrer, infer and generate unconstrained future trajectories, perform bone length constraint processing based on the bone segment relationship list and bone length benchmark, and generate constrained future trajectories. Collect standard motion videos, generate standard gating joint trajectories and standard constrained future trajectory sets, obtain correction trigger thresholds and perform phase alignment, calculate correction risk index, and generate correction prompts and visual overlay content; The training sample set for constructing the timing inferr includes reading the start and end frame numbers of the action segment, extracting the key point pixel coordinate sequence, timestamp sequence, and joint optical flow vector sequence, and generating a frame-by-frame processing sequence for the action segment. Based on the frame-by-frame processing sequence of action segments, the consistency angle of key point displacement and optical flow direction is calculated. Combined with the occlusion gating threshold, a reliable update or propagation update is performed, an update marker is written, and a gating joint trajectory is generated. Based on the gating joint trajectory, according to the review duration and prediction duration, historical windows and future windows are extracted to form training sample pairs. Relativization processing is performed, and samples are filtered through action start and end thresholds to generate a training sample pair set for the time series inferr. The process of generating standard gated joint trajectories and standard constrained future trajectory sets includes: acquiring standard motion videos, calling the trained keypoint regressor, inferring and generating standard keypoint pixel coordinate sequences, extracting standard motion segments, generating standard joint optical flow vector sequences, performing gating updates, and obtaining standard gated joint trajectories. Filter the predicted starting frame sequence number within the standard action segment, extract the future window from the standard gated joint trajectory, obtain the standard unconstrained future trajectory, perform bone length constraint processing, and generate a set of standard constrained future trajectories.

2. The motion trajectory prediction method based on deep learning as described in claim 1, characterized in that: The process of creating the bone segment relationship list includes fixing acquisition conditions, setting camera parameters, acquiring acquisition parameter records, acquiring color video streams according to the script, forming an acquired video set, generating a timestamp sequence and verifying it. Candidate labeled frames are extracted from the acquired video set, key point pixel coordinates are labeled and consistency is verified to generate a set of labeled frames; Based on the annotated frame set, a list of bone segment relationships is created and archived according to the human joint connection relationship, and the playback duration and prediction duration are fixed.

3. The motion trajectory prediction method based on deep learning as described in claim 1, characterized in that: The process of obtaining the start and end thresholds of actions and segmenting action segments includes generating a training sample set based on the labeled frame set, training a keypoint regressor, unifying the inference criteria, and obtaining the trained keypoint regressor and inference criteria files. Call the trained keypoint regressor and inference caliber file, perform frame-by-frame inference on the acquired video set, inversely transform the coordinates to the original resolution, and generate a sequence of keypoint pixel coordinates. Based on the keypoint pixel coordinate sequence and the bone segment relationship list, calculate the bone length benchmark and scale benchmark, count the frame velocity of the static segment, generate the motion start and end thresholds, and segment the motion segment.

4. The motion trajectory prediction method based on deep learning as described in claim 3, characterized in that: The process of obtaining the occlusion gating threshold includes: calculating a dense optical flow field within the action segment, sampling to generate a joint optical flow vector sequence, obtaining a key point displacement vector based on the key point pixel coordinate sequence, calculating a consistency angle, and generating an occlusion gating threshold based on the consistency angle.

5. The motion trajectory prediction method based on deep learning as described in claim 3, characterized in that: The process of generating constrained future trajectories includes dividing the training sample set of the time series inferr into a training set and a validation set, training the time series inferr, and generating the trained time series inferr. The trained temporal inferrer infers and generates unconstrained future trajectories. By combining the list of bone segment relationships and bone length benchmarks, it performs segment-by-segment constraint processing to generate constrained future trajectories.

6. The motion trajectory prediction method based on deep learning as described in claim 1, characterized in that: The generation of correction prompts and visualization overlay content includes: determining the allowable pixel deviation of the action, calculating the correction trigger threshold in combination with the scale reference, reading the user's gating joint trajectory to generate the user's current posture, matching the user's current posture with the standard candidate posture sequence for phase alignment, indexing the standard constrained future trajectory set, and generating a standard future window. Read the user's constrained future trajectory and standard future window, calculate the key point pixel coordinate difference frame by frame, generate a correction risk index, compare it with the correction trigger threshold, and generate a correction judgment conclusion. Based on the correction judgment conclusion and correction risk index, locate the moment of maximum deviation and key points for prompting, generate correction prompts, draw arrows, and generate visual overlay content.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the deep learning-based motion trajectory prediction method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the deep learning-based motion trajectory prediction method according to any one of claims 1 to 6.

Images