Method for recognizing motion posture and fusing semantic features based on deep learning

Abstract

This invention provides a method for motion posture recognition and semantic feature fusion based on deep learning, comprising: acquiring videos of ball sports in colleges and universities; after unified preprocessing, generating student target tracking sequences and motion target tracking sequences based on a deep learning image recognition model, and extracting human key point sequences, motion target trajectories, and spatiotemporal relationships between humans and the ball; determining candidate time windows before contact based on the spatiotemporal relationships between humans and the ball, identifying body orientation, supporting foot direction, arm trajectory, center of gravity preset state, and force chain initiation sequence, and generating semantic features of pre-contact intent; within the contact search time window, determining the moment of contact with the ball based on distance convergence, velocity change, direction change, and nearest neighbor contact relationships, and generating contact semantic features at the moment of contact; determining the post-contact result time window based on the moment of contact with the ball, extracting motion target result features and human body reset result features, and generating semantic features of the post-contact result.

Description

Technical Field

[0001] This invention relates to the field of video image recognition technology, and in particular to a method for motion pose recognition and semantic feature fusion based on deep learning. Background Technology

[0002] With the digital development of college sports, motion posture recognition methods based on video images are gradually being used in ball game teaching scenarios. Typically, video cameras are used to capture student movement videos, and image recognition models are used to extract key points on the human body or the trajectory of the moving target to determine whether the student has completed actions such as shooting, passing, or swinging the ball. However, in ball games, the correctness of movement technique depends not only on the posture at the moment of contact with the ball, nor solely on whether the moving target has entered the valid area.

[0003] The student's body orientation, supporting foot direction, center of gravity pre-position, and power chain activation sequence before contacting the ball will affect the contact point, contact direction, and contact intensity at the moment of contact. The ball's trajectory, landing point, and the student's recovery posture after contact also reflect whether the action was completed using the correct technical path.

[0004] Existing technologies often process human posture recognition, moving target trajectory recognition, and action result judgment separately, or evaluate solely based on ball contact events, ball trajectory results, and action counts. This can easily lead to misinterpreting a successful result as correct technique, and makes it difficult to identify accidental successful actions resulting from wrist compensation, center of gravity shift, or abnormal force chain. Therefore, this invention proposes a deep learning-based method for fusing motion posture recognition and semantic features. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a deep learning-based method for motion pose recognition and semantic feature fusion, thereby solving the technical problems mentioned in the background section.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The deep learning-based motion pose recognition and semantic feature fusion method includes the following steps: S1. Collect videos of ball sports, perform unified preprocessing on the videos to obtain a sequence of video frames to be identified, generate student target tracking sequence and moving target tracking sequence based on a deep learning image recognition model, and extract human key point sequence, moving target trajectory and spatiotemporal relationship between human and ball; S2. Determine the candidate time window before contact based on the spatiotemporal relationship between the human and the ball. Within the candidate time window before contact, identify the student target's body orientation, supporting foot direction, arm trajectory, center of gravity preset state, and force chain activation sequence to generate semantic features of pre-contact intent. S3. After the candidate time window before contact, establish the contact search time window. Based on the convergence of the distance between the candidate action area and the moving target, the speed change and direction change of the moving target trajectory, determine the moment of contact with the ball and generate the contact semantic features at the moment of contact. S4. Determine the post-contact result time window based on the instant of contact with the ball, extract the result features of the moving target and the result features of the human body repositioning, and generate semantic features of the post-contact result. S5. Perform time alignment and target alignment on the semantic features of pre-contact intention, contact moment, and post-contact result. Input the data into a deep semantic fusion network, calculate the consistency of the three semantic loops, and output the identification results of technically correct actions, accidental successful actions, or technically incorrect actions.

[0007] S1 specifically includes: collecting continuous motion videos of ball games in colleges and universities, unifying the frame rate according to the timestamp, and performing image frame stabilization, brightness normalization, and invalid edge cropping on the image frames to obtain a sequence of video frames to be identified; inputting the sequence of video frames to be identified into a deep learning image recognition model to detect student targets, moving targets, and shooting face targets, outputting target categories, detection box coordinates, target center point coordinates, and target category confidence, and generating student target tracking sequences and moving target tracking sequences; extracting human key point sequences based on student target tracking sequences, extracting moving target trajectories based on moving target tracking sequences, establishing the spatiotemporal relationship between humans and balls according to the same timestamp, and outputting a set of 3 recognition input segments.

[0008] S2 specifically includes: calling three sets of recognition inputs; calculating the minimum relative distance between the candidate action area and the moving target based on the sequence of human key points of the same student target, the trajectory of the moving target of the same moving target, and the spatiotemporal relationship between the human and the ball; and determining the candidate time window before contact by combining the continuous proximity relationship and the direction of movement; within the candidate time window before contact, recognizing the student target's body orientation, supporting foot direction, arm trajectory, torso rotation trend, center of gravity preset state, and force chain initiation sequence to obtain body preparation state data; performing structured semantic encoding on the body preparation state data to generate semantic features of pre-contact intent, which are used as inputs to determine the moment of contact with the ball and to calculate the consistency of the three semantic closed loops.

[0009] S3 specifically includes: invoking the spatiotemporal relationship between the human and the ball and the semantic features of the pre-contact intent; establishing a contact search time window after the pre-contact candidate time window; determining the initial candidate contact frame based on the minimum relative distance between the candidate action area and the moving target and its continuously decreasing trend; verifying the changes in the speed and direction of the moving target and the nearest neighbor contact relationship of the candidate action area based on the trajectory of the moving target before and after the initial candidate contact frame, and generating contact verification results; determining the moment of contact with the ball based on the initial candidate contact frame and the contact verification results, and generating contact semantic features of the moment of contact including contact time, contact location, contact direction, contact intensity, and contact effectiveness.

[0010] S4 specifically includes: determining the post-contact result time window by calling the contact time in the contact semantic features at the moment of contact, and reading the trajectory of the moving target and the sequence of human key points within this time window; extracting the flight direction, speed change, landing point or net crossing trend based on the trajectory of the moving target, and extracting the trunk return state, lower limb support recovery state, arm follow-through state and body stability state based on the sequence of human key points, thus obtaining the moving target result features and the human body reset result features; performing structured semantic fusion on the moving target result features and the human body reset result features to generate post-contact result semantic features including ball trajectory direction matching value, landing point or net crossing trend matching value, human body reset stability value, follow-through integrity value and result validity.

[0011] S5 specifically includes: calling the semantic features of intent before contact, the semantic features of contact at the moment of contact, and the semantic features of result after contact; performing time alignment and target alignment according to the same student target, the same movement target, and the same contact time to generate three semantic fusion samples of the same ball-touching action; inputting the three semantic fusion samples into a deep semantic fusion network to calculate the consistency of intent contact, the consistency of contact result, and the consistency of intent result, and fusing them to obtain the consistency of the three semantic closed loops; combining the consistency of the three semantic closed loops and the result validity in the semantic features of result after contact, outputting the technically correct action, the accidentally successful action, or the technically incorrect action, and simultaneously outputting the inconsistent semantic position.

[0012] The beneficial effects of this invention are as follows: This invention extracts the spatiotemporal relationship between the student target, the moving target, and the ball, enabling human posture recognition to move beyond isolated reliance on key points. This provides a unified data foundation for subsequent pre-contact, instant-contact, and post-contact analysis, improving the stability of video recognition in college ball sports. By generating semantic features of pre-contact intent, it can identify the student's body orientation, supporting foot direction, arm trajectory, center of gravity pre-set state, and force chain initiation sequence before contact with the ball, thereby determining the rationality of the action preparation and avoiding evaluation of action quality solely based on the ball contact result.

[0013] This invention determines the instant of ball contact by using a contact search time window, distance convergence, velocity changes, direction changes, and nearest neighbor contact relationships. This reduces misjudgments caused by situations such as the ball passing by, being obstructed, or multiple people approaching, thus improving the accuracy of contact event identification. By extracting semantic features of the post-contact result, and combining the flight direction, landing point, or net-crossing trend of the moving target with the student's target reset posture and body stability, it can identify whether the ball trajectory result after contact matches the human action result.

[0014] This invention distinguishes between technically correct actions, accidentally successful actions, and technically incorrect actions by calculating the consistency of three semantic loops: pre-contact intention semantic features, instantaneous contact semantic features, and post-contact result semantic features. This avoids directly assuming a successful result is technically correct. The invention outputs the semantic location of inconsistencies along with the action recognition result, allowing teachers to identify whether the error originated from pre-contact preparation, the action mode at the moment of contact, or the post-contact reset result, thus improving the targetedness and interpretability of action correction in college ball sports instruction. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the motion pose recognition and semantic feature fusion method based on deep learning of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example: Figure 1 As shown, this embodiment provides a method for motion pose recognition and semantic feature fusion based on deep learning, including the following steps: S1. Collect videos of ball sports, perform unified preprocessing on the videos to obtain a sequence of video frames to be identified, generate student target tracking sequence and moving target tracking sequence based on a deep learning image recognition model, and extract human key point sequence, moving target trajectory and spatiotemporal relationship between human and ball; S2. Determine the candidate time window before contact based on the spatiotemporal relationship between the human and the ball. Within the candidate time window before contact, identify the student target's body orientation, supporting foot direction, arm trajectory, center of gravity preset state, and force chain activation sequence to generate semantic features of pre-contact intent. S3. After the candidate time window before contact, establish the contact search time window. Based on the convergence of the distance between the candidate action area and the moving target, the speed change and direction change of the moving target trajectory, determine the moment of contact with the ball and generate the contact semantic features at the moment of contact. S4. Determine the post-contact result time window based on the instant of contact with the ball, extract the result features of the moving target and the result features of the human body repositioning, and generate semantic features of the post-contact result. S5. Perform time alignment and target alignment on the semantic features of pre-contact intention, contact moment, and post-contact result. Input the data into a deep semantic fusion network, calculate the consistency of the three semantic loops, and output the identification results of technically correct actions, accidental successful actions, or technically incorrect actions.

[0018] S1 specifically includes the following sub-steps: S110. Collect continuous motion videos of ball sports scenes in colleges and universities, and perform unified preprocessing on the continuous motion videos to obtain the video frame sequence to be identified. The continuous motion videos come from fixed cameras, venue monitoring cameras, or mobile shooting devices set up in basketball courts, volleyball courts, badminton courts, table tennis training areas, or other college ball sports teaching venues; the continuous motion videos must at least include student targets, sports targets, and effective sports fields.

[0019] Student target refers to the human body of a student participating in ball games in the video; motion target refers to the ball, shuttlecock, or table tennis ball that is hit, passed, or controlled in ball games; racket face target refers to the area of ​​action of the equipment used to contact the motion target in racket games, with the racket face target being a component of the candidate area of ​​action.

[0020] When preprocessing continuous motion videos, the frame rate is first unified according to the video timestamp, converting videos from different sources into image frames with the same time interval; then, the image is de-shaken according to the displacement of background feature points of the field edge lines, ground textures, or fixed equipment in adjacent image frames, so that the coordinates of the same field position remain stable in continuous frames; then, brightness normalization is performed on the effective image area of ​​each frame to reduce the impact of indoor lighting, backlighting by windows, or venue shadows on image recognition.

[0021] Brightness normalization can be performed using the following formula:

[0022] in, Represents the pixel position in frame t. Normalized brightness value Represents the pixel position in frame t. The original brightness value, This represents the average brightness of the effective image region in frame t. This represents the standard deviation of the brightness of the effective image region in frame t. This represents a preset constant to prevent the denominator from being zero. After completing brightness normalization, the stands, walls, irrelevant passages, and black borders of the screen are cropped based on the area enclosed by the field boundary lines, the area where student targets appear, and the area covered by the trajectory of moving targets, resulting in the video frame sequence to be identified.

[0023] For example, in college volleyball teaching videos, the original video is 60fps, and the system resamples it to 30fps according to the timestamp. When the average displacement of the volleyball court sideline feature points in adjacent frames exceeds 2px, translation registration is performed on the next frame, and the off-court spectator area and wall area are cropped. The output is a sequence of video frames to be identified that only contains student targets, volleyballs and valid court area.

[0024] The sequence of video frames to be identified serves as input for S120 target detection and temporal tracking.

[0025] S120. Input the video frame sequence to be identified obtained in S110 into the deep learning image recognition model, detect the student target and moving target in each frame, and perform temporal tracking of the position changes of the same student target and the same moving target in consecutive frames to generate student target tracking sequence and moving target tracking sequence.

[0026] The data output by a deep learning image recognition model includes at least the target category, detection box coordinates, target center point coordinates, and target category confidence score. Among them, the target category is used to distinguish between student targets, moving targets, and targets that are hit in the face; the detection box coordinates are used to determine the image region where the target is located; the target center point coordinates are used to calculate the displacement of the target in consecutive frames; and the target category confidence score is used to eliminate falsely detected targets.

[0027] When performing temporal tracking of student targets, the system determines the consecutive numbers of the same student target based on the intersection-union ratio, center point displacement, and human appearance features of the student target detection boxes in adjacent frames. When a student target is temporarily occluded or crosses over, the system further uses the matching relationship between the predicted position of human key points in the previous frame and the detection box in the current frame to maintain the student target number, thus avoiding the continuation of the limb regions of different students into the same student target tracking sequence.

[0028] The student's target matching value can be calculated using the following formula:

[0029] in, Indicates the first The matching value between the i-th student target in frame 1 and the j-th student target in frame t. This represents the intersection-union ratio (IUU) of two detection boxes. Indicates the first The i-th student target detection box in the frame. This represents the bounding box for the j-th student target in frame t. Indicates the first The center point of the i-th student target in the frame. This represents the center point of the j-th student target in frame t. This represents the preset distance normalization parameter. and represents the preset weight, and e represents the natural constant.

[0030] When performing temporal tracking of moving targets, the system generates a moving target tracking sequence based on the continuity of the target's center point, velocity direction, and detection confidence. For example, in a college basketball shooting video, if three students appear in the frame simultaneously, the system first detects the three student targets and one basketball target. Then, it assigns a continuous number to each student based on the intersection-union ratio of the detection boxes, the displacement of the center point, and human appearance features. When two students overlap, the system uses the predicted positions of the shoulder, hip, and knee keypoints from the previous frame to verify the current detection box, ensuring that the student target number does not change.

[0031] The student target tracking sequence and the moving target tracking sequence are used as inputs for the S130 to extract the human body key point sequence and the moving target trajectory.

[0032] In this embodiment, the deep learning image recognition model can employ, but is not limited to, the YOLOv8 object detection network combined with the DeepSORT multi-object tracking algorithm. Specifically, YOLOv8 is used to extract feature maps of student targets and moving targets and output detection boxes and confidence scores. These are then fed into DeepSORT's Kalman filter to predict the target's position in the next frame, and combined with the Hungarian algorithm for matching based on the intersection-over-union ratio and appearance features, thereby generating continuous student target tracking sequences and moving target tracking sequences.

[0033] S130: Extract the human body key point sequence of the student target based on the student target tracking sequence obtained in S120, extract the trajectory of the moving target based on the moving target tracking sequence obtained in S120, and establish the spatiotemporal relationship between the human body key point sequence and the moving target trajectory according to the same timestamp, and output 3 segments of recognition input set.

[0034] Human keypoint sequence refers to the coordinates and corresponding confidence scores of keypoints of the same student target in consecutive frames, including the head, shoulders, elbows, wrists, hips, knees, and ankles. In racket-holding projects, the relative positions of hand keypoints and racket face targets are also extracted. Moving target trajectory refers to the coordinates of the center point, velocity direction, velocity amplitude, and detection confidence score of the moving target in consecutive frames. Human-ball spatiotemporal correlation refers to the relative distance, relative direction, approach velocity, and candidate action area between human keypoints and the moving target at the same timestamp.

[0035] The relative distance can be calculated using the following formula:

[0036] in, This represents the relative distance between the k-th human keypoint and the moving target in the t-th frame. This represents the coordinates of the k-th human body keypoint in the t-th frame. This represents the coordinates of the center point of the moving target in frame t. The system obtains the approach velocity based on the relative distance changes between adjacent frames and determines a preset range of pixels around the hand, wrist, forearm, or racket face as the candidate action area.

[0037] For example, in a badminton swing video from a university, the system extracts key points of the shoulder, elbow, wrist, hip, knee, and ankle from the student target tracking sequence, extracts the trajectory of the badminton center point from the moving target tracking sequence, and extracts the trajectory of the racket face target center point. Then, it calculates the relative distance and relative direction between the wrist key points, the racket face target center point, and the badminton center point frame by frame. When the relative distance between the racket face target center point and the badminton center point continuously decreases and the badminton trajectory points to the area where the racket face target is located, the result is recorded as the spatiotemporal correlation between the person and the ball.

[0038] The three-segment recognition input set includes the sequence of human key points, the trajectory of the moving target, and the spatiotemporal relationship between the human and the ball. It is used by S210 to determine the candidate time window before contact, by S310 to determine the moment of contact with the ball, and by S410 to continue reading the trajectory of the moving target and the sequence of human key points within the result time window after contact.

[0039] Furthermore, when extracting the human keypoint sequence based on the student target tracking sequence, this embodiment adopts a human pose estimation model based on a high-resolution network (HRNet). This model maintains high-resolution feature representation in parallel multi-resolution subnetworks and outputs two-dimensional coordinates and corresponding confidence scores of 17 human keypoints, including the head, shoulders, elbows, wrists, hips, knees, and ankles, to meet the spatial accuracy requirements of subsequent motion pose analysis.

[0040] S2 specifically includes the following sub-steps: S210 calls the three-segment recognition input set output by S130, and determines the candidate time window before contact based on the human body key point sequence of the same student target, the motion target trajectory of the same moving target, and the spatiotemporal relationship between the two.

[0041] The pre-contact candidate time window refers to a continuous frame interval used to analyze the student's target action preparation process before the instant of contact with the ball. It is not an arbitrarily selected fixed segment, but is determined by the continuous proximity relationship between the student target's candidate action area and the moving target. The candidate action area refers to the image region where the student target may interact with the moving target; in unarmed sports such as basketball and volleyball, the candidate action area is defined by key points of the hand, wrist, and forearm; in racket sports such as badminton and table tennis, the candidate action area also includes the area where the racket face target is located.

[0042] The system reads the relative distances between the candidate action point set and the center point of the moving target from the spatiotemporal correlation between the human and the ball, and calculates the minimum relative distance for consecutive frames:

[0043] in, This represents the minimum relative distance between the candidate action point of the student target and the moving target in frame t. Represents the set of candidate action points. This represents the coordinates of the k-th candidate action point in the t-th frame. This represents the coordinates of the center point of the moving target in frame t.

[0044] If the minimum relative distance continuously decreases within a preset number of consecutive frames, and the direction of motion of the moving target points towards the candidate action area of ​​the student target, then the image frame in which the minimum relative distance reaches a local minimum is determined as a candidate approach frame, and the candidate time window before contact is determined as follows:

[0045] in, This indicates the candidate time window before contact. Indicates candidate close frames. Indicates the preset number of frames before contact. This indicates the frame preceding the candidate approach frame. This pre-contact candidate time window serves as the image recognition analysis range for S220 to identify the body's readiness state.

[0046] For example, in a college volleyball passing video, the system reads the relative distance between the key points of the student's wrists and the center point of the volleyball. When the minimum relative distance decreases from 120px to 35px within 8 consecutive frames, and the direction of the volleyball movement points to the candidate action area between the two forearms, the 20 frames before the candidate approach frame are determined as the candidate time window before contact. If the volleyball only passes by the side of the student's body and the direction of movement does not point to the candidate action area between the two forearms, no candidate time window before contact is generated for the student, so as to avoid mistaking irrelevant passing as a ball-touching preparation action.

[0047] S220. Within the candidate time window before contact determined in S210, identify the student target's physical readiness state based on the sequence of human key points, the trajectory of the moving target, and the spatiotemporal relationship between the person and the ball. The physical readiness state refers to the structured posture state formed by the student target before contacting the ball, which can affect the subsequent contact action with the ball. It includes at least the body orientation, the direction of the supporting foot, the trajectory of the arm, the tendency of the torso rotation, the pre-positioned state of the center of gravity, and the starting sequence of the force chain.

[0048] Body orientation is determined by the angle between the lines connecting the shoulder and hip key points and the direction of the moving target; the supporting foot direction is determined by the ankle key point, the approximate foot direction, and the field coordinate direction; the arm trajectory is determined by the displacement sequence of the wrist key point or the center point of the racket face target within the candidate time window before contact; the trunk rotation trend is determined by the angle change between the shoulder key point line and the hip key point line; the pre-positioned center of gravity is determined by the weighted position of the hip, knee, and ankle key points.

[0049] in, This represents the preset centroid point in frame t. This represents the center coordinates of the hip keypoint in frame t. This represents the center coordinates of the knee keypoint in frame t. This represents the center coordinates of the bi-ankle keypoints in frame t. , , This indicates a preset weight. In this embodiment, since lower limb support contributes the most to the center of gravity in ball sports, the preset weight can be set based on human biomechanical experience, for example, by setting... =0.4 (hip area) =0.3 (knee) =0.3 (ankle); or it can be obtained by using the analytic hierarchy process (AHP) to quantify the impact factors of various key point displacements on the overall motion stability.

[0050] The power chain activation sequence is determined based on the time frame at which the lower limb support change, trunk rotation change, and upper limb guidance or racket-holding guidance change first reach the preset change conditions:

[0051] in, Indicates the starting sequence of the power chain. This indicates the time frame at which the lower limb support change first reaches the preset change condition. This indicates the time frame at which the torso rotation change first reaches the preset change condition. This indicates the time frame at which the upper limb guidance or hand-held guidance first reaches the preset change condition. The system maps the above body readiness state to the trajectory of the moving target, determines whether the student's body readiness is pointing towards the expected contact position of the moving target, and outputs the body readiness state data.

[0052] For example, in a college basketball shooting video, if the foot support area remains stable, the center of the hip first shifts upward, then the line connecting the shoulder key points rises, and finally the wrist key point accelerates along the direction of the basket, it is recorded as the lower limb support, trunk drive, and upper limb release starting sequentially; if the wrist key point accelerates first, while the center of the hip and the line connecting the shoulder key points change lagging behind, it is recorded as an abnormal force chain, and this result is entered into S230 for semantic encoding.

[0053] S230: Perform structured semantic encoding on the body readiness state data output by S220 to generate pre-contact intention semantic features. Pre-contact intention semantic features refer to computable features used to indicate whether the student target has formed a movement preparation that matches the moving target before contacting the ball. They are not textual evaluation results, but structured feature vectors that can be called by S310 and S520.

[0054] The structured feature vector includes at least the following: body orientation matching value, supporting foot direction matching value, arm trajectory integrity value, center of gravity preset stability value, force chain sequence matching value, and expected contact position matching value.

[0055] in, This indicates semantic features representing pre-contact intent. This indicates the body orientation matching value. Indicates the support leg direction matching value. This represents the complete value of the lever trajectory. This indicates the preset stable value of the center of gravity. Indicates the power chain sequence matching value. This indicates the expected contact location matching value.

[0056] The above matching values ​​are obtained by normalizing the correspondence between the body's ready state and the expected contact position of the moving target obtained in S220; among them, the force chain sequence matching value can be determined as follows:

[0057] in, Indicates the power chain sequence matching value. This indicates the time frame at which the lower limb support change first reaches the preset change condition. This indicates the time frame at which the torso rotation change first reaches the preset change condition. This indicates the time frame at which the upper limb guidance or shot guidance first reaches the preset change condition.

[0058] For example, in a badminton forehand shot video in college, if the student's torso is facing the same direction as the incoming ball, the supporting foot is pointing towards the expected hitting area, the racket face forms a complete backswing trajectory from back to front before contact, the center of gravity is preset within the supporting area formed by the line connecting the supporting feet, and the change in lower limb support precedes the change in torso rotation, and the change in torso rotation precedes the change in racket face acceleration, then the corresponding field will be encoded with a higher matching value. If a student hits the ball directly with wrist compensation before completing the backswing, the complete backswing trajectory value and the power chain sequence matching value will be encoded as low values. The semantic features of the pre-contact intention output by S230 serve as the pre-constraint for S310 to determine the instant of contact with the ball, and as the first input for S510 and S520 to calculate the consistency of the three-segment semantic closed loop.

[0059] S3 specifically includes the following sub-steps: S310 calls the spatiotemporal relationship between the human and the ball output by S130 and the semantic features of pre-contact intent output by S230 to establish a contact search time window after the pre-contact candidate time window determined by S210, and determines the initial candidate contact frame within the contact search time window. The contact search time window refers to the continuous frame interval used to find the contact between the student target and the moving target. Its starting point is the frame after the last frame of the pre-contact candidate time window, and its ending point is the preset continuous frames after the starting point. The preset continuous frames can be set according to the project type and video frame rate. For example, in a 30fps video, it can be set to 10 to 20 frames.

[0060] The system reads the minimum relative distance between the candidate action area of ​​the same student target and the same moving target from the spatiotemporal relationship of the human and ball output by S130. The candidate action area follows the definition in S210, including the area defined by the key points of the hand, wrist and forearm in the freehand project, and also the area where the target of the racket face is located in the racket project.

[0061] When determining the initial candidate contact frame, the system not only selects the image frame with the smallest distance, but also requires that the minimum relative distance in the previous several consecutive frames shows a decreasing trend. Furthermore, the student target has already generated contact pre-contact intention semantic features corresponding to the current moving target trajectory by S230, so as to exclude interference from other students who are close to the moving target but have not formed a contact preparation.

[0062] The initial candidate contact frame can be determined by the following formula:

[0063] in, Indicates the initial candidate contact frame. This indicates that the search time window has been accessed. This represents the minimum relative distance between the candidate action point of the student target and the moving target in frame t. This indicates that the image frame that minimizes the minimum relative distance is selected within the contact search time window.

[0064] For example, in a video of a forehand shot in table tennis at a university, the system first confirms, based on S230, that the student target has semantic features of pre-contact intent with the racket face facing the direction of the incoming ball, a complete backswing trajectory, and a stable center of gravity. Then, within the contact search time window, the system calculates the distance between the center point of the racket face target and the center point of the table tennis ball. When this distance decreases from 42px to 8px within 6 consecutive frames and increases to 14px in the next frame, the image frame corresponding to 8px is determined as the initial candidate contact frame.

[0065] S320: Based on the initial candidate contact frame determined by S310, call the moving target trajectory output by S130 to verify the changes in the moving target's speed and direction of motion before and after the initial candidate contact frame, as well as the nearest contact relationship of the candidate action area, and generate contact verification results.

[0066] The contact verification result is used to determine whether the initial candidate contact frame is a real contact frame, rather than an invalid approach frame formed by a moving target passing near the student target. The system calculates the displacement change of the moving target before and after contact based on the coordinates of the center point of the moving target in one or more frames before and after the initial candidate contact frame; when the displacement amplitude after contact changes relative to before contact, it is recorded as a velocity change that meets the condition.

[0067] The change in velocity can be calculated using the following formula:

[0068] in, This represents the change in velocity of the moving target corresponding to frame t. , , They represent the first Frame, t-th frame and t-th frame The coordinates of the center point of the moving target in the frame. This represents the Euclidean distance in the image plane coordinates.

[0069] The system simultaneously calculates the direction change angle based on the trajectory direction vectors of the moving target before and after the initial candidate contact frame, and determines whether the center point of the moving target or the detection box forms spatial overlap or close proximity contact with the candidate action area corresponding to the hand, wrist, forearm, or clapping face target. If the initial candidate contact frame satisfies the proximity relationship and the speed change or direction change meets the preset contact conditions, the frame is retained; if there is only proximity but the moving target trajectory does not change speed or direction, the frame is excluded.

[0070] For example, in a badminton hitting video in college, if the center point of the racket face and the center point of the shuttlecock are 6px apart at the initial candidate contact frame, the system further calculates and finds that the shuttlecock moves towards the student target before the frame and moves towards the opposite side of the court after the frame. Furthermore, if the area where the racket face target is located forms a close neighbor contact relationship with the shuttlecock detection box, then a valid contact verification result is generated. If the shuttlecock only passes in front of the racket face and its trajectory direction does not change, then an invalid contact verification result is generated.

[0071] S330: Based on the initial candidate contact frames obtained in S310 and the contact verification results obtained in S320, determine the instant of contact with the ball and generate contact semantic features at the instant of contact. The instant of contact with the ball refers to the time point of the image frame jointly confirmed by distance convergence, trajectory change, and nearest neighbor contact in the candidate action area; the contact semantic features at the instant of contact refer to structured data characterizing the interaction between the student target and the moving target at the instant of contact with the ball. It is used to limit the start time of the result time window after contact in S410 and serves as the second input for S510 and S520 to calculate the consistency of the three-segment semantic closed loop.

[0072] The semantic features of contact at the moment of contact include at least contact time, contact location, contact direction, contact intensity, and contact effectiveness. Among them, contact time is determined by the timestamp of the confirmed contact frame, contact location is determined by the human body part or the target face that is closest to the moving target and belongs to the candidate action area, contact direction is determined by the change in the trajectory direction of the moving target before and after the confirmed contact frame, contact intensity is obtained by normalizing the change in the velocity or the angle of change in the direction of the moving target, and contact effectiveness is jointly determined by the nearest neighbor relationship, the velocity change relationship, and the direction change relationship.

[0073] The semantic features of contact at the moment of contact can be represented as:

[0074] in, Indicates the semantic features of contact at the moment of contact. This indicates the contact frame number corresponding to the contact time. Indicates the contact area. Indicates the direction of contact. Indicates contact strength. Indicates the effectiveness of the contact.

[0075] For example, in a college basketball passing video, the system determines the initial candidate contact frame where the distance between the hands area and the center point of the basketball reaches the local minimum according to S310, and then confirms according to S320 that the basketball changes from a low-speed ball-holding state to a flying state in the direction of receiving the ball before and after the frame, and the center point of the basketball and the candidate action area of ​​the hands form a close contact relationship. Based on this, the system identifies the frame as the moment of contact with the ball, encodes the frame's timestamp as the contact time, encodes the hand area as the contact point, encodes the basketball's flight direction after leaving the hands as the contact direction, encodes the normalized value corresponding to the change in basketball speed as the contact intensity, and encodes the contact validity as valid. The contact semantic features at the moment of contact output by S330 are then called by S410 to extract the post-contact result time window after the contact time, and are then called by S520 to determine whether the contact direction, contact intensity, and post-contact ball trajectory result are consistent.

[0076] S4 specifically includes the following sub-steps: S410: Call the contact semantic features output by S330 at the moment of contact, determine the post-contact result time window based on the contact time, and continue to read the trajectory of the moving target and the human body key point sequence of the student target in S130 within the post-contact result time window.

[0077] The post-contact result time window refers to the continuous frame interval starting from the moment of contact with the ball, used to determine the ball path result after the moving target leaves the candidate action area and the student target's body reset result; its starting point is determined by the contact time confirmed by S330, and the ending point is determined according to the ball game type, video frame rate, moving target speed, and preset result observation duration.

[0078] The post-contact result time window can be determined as follows:

[0079] in, Indicates the time window for results after contact. This indicates the contact frame number corresponding to the contact time determined by S330. This indicates the preset number of frames to observe after contact. Indicates the frame following the contact time. This indicates the last frame of the result time window after contact.

[0080] For basketball passing, the post-contact result time window covers the main flight process from when the ball leaves the hand until it is caught; for volleyball passing, the post-contact result time window covers the process of the volleyball stabilizing its flight direction and net crossing tendency after leaving the forearm; for badminton or table tennis hitting, the post-contact result time window covers the process of the moving target stabilizing its initial flight direction after leaving the racket face.

[0081] If a moving target is briefly lost within the result time window after contact due to occlusion, ghosting, or excessive speed, the system performs trajectory replacement based on the coordinates of the target's center point, direction of motion, and speed continuity before and after the loss. The detection confidence corresponding to the replacement point is marked as lower than that of the directly detected frame for weighted calculation by S420.

[0082] For example, in a college volleyball passing video, S330 determines the 150th frame as the moment of contact with the ball, with a video frame rate of 30fps, and the system determines the time window from the 151st frame to the 180th frame as the result time window after contact; If the volleyball is obscured by a student's arm in frames 158 to 160, the system completes the missing position based on the volleyball center point trajectory in frames 151 to 157 and 161 to 165, and marks the completed position as low-confidence data. The post-contact result time window, post-contact moving target trajectory, and post-contact human key point sequence output by S410 are used as inputs to S420.

[0083] S420. Within the time window of the contact result determined in S410, extract the motion target result features based on the trajectory of the moving target after contact, and extract the human body reset result features based on the sequence of human body key points after contact.

[0084] The motion target result characteristics refer to the motion state data of the motion target after leaving the student target candidate action area, including at least the flight direction, speed change, landing point trend or net crossing trend; among them, the flight direction is obtained by fitting the coordinates of the center point of the motion target within the result time window after contact, the speed change is calculated by the displacement of the center point of the motion target in adjacent frames, the landing point trend is obtained by the correspondence between the extension direction of the motion target trajectory and the field coordinate area, and the net crossing trend is determined by the change of the motion target height, the position of the ball and net lines and the motion direction.

[0085] The characteristics of human body reset result refer to the posture data of the student target after regaining body control after contact with the ball, including at least the trunk return to center, the lower limb support recovery state, the arm follow-through state, and the body stability state. Among them, the trunk return to center is determined by the relative angle change of the line connecting the shoulder key points and the line connecting the hip key points; the lower limb support recovery state is determined by whether the ankle key points, knee key points, and the center of gravity preset point return to the support area; and the arm follow-through state is determined by the wrist key point or the continued trajectory of the target after contact.

[0086] A stable state of body health can be calculated as follows:

[0087] in, This represents the stable change value of the body in frame t. This represents the preset centroid point in frame t. Indicates the first Preset centroid in the frame This represents the coordinates of the left ankle keypoint in frame t. This represents the coordinates of the right ankle keypoint in frame t. This represents a preset constant to prevent the denominator from being zero. This body stability variation value is used to represent the degree of inter-frame fluctuation of the preset center of gravity relative to the width of the foot support; the smaller the value, the higher the body stability after contact.

[0088] For example, in a video of a badminton forehand shot in college, the system reads the trajectory of the center point of the shuttlecock within the result time window after contact. If the center point of the shuttlecock moves continuously to the opposite side of the court and the height change is consistent with the direction of crossing the net, the result characteristic of the moving target is recorded as an effective return trend. Simultaneously, key points of the student's shoulder, hip, knee, ankle, and wrist are read. If, after the ball is struck, the wrist key point continues along the direction of the strike, the torso returns from a rotating state to the direction facing the court, and the center of gravity does not continuously deviate from the support area formed by the two ankles, then the human body reset result feature is recorded as a stable reset. The motion target result feature and human body reset result feature output by S420 serve as inputs for semantic fusion of S430.

[0089] S430: Perform structured semantic fusion on the motion target result features and human body reset result features output by S420 to generate post-contact result semantic features. Post-contact result semantic features refer to the computable feature vectors used to represent the ball trajectory result and human body reset result after contact with the ball, and are not natural language evaluation content; they include at least the ball trajectory direction matching value, ball trajectory speed stability value, landing point or net crossing trend matching value, human body reset stability value, follow-through integrity value, and result validity.

[0090] The ball trajectory direction matching value is obtained by normalizing the angle between the flight direction of the moving target after contact and the contact direction in S330; the ball trajectory velocity stability value is obtained by normalizing the velocity change amplitude of the moving target in consecutive frames after contact; the landing point or net crossing trend matching value is obtained by the correspondence between the trajectory extension result of the moving target and the target area of ​​the event; the body reset stability value is obtained by the body stability change value calculated in S420; the follow-through integrity value is obtained by whether the wrist key point or the target on the racket face continues to move along the contact direction after contact; the result validity is used to indicate whether the ball trajectory result after contact with the moving target meets the action target of the corresponding event.

[0091] The semantic features of the result after contact can be represented as:

[0092] in, Indicates the semantic features of the result after contact. Indicates the ball trajectory direction matching value. This indicates the stable value of the ball's trajectory speed. Indicates the landing point or net crossing trend matching value. Indicates the stable value of human body repositioning. Indicates the complete value of the wave. Indicates the validity of the result.

[0093] For example, in a college basketball passing video, if the flight direction of the basketball after leaving the hand is consistent with the contact direction obtained by S330, the basketball speed does not show an abnormal decrease in consecutive frames after contact, the trajectory extension direction points to the chest area of ​​the receiving student, the passing student's feet support is restored and the arm follows through completely, then the ball path direction matching value, ball path speed stability value, landing point trend matching value, human body reset stability value and follow-through value are encoded as higher values, and the validity of the result is encoded as valid; If the basketball's flight direction deviates from the receiving student's area, and the passing student's center of gravity continues to move forward after contact, the landing point trend matching value and the human body reset stability value are encoded as low values. The semantic features of the contact result output by S430 serve as the third input for S510 to perform time alignment and target alignment, and as the basis for S520 to calculate the consistency of contact result and the consistency of intention result.

[0094] S5 specifically includes the following sub-steps: S510 calls the semantic features of pre-contact intent output by S230, the semantic features of contact moment output by S330, and the semantic features of post-contact result output by S430. Time alignment and target alignment are performed according to the same student target, the same motion target, and the same contact time to generate three semantic fusion samples.

[0095] Time alignment refers to using the contact time determined by S330 as the anchor point, mapping the candidate time window before the contact time to the semantic features of the intention before contact, mapping the confirmed contact frame where the contact time is located to the semantic features of the contact moment, and mapping the time window of the result after the contact time to the semantic features of the result after contact.

[0096] Target alignment refers to confirming that the three types of semantic features all originate from the same student target and the same motion target based on the consecutive student target number and motion target number generated by S120, thus avoiding the erroneous merging of data from different students or different ball-touching actions.

[0097] Three-segment semantic fusion samples refer to the structured input data used for subsequent consistency calculations. They consist of three types of semantic features corresponding to the same ball-touching action, and can be represented as:

[0098] in, This represents the three semantic fusion samples corresponding to the m-th ball touch action, where m represents the ball touch action number. The semantic features representing the pre-contact intention of the m-th touch are: The semantic features of contact at the moment of contact during the m-th ball contact action. The semantic features representing the result of the m-th ball contact action.

[0099] For example, in a college basketball passing video, student A03 completes a passing action in frame 220. S230 outputs the semantic features of pre-contact intention from frame 200 to frame 219, S330 outputs the semantic features of contact at the moment of contact in frame 220, and S430 outputs the semantic features of post-contact result from frame 221 to frame 250. When the above three types of semantic features all correspond to student A03, the same basketball target, and the contact time in frame 220, the system generates a three-segment semantic fusion sample and uses it as the input of S520.

[0100] S520. Input the three semantic fusion samples generated in S510 into the deep semantic fusion network, calculate the consistency of intent contact, the consistency of contact result, and the consistency of intent result, and fuse them to obtain the consistency of the three semantic closed loops. The deep semantic fusion network is a neural network used to perform feature mapping, pairing, and fusion discrimination of three structured semantic features. Its training samples come from labeled videos of college ball sports, and the labeled fields include at least the pre-contact action preparation state, contact point, contact direction, ball trajectory result after contact, and action recognition result.

[0101] Intent-contact consistency is used to determine whether the pre-contact action preparation can explain the contact point, contact direction, and contact intensity at the moment of contact; contact-result consistency is used to determine whether the contact direction and contact intensity at the moment of contact can explain the ball's trajectory direction, speed change, and landing point or net crossing trend after contact; intent-result consistency is used to determine whether the pre-contact action intent matches the ball's trajectory result and body reset result after contact.

[0102] To compute the consistency between two feature segments, the deep semantic fusion network consists of two sets of structurally identical but weight-sharing feature projection modules. Since various semantic features belong to structured data of different dimensions, each feature projection module employs a multilayer perceptron (MLP) as its feature mapping function. and .

[0103] Specifically, the multilayer perceptron consists of an input layer, two hidden layers (64 and 128 nodes respectively), and an output layer (64 nodes), using ReLU as the activation function. Any two semantic features to be compared... and After mapping, they are projected into the same 64-dimensional implicit metric space.

[0104] Subsequently, the consistency score is obtained by calculating the inner product of the mapped features and performing Sigmoid normalization:

[0105] in, This indicates the consistency between two semantic features. and This represents the mapping function of the multilayer perceptron. This represents the transpose of a vector. This represents the Sigmoid function, which maps the inner product result to the range of 0 to 1.

[0106] The consistency of the three semantic loop closures can be obtained by fusing them as follows:

[0107] in, This indicates that the three segments share semantic loop consistency. Indicates consistency of intent to contact, This indicates consistency in the contact results. The intention and the result are consistent. , , This represents a preset weight. The preset weight can be a fixed value (e.g., set to a value specified in this embodiment). =0.3, =0.3, =0.4, highlighting the weight of the judgment on consistency between intention and result. In another preferred embodiment, , , It can also be used as a learnable parameter to participate in the joint training of the aforementioned deep semantic fusion network. As the network backpropagates, it is automatically updated to the optimal solution to adapt to the different degrees of dependence of different ball games on the consistency of the three stages.

[0108] For example, in a college badminton forehand shot video, if the semantic features of intent before contact show a complete backswing, the supporting foot pointing towards the expected hitting area, and the correct power chain sequence, and the semantic features of contact at the moment of contact show that the contact point is the racket face target and the contact direction is towards the opposite side of the court, and the semantic features of result after contact show that the shuttlecock flies to the opposite side of the effective area and the student's target returns to a stable position after the follow-through, then all three types of consistency are high values. If the backswing is insufficient but the shuttlecock still enters the opposite side of the effective area, then the consistency of intent and result decreases, and this result is entered into S530 for action category determination.

[0109] The aforementioned deep semantic fusion network requires supervised training before it can be used for inference. The training steps are as follows: Obtain labeled videos of college ball sports, extract the semantic features of the pre-contact intention of the same correct technical action as anchor samples, extract the corresponding contact semantic features at the moment of contact or the semantic features of the result after contact as positive samples, and extract the corresponding features of the same type of technically incorrect action as negative samples, and construct a triple training set.

[0110] The triplet loss function is used for gradient backpropagation and parameter update of the deep semantic fusion network. Its loss function is defined as follows:

[0111] Where L is the loss value, A is the anchor sample vector after feature mapping, P is the positive sample vector, and N is the negative sample vector. represents the Euclidean distance between vectors, and margin is a preset boundary threshold. By minimizing this loss function, the network is forced to narrow the semantic feature distance before and after the matching action in the implicit metric space, and widen the distance of the mismatched features, thereby enabling the network to have the ability to compute consistency.

[0112] S530: Based on the consistency of intent contact, consistency of contact result, consistency of intent result, and consistency of the three-segment semantic loop obtained in S520, and combined with the result validity contained in the semantic features of the contact result in S430, the action recognition result is output. Result validity refers to whether the motion target result after contact meets the action target of the corresponding ball game, such as whether the basketball pass points to the receiving area, whether the volleyball pass forms a tendency to cross the net, and whether the badminton hit enters the effective area on the opposite side.

[0113] Action recognition results include technically correct actions, accidentally successful actions, and technically incorrect actions. Among them, a technically correct action means that the intention before contact, the contact moment, and the result after contact all meet the corresponding consistency requirements and the result is valid. An accidentally successful action means that the result after contact is valid, but the intention before contact or the contact moment does not meet the corresponding consistency requirements. A technically incorrect action means that the result after contact is invalid.

[0114] Action recognition results can be determined as follows:

[0115] Where Y represents the action recognition result, Indicates the correct technical action. Indicates an action that was accomplished by chance. Indicates a technical error action. , , This indicates the corresponding consistency threshold. This indicates the validity of the result. When outputting the action recognition result, the system simultaneously outputs the inconsistent semantic locations; these locations include abnormal locations of intent before contact, abnormal locations of contact at the moment of contact, and abnormal locations of results after contact, which are used for subsequent action error correction.

[0116] For example, in a college volleyball passing video, if the student's forearm platform is fully formed and the center of gravity is pre-positioned and stable before contact, and the volleyball makes effective contact with the candidate action area of ​​the forearms at the moment of contact, and the volleyball flies towards the expected net crossing area after contact and the student's target is reset and stable, then the correct technical action is output; if the volleyball eventually crosses the net, but the forearm platform is not formed before contact and the ball trajectory is changed by wrist compensation at the moment of contact, then an accidental success action is output, and abnormal intention before contact and abnormal contact at the moment of contact are marked; if the volleyball does not form an effective net crossing trend, then an incorrect technical action is output, and the source of error is marked according to the semantic segment with the lowest consistency.

[0117] All the above formulas are performed using dimensionless numerical calculations; the relevant formulas are based on empirical models that approximate the real situation, obtained through extensive data collection and software simulation fitting. The preset parameters and thresholds involved in the formulas can be conventionally set and adjusted by those skilled in the art according to the physical constraints of the actual application scenario.

[0118] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0119] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0120] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for motion pose recognition and semantic feature fusion based on deep learning, characterized in that, Includes the following steps: S1. Collect videos of ball sports, perform unified preprocessing on the videos to obtain a sequence of video frames to be identified, generate student target tracking sequence and moving target tracking sequence based on a deep learning image recognition model, and extract human key point sequence, moving target trajectory and spatiotemporal relationship between human and ball; S2. Determine the candidate time window before contact based on the spatiotemporal relationship between the human and the ball. Within the candidate time window before contact, identify the student target's body orientation, supporting foot direction, arm trajectory, center of gravity preset state, and force chain activation sequence to generate semantic features of pre-contact intent. S3. After the candidate time window before contact, establish the contact search time window. Based on the convergence of the distance between the candidate action area and the moving target, the speed change and direction change of the moving target trajectory, determine the moment of contact with the ball and generate the contact semantic features at the moment of contact. S4. Determine the post-contact result time window based on the instant of contact with the ball, extract the result features of the moving target and the result features of the human body repositioning, and generate semantic features of the post-contact result.

2. The method for motion pose recognition and semantic feature fusion based on deep learning according to claim 1, characterized in that, It also includes: S5, which performs time alignment and target alignment on the semantic features of pre-contact intention, contact moment, and post-contact result, inputs them into a deep semantic fusion network, calculates the consistency of the three-segment semantic loop, and outputs the identification results of technically correct actions, accidental successful actions, or technically incorrect actions.

3. The method for motion pose recognition and semantic feature fusion based on deep learning according to claim 1, characterized in that, S1 specifically includes: Collect continuous motion videos of ball sports scenes in colleges and universities, unify the frame rate according to the timestamp, and perform image frame stabilization, brightness normalization and invalid edge cropping on the image frames to obtain the video frame sequence to be identified. The video frame sequence to be identified is input into the deep learning image recognition model to detect student targets, moving targets, and shooting targets. The model outputs the target category, detection box coordinates, target center point coordinates, and target category confidence, and generates student target tracking sequence and moving target tracking sequence. Human key point sequences are extracted based on student target tracking sequences, and moving target trajectories are extracted based on moving target tracking sequences. Human-sphere spatiotemporal correlation is established according to the same timestamp, and three sets of recognition inputs are output.

4. The deep learning-based motion pose recognition and semantic feature fusion method according to claim 1, characterized in that, S2 specifically includes: The system calls three sets of recognition inputs, calculates the minimum relative distance between the candidate action area and the moving target based on the sequence of human key points of the same student target, the trajectory of the moving target of the same moving target, and the spatiotemporal relationship between the human and the ball, and determines the candidate time window before contact by combining the continuous proximity relationship and the direction of movement. Within the pre-contact candidate time window, identify the student's body orientation, supporting foot direction, arm trajectory, torso rotation trend, center of gravity preset state, and force chain activation sequence to obtain body readiness state data.

5. The deep learning-based motion pose recognition and semantic feature fusion method according to claim 4, characterized in that, Also includes: Structured semantic encoding is performed on the body readiness state data to generate semantic features of pre-contact intent, which are then used as inputs to determine the moment of contact with the ball and to calculate the consistency of the three semantic loops.

6. The method for motion pose recognition and semantic feature fusion based on deep learning according to claim 1, characterized in that, S3 specifically includes: By invoking the spatiotemporal relationship between humans and the ball and the semantic features of pre-contact intent, a contact search time window is established after the pre-contact candidate time window. The initial candidate contact frame is determined based on the minimum relative distance between the candidate action area and the moving target and the continuous decreasing trend. Based on the trajectory of the moving target before and after the initial candidate contact frame, the changes in the speed and direction of the moving target and the nearest contact relationship of the candidate action area are verified to generate contact verification results. The moment of contact with the ball is determined based on the initial candidate contact frames and contact verification results, and contact semantic features of the moment of contact, including contact time, contact location, contact direction, contact intensity, and contact effectiveness, are generated.

7. The method for motion pose recognition and semantic feature fusion based on deep learning according to claim 1, characterized in that, S4 specifically includes: The contact time in the contact semantic features at the moment of contact is used to determine the time window of the result after contact, and the trajectory of the moving target and the sequence of human key points are read within the time window; Based on the trajectory of the moving target, the flight direction, speed change, landing point or net crossing trend are extracted. Based on the sequence of human body key points, the trunk return to center state, the lower limb support recovery state, the arm swing state and the body stability state are extracted to obtain the characteristics of the moving target result and the characteristics of the human body reset result.

8. The method for motion pose recognition and semantic feature fusion based on deep learning according to claim 7, characterized in that, Also includes: Structured semantic fusion is performed on the characteristics of the moving target result and the characteristics of the human body reset result to generate post-contact result semantic features, including ball trajectory direction matching value, landing point or net crossing trend matching value, human body reset stability value, follow-through integrity value, and result validity.

9. The method for motion pose recognition and semantic feature fusion based on deep learning according to claim 2, characterized in that, S5 specifically includes: The semantic features of intent before contact, contact semantic features at the moment of contact, and result semantic features after contact are invoked. Time alignment and target alignment are performed according to the same student target, the same movement target, and the same contact time to generate three semantic fusion samples of the same ball-touching action. Three semantic fusion samples are input into a deep semantic fusion network to calculate intent contact consistency, contact result consistency, and intent result consistency, and then fused to obtain the three semantic closed-loop consistency.

10. The method for motion pose recognition and semantic feature fusion based on deep learning according to claim 9, characterized in that, Also includes: Combining the consistency of the three semantic loops and the validity of the results in the semantic features of the post-contact results, the system outputs the technically correct action, the accidentally successful action, or the technically incorrect action, and simultaneously outputs the inconsistent semantic positions.

Images