Tennis posture evaluation system and method based on multi-modal data fusion

Abstract

The invention is suitable for the technical field of crossing of computer vision, artificial intelligence and sports science, and provides a tennis posture evaluation system based on multi-modal data fusion, which comprises a three-dimensional posture extraction module, an action recognition module, an index calculation module and a natural language evaluation generation module, the three-dimensional posture extraction module is used for extracting a human body three-dimensional key point sequence from a monocular video according to an STCFormer model; the action recognition module is used for performing standardization processing, Bi-LSTM network processing and full-connection classification network processing on the human body three-dimensional key point sequence to obtain tennis action types; the index calculation module is used for calculating biomechanical parameters according to the action type and the human body three-dimensional key point sequence; and the natural language evaluation generation module is used for inputting the action type and the biomechanical parameters into a preset large language model to generate feedback information in a natural language form. Therefore, end-to-end automatic processing from original video input to final professional evaluation report generation is realized, and objectivity and accuracy of evaluation results are improved.

Description

Technical Field

[0001] This application belongs to the interdisciplinary fields of computer vision, artificial intelligence and sports science, and in particular relates to a tennis posture evaluation system and method based on multimodal data fusion. Background Technology

[0002] With the development of artificial intelligence technology, computer vision-based motion recognition and evaluation systems have been initially applied in tennis training. Existing solutions mostly focus on two levels: 1. Motion classification and comparison: Through two-dimensional or three-dimensional posture estimation, the user's motion is simply compared with a standard motion template, or only coarse-grained classifications such as "serve" and "forehand shot" are completed. These methods lack in-depth analysis of motion details, and the feedback is usually "the motion is not standard," but it cannot explain "why it is not standard" or "how to improve." 2. Simple parameter measurement: Some systems can calculate simple parameters such as joint angles, but usually only the raw data is presented to the user, lacking the ability to transform the data into semantic feedback that is instructive and conforms to the thinking habits of human coaches.

[0003] Therefore, current tennis posture evaluation methods suffer from low objectivity and accuracy. Summary of the Invention

[0004] This application provides a tennis posture evaluation system and method based on multimodal data fusion, which can solve the problem of low objectivity and accuracy of current tennis posture evaluation methods.

[0005] In a first aspect, embodiments of this application provide a tennis posture evaluation system based on multimodal data fusion. The system includes a three-dimensional posture extraction module, an action recognition module, an index calculation module, and a natural language evaluation generation module. The three-dimensional posture extraction module is used to extract the sequence of three-dimensional key points of the human body from monocular video according to the STCFormer model. The action recognition module is used to standardize the sequence of three-dimensional key points of the human body and then input it sequentially into a bidirectional long short-term memory (Bi-LSTM) network and a fully connected classification network to obtain the action type of tennis. The index calculation module is used to calculate the corresponding biomechanical parameters based on the identified action type and the sequence of three-dimensional key points of the human body. The natural language evaluation generation module is used to input the action type and biomechanical parameters into a preset large language model to generate feedback information in natural language form.

[0006] In one possible implementation of the first aspect, the extraction of the human body 3D keypoint sequence from monocular video based on the STCFormer model is as follows:

[0007] The STCFormer model was used to analyze the video frame sequence of monocular video, and the coordinates of the three-dimensional key points of the human body in each frame were extracted using the spatiotemporal attention mechanism.

[0008] The coordinates of the three-dimensional human body key points in each frame are organized in chronological order into a sequence of (T, 17, 3) three-dimensional human body key points, where T is the frame number, 17 is the number of human body key points, and 3 is the spatial coordinate dimension.

[0009] Save the human body 3D key point sequence as an NPZ format file.

[0010] Optionally, in another possible implementation of the first aspect, the extracted three-dimensional keypoint sequence is classified to identify the action type of tennis, as follows:

[0011] The human body 3D key point sequence is standardized to obtain a standardized sequence, including the following steps: centering the sequence with the center of the left and right hip joints as the origin; normalizing the scale with the shoulder width as the unit length; and unifying the sequence length to a fixed frame length by truncation or zero padding.

[0012] The standardized sequence is input into the Bi-LSTM network to extract time-dimensional features;

[0013] An adaptive average pooling layer is used to aggregate the time-dimensional features, compressing the time-dimensional features from a variable-length sequence into a fixed-dimensional feature vector;

[0014] The fixed-dimensional feature vectors are input into a fully connected classification network with a structure of 512→128→3. The hidden layers are activated using ReLU and Dropout=0.4 is set to prevent overfitting. The output layer uses the Softmax activation function to obtain the probability distribution of each action type.

[0015] The action type corresponding to the highest probability is determined based on the probability distribution, and the recognition result is transmitted to the indicator calculation module.

[0016] Optionally, in another possible implementation of the first aspect, the Bi-LSTM network is a two-layer stacked structure, with each layer containing 256 hidden units. The Bi-LSTM network simultaneously extracts forward and backward features from the input sequence in the time dimension, and concatenates the extracted forward and backward outputs to form a 512-dimensional feature representation.

[0017] Optionally, in another possible implementation of the first aspect, the corresponding biomechanical parameters are calculated based on the identified action type and the sequence of three-dimensional key points of the human body, as follows:

[0018] Anomaly detection is performed on the three-dimensional keypoint sequence of the human body. Anomaly detection includes detecting limb length constraints, detecting joint velocity upper limit constraints, and detecting coordinate Z value.

[0019] The detected outliers are corrected using time-neighborhood weighted interpolation to ensure the temporal continuity of the key point trajectories;

[0020] Differentiated smoothing strategies were adopted based on the key point categories: 7-frame windows were used for trunk key points, 5-frame windows for limb key points, and 3-frame windows for wrist key points. Moving average filtering was used for smoothing to obtain a smoothed human 3D key point sequence.

[0021] Based on the smoothed three-dimensional key point sequence of the human body and the action type provided by the action recognition module, the corresponding biomechanical parameters are calculated.

[0022] Optionally, in another possible implementation of the first aspect, the action type and biomechanical parameters are input into a preset large language model to generate feedback information in natural language form, as follows:

[0023] The pre-set large language model is trained based on a corpus of tennis professional domains. The pre-set large language model adopts the Transformer architecture.

[0024] The action type and biomechanical parameters are combined into prompt words and input into the pre-trained large language model, and the feedback information is output in natural language form.

[0025] Perform quality assessment and semantic consistency verification on the feedback information.

[0026] Secondly, embodiments of this application provide a tennis posture evaluation method based on multimodal data fusion, comprising the following steps: extracting a sequence of three-dimensional key points of the human body from a monocular video according to the STCFormer model; standardizing the sequence of three-dimensional key points of the human body and then inputting it sequentially into a bidirectional long short-term memory (Bi-LSTM) network and a fully connected classification network to obtain the action type of tennis movement; calculating the corresponding biomechanical parameters based on the identified action type and the sequence of three-dimensional key points of the human body; and inputting the action type and biomechanical parameters into a preset large language model to generate feedback information in natural language form.

[0027] Thirdly, embodiments of this application provide a terminal device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a tennis posture evaluation method based on multimodal data fusion as described above.

[0028] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a tennis posture evaluation method based on multimodal data fusion as described above.

[0029] Beneficial Effects: This application provides a tennis posture evaluation system based on multimodal data fusion, including a 3D posture extraction module, a motion recognition module, an index calculation module, and a natural language evaluation generation module. The 3D posture extraction module extracts the sequence of 3D key points of the human body from monocular video using the STCFormer model. The motion recognition module standardizes the sequence of 3D key points and then sequentially inputs it into a bidirectional long short-term memory (Bi-LSTM) network and a fully connected classification network to obtain the motion type of tennis. The index calculation module calculates the corresponding biomechanical parameters based on the identified motion type and the sequence of 3D key points. The natural language evaluation generation module inputs the motion type and biomechanical parameters into a preset large language model to generate feedback information in natural language form. This achieves end-to-end automated processing from raw video input to the final generation of a professional evaluation report, improving the objectivity and accuracy of the evaluation results. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of the structure of a tennis posture evaluation system based on multimodal data fusion provided in an embodiment of this application;

[0032] Figure 2 This is a schematic flowchart of a tennis posture evaluation method provided in an embodiment of this application;

[0033] Figure 3 This is a schematic flowchart illustrating the calculation of biomechanical parameters according to an embodiment of this application;

[0034] Figure 4 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0035] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0036] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0037] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0038] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0039] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0040] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0041] The following is a detailed description of a tennis posture evaluation system and method based on multimodal data fusion provided in this application, with reference to the accompanying drawings.

[0042] Figure 1 The diagram shows a schematic representation of a tennis posture evaluation system based on multimodal data fusion, as provided in an embodiment of this application.

[0043] like Figure 1 As shown, the tennis posture evaluation system 100 based on multimodal data fusion includes a three-dimensional posture extraction module 101, an action recognition module 102, an index calculation module 103, and a natural language evaluation generation module 104.

[0044] The 3D pose extraction module 101 is used to extract the sequence of 3D key points of the human body from monocular video based on the STCFormer model.

[0045] The action recognition module 102 is used to standardize the three-dimensional key point sequence of the human body and then input it sequentially into the bidirectional long short-term memory Bi-LSTM network and the fully connected classification network to obtain the action type of tennis.

[0046] The index calculation module 103 is used to calculate the corresponding biomechanical parameters based on the identified action type and the sequence of three-dimensional key points of the human body.

[0047] The natural language evaluation generation module 104 is used to input action type and biomechanical parameters into a preset large language model to generate feedback information in natural language form.

[0048] Furthermore, in this embodiment of the application, the extraction of the human body 3D key point sequence from monocular video based on the STCFormer model is as follows:

[0049] The STCFormer model was used to analyze the video frame sequence of monocular video, and the coordinates of the three-dimensional key points of the human body in each frame were extracted using the spatiotemporal attention mechanism.

[0050] The coordinates of the three-dimensional human body key points in each frame are organized in chronological order into a sequence of (T, 17, 3) three-dimensional human body key points, where T is the frame number, 17 is the number of human body key points, and 3 is the spatial coordinate dimension.

[0051] Save the human body 3D key point sequence as an NPZ format file.

[0052] Furthermore, in this embodiment of the application, the above-mentioned classification of the extracted three-dimensional keypoint sequence to identify the action type of tennis is as follows:

[0053] The human body 3D key point sequence is standardized to obtain a standardized sequence, including the following steps: centering the sequence with the center of the left and right hip joints as the origin; normalizing the scale with the shoulder width as the unit length; and unifying the sequence length to a fixed frame length by truncation or zero padding.

[0054] The standardized sequence is input into the Bi-LSTM network to extract time-dimensional features;

[0055] An adaptive average pooling layer is used to aggregate the time-dimensional features, compressing the time-dimensional features from a variable-length sequence into a fixed-dimensional feature vector;

[0056] The fixed-dimensional feature vectors are input into a fully connected classification network with a structure of 512→128→3. The hidden layers are activated using ReLU and Dropout=0.4 is set to prevent overfitting. The output layer uses the Softmax activation function to obtain the probability distribution of each action type.

[0057] The action type corresponding to the highest probability is determined based on the probability distribution, and the recognition result is transmitted to the indicator calculation module.

[0058] Furthermore, in the embodiments of this application, the above-mentioned Bi-LSTM network is a two-layer stacked structure, each layer containing 256 hidden units. The Bi-LSTM network simultaneously extracts forward and backward features from the input sequence in the time dimension, and concatenates the extracted forward and backward outputs to form a 512-dimensional feature representation.

[0059] Furthermore, in this embodiment, the calculation of corresponding biomechanical parameters based on the identified action type and the sequence of three-dimensional key points of the human body is as follows:

[0060] Anomaly detection is performed on the three-dimensional keypoint sequence of the human body. Anomaly detection includes detecting limb length constraints, detecting joint velocity upper limit constraints, and detecting coordinate Z value.

[0061] The detected outliers are corrected using time-neighborhood weighted interpolation to ensure the temporal continuity of the key point trajectories;

[0062] Differentiated smoothing strategies were adopted based on the key point categories: 7-frame windows were used for trunk key points, 5-frame windows for limb key points, and 3-frame windows for wrist key points. Moving average filtering was used for smoothing to obtain a smoothed human 3D key point sequence.

[0063] Based on the smoothed three-dimensional key point sequence of the human body and the action type provided by the action recognition module, the corresponding biomechanical parameters are calculated.

[0064] For example, biomechanical parameters include joint angle indices, kinematic indices, body posture indices, spatial position indices, and movement coordination indices; calculated joint angle indices include right elbow angle, right knee angle, hip and shoulder twist angle, and trunk forward tilt angle; kinematic indices include right wrist linear velocity, right wrist acceleration, and right elbow angular velocity; body posture indices include body extension height, support base width, and body center of mass trajectory; spatial position indices include hitting point height and ball toss height; and movement coordination indices include hip and shoulder speed difference and leg drive force index.

[0065] Furthermore, in this embodiment, the above-mentioned input of action type and biomechanical parameters into a preset large language model to generate feedback information in natural language form is as follows:

[0066] The pre-set large language model is trained based on a corpus of tennis professional domains. The pre-set large language model adopts the Transformer architecture.

[0067] The action type and biomechanical parameters are combined into prompt words and input into the pre-trained large language model, and the feedback information is output in natural language form.

[0068] Perform quality assessment and semantic consistency verification on the feedback information.

[0069] This application provides a tennis posture evaluation system based on multimodal data fusion, including a 3D posture extraction module, a motion recognition module, an index calculation module, and a natural language evaluation generation module. The 3D posture extraction module extracts the sequence of 3D key points of the human body from monocular video using the STCFormer model. The motion recognition module standardizes the sequence of 3D key points and then sequentially inputs it into a bidirectional long short-term memory (Bi-LSTM) network and a fully connected classification network to obtain the motion type of tennis. The index calculation module calculates the corresponding biomechanical parameters based on the identified motion type and the sequence of 3D key points. The natural language evaluation generation module inputs the motion type and biomechanical parameters into a preset large language model to generate feedback information in natural language form. This achieves end-to-end automated processing from raw video input to the final generation of a professional evaluation report, improving the objectivity and accuracy of the evaluation results.

[0070] Figure 2 The diagram shows a flowchart of a tennis posture evaluation method based on multimodal data fusion provided in an embodiment of this application.

[0071] like Figure 2 As shown, this tennis posture evaluation method based on multimodal data fusion includes:

[0072] S201. Extract the sequence of three-dimensional human body key points from monocular video based on the STCFormer model;

[0073] S202. After standardizing the three-dimensional key point sequence of the human body, the sequence is input into a bidirectional long short-term memory (Bi-LSTM) network and a fully connected classification network to obtain the action type of tennis.

[0074] S203. Calculate the corresponding biomechanical parameters based on the identified motion type and the sequence of three-dimensional key points of the human body;

[0075] S204. Input the action type and biomechanical parameters into the preset large language model to generate feedback information in natural language form.

[0076] This application provides a tennis posture evaluation method based on multimodal data fusion. First, it extracts a sequence of three-dimensional keypoints of the human body from monocular video using the STCFormer model. Then, after standardization, the sequence is sequentially input into a bidirectional long short-term memory (Bi-LSTM) network and a fully connected classification network to obtain the tennis motion type. Next, it calculates the corresponding biomechanical parameters based on the identified motion type and the three-dimensional keypoint sequence. Finally, it inputs the motion type and biomechanical parameters into a pre-defined large language model to generate feedback information in natural language. This achieves end-to-end automated processing from raw video input to the final generation of a professional evaluation report, improving the objectivity and accuracy of the evaluation results.

[0077] The solution provided in this application will be illustrated below with a specific embodiment.

[0078] Step 1: Analyze the input video using the STCFormer model, output the corresponding 3D coordinates, and save them as an NPZ file. The NPZ file contains 3D data of 17 keypoints × 3D coordinates × 1000 frames. The index of the 17 keypoints is shown in Table 1.

[0079] Table 1

[0080]

[0081] Step 2: Build a pose classifier and perform action classification processing on the extracted NPZ file in the format (T, 17, 3).

[0082] Step 2.1: Standardize the attitude sequence data output in Step 1.

[0083] Step 2.2: Input the sequence into the bidirectional LSTM network.

[0084] Step 2.3, use adaptive average pooling:

[0085]

[0086] in:

[0087] p: Fixed-dimensional feature vector after adaptive average pooling, used as input for subsequent classification networks.

[0088] : Fixed frame length after standardization (value is 128).

[0089] }: Temporal features output by the Bi-LSTM network in frame t, with dimensions of .

[0090] B: Batch size, which is the number of video sequences input into the network in a single operation.

[0091] 2H: Feature dimensions after concatenation of bidirectional outputs from the Bi-LSTM network.

[0092] The temporal dimension features are extracted and the action representation is obtained through global pooling, compressing the entire temporal dimension into a single feature vector; the temporal aggregation of the overall action features is realized, ensuring the consistency of sequences of different lengths; the final output feature dimension is 512.

[0093] Step 2.4: Use a fully connected network to output the classification results. The fully connected classification layer uses two linear mapping layers: 512->128->3; reducing the 512-dimensional features to 3 classification states, and the final output is:

[0094]

[0095] in:

[0096] z: The raw logical value output by the fully connected network (without Softmax activation).

[0097] B: Batch size, consistent with the previous definition.

[0098] : Number of output motion categories (corresponding to the 3 core motion categories in tennis).

[0099] The hidden layer uses ReLU activation and Dropout (Dropout=0.4) to prevent overfitting;

[0100] The output layer uses Softmax activation to obtain the probability distribution of the three types of actions:

[0101]

[0102] in:

[0103] : The probability value of the i-th type of action, ranging from [0,1].

[0104] : The original logical value of the i-th type of action output by the fully connected network.

[0105] : The original logical value of the j-th action output by the fully connected network.

[0106] : Number of action categories, consistent with the previous definition (value is 3).

[0107] e: natural constant, approximately equal to 2.71828.

[0108] The final output is the category with the highest probability, which is used to predict the action type.

[0109]

[0110] in:

[0111] The final identified action type (values ​​1, 2, and 3, corresponding to 3 types of core tennis actions).

[0112] : The probability value of the i-th type of action, consistent with the previous definition.

[0113] : A function that retrieves the index corresponding to the maximum value.

[0114] Step 2.5: The recognition results are passed to the parameter calculation module to perform subsequent biomechanical index calculation and language generation.

[0115] Furthermore, the specific method for step 2.1 above is as follows: Step 2.1-1: Center the sequence using the center points of the left and right hips as the origin. Denote the original keypoint sequence of a given single video. ,in , No. Frame number The coordinates of the key points are: The indexes of the key points on the left and right hips are: ,frame The center point of the hip is:

[0116] in: : No. The coordinates of the hip center point of the frame are used for subsequent centering processing.

[0117] : No. 3D coordinates of the key point on the left hip of the frame.

[0118] : No. 3D coordinates of the right hip key point in the frame.

[0119] : Video frame number, ranging from [1, T] (T is the original video frame number).

[0120] A set of three-dimensional spatial coordinates (x, y, z dimensions).

[0121] After centralization:

[0122] in: After centralized processing, the first Frame number The coordinates of the key points.

[0123] Before centralized processing Frame number The original coordinates of the key points.

[0124] : No. The coordinates of the hip center point of the frame are consistent with the aforementioned definition.

[0125] j: Human body key point index, ranging from [0, 16] (corresponding to 17 human body key points).

[0126] For all frame sequence numbers and key point index Both are valid.

[0127] Step 2.1-2: Normalize the sequence using shoulder width as the scale, and denote the indices of the left and right shoulders as follows: Calculate the shoulder width for each frame:

[0128]

[0129] in:

[0130] : Shoulder width in frame t (Euclidean distance between the two shoulder key points).

[0131] : Coordinates of the left shoulder keypoint in frame t after centering.

[0132] The coordinates of the right shoulder keypoint in frame t after centering, using the average scale of the entire sequence:

[0133]

[0134] in:

[0135] : Average shoulder width of the entire sequence, used for scale normalization.

[0136] T: Number of original video frames.

[0137] The shoulder width of frame t is consistent with the aforementioned definition.

[0138] : To prevent small constants with denominators of 0 (the original text does not specify the value, but it is usually taken as 10). -6 ).

[0139] After normalization:

[0140]

[0141] in:

[0142] : The coordinates of the j-th keypoint in the t-th frame after scale normalization.

[0143] : The coordinates of the j-th key point in the t-th frame after centralization, consistent with the aforementioned definition.

[0144] The average shoulder width across the entire sequence is consistent with the aforementioned definition.

[0145] Steps 2.1-3: Unify all sequences to a fixed frame length by truncation or zero-padding. .

[0146] Furthermore, the specific method for step 2.2 above is as follows:

[0147] Step 2.2-1: Set the input layer to flatten the pose keypoints of each frame into a format with a length of [length missing]. The feature vectors are used to obtain the input tensor with the shape of... , where B is the batch size.

[0148] Step 2.2-2 employs a two-layer stacked LSTM network, with H hidden units in each layer. During training, the network propagates bidirectionally in the time dimension to capture the dynamic dependencies between consecutive frames. This structure can simultaneously model the overall temporal rhythm of the action and the keyframe change features. The bidirectional outputs, after concatenation, have a dimension of [missing information]. .

[0149] This layer is calculated as follows:

[0150]

[0151] in:

[0152] The hidden state of the Bi-LSTM network during forward propagation in frame t has a dimension of H.

[0153] The backpropagation of the Bi-LSTM network is performed on the hidden state of frame t, with dimension H.

[0154] LSTM: Long Short-Term Memory network computing unit.

[0155] : The input feature vector of frame t (obtained by flattening the three-dimensional coordinates of 17 key points, with a dimension of 51).

[0156] : The hidden state of the t-th frame after splicing, with a dimension of 2H.

[0157] The semicolon ";" indicates a vector concatenation operation, which connects the forward and backward hidden states along the feature dimension.

[0158] Step 3: Based on the detected classification actions, the corresponding analysis method is used to preprocess the collected data and calculate the corresponding indicators, and so on. Figure 3 As shown.

[0159] Step 3.1: Detect whether there are outliers in limb length, movement speed, and deviations between key points, and correct them using time-distance weighted interpolation.

[0160] Step 3.2: Based on the movement characteristics of different parts of the body in tennis, key points are divided into three categories and different smoothing window sizes are set (window sizes are odd numbers to ensure temporal symmetry): Trunk key points (including hip center, spine, chest, neck, and head, indices 0-4) are smoothed using a 7-frame window due to their gentle movement; Limb key points (including right hip, right knee, left hip, left knee, right shoulder, right elbow, left shoulder, and left elbow, indices 5-6, 8-9, 11-12, 14-15) have moderate movement amplitude and are smoothed using a 5-frame window; Wrist key points (including right and left wrists, indices 13 and 16) have intense movement, such as sudden speed changes during ball impact, and are smoothed using a 3-frame window. The smoothing calculation uses a moving average filter, i.e.

[0161]

[0162] in:

[0163] t: Current frame number.

[0164] j: Key point index, ranging from [0, 16] (corresponding to 17 human body key points).

[0165] k: Coordinate dimension index, with values ​​{0, 1, 2}, corresponding to the three spatial dimensions x, y, and z respectively.

[0166] : The value of the j-th key point of the t-th frame in the k-th coordinate dimension (x / y / z) after smoothing.

[0167] W: Smooth window size (Torso keypoint W=7, Limb keypoint W=5, W wrist keypoint W=3).

[0168] w: Window half-width, derived from window size W (ensuring the window size is odd to guarantee timing symmetry).

[0169] m: Frame offset within the window, ranging from [-k, k].

[0170] : The value of the j-th key point of the j-th class in the k-th coordinate dimension in the t+m-th frame before smoothing.

[0171] Step 3.3: Based on the preprocessed 3D posture data and combined with the biomechanical principles of tennis movements, calculate quantitative indicators.

[0172] Furthermore, the specific method for step 3.1 above is as follows:

[0173] Step 3.1-1: Detect and constrain limb length. Based on the average adult limb length, set the maximum reasonable limb length. For preset limb key point pairs (e.g., (right hip, right knee), (right knee, right foot), (left shoulder, left elbow)), calculate the limb length for each frame: ,in , Let L be the 3D coordinates of keypoints i and j in frame t, and L be the limb length between keypoints i and j in frame t. Then mark , This is an outlier.

[0174] Step 3.1-2: Detect and constrain motion speed, setting an upper limit for human joint movement speed, such as a maximum wrist swing speed of approximately 15 m / s. Calculate the displacement velocity of key points in adjacent frames: ,in Let i be the motion velocity of key point i in frame t. Let i be the 3D coordinates of keypoint i in frame t. Let i be the 3D coordinates of keypoint i in frame t-1. If the time interval between two adjacent frames is Then mark This is an outlier.

[0175] Steps 3.1-3: Based on the statistical characteristics of the time-series data, detect coordinate values ​​that deviate too far from the mean and calculate the Z-value of each keypoint coordinate: ,in Let Z be the Z value of a certain coordinate dimension of key point i in frame t (used for outlier detection). Let i be the value of keypoint i in frame t in this coordinate dimension. Let be the mean of a certain coordinate dimension of keypoint i. The standard deviation is denoted as ; if If the value is greater than 3.0, then x_k(t) is marked as an outlier.

[0176] Steps 3.1-4 correct outliers based on "temporal neighborhood weighted interpolation" to ensure temporal continuity: For key point k in outlier frame t, the forward search frame sequence is t-1, t-2, ..., t-5; the backward search frame sequence is t+1, t+2, ..., t-5; for each valid neighborhood frame... ∈ S, calculate its time distance Weight .

[0177] The weights are normalized to obtain normalized weights: Where K is the number of valid neighboring frames. For the corresponding weights, For the first The coordinates of frame keypoint i; if there are no valid neighboring frames, the global mean of the keypoint is used for padding: ,in Let i be the coordinates of keypoint i in frame t after correction, and T be the number of frames in the entire sequence.

[0178] Furthermore, the specific method for step 3.3 above is as follows:

[0179] Step 3.3-1: Calculate joint angle indices to reflect the standardization of the movement. The calculations include: right elbow joint angle, the angle formed by the right shoulder, right elbow, and right wrist; a value close to 180 degrees at the time of striking the ball ensures sufficient power; right knee joint angle, formed by the right hip, right knee, and right foot; approximately 90 degrees when flexing the knee to build power, and close to 180 degrees when extending the knee; hip-shoulder twist angle, the angle between the hip vector and the shoulder vector in the horizontal plane, reflecting the efficiency of trunk rotation for power generation; an ideal value of 30 to 45 degrees at the time of serving; and trunk forward tilt angle, the angle between the spinal vector and the vertical axis; an ideal value of 15 to 20 degrees at the time of striking the ball.

[0180] Step 3.3-2: Calculate kinematic parameters to reflect the intensity and rhythm of the swing. The calculation includes: right wrist linear velocity, calculated from the displacement of the right wrist in adjacent frames, whose peak value at the moment of impact directly determines the ball speed, with an ideal value of not less than 25 m / s; right wrist acceleration, i.e., the rate of change of the right wrist linear velocity, reflecting the explosive power of the swing; and right elbow angular velocity, i.e., the rate of change of the right elbow joint angle, measuring the power efficiency during the push-off phase.

[0181] Step 3.3-3: Calculate body posture indices to reflect movement stability and body extension. Calculations include: body extension height, the vertical distance from the head to the average position of both feet, ideally 1.1 to 1.2 times the height at the time of impact; support base width, the distance between the left and right feet on the horizontal plane, ideally 0.8 to 1.0 times the shoulder width to ensure balance; and body center of mass, obtained by weighted averaging of all key point coordinates, used to assess the overall trajectory of the center of gravity during impact.

[0182] Steps 3.3-4 involve calculating spatial position indicators to directly reflect the quality of the shot. The calculations include: the height of the hitting point, estimated by extrapolating from the right wrist position and racket length; ideally, this should be no less than 2.5 meters during the serve to ensure a good net clearance angle; and the height of the ball toss, taken as the maximum vertical height of the left wrist during the toss phase; ideally, this should be no less than 2.0 meters to ensure sufficient opportunity to hit the ball.

[0183] Steps 3.3-5 involve calculating the motion coordination index, which reflects the force sequence of the kinetic chain. The calculation includes: hip-shoulder velocity difference, i.e., the difference between the hip torsional angular velocity and the shoulder torsional angular velocity; ideally, this should be a positive value, reflecting the correct force sequence of "hip rotation first, shoulder rotation later"; and leg drive force index, obtained by averaging the absolute values ​​of the rate of change of the angles of both knee joints, reflecting the intensity of the lower limb extension.

[0184] Step 4: Input the action category and the biomechanical parameters calculated in Step 3 as prompt words into the fine-tuned large language model. The model uses internally learned motion knowledge to reason and generate professional, natural and operationally instructive feedback statements.

[0185] Step 4.1: The system receives "action category" information from the action recognition module and a standardized set of "biomechanical parameters" from the index calculation module. This information is then structured according to a preset prompt word template to form prompt words that conform to the input specifications of the large language model.

[0186] Step 4.2: Input the constructed prompts into the large language model that has been fine-tuned with professional knowledge in the tennis field. Utilize the principles of sports biomechanics, technical points and coaching experience learned internally to perform deep reasoning and analyze the causal relationship between abnormal parameters and movement problems.

[0187] Step 4.3: Based on the reasoning results, the model generates professional, natural, and operationally instructive feedback statements. It explains the causes of the problem using biomechanical principles and provides specific, actionable improvement suggestions.

[0188] Step 4.4: Output the generated natural language evaluation to the user interface. The system can also set a feedback quality verification mechanism to ensure the accuracy and usability of the generated content, and ultimately provide users with a clear and professional action improvement guidance report.

[0189] To implement the above embodiments, this application also proposes a terminal device.

[0190] Figure 4 This is a schematic diagram of the structure of a terminal device according to an embodiment of this application.

[0191] like Figure 4 As shown, the terminal device 200 includes:

[0192] The system includes a memory 210 and at least one processor 220, and a bus 230 connecting different components (including the memory 210 and the processor 220). The memory 210 stores a computer program, which, when executed by the processor 220, implements the tennis posture evaluation method described in the embodiments of this application.

[0193] Bus 230 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0194] Terminal device 200 typically includes various electronically readable media. These media can be any available media that can be accessed by terminal device 200, including volatile and non-volatile media, removable and non-removable media.

[0195] Memory 210 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 240 and / or cache memory 250. Terminal device 200 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 260 may be used to read and write non-removable, non-volatile magnetic media (… Figure 4 Not shown; usually referred to as a "hard drive"). Although Figure 4As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 230 via one or more data media interfaces. Memory 210 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0196] A program / utility 280 having a set (at least one) of program modules 270 may be stored in, for example, memory 210. Such program modules 270 include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 270 typically perform the functions and / or methods described in the embodiments of this application.

[0197] Terminal device 200 can also communicate with one or more external devices 290 (e.g., keyboard, pointing device, display 291, etc.), and with one or more devices that enable a user to interact with terminal device 200, and / or with any device that enables terminal device 200 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 292. Furthermore, terminal device 200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 293. As shown, network adapter 293 communicates with other modules of terminal device 200 via bus 230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with terminal device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0198] The processor 220 performs various functional applications and data processing by running programs stored in the memory 210.

[0199] It should be noted that the implementation process and technical principles of the terminal device in this embodiment are explained in the foregoing description of the tennis posture evaluation method in this application embodiment, and will not be repeated here.

[0200] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0201] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the above-described method embodiments.

[0202] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0203] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0204] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0205] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0206] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0207] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A tennis posture evaluation system based on multimodal data fusion, characterized in that, The system includes a 3D pose extraction module, an action recognition module, an index calculation module, and a natural language evaluation generation module; The three-dimensional pose extraction module is used to extract the sequence of three-dimensional key points of the human body from monocular video based on the STCFormer model; The action recognition module is used to standardize the human body's three-dimensional key point sequence and then input it sequentially into a bidirectional long short-term memory (Bi-LSTM) network and a fully connected classification network to obtain the action type of tennis. The index calculation module is used to calculate the corresponding biomechanical parameters based on the identified action type and the sequence of three-dimensional key points of the human body. The natural language evaluation generation module is used to input action type and biomechanical parameters into a preset large language model and generate feedback information in natural language form.

2. The system according to claim 1, characterized in that, The extraction of the human body 3D key point sequence from monocular video based on the STCFormer model is as follows: The STCFormer model was used to analyze the video frame sequence of monocular video, and the coordinates of the three-dimensional key points of the human body in each frame were extracted using the spatiotemporal attention mechanism. The coordinates of the three-dimensional human body key points in each frame are organized in chronological order into a sequence of (T, 17, 3) three-dimensional human body key points, where T is the frame number, 17 is the number of human body key points, and 3 is the spatial coordinate dimension. Save the human body 3D key point sequence as an NPZ format file.

3. The system according to claim 2, characterized in that, The extracted 3D keypoint sequence is classified to identify the movement types in tennis, as follows: The human body 3D key point sequence is standardized to obtain a standardized sequence, including the following steps: centering the sequence with the center of the left and right hip joints as the origin; normalizing the scale with the shoulder width as the unit length; and unifying the sequence length to a fixed frame length by truncation or zero padding. The standardized sequence is input into the Bi-LSTM network to extract time-dimensional features; An adaptive average pooling layer is used to aggregate the time-dimensional features, compressing the time-dimensional features from a variable-length sequence into a fixed-dimensional feature vector; A fixed-dimensional feature vector is input into a fully connected classification network with a structure of 512→128→3. The hidden layer uses ReLU activation and Dropout=0.4 is set to prevent overfitting. The output layer uses the Softmax activation function to obtain the probability distribution of each action type. The action type corresponding to the highest probability is determined based on the probability distribution, and the recognition result is transmitted to the indicator calculation module.

4. The system according to claim 3, characterized in that, The Bi-LSTM network is a two-layer stacked structure, with each layer containing 256 hidden units. The Bi-LSTM network simultaneously extracts forward and backward features from the input sequence in the time dimension, and concatenates the extracted forward and backward outputs to form a 512-dimensional feature representation.

5. The system according to claim 4, characterized in that, The corresponding biomechanical parameters are calculated based on the identified action type and the sequence of three-dimensional key points of the human body, as follows: Anomaly detection is performed on the three-dimensional key point sequence of the human body. The anomaly detection includes detecting limb length constraints, detecting joint velocity upper limit constraints, and detecting coordinate Z value. The detected outliers are corrected using time-neighborhood weighted interpolation to ensure the temporal continuity of the key point trajectories; Differentiated smoothing strategies are adopted based on the key point categories: 7-frame windows are used for trunk key points, 5-frame windows are used for limb key points, and 3-frame windows are used for wrist key points. Moving average filtering is used for smoothing to obtain a smoothed human 3D key point sequence. Based on the smoothed three-dimensional key point sequence of the human body, and based on the action type provided by the action recognition module, the corresponding biomechanical parameters are calculated.

6. A tennis posture evaluation system based on multimodal data fusion according to claim 5, characterized in that, The process of inputting the action type and biomechanical parameters into a preset large language model to generate feedback information in natural language form is as follows: A pre-defined large language model is trained based on a corpus of tennis-related language, and the pre-defined large language model adopts the Transformer architecture. The action type and biomechanical parameters are combined into prompt words and input into the pre-trained large language model, and the feedback information is output in natural language form. Perform quality assessment and semantic consistency verification on the feedback information.

7. A method for evaluating tennis posture, characterized in that, Includes the following steps: Extracting human 3D key point sequences from monocular video based on the STCFormer model; After standardizing the human body's three-dimensional keypoint sequence, it is sequentially input into a bidirectional long short-term memory (Bi-LSTM) network and a fully connected classification network to obtain the action types of tennis. Calculate the corresponding biomechanical parameters based on the identified motion type and the sequence of three-dimensional key points of the human body; Input the action type and biomechanical parameters into a pre-set large language model to generate feedback information in natural language form.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in claim 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in claim 7.

Images