Motion action correction method, system, storage medium, computer device and terminal

Abstract

The application belongs to the technical field of computer vision, and discloses a motion action correction method, system, storage medium, computer device and terminal, the method comprising: collection of human motion posture recognition data; design of a motion action classification neural network model and training of the model; comparison of single-frame image postures and DTW (Dynamic Time Warping) distance comparison of action time sequences.The application uses a common USB camera to acquire human motion images, uses a notebook computer to perform inference on the structure of a human posture recognition neural network model, extracts a skeleton and classifies postures, and completes error action comparison and correction, and has high universality, small calculation amount, high accuracy, and well meets daily life requirements.The application uses a double comparison strategy of key frame action and time sequence comparison, first, key actions are used to correct actions of parts, and second, a complete action segment is compared.The correction strategy is more accurate and reasonable than general systems.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and particularly relates to a motion correction method, system, storage medium, computer device and terminal. Background Technology

[0002] In today's booming social and economic development, some people need more professional sports guidance, but private coaches are too expensive and the teaching quality varies. Therefore, many sports movement correction systems have emerged on the market.

[0003] Existing motion assistance systems commonly employ two methods: inertial motion capture and optical motion capture. Inertial motion capture systems consist of attitude sensors, signal receivers, and data processing systems. The attitude is fixed to major limbs, and the attitude signals are transmitted wirelessly via Bluetooth or other methods to the data processing system for motion calculation. The attitude sensors integrate inertial sensors, gravity sensors, accelerometers, and other elements to obtain the attitude information of each limb. This information is then combined with skeletal length information and skeletal hierarchical connections to calculate the spatial position information of joints. Optical motion capture, on the other hand, is based on computer vision principles. Multiple high-speed cameras monitor and track target feature points from different angles, while simultaneously using skeletal algorithms to complete motion capture. Theoretically, for any point in space, if it can be seen by two or more cameras simultaneously, its 3D position in space at that moment can be determined. When the cameras continuously capture images at a high frame rate, the motion trajectory of that point can be obtained from the image sequence, yielding meaningful metrics and other information.

[0004] However, the optical motion capture systems mentioned above also have many drawbacks, hindering their use and widespread adoption by everyday users and making deployment more convenient and efficient: 1) Difficult multi-camera setup. High cost and large space requirements. 2) Complex frame synchronization technology. The design of redundant hardware further complicates the system and increases maintenance costs. 3) High computational demands on computer vision and machine learning equipment. This makes it difficult to guarantee the portability and real-time processing of the entire system. 4) Poor portability. Current professional motion correction software is specifically designed for particular sports; if new movements are involved, porting becomes even more difficult. Therefore, there is an urgent need to design a new motion correction method and system.

[0005] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0006] (1) Existing optical motion capture systems are difficult to set up in multiple locations, costly, and require a large space, which is not conducive to daily use and popularization, and cannot be deployed more conveniently and quickly.

[0007] (2) The frame synchronization technology of existing optical motion capture systems is complex; the design of redundant hardware devices further complicates the system and increases maintenance costs.

[0008] (3) Existing computer vision and machine learning equipment have high computing power requirements, which makes it impossible to guarantee the portability and real-time processing of the entire system.

[0009] (4) Existing optical motion capture systems have poor portability. Currently, professional motion correction software is specially designed for specific sports. If new movements are involved, there will be greater difficulty in porting them.

[0010] The difficulty in addressing the above problems and shortcomings lies in the fact that the solutions adopted are currently the mainstream solutions for human movement correction systems, but their application scenarios are concentrated in professional settings. Since professional sports require professional guidance, it is essential to ensure good hardware and other characteristic conditions. Therefore, this invention faces significant challenges in reducing hardware and site costs while simultaneously ensuring the effectiveness and adaptability of the algorithm.

[0011] The significance of solving the above problems and defects is that it controls the development cost of the correction system, reduces site and hardware limitations, expands the application scenarios and business scope, is easy to use, and has a short iteration cycle. Summary of the Invention

[0012] To address the problems existing in the prior art, the present invention provides a motion correction method, system, storage medium, computer device, and terminal, and particularly relates to a motion correction method, system, storage medium, computer device, and terminal based on Cascade PoseNet human pose recognition.

[0013] This invention is implemented as follows: a method for correcting movement, the method comprising the following steps:

[0014] Step 1: Collect human motion posture recognition data in a targeted manner;

[0015] The purpose of this step is to acquire images of human movement. Because the features of images from sports activities vary greatly, unlike general human posture image acquisition, in addition to capturing normal movement postures, certain specific situations need to be considered, such as keypoint occlusion, complex backgrounds, motion blur, and poor lighting conditions. The dataset collected in this step contains more comprehensive human movement images, more diverse features, and considers a more specific range of posture categories. These dataset characteristics differ from traditional human posture estimation datasets, making them more conducive to training models for human posture detection and recognition in motion.

[0016] Step two: Design and train the neural network model for motion classification.

[0017] The purpose of this step is to design a lightweight human pose detection and recognition network, which performs two functions: skeletal keypoint location detection and human pose classification. The lightweight network, as the core of the entire algorithm, ensures the real-time performance of the system for deployment in embedded devices. Skeletal keypoint location detection obtains the 2D spatial coordinates of key points on the human skeleton. Since the spatial coordinates of standard movements are consistent, this information can be used to indicate whether the movement is correct or incorrect. Human pose classification determines the type of movement based on the keypoint coordinates, and can be used to match standard movements with movements to be corrected.

[0018] Step 3: Compare the pose of single-frame images and perform DTW distance comparison of the action time series.

[0019] The single-frame image posture comparison in this step specifically refers to comparing the standard movements of athletes with the movements to be corrected in ordinary people within a single static image. Its main function is to correct certain key movements in the movement of ordinary people by comparing the 2D positions of relevant skeletal keypoints in the static image with those of the incorrect ones. The DTW distance comparison of the action time series specifically refers to comparing a continuous movement between an athlete and an ordinary person in consecutive image frames. Its main function is to construct a sequence of the 2D spatial locations of the skeletal keypoints in consecutive image frames along the time direction, and then use the DTW algorithm to compare the similarity between the standard action sequence and the action sequence to be corrected, judging and scoring the action based on the similarity of the action sequences.

[0020] Furthermore, the collection of human motion posture recognition data in step one includes:

[0021] The dataset information is collected from everyday images and video data of athletes collected from the Internet. The video data is captured by extracting a screenshot of a specific action of an athlete from the video. The video data includes the movement process of a single tennis player.

[0022] All static pose images are three-channel color RGB images in JPG format. A dataset of action-labeled tennis key motion images was obtained from a video of appropriate length.

[0023] During training, the image dataset is divided into a training set, a validation set, and a test set. The training set is used to input the neural network for training during the training process. The validation set is used to periodically verify the rationality of the method during the training process. The test set is used to evaluate the performance of the method upon completion.

[0024] Furthermore, after collecting the human motion posture recognition data in step one, the process also includes image input and image enhancement; wherein, the methods for image input and image enhancement include:

[0025] Seven image enhancement strategies, including horizontal flipping, vertical flipping, random rotation of the image from 0 to 10 degrees, random variation of image brightness, random variation of image contrast, image distortion, and image scaling, are used to enhance the training images of the input model in pairs.

[0026] Furthermore, the design of the neural network for human pose recognition in step two includes:

[0027] The Cascade PoseNet neural network is used to train the collected human motion posture dataset. The input is a single person motion posture image, and the output is the coordinates of human posture key points in the 2D image and the category of the action in the image.

[0028] The network consists of a PoseNet human pose estimation network as the backbone and a classification network. The PoseNet part of the entire network outputs 17 human pose keypoints, including the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. At the same time, these 17 points are used as features and input into the subsequent classification network for classification to obtain the category of the action in the image.

[0029] The PoseNet part of Cascade PoseNet is a human motion pose estimation network. The core architecture of PoseNet is MobileNetv1, consisting of 28 layers. The first layer uses a standard convolutional kernel, and the remaining convolutional layers all use depthwise separable convolutions.

[0030] The convolutional layer is used to perform convolution operations on the input image data. During the analysis and training process of the input image, the neural network gradually extracts features from the gesture image dataset for analysis, progressing from shallow to deep. The basic convolution operation is as follows:

[0031]

[0032] Where x is the input image for convolution, h is the convolution kernel, and y is the result after convolution. Convolution is a basic computational method in image processing based on deep learning. By updating the parameters of the convolution kernel, the effect of extracting features from the input image can be achieved.

[0033] The batch standard layer is used to forcibly pull the distribution of input values, which gradually approach the saturation region of the value range after being mapped to a nonlinear function by each neuron in the neural network, back to a relatively standard normal distribution with a mean of 0 and a variance of 1. This ensures that the input values ​​of the nonlinear transformation function fall into a region that is more sensitive to the input.

[0034] In the final part of the neural network, a dropout layer is used for regularization, and a fully connected layer is used for classification to obtain the neural network prediction results. Finally, a 17-dimensional vector is output, which represents the x and y coordinates of the human pose key points predicted for the test image, the confidence level, and the category of the action.

[0035] Furthermore, the training of the motion classification neural network model in step two includes:

[0036] After designing the pose estimation neural network for training, the training set in the dataset is input into the network for computation. The training process uses a 5-fold cross-validation method, which divides the training set into 5 mutually exclusive subsets of the same size, and the ratio of the number of 7 types of images in each subset is close to 1:1:1.

[0037] Choose one of the five subsets as the validation set during training, and use the remaining four subsets as the training set; and so on, for a total of five batches of training, with each subset serving as the validation set once, and each batch being trained for 20 rounds.

[0038] During training, a batch size of 32 was used, and the Adam optimizer was employed as the optimization function. The momentum parameters were 0.9 and 0.99, and the initial learning rate was 0.01. The learning rate was intentionally decayed in each round until it reached 0.00001 in the last round.

[0039] An early stopping strategy is incorporated into the training process. The error of the model on the validation set is calculated every 15 epochs. Training stops when the error of the model on the validation set is worse than the previous training result, and the parameters from the previous iteration are used as the final parameters of the model.

[0040] Furthermore, the comparison of pose in single-frame images and the DTW distance comparison of action time series in step three include:

[0041] (1) Extraction of action segments from video stream

[0042] For each frame of the video stream captured by the user using the camera in real time, the image is input into the CascadePoseNet network to extract the coordinates of human pose key points and perform classification to obtain a single action frame that has been classified. The single action frame is then matched with its corresponding frame in the video stream, and a time sequence between the matched action frames is extracted.

[0043] The matching action combinations are forehand backswing and forehand swing, and backhand backswing and backhand swing. The remaining actions mainly involve correcting a single frame image.

[0044] The standard image dataset is input into the Cascade PoseNet network, and action frame capture and action frame time series matching are used. Then, the video stream read by the user through the camera is used to extract frames, resulting in a standard control group and a user test data group.

[0045] (2) Single-frame motion matching and DTW motion segment distance matching

[0046] Both single-frame image contrast correction and time-series DTW contrast correction were performed using a voting method. The "vote" method represents a voting process, including:

[0047] Identify key action points, compare the coordinate values ​​of each key point one by one, determine the parts where the movement is not in place, select the suggestion with the largest proportion, and vote to obtain the most reasonable correction suggestion.

[0048] For DTW time series motion comparison, the function of time series is explained first. Such a time series is a one-dimensional signal, where the horizontal axis represents time in milliseconds, and the vertical axis represents the x-value or y-value of a key point on the human body. This spatiotemporal signal reflects the changes in a specific part of the body during a complete single motion. It includes two elements: spatial information and temporal information. The spatial information includes coordinate values, reflecting the amplitude and range of a motion, the relative positions of key points, and whether other indicators are met. The temporal information includes whether a motion segment is too long or too short, and uses the DTW algorithm to test the similarity to a standard motion time series.

[0049] The system performs position comparison of key joints in a single action frame, performs fuzzy comparison of the (x,y) coordinates of key points in the two-dimensional image between the benchmark dataset and the user dataset to determine the movement range of a certain part; and performs DTW distance comparison of the action time series between a benchmark dataset and the user dataset to determine whether the timing of an action segment is appropriate.

[0050] Another object of the present invention is to provide a motion correction system for implementing the aforementioned motion correction method, the motion correction system comprising:

[0051] The motion posture recognition data acquisition module is used to collect human motion posture recognition data;

[0052] The network model building and training module is used to design and train neural network models for motion classification.

[0053] The action sequence capture, comparison, and correction module is used to compare the pose of single-frame images and perform DTW distance comparison of action time sequences.

[0054] Another object of the present invention is to provide a computer device comprising a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the following steps:

[0055] The process involves collecting human motion posture recognition data; designing and training a neural network model for motion classification; comparing postures in single-frame images and performing DTW distance comparisons of motion time series.

[0056] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the following steps:

[0057] The process involves collecting human motion posture recognition data; designing and training a neural network model for motion classification; comparing postures in single-frame images and performing DTW distance comparisons of motion time series.

[0058] Another objective of the present invention is to provide an information data processing terminal for implementing the aforementioned motion correction system.

[0059] Traditional motion correction systems are mostly used in professional settings such as national team training, resulting in high limitations in application scenarios. These systems consist of wearable devices, multi-camera high-definition, high-frame-rate cameras, high-performance hardware, and large-scale algorithms. All these factors directly lead to numerous problems, including high cost, cumbersome processes, significant space limitations, and high learning curve for the hardware and software systems used in motion correction. Due to these issues, such motion correction systems are currently difficult to popularize in everyday life, and a growing number of home users urgently need a suitable motion correction system to assist them in their physical exercise. The motion correction system proposed in this invention employs a single camera, a single-position, edge-embedded device, and a simple algorithm process, making it easy to learn and requiring minimal space constraints, thus specifically addressing the gap in the current market for this type of product.

[0060] Combining all the above technical solutions, the advantages and positive effects of this invention are as follows: The motion correction method provided by this invention is based on an improved optical motion capture system. It proposes a machine learning algorithm based on Cascade PoseNet for human pose recognition and DTW distance comparison to complete motion correction feedback. This algorithm requires less hardware, lower computing power, and has a fast processing speed and high accuracy. The entire system is portable and can complete motion correction and timely feedback tasks in most home and outdoor scenarios, thereby meeting the needs of most users.

[0061] The Cascade PoseNet cascaded network and the DTW distance comparison algorithm designed in this invention have achieved an 85% accuracy rate in correcting human motion postures, with a frame rate exceeding 20fps, according to experimental tests. This level of accuracy, achieved on a handheld device with a complex algorithm stack, is sufficient for the needs of everyday users. Therefore, this invention can provide motion correction feedback for different motion postures, achieving high accuracy and high-speed recognition.

[0062] In addition, the present invention also has the following beneficial effects:

[0063] (1) This invention uses a regular USB camera to acquire human motion images, and uses a laptop computer to infer the structure of a neural network model for human posture recognition. Through skeleton extraction and posture classification, it completes the comparison and correction of incorrect actions. It has high versatility, low computational load, and high accuracy, and well meets the needs of daily life.

[0064] (2) The present invention uses a dual comparison strategy of keyframe action and time series comparison. First, it corrects the action of the target part by comparing key actions, and second, it compares a complete action segment. Such a correction strategy is more accurate and reasonable than general systems.

[0065] (3) This invention proposes a Cascade algorithm, in which PoseNet performs pose recognition, and then activates a subsequent pose classification network to identify the human pose type in the current state. This cascade approach reduces the computational requirements of the system. Attached Figure Description

[0066] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a flowchart of the motion correction method provided in an embodiment of the present invention.

[0068] Figure 2 This is a structural block diagram of the motion correction system provided in an embodiment of the present invention;

[0069] In the diagram: 1. Motion posture recognition data acquisition module; 2. Network model construction and training module; 3. Action sequence extraction, comparison and correction module.

[0070] Figure 3 This is a schematic diagram illustrating the improvements brought about by the basic depthwise separable convolution in the PoseNet part provided in this embodiment of the invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0072] To address the problems existing in the prior art, the present invention provides a method, system, storage medium, computer device, and terminal for correcting movement. The present invention will now be described in detail with reference to the accompanying drawings.

[0073] like Figure 1 As shown, the motion correction method provided in this embodiment of the invention includes the following steps:

[0074] S101, collect human motion posture recognition data;

[0075] S102, Design and train the neural network model for motion classification;

[0076] S103 performs pose comparison of single-frame images and DTW distance comparison of action time series.

[0077] like Figure 2 As shown, the motion correction system provided in this embodiment of the invention includes:

[0078] The motion posture recognition data acquisition module 1 is used to collect human motion posture recognition data.

[0079] Module 2, which is used to design and train a neural network model for motion classification;

[0080] The action sequence capture, comparison and correction module 3 is used to compare the pose of a single frame image and the DTW distance of the action time sequence.

[0081] The technical solution of the present invention will be further described below with reference to specific embodiments.

[0082] The technical problem to be solved by this invention is to propose a machine learning algorithm based on Cascade PoseNet for human pose recognition and DTW distance comparison to complete the correction and feedback of movement. This algorithm requires less hardware, lower computing power, and has a fast processing speed and high accuracy. The whole system is portable and can complete the task of movement correction and timely feedback in most home and outdoor scenarios, thereby meeting the needs of most users.

[0083] The motion correction method based on human posture recognition and machine learning in this embodiment includes the following two stages: the establishment and training of a motion classification neural network, and the extraction, comparison and correction of motion sequences.

[0084] The first stage includes the following steps: collection of human motion posture recognition data, design of neural network models, and training of models.

[0085] I. Collection of Standard Motion Datasets

[0086] The dataset used in this invention is collected from everyday images, specifically five video clips of athletes gathered from the internet, ensuring that the athletes were placed in different backgrounds and lighting conditions as much as possible. The video clips contain the individual movement of a tennis player.

[0087] Using video frame extraction, a screenshot is taken of a specific action of the athlete in the video obtained above, such as: forehand backswing, forehand swing completion, backhand backswing, backhand swing completion, tennis volley, smash, and waiting. The specific examples are the above 7 types.

[0088] Five professional athletes were selected, and 2,000 images were extracted from each athlete's sports videos, for a total of 10,000 images. Each athlete's 2,000 images included seven different static sports images in equal proportions.

[0089] All static pose images are three-channel color RGB images in JPG format. A dataset of images representing seven key tennis movements with action labels can be obtained from a video of suitable length.

[0090] In actual training, the image dataset first needs to be divided into three parts: a training set, a validation set, and a test set. The training set is used to train the neural network during training, the validation set is used to periodically verify the method's effectiveness during training, and the test set is used to evaluate the method's performance upon completion. Of the total 10,000 images, 7,500 are selected for the training set, 500 for the validation set, and the remaining 2,000 for the test set, resulting in a data distribution ratio of 15:1:4, which conforms to deep learning training standards.

[0091] II. Image Input and Image Enhancement

[0092] The main task in this step is to train the human pose recognition neural network from the previous step on the static motion image recognition dataset. Before training the neural network, image enhancement processing is needed on the images to be trained. This increases the difficulty for the neural network to learn image features and appropriately expands the dataset. The final algorithm allows the entire network to more deeply mine the feature information of the images, achieving accurate classification results.

[0093] In response to the characteristics of the collected motion posture image dataset, this invention adopts the following seven image enhancement methods: horizontal image flipping, vertical image flipping, random image rotation of 0 to 10 degrees, random image brightness variation, random image contrast variation, image distortion, and image scaling. Finally, these methods are combined in pairs to enhance the training images of the input model.

[0094] III. Neural Network Structure for Human Pose Recognition

[0095] This invention uses the Cascade PoseNet neural network to train a collected human motion posture dataset. The input is a single person's motion posture image, and the output is the coordinates of 17 human posture key points in the 2D image and the category of the action in the image.

[0096] The network consists of a PoseNet human pose estimation network as the backbone and a classification network. The PoseNet part of the entire network outputs 17 human pose key points, including the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle, as shown in Table 1. At the same time, these 17 points are used as features and input into the subsequent classification network for classification to obtain the category of the action in the image.

[0097] Table 1 Key Points of Human Posture

[0098]

[0099]

[0100] The PoseNet part of Cascade PoseNet is a human motion pose estimation network. The core architecture of PoseNet is MobileNetv1, and its network structure is shown in Table 2. It consists of 28 layers (excluding Avg Pool and FC layers, and depthwise convolution and pointwise convolution are calculated separately). Except for the first layer, which uses a standard convolution kernel, the remaining convolutional layers all use depthwise separable convolution. Figure 3 This demonstrates the improvements brought by the basic depthwise separable convolution in PoseNet, which can reduce computation by about 9 times compared to standard convolution.

[0101] Table 2. Human motion posture estimation network structure

[0102]

[0103] The function of the convolutional layer is to perform convolution operations on the input image data, similar to the operation of traditional filters. During the analysis and training process of the input image, the neural network gradually extracts features from the gesture image dataset from shallow to deep for analysis. The basic convolution operation is as follows:

[0104]

[0105] Where x is the input image for convolution, h is the convolution kernel, and y is the result after convolution. Convolution is a basic computational method in image processing based on deep learning. By updating the parameters of the convolution kernel, the effect of extracting features from the input image can be achieved.

[0106] The function of the standard layer is to forcibly pull the distribution of input values—which gradually approach the saturation region of the value range after being mapped to a nonlinear function in each layer of the neural network—back to a relatively standard normal distribution with a mean of 0 and a variance of 1, through certain normalization methods. This ensures that the input values ​​of the nonlinear transformation function fall into a region that is more sensitive to the input, thereby avoiding the gradient vanishing problem. At the same time, the increase in gradient is equivalent to an increase in the convergence rate of learning, which can greatly reduce the training time.

[0107] To avoid overfitting in deep neural networks, this invention uses a dropout layer for regularization in the final part of the neural network and a fully connected layer for classification to obtain the neural network prediction results. Finally, a 17-dimensional vector is output, representing the x and y coordinates and confidence of the human pose key points predicted from the test image, as well as the category of the action.

[0108] IV. Cross-validation training

[0109] After designing the pose estimation neural network for training, the training set from the dataset needs to be input into the network for computation. The training process uses a 5-fold cross-validation method, dividing the training set into 5 mutually exclusive subsets of the same size. Each subset contains 1600 pose images, and the ratio of the 7 types of images in each subset is approximately 1:1:1.

[0110] First, one subset is randomly selected from the five subsets as the validation set during training, and the remaining four subsets are used as the training set. This process is repeated for a total of five batches, with each subset serving as the validation set once, and each batch undergoing 20 training iterations. This cross-validation training method fully utilizes the dataset, enabling the neural network to learn the full feature information of the images and effectively avoiding overfitting.

[0111] During training, a batch size of 32 was used, and the Adam optimizer was employed with momentum parameters of 0.9 and 0.99. The initial learning rate was 0.01. The learning rate was intentionally decayed in each round, reaching a final learning rate of 0.00001 in the last round.

[0112] To prevent overfitting, this invention incorporates an early stopping strategy during training. The error of the model on the validation set is calculated every 15 epochs. Training stops when the error on the validation set is worse than the previous training result, and the parameters from the previous iteration are used as the final parameters of the model.

[0113] Finally, after training the entire neural network model for 180 rounds, the model accuracy reached 98.9%. At this point, the inference speed of the model on the handheld computer was 23ms / frame, and the entire model occupied a size of 5Mb.

[0114] The second stage includes comparing poses in single-frame images and comparing DTW distances in action time series.

[0115] I. Extracting Action Clips from Video Streams

[0116] For each frame of the video stream captured by the user using the camera in real time, the image is input into the CascadePoseNet network to extract the coordinates of human pose key points and perform classification, resulting in a single action frame that has been classified. The obtained single action frame is then matched with its corresponding frame in the video stream (for example, the forehand backswing and the completion of the forehand swing are a matching action), and a time sequence between the matched action frames is extracted.

[0117] The specific motion combinations to be matched are the forehand backswing and forehand swing completion, and the backhand backswing and backhand swing completion. These two sets of movements are the most fundamental in tennis and have significant corrective value. The remaining movements mainly require correction of single frames.

[0118] The standard image dataset is input into the Cascade PoseNet network, and action frame capture and action frame time series matching are used. Then, the video stream read by the user through the camera is used to extract frames, which also completes the above process. In this way, a standard control group and a user test data group are obtained.

[0119] II. Single-frame motion comparison and DTW motion segment distance comparison

[0120] Both single-frame image comparison correction and time-series DTW comparison correction employ a voting method. The voting method is used because for a user's specific movement, there are over 1000 standard movements in the standard dataset. While most of these standard movements are relatively correct, even professional athletes may make mistakes. Therefore, in single-frame image movement comparison, this invention uses the following approach: First, it identifies key movement points. For example, the main points of a racket swing are the right wrist, right elbow, and right shoulder, along with the auxiliary left arm system. This invention compares these six key points, totaling 12 coordinate values, one by one to determine if a particular part of the movement is inaccurate. The voting concept is reflected in the fact that among the over 1000 correction suggestions for a certain part of the movement, this invention selects the suggestion with the highest proportion and votes for the most reasonable correction suggestion, which is more reasonable.

[0121] For DTW time series motion comparison, let's first explain the function of time series. Such a time series is a one-dimensional signal, where the horizontal axis represents time in milliseconds, and the vertical axis represents the x-value (or y-value) of a key point on the human body. This spatiotemporal signal reflects the changes in a specific part of a body during a complete, single motion. It contains two elements: simple spatial information and temporal information. Spatial information includes the coordinate values, reflecting the amplitude and range of a motion, the relative positions of key points, and whether other indicators are in place. Temporal information includes whether a motion segment is too long or too short. The DTW algorithm is used to test the similarity to a standard motion time series, especially for the x or y coordinates of a specific body part, providing more precise and detailed analysis.

[0122] The system performs position comparison of key joints in a single action frame. Specifically, this involves comparing the (x,y) coordinates of key points in a 2D image between the benchmark dataset and the user dataset to determine the range of motion of a certain part. It also performs DTW distance comparison of the action time series between a benchmark dataset and the user dataset to determine whether the timing of an action segment is appropriate.

[0123] This invention uses a regular USB camera to acquire images of human motion, and then uses a laptop computer to infer the structure of a neural network model for human posture recognition. Through skeleton extraction and posture classification, it completes the comparison and correction of erroneous movements. It has high versatility, low computational load, and high accuracy, and well meets the needs of daily life.

[0124] This invention employs a dual comparison strategy: keyframe action and time series comparison. First, it corrects actions targeting specific body parts using key actions; second, it compares complete action segments. This correction strategy is more accurate and reasonable compared to conventional systems. This invention also proposes a Cascade algorithm, where PoseNet performs pose recognition before activating a subsequent pose classification network to identify the current human pose type. This cascade approach reduces the computational requirements of the system.

[0125] The technical effects of the present invention will be described in detail below with reference to simulation experiments.

[0126] 1. Experimental conditions:

[0127] The test platform for this invention is the LattePanda Delta, configured with an 8th generation Intel Celeron N4100 processor. This processor is the perfect choice in terms of both price and performance when used as a robot controller, the core of interactive projects, an IoT edge device, and an AI brain. The operating system used is Ubuntu 16.04.

[0128] The camera used is a Logitech Webcam C270, with a resolution and frame rate of 720p / 30fps, fixed focal length, and a field of view of 60°.

[0129] The software platforms used were VS Code, OpenCV, and Electron, and a three-channel RGB image dataset with a resolution of 640px×480px was tested.

[0130] 2. Experimental Results:

[0131] The Cascade PoseNet cascaded network and the DTW distance comparison algorithm designed in this invention have achieved an 85% accuracy rate in correcting human motion, with a frame rate exceeding 20fps, according to experimental tests. This level of accuracy, achieved on a handheld device with a complex algorithm stack, is generally sufficient for the needs of everyday users.

[0132] In summary, this invention can provide corrective feedback for different motion postures, achieving high accuracy and high recognition speed.

[0133] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented, in whole or in part, as a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0134] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for correcting movement, characterized in that, The method for correcting movement includes the following steps: Step 1: Collect human motion posture recognition data; Step two: Design and train the neural network model for motion classification. Step 3: Compare the pose of single-frame images and perform DTW distance comparison of the action time series; The collection of human motion posture recognition data in step one includes: The dataset information is collected from everyday images and video data of athletes from the Internet. The video data is extracted by taking screenshots of a specific action of an athlete from the video. The video data includes the movement process of a single tennis player. All static pose images are three-channel color RGB images in JPG format; a dataset of action-labeled tennis key motion images was obtained from a video of appropriate length. During training, the image dataset is divided into a training set, a validation set, and a test set. The training set is used to input data into the neural network for training. The validation set is used to periodically verify the rationality of the method during training. The test set is used to evaluate the performance of the method upon completion. The comparison of pose in single-frame images and the DTW distance comparison of action time series in step three include: (1) Extraction of action segments from video stream For each frame of the video stream captured by the user using the camera in real time, the image is input into the Cascade PoseNet network to extract the coordinates of human pose key points and complete the classification to obtain a single action frame that has been classified. The single action frame is then matched with its corresponding frame in the video stream, and a time sequence between the matched action frames is extracted. The matching action combinations are forehand backswing and forehand swing, and backhand backswing and backhand swing. The remaining actions mainly correct single-frame images. The standard image dataset is input into the Cascade PoseNet network, and action frame capture and action frame time series matching are used. Then, the frame is extracted from the video stream read by the user through the camera to obtain the standard control group and the user test data group. (2) Single-frame motion matching and DTW motion segment distance matching Both single-frame image contrast correction and time-series DTW contrast correction are performed using a vote method; the vote method represents a voting process, including: Identify key action points, compare the coordinate values ​​of each key point one by one, determine the parts where the movement is not in place, select the suggestion with the largest proportion, and vote to obtain the most reasonable correction suggestion; For DTW time series motion comparison, the role of the time series is first explained. This time series is a one-dimensional signal, where the horizontal axis represents time in milliseconds, and the vertical axis represents the x or y value of a key point on the human body. This spatiotemporal signal reflects the changes in a specific part of the body during a complete single motion. It includes two elements: spatial information and temporal information. The spatial information includes coordinate values, reflecting the amplitude and range of a motion and whether the relative positions of the joints are correct. The temporal information includes whether a motion segment is too long or too short, and uses the DTW algorithm to test the similarity to a standard motion time series. The system performs position comparison of key joints in a single action frame, compares the (x,y) coordinates of key points in the two-dimensional image between the benchmark dataset and the user dataset to determine the movement range of a certain part; and compares the DTW distance between the action time series of a benchmark dataset and the user dataset to determine whether the timing of an action segment is appropriate.

2. The motion correction method as described in claim 1, characterized in that, After collecting human motion posture recognition data in step one, the process further includes image input and image enhancement; wherein, the methods for image input and image enhancement include: Seven image enhancement strategies, including horizontal flipping, vertical flipping, random rotation of the image from 0 to 10 degrees, random variation of image brightness, random variation of image contrast, image distortion, and image scaling, are used to enhance the training images of the input model in pairs.

3. The motion correction method as described in claim 1, characterized in that, The design of the neural network for human pose recognition in step two includes: The Cascade PoseNet neural network is used to train the collected human motion posture dataset. The input is a single person motion posture image, and the output is the coordinates of human posture key points in the 2D image and the category of the action in the image. The network consists of a PoseNet human pose estimation network as the backbone and a classification network. The PoseNet part of the entire network outputs 17 human pose key points, including nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. At the same time, these 17 points are used as features and input into the subsequent classification network part for classification to obtain the category of the action in the image. The PoseNet part of Cascade PoseNet is a human motion pose estimation network. The core architecture of PoseNet is MobileNetv1, consisting of 28 layers. The first layer uses a standard convolutional kernel, and the remaining convolutional layers all use depthwise separable convolutions. The convolutional layer is used to perform convolution operations on the input image data. During the analysis and training process of the input image, the neural network gradually extracts and analyzes the features of the gesture image dataset from shallow to deep. The convolution operation is as follows: ； in, x The image is the input for convolution. h For convolution kernel, y The result after convolution is the image. Convolution is a computational method in image processing based on deep learning. By updating the parameters of the convolution kernel, the effect of extracting features from the input image is achieved. The batch standard layer is used to force the distribution of input values, which are gradually approaching the limit saturation region of the value range after being mapped to a nonlinear function by each neuron in the neural network, back to a relatively standard normal distribution with a mean of 0 and a variance of 1; so that the input value of the nonlinear transformation function falls into a region that is more sensitive to the input. In the final part of the neural network, a dropout layer is used for regularization, and a fully connected layer is used for classification to obtain the neural network prediction results. Finally, a 17-dimensional vector is output, which represents the x and y coordinates of the human pose key points predicted for the test image, the confidence level, and the category of the action.

4. The motion correction method as described in claim 1, characterized in that, The training of the motion classification neural network model in step two includes: After designing the pose and action estimation neural network for training, the training set in the dataset is input into the network for computation. The training process uses a 5-fold cross-validation method, which divides the training set into 5 mutually exclusive subsets of the same size, and the ratio of the number of 7 types of images in each subset is close to 1:1:

1. Choose one subset from the five subsets as the validation set during training, and use the remaining four subsets as the training set; and so on, for a total of five batches of training, with each subset serving as the validation set once, and each batch training for 20 rounds; During training, a batch size of 32 was used, and the Adam optimizer was employed as the optimization function. The momentum parameters were 0.9 and 0.99, and the initial learning rate was 0.

01. The learning rate was intentionally decayed in each round until it reached 0.00001 in the last round. An early stopping strategy is incorporated into the training process. The error of the model on the validation set is calculated every 15 epochs. Training stops when the error of the model on the validation set is worse than the previous training result, and the parameters from the previous iteration are used as the final parameters of the model.

5. A motion correction system implementing the motion correction method according to any one of claims 1 to 4, characterized in that, The motion correction system includes: The motion posture recognition data acquisition module is used to collect human motion posture recognition data; The network model building and training module is used to design and train neural network models for motion classification. The action sequence capture, comparison, and correction module is used to compare the pose of single-frame images and perform DTW distance comparison of action time sequences.

6. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the motion correction method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the motion correction method as described in any one of claims 1 to 4.

8. An information data processing terminal, characterized in that, The information data processing terminal is used to implement the motion correction system as described in claim 5.

Images