A Tai Chi movement correction method and system based on generative adversarial network
Through generation and adversarial network technology, detecting and identifying human joints and correcting Tai Chi movements in real time, solving the problems of traditional equipment limiting the range of motion and high cost, achieving low-cost and accurate Tai Chi movement correction and convenient practice experience.
Patent Information
- Application Number
- CN202111201371.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-15
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-10-15
AI Technical Summary
The existing indoor Tai Chi training devices rely on wearable devices, limiting the user's range of movement and high cost, making it difficult to promote.
Using a method based on generative adversarial networks, through computer vision and deep learning technology, we detect and identify human joints, generate standard Tai Chi action sequences, correct users' Tai Chi action in real time, reduce costs and improve practice convenience.
It realizes low-cost and accurate Tai Chi movement correction, improves the convenience of practice and the accuracy of movement evaluation, and avoids the limitation of movement amplitude of traditional equipment.
Smart Images

Figure CN113947811B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence and image recognition technology, and in particular to a Tai Chi action correction method and system based on a generative adversarial network. Background Art
[0002] At present, existing indoor Tai Chi training devices mainly use wearable devices fixed on the user to collect the user's movement data. Although this method can correct the user's Tai Chi movements, it limits the user's range of motion. Moreover, this wearable device is expensive, which is not conducive to user use and Tai Chi promotion. Summary of the invention
[0003] The purpose of the present invention is to provide a Tai Chi movement correction method and system based on a generative adversarial network, which can correct the user's Tai Chi movements at a low cost and realize Tai Chi teaching.
[0004] To achieve the above object, the present invention provides the following solutions:
[0005] A Tai Chi action correction method based on a generative adversarial network, comprising:
[0006] Acquire a human body motion image of the user at the current stage; the human body motion image is a motion image captured when practicing Tai Chi;
[0007] Based on the human body motion image of the user in the current stage and the trained generative adversarial network model, a standard motion sequence of the user in the next stage is determined; the data in the standard motion sequence is standard Tai Chi skeleton joint point data;
[0008] Obtaining the user's human body action image for the next stage;
[0009] Calculating the similarity between the standard action sequence of the user at the next stage and the human body action image of the user at the next stage;
[0010] According to the similarity, the user's Tai Chi exercises are corrected.
[0011] Optionally, before acquiring the human body movement image of the user at the current stage, the Tai Chi movement correction method further includes a preprocessing part; the preprocessing part is:
[0012] Acquire a human body motion image of a user in an initial stage; the initial stage is a stage before the user practices Tai Chi;
[0013] Determining whether the human body motion image of the user in the initial stage includes a set human body joint point;
[0014] If yes, then obtain the user's human body motion image at the current stage;
[0015] If not, the user position adjustment information is output, and after the user adjusts the position, the human body motion image of the user in the initial stage is re-photographed, and the process returns to determine whether the human body motion image of the user in the initial stage includes the set human body joint points.
[0016] Optionally, the step of determining whether the user's human body motion image in the initial stage includes all human body joints includes:
[0017] The openpose model is used to process the human body motion image of the user in the initial stage to obtain human body joint data in the initial stage;
[0018] Calculating the confidence of each joint point of the human joint data in the initial stage;
[0019] Determine whether the confidence level of each joint point is greater than 0;
[0020] Among them, when the confidence of each of the joint points is greater than 0, the human body motion image of the user in the current stage is obtained; when the confidence of the joint point is less than or equal to 0, the adjusted user position information is output, and after the user adjusts the position, the human body motion image of the user in the initial stage is re-shot, and the step of returning to determine whether the human body motion image of the user in the initial stage includes the set human body joint points.
[0021] Optionally, determining a standard action sequence of the user in the next stage based on the human action image of the user in the current stage and the trained generative adversarial network model specifically includes:
[0022] The openpose model is used to process the human body motion image of the user at the current stage to obtain human body joint data at the current stage;
[0023] The human body joint data of the current stage is input into the trained generative adversarial network model to determine the standard action sequence of the user in the next stage.
[0024] Optionally, calculating the similarity between the standard action sequence of the user at the next stage and the human body action image of the user at the next stage specifically includes:
[0025] The openpose model is used to process the human body motion image of the user in the next stage to obtain human body joint data in the next stage;
[0026] Based on the standard action sequence of the user in the next stage and the human joint data in the next stage, the similarity between each group of standard joint points and the user's human joint points matching the standard joint points is calculated; the standard action sequence of the user in the next stage includes action data of multiple standard joint points, and the human joint data in the next stage includes action data of multiple user's human joint points.
[0027] Optionally, according to the similarity, correcting the user's Tai Chi exercises, specifically including:
[0028] Determine in turn whether each of the similarities is greater than a set threshold;
[0029] If so, the region of the user's body joint points corresponding to the similarity is determined as a qualified region;
[0030] If not, the area of the user's body joint points corresponding to the similarity is determined as an unqualified area, and the Tai Chi movements in the unqualified area are corrected.
[0031] Optionally, the generative adversarial network model includes a skeleton sequence-based discriminator, a frame-based discriminator and an action generator; wherein the action generator includes a GRU layer, a temporal attention module and a GCN network.
[0032] A Tai Chi action correction system based on a generative adversarial network, comprising:
[0033] The current stage data acquisition module is used to acquire the human body motion image of the user in the current stage; the human body motion image is the motion image captured when practicing Tai Chi;
[0034] A standard action sequence determination module for the user at the next stage, used to determine the standard action sequence for the user at the next stage based on the human action image of the user at the current stage and the trained generative adversarial network model; the data in the standard action sequence is standard Tai Chi skeleton joint point data;
[0035] The next stage data acquisition module is used to acquire the human body motion images of the user in the next stage;
[0036] A similarity calculation module, used for calculating the similarity between the standard action sequence of the user in the next stage and the human body action image of the user in the next stage;
[0037] The correction module is used to correct the user's Tai Chi exercises according to the similarity.
[0038] A Tai Chi action correction system based on a generative adversarial network comprises: a background server and a mirror-surface integrated machine; wherein the mirror-surface integrated machine is provided with a camera and a display;
[0039] The camera is used to capture the user's human body motion image in real time;
[0040] The display is used to play Tai Chi demonstration movements practiced by the user;
[0041] The backend server is used for:
[0042] Acquire a human body motion image of the user at the current stage; the human body motion image is a motion image captured when practicing Tai Chi;
[0043] Based on the human body motion image of the user in the current stage and the trained generative adversarial network model, a standard motion sequence of the user in the next stage is determined; the data in the standard motion sequence is standard Tai Chi skeleton joint point data;
[0044] Obtaining the user's human body action image for the next stage;
[0045] Calculating the similarity between the standard action sequence of the user at the next stage and the human body action image of the user at the next stage;
[0046] Outputting a correction action instruction according to the similarity; the correction action instruction is a Tai Chi action correction instruction for the unqualified area;
[0047] The display is also used to display the unqualified area and the Tai Chi correction movements of the unqualified area.
[0048] Optionally, a user end is also included;
[0049] The user terminal is used to input the user's physical data and select the Tai Chi movements to be practiced, and upload the user's physical data and the Tai Chi movements to be practiced to the background server.
[0050] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0051] The present invention uses computer vision and deep learning technology to detect human joints and identify and evaluate movements. Compared with the traditional use of sensors and wearable devices, it reduces costs, improves the convenience of Tai Chi practice, and improves the accuracy of movement evaluation. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0053] Figure 1A schematic diagram of a flow chart of a Tai Chi action correction method based on a generative adversarial network according to the present invention;
[0054] Figure 2 A schematic diagram of a specific process of a Tai Chi action correction method based on a generative adversarial network according to the present invention;
[0055] Figure 3 A model architecture diagram of a generative adversarial network model of the present invention;
[0056] Figure 4 is a network structure diagram of the action generator of the present invention;
[0057] Figure 5 The network structure diagram of the skeleton sequence-based discriminator of the present invention;
[0058] Figure 6 is a network structure diagram of a frame-based discriminator of the present invention;
[0059] Figure 7 is a flow chart of the action scoring algorithm of the present invention;
[0060] Figure 8 It is a schematic diagram of human joint points of the present invention;
[0061] Fig. 9 A schematic diagram of the structure of a Tai Chi action correction system based on a generative adversarial network according to the present invention;
[0062] Fig.10 This is an information interaction diagram of a Tai Chi movement correction system based on a generative adversarial network according to the present invention. DETAILED DESCRIPTION
[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0064] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0065] Embodiment 1
[0066] like Figure 1 As shown, this embodiment provides a Tai Chi action correction method based on a generative adversarial network, including:
[0067] Step 101: Acquire a human body motion image of the user at the current stage; the human body motion image is a motion image captured when practicing Tai Chi.
[0068] Step 102: Based on the human body motion image of the user in the current stage and the trained generative adversarial network model, determine the standard motion sequence of the user in the next stage; the data in the standard motion sequence is standard Tai Chi skeleton joint point data.
[0069] Step 103: Obtain the user's human body motion image in the next stage.
[0070] Step 104: Calculate the similarity between the standard action sequence of the user at the next stage and the human body action image of the user at the next stage.
[0071] Step 105: Correct the user's Tai Chi exercises according to the similarity.
[0072] Furthermore, before executing step 101, the Tai Chi movement correction method further includes a preprocessing part; the preprocessing part is:
[0073] Acquire a human body motion image of the user in the initial stage; the initial stage is the stage before the user practices Tai Chi; determine whether the human body motion image of the user in the initial stage includes set human body joints; if so, acquire the human body motion image of the user in the current stage; if not, output the adjusted user position information, and when the user adjusts the position, re-shoot the human body motion image of the user in the initial stage, and return to the step of determining whether the human body motion image of the user in the initial stage includes the set human body joints.
[0074] The step: determining whether the user's human body motion image in the initial stage includes all the joints of the human body, specifically including:
[0075] The human body motion image of the user in the initial stage is processed using the openpose model to obtain the human body joint data in the initial stage; the confidence of each joint point of the human body joint data in the initial stage is calculated; and it is determined whether the confidence of each joint point is greater than 0.
[0076] When the confidence of each of the joint points is greater than 0, the human body motion image of the user in the current stage is obtained; when the confidence of the joint point is less than or equal to 0, the adjusted user position information is output, and after the user adjusts the position, the human body motion image of the user in the initial stage is re-shot, and the step of returning to determine whether the human body motion image of the user in the initial stage includes the set human body joint points.
[0077] The step 102 specifically includes:
[0078] The openpose model is used to process the human body motion image of the user in the current stage to obtain the human body joint data in the current stage; the human body joint data in the current stage is input into the trained generative adversarial network model to determine the standard motion sequence of the user in the next stage.
[0079] The step 104 specifically includes:
[0080] The human body motion image of the user in the next stage is processed by using an openpose model to obtain the human body joint data of the next stage; based on the standard motion sequence of the user in the next stage and the human body joint data of the next stage, the similarity between each group of standard joint points and the user's human body joint points matching the standard joint points is calculated; the standard motion sequence of the user in the next stage includes the motion data of multiple standard joint points, and the human body joint data of the next stage includes the motion data of multiple user's human body joint points.
[0081] The step 105 specifically includes:
[0082] Determine in turn whether each of the similarities is greater than a set threshold; if so, determine the area of the user's body joints corresponding to the similarity as a qualified area; if not, determine the area of the user's body joints corresponding to the similarity as an unqualified area, and correct the Tai Chi movements in the unqualified area.
[0083] The generative adversarial network model includes a skeleton sequence-based discriminator, a frame-based discriminator and an action generator; wherein the action generator includes a GRU layer, a temporal attention module and a GCN network.
[0084] Embodiment 2
[0085] See also Figure 2 This embodiment provides a Tai Chi action correction method based on a generative adversarial network, and the main process is as follows:
[0086] Step 1: Log in to the WeChat applet and enter the user's physical data.
[0087] Step 2: Select the Tai Chi moves you want to practice through the WeChat applet, and the WeChat applet transmits the Tai Chi moves selected by the user to the backend server.
[0088] Step 3: The user stands in front of the mirror-mounted machine and in a suitable position so that the mirror-mounted machine can completely capture the user's human body movement image; wherein the mirror-mounted machine is composed of a display, a plane mirror and a camera, the display is used to play the demonstration movement, and the plane mirror is used for the user to view his own movement.
[0089] The camera will take a human action image every second (the human action image here is the human image when Tai Chi training is not performed), and send the human action image to the background server for detection. The background server determines whether the human action image contains all the user's human joints according to the corresponding algorithm. The main judgment process is: use the openpose model to identify the human joint data of the captured human action image, traverse the 25 joints of the human body (including 0-nose, 1-neck, 2-right shoulder, 3-right elbow, 4-right wrist, 5-left shoulder, 6-left elbow, 7-left wrist, 8-mid-hip, 9-right hip , 10-right knee, 11-right ankle, 12-left hip, 13-left knee, 14-left ankle, 15-right eye, 16-left eye, 17-right ear, 18-left ear, 19-left big toe, 20-left little toe, 21-left heel, 22-right big toe, 23-right little toe, 24-right heel), determine whether the confidence of each joint point in the recognition result is greater than 0. If so, it is determined that the complete human body is successfully detected. The background server controls the display to play the corresponding Tai Chi moves for training according to the Tai Chi moves transmitted by the WeChat applet. Otherwise, the user will be prompted by voice to stand in the appropriate position and continue to capture human action images.
[0090] Step 4: After reaching the teaching page, the monitor will play the Tai Chi movements demonstrated by the coach, and users can practice along.
[0091] The camera still takes a human action image every second (the human action image at this time is an image of a human body doing Tai Chi movements) and sends it to the background server for action detection and scoring. The background server uses the openpose model to identify the human joint data in the human action image, inputs it into the trained generative adversarial network model, and outputs the standard Tai Chi skeleton joint data for the next few seconds. This standard Tai Chi skeleton joint data is the subsequent prediction based on the user's current human joint data and serves as the standard data for action evaluation. When the user moves to the next second, the user's human joint data in the next second is compared with the previously generated standard Tai Chi skeleton joint data for similarity to calculate the action score of each joint. When the score is lower than the threshold, it is determined to be an incorrect action. At this time, the user is prompted to correct the action through voice. At the same time, the user's human joint data in the next second is also input into the trained generative adversarial network model to generate standard data for subsequent action evaluation. The above process is repeated.
[0092] The generative adversarial network model used to generate standard Tai Chi skeleton joint data mainly consists of two discriminators and an action generator G. The model architecture is shown in the figure below. Figure 3 As shown in Figure 2, the two discriminators are: the discriminator D based on the skeleton sequence V and the frame-based discriminator D F .
[0093] The action generator G is used to generate a series of subsequent standard actions based on the current human action image. Assume that the skeleton data consists of 2D coordinates of N joints, the data set is divided into n types of actions, and predicts the subsequent T frames of human actions. The input of the action generator G consists of three parts: 1. The initial frame human skeleton A 2×N vector represents the two-dimensional coordinates of the joint, with a data range of [-1,1]; 2. Random Gaussian noise Generate C for each frame through Gaussian noise z dimensional latent vector; 3. Label representing the action category An n-dimensional One-hot vector. The network structure of the action generator G is mainly composed of GRU layer, temporal attention module, and GCN network; the GRU layer enables the action generator G to learn the temporal feature distribution of the skeleton sequence, the temporal attention module can select similar skeleton aggregation features, and the GCN network converts high-dimensional features into structured human skeletons.
[0094] The network structure of the action generator G is as follows Figure 4 First, a 1D convolution with a convolution kernel of 1 is used to transform the label vector l, and the feature dimension is enlarged from n to C. l The convolutional label vector l′ is connected in series with the noise z, and the two features are fused using 1D convolution; the convolutional features are used as the input of the RNN layer. The RNN layer has a cyclic structure and can remember the previous and next inputs and extract the global features of the time series. Therefore, using a single GRU layer enables the action generator G to map the Gaussian noise sequence to a skeleton sequence. Assume that the sequence length is T and the GRU layer input is C h ×T size feature tensor, output is C G ×T size feature tensor, represented as [o0,o1,o2,...,o T-1 ]. In order to enhance the network's memory of action categories, the output of the GRU layer is connected in series with the label vector l' on the feature channel. The time series has continuity. The action of the current frame will be affected by all previous frames. Changes that occur before time t will affect time t. In order to make full use of the temporal information, the output of the GRU layer is residually accumulated. The output after residual connection is [c0,c1,c2,...,c T-1 ], where c0=o0, c1=o1+c0,...,c T-1 =o T-1 +c T-2. The GRU layer is followed by two temporal attention modules, which are used to enhance the use of temporal information. After the temporal attention module, a feature tensor of size 2V×T is obtained, which is deformed into 2×1×T×V and input into a 5-layer GCN network, where 2 represents the two channels of the joint 2D coordinates X and Y. The GCN network learns the topological structure of the human body, assigns weights to different joint connections, fuses the features of adjacent joints, and outputs a tensor of size 2×128×T×V. Finally, a 2D convolution with a convolution kernel of 1×1 is used to compress the feature channel from 128 to 1, and then the value range is normalized to -1~1 through the Tanh function to obtain the joint coordinates of the skeleton sequence.
[0095] Figure 5 is the skeleton sequence based discriminator D V . Discriminator D V Take the complete skeleton sequence and label as input, first use two "1D convolution with convolution kernel 1 + LeakyReLU" to expand the feature dimension of the skeleton sequence to 1000, then concatenate the label and the feature, use two "1D convolution with convolution kernel 1 + LeakyReLU" to compress the feature dimension to 128, and then use 1D convolution with convolution kernel 3 to compress 128 dimensions to 1 dimension, and finally use two fully connected layers for pooling. One fully connected layer is pooled to n dimensions, and then Softmax is used to obtain the probability that the skeleton sequence belongs to each type of action; the other fully connected layer is pooled to 1 dimension, and the Sigmoid function is used to obtain the probability that the skeleton sequence is real data.
[0096] Figure 6 is the frame-based discriminator D F , whose network structure is similar to the discriminator D V Similar. Discriminator D V It not only determines whether the output sequence is real data or generated by the action generator G, but also outputs the action category judgment of the sequence, while the discriminator D F Only judge the authenticity of the input frame. Randomly select k from the complete skeleton sequence frame Frame Input Discriminator D F First, two "1D convolution with kernel 1 + LeakyReLU" are used to expand the input feature dimension to 512. In order not to lose information, the input is used as the residual after passing through a fully connected layer and is residually connected with the 1D convolution output; then the label and feature are connected in series, and then two "1D convolution with kernel 1 + LeakyReLU" are used to compress the feature dimension to 128; finally, two fully connected layers are used to pool the feature channels and the number of frames to 1, and the probability of whether the input is true is obtained through Sigmoid.
[0097] The action generator G generates the most realistic action possible based on the Tai Chi joint data of the user in the current frame, and the discriminator DV and frame discriminator D F To distinguish generated data from real data as much as possible, the objective function of the action prediction network is:
[0098]
[0099] Among them, the discriminator D V and discriminator D F The loss of L F and loss L V They are:
[0100] L F =E x~p(x) [log(D F (x|y))]+E z~p(z) [log(1-D F (G(z|y)))];
[0101] L V =E x~p(x) [log(D V (x|y))]+E z~p(z) [log(1-D V (G(z|y)))]+log(c x )+log(c g );
[0102] Loss L F The discriminator D F The cross entropy loss L is used to distinguish true and false data from generated data. V It consists of two parts: the cross entropy for distinguishing true from false and the cross entropy for distinguishing action categories. x and c g The softmax probability values of the action category labels corresponding to the real action skeleton sequence and the action skeleton sequence generated by the generator are respectively. The network is trained using an alternating update strategy, for the action generator G and the discriminator D V , discriminator D F There are three networks. The parameters of two networks are fixed each time and only the remaining network is updated. The training algorithm is as follows.
[0103] Algorithm 1 GAN training algorithm
[0104] Input: data x, initial frame I, category label l, true or false label y
[0105]
[0106]
[0107] The dataset used in this embodiment comes from a real Tai Chi video dataset, which is collected from videos of 132 Tai Chi coaches and contains 2956 video clips. The main content is Chen-style Tai Chi Old Frame 1 to 6 and the closing posture, which is subdivided into 25 types of movements. The collected Tai Chi videos are all shot from the front, and the subjects always remain in the picture; the Tai Chi movements of the subjects are standard, and the generative adversarial network model can extract features well.
[0108] Dataset processing steps: First, use OpenPose to extract the human skeleton joint coordinates in the video, and process missing data and illegal values through data cleaning. In order to enhance the generalization ability, the joint coordinates are normalized, the camera movement is simulated, and the data is filled in for data enhancement.
[0109] Figure 7 The flowchart of the action scoring algorithm of the present invention is shown in Figure 1. According to the movement characteristics of Tai Chi and the different contributions of each joint point to the description of the action state, the joint point data information is divided into six parts, namely: left hand area, right hand area, left leg area, right leg area, leg spacing, and overall coordination. For different groups of joint point data in each part, the scores of the relative distance difference of the joint point and the joint angle difference are obtained respectively.
[0110] The method for calculating the joint point distance difference score is as follows:
[0111] First, the Euclidean distance of each pair of joint points is calculated; since the proportions of different images may be different, directly using absolute distance calculation may cause errors, and the proportion of the human spine length in different images is basically fixed, that is, Figure 8 The distance between the two joints 1 and 8 is the distance between the two joints 1 and 8. Therefore, the improved formula divides the absolute distance by the length of the spine to get the relative distance. Let the two joints be A(x i ,y i )、B(x j ,y j ), i and j are the indexes of the two nodes respectively, and the spine length is d0, then the relative distance is:
[0112]
[0113] Secondly, calculate the relative distance score; let the relative distance of the standard data be d std (i,j), the relative distance of user data is d usr (i, j), the maximum relative distance between the standard data and the user data is MaxError, and the final score is:
[0114]
[0115] Next, calculate the angle; assuming that the three points are [A, C, B], use the above formula to find the distances between the three points, AC is b, AB is c, BC is a, and according to the cosine theorem, the cosine value of angle C can be found as follows:
[0116]
[0117] Since the angle is larger than the radian range and easier to calculate, the angle value angle is calculated using the following formula.
[0118]
[0119] Then calculate the angle score; the range of joint angle change is 0-180°, that is, the maximum difference between the standard action and the user action is 180, and the standard data is angle std , user data is angle usr , so the angle score can be obtained using the following formula.
[0120]
[0121] Finally, the average of the distance and angle scores is the total score for this part. The score of each part is obtained by traversing the six major parts, and the average of the scores of all groups of data is obtained to get the final score of this action.
[0122] The user's actions recorded every second and their corresponding standard actions will be displayed in real time on the monitor of the mirror all-in-one machine. Incorrect actions will be marked with a conspicuous red frame for easy viewing. After the training is completed, the system will generate a training analysis report based on the score and action errors.
[0123] Embodiment 3
[0124] See also Fig. 9 , this embodiment provides a Tai Chi action correction system based on a generative adversarial network, including:
[0125] The current stage data acquisition module 901 is used to acquire the human body motion image of the user in the current stage; the human body motion image is the motion image captured when practicing Tai Chi.
[0126] The standard action sequence determination module 902 for the user in the next stage is used to determine the standard action sequence for the user in the next stage based on the human body action image of the user in the current stage and the trained generative adversarial network model; the data in the standard action sequence is the standard Tai Chi skeletal joint point data.
[0127] The next stage data acquisition module 903 is used to acquire the human body motion image of the user in the next stage.
[0128] The similarity calculation module 904 is used to calculate the similarity between the standard action sequence of the user in the next stage and the human body action image of the user in the next stage.
[0129] The correction module 905 is used to correct the user's Tai Chi exercises according to the similarity.
[0130] Embodiment 4
[0131] like Fig.10 As shown, the present embodiment provides a Tai Chi action correction system based on a generative adversarial network, comprising: a background server, a mirror-surface integrated machine and a user terminal; wherein the mirror-surface integrated machine is provided with a camera and a display;
[0132] The user terminal is used to input the user's physical data and select the Tai Chi movements to be practiced, and upload the user's physical data and the Tai Chi movements to be practiced to the background server.
[0133] The camera is used to capture the user's body motion images in real time.
[0134] The display is used to play Tai Chi demonstration movements practiced by the user.
[0135] The backend server is used for:
[0136] Acquire a human body motion image of the user at the current stage; the human body motion image is a motion image captured when practicing Tai Chi;
[0137] Based on the human body motion image of the user in the current stage and the trained generative adversarial network model, a standard motion sequence of the user in the next stage is determined; the data in the standard motion sequence is standard Tai Chi skeleton joint point data;
[0138] Obtaining the user's human body action image for the next stage;
[0139] Calculating the similarity between the standard action sequence of the user at the next stage and the human body action image of the user at the next stage;
[0140] Outputting a correction action instruction according to the similarity; the correction action instruction is a Tai Chi action correction instruction for the unqualified area;
[0141] The display is also used to display the unqualified area and the Tai Chi correction movements of the unqualified area.
[0142] Compared with the prior art, the present invention has the following beneficial effects:
[0143] (1) Generative adversarial network models are used to generate standard Tai Chi action sequences as a criterion for judging the accuracy of actions, thereby realizing Tai Chi teaching. When joints that deviate greatly from the standard action are detected, voice prompts are used to correct the action. (2) The present invention uses a generative adversarial network model to predict standard subsequent Tai Chi actions based on the user's current action, which can solve the action evaluation errors caused by different human body orientations when only two-dimensional human joint data is available. When using the device, users have no restrictions on the shooting angle and shooting position of the image obtained by the device. (3) An Android display is used in combination with a dressing mirror to design an intelligent Tai Chi training mirror-in-one machine. The system developed based on this device and the algorithm model enables users to observe their own movements in real time while practicing Tai Chi, and provide real-time feedback on the training effect.
[0144] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0145] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of the present invention. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A Tai Chi action correction method based on generative adversarial network, characterized in that: include: Acquire a human body motion image of the user at the current stage; the human body motion image is a motion image captured when practicing Tai Chi; Based on the human body motion image of the user in the current stage and the trained generative adversarial network model, the standard motion sequence of the user in the next stage is determined; the data in the standard motion sequence is the standard Tai Chi skeleton joint point data; wherein the generative adversarial network model includes two discriminators and an action generator G, and the two discriminators are respectively a discriminator D based on the skeleton sequence V and the frame-based discriminator D F ; The network structure of the action generator G includes GRU layer, temporal attention module, and GCN network; the GRU layer enables the action generator G to learn the temporal feature distribution of the skeleton sequence, the temporal attention module can select similar skeleton aggregation features, and the GCN network converts high-dimensional features into structured human skeletons; The objective function of the action prediction network in the adversarial network model is: Among them, G is the action generator, D F is the frame-based discriminator, D V Skeleton sequence based discriminator, loss L F The discriminator D F The cross entropy for distinguishing true and false data from generated data; loss L V It consists of two parts: the cross entropy for distinguishing true from false and the cross entropy for distinguishing action categories; Obtaining the user's human body action image for the next stage; Calculating the similarity between the standard action sequence of the user at the next stage and the human body action image of the user at the next stage; According to the similarity, the user's Tai Chi exercises are corrected.
2. A Tai Chi action correction method based on a generative adversarial network according to claim 1, characterized in that: Before obtaining the human body movement image of the user at the current stage, the Tai Chi movement correction method further includes a preprocessing part; the preprocessing part is: Acquire a human body motion image of a user in an initial stage; the initial stage is a stage before the user practices Tai Chi; Determining whether the human body motion image of the user in the initial stage includes a set human body joint point; If yes, then obtain the user's human body motion image at the current stage; If not, the user position adjustment information is output, and after the user adjusts the position, the human body motion image of the user in the initial stage is re-photographed, and the process returns to determine whether the human body motion image of the user in the initial stage includes the set human body joint points.
3. A Tai Chi action correction method based on generative adversarial network according to claim 2, characterized in that: The determination of whether the human body motion image of the user in the initial stage includes all the joints of the human body specifically includes: The openpose model is used to process the human body motion image of the user in the initial stage to obtain human body joint data in the initial stage; Calculating the confidence of each joint point of the human joint data in the initial stage; Determine whether the confidence level of each joint point is greater than 0; Among them, when the confidence of each of the joint points is greater than 0, the human body motion image of the user in the current stage is obtained; when the confidence of the joint point is less than or equal to 0, the adjusted user position information is output, and after the user adjusts the position, the human body motion image of the user in the initial stage is re-shot, and the step of returning to determine whether the human body motion image of the user in the initial stage includes the set human body joint points.
4. The Tai Chi movement correction method based on generative adversarial network according to claim 1, characterized in that: The step of determining a standard action sequence of the user in the next stage based on the human action image of the user in the current stage and the trained generative adversarial network model specifically includes: The openpose model is used to process the human body motion image of the user at the current stage to obtain human body joint data at the current stage; The human body joint data of the current stage is input into the trained generative adversarial network model to determine the standard action sequence of the user in the next stage.
5. The Tai Chi movement correction method based on generative adversarial network according to claim 1, characterized in that: Calculating the similarity between the standard action sequence of the user at the next stage and the human body action image of the user at the next stage specifically includes: The openpose model is used to process the human body motion image of the user in the next stage to obtain human body joint data in the next stage; Based on the standard action sequence of the user in the next stage and the human joint data in the next stage, the similarity between each group of standard joint points and the user's human joint points matching the standard joint points is calculated; the standard action sequence of the user in the next stage includes action data of multiple standard joint points, and the human joint data in the next stage includes action data of multiple user's human joint points.
6. The Tai Chi movement correction method based on generative adversarial network according to claim 5, characterized in that: Correcting the user's Tai Chi exercises according to the similarity, specifically including: Determine in turn whether each of the similarities is greater than a set threshold; If so, the region of the user's body joint points corresponding to the similarity is determined as a qualified region; If not, the area of the user's body joint points corresponding to the similarity is determined as an unqualified area, and the Tai Chi movements in the unqualified area are corrected.
7. A Tai Chi action correction system based on generative adversarial network, characterized in that: include: The current stage data acquisition module is used to acquire the human body motion image of the user in the current stage; the human body motion image is the motion image captured when practicing Tai Chi; The standard action sequence determination module for the user in the next stage is used to determine the standard action sequence of the user in the next stage based on the human action image of the user in the current stage and the trained generative adversarial network model; the data in the standard action sequence is the standard Tai Chi skeleton joint point data; wherein the generative adversarial network model includes two discriminators and an action generator G, and the two discriminators are respectively a discriminator D based on the skeleton sequence V and the frame-based discriminator D F ; The network structure of the action generator G includes GRU layer, temporal attention module, and GCN network; the GRU layer enables the action generator G to learn the temporal feature distribution of the skeleton sequence, the temporal attention module can select similar skeleton aggregation features, and the GCN network converts high-dimensional features into structured human skeletons; The objective function of the action prediction network in the adversarial network model is: Among them, G is the action generator, D F is the frame-based discriminator, D V Skeleton sequence based discriminator, loss L F The discriminator D F The cross entropy for distinguishing true and false data from generated data; loss L V It consists of two parts: the cross entropy for distinguishing true from false and the cross entropy for distinguishing action categories; The next stage data acquisition module is used to acquire the human body motion images of the user in the next stage; A similarity calculation module, used for calculating the similarity between the standard action sequence of the user in the next stage and the human body action image of the user in the next stage; The correction module is used to correct the user's Tai Chi exercises according to the similarity.
8. A Tai Chi action correction system based on generative adversarial network, characterized in that: A Tai Chi action correction method based on a generative adversarial network applied to any one of claims 1 to 6, comprising: a background server and a mirror-surface integrated machine; wherein the mirror-surface integrated machine is provided with a camera and a display; The camera is used to capture the user's human body motion image in real time; The display is used to play Tai Chi demonstration movements practiced by the user; The backend server is used for: Acquire a human body motion image of the user at the current stage; the human body motion image is a motion image captured when practicing Tai Chi; Based on the human body motion image of the user in the current stage and the trained generative adversarial network model, a standard motion sequence of the user in the next stage is determined; the data in the standard motion sequence is standard Tai Chi skeleton joint point data; Obtaining the user's human body action image for the next stage; Calculating the similarity between the standard action sequence of the user at the next stage and the human body action image of the user at the next stage; Outputting a correction action instruction according to the similarity; the correction action instruction is a Tai Chi action correction instruction for the unqualified area; The display is also used to display the unqualified area and the Tai Chi correction movements of the unqualified area.
9. A Tai Chi action correction system based on generative adversarial network according to claim 8, characterized in that: It also includes the user side; The user terminal is used to input the user's physical data and select the Tai Chi movements to be practiced, and upload the user's physical data and the Tai Chi movements to be practiced to the background server.
Citation Information
Patent Citations
Shadowboxing learning intelligent motion diagnostic feedback system
CN108905136A
Human body motion attitude identification and evaluation method and system therefor
WO2021184619A1
