A Tai Chi teaching system based on action perception

Abstract

The present invention belongs to the technical field of Tai Chi teaching, and specifically refers to a Tai Chi teaching system based on motion perception, including a video acquisition module, a central processing module, and an interaction module. The central processing module of this system uses the OvR strategy CNN fusion fully connected neural network algorithm to perform action recognition on the user's video, and combines the Tai Chi teaching rule base for secondary determination and gives evaluation opinions. The prior knowledge introduced by the rule base improves the accuracy of the deep learning model, improves the accuracy of action recognition, and further improves the scalability and interpretability of the program. The low-cost and low-volume hardware improves the overall economy and portability of the product, effectively solves the problems of high cost and heaviness of the existing Tai Chi teaching system, and enables the elderly to learn Tai Chi at low cost anytime and anywhere.

Description

Technical Field

[0001] The present invention belongs to the field of teaching systems, and specifically refers to a Tai Chi teaching system based on motion perception. Background Art

[0002] With the aggravation of population aging, there are more and more elderly people. Many elderly people attach great importance to physical exercise, and especially love Tai Chi. Because practicing Tai Chi can cultivate the body and mind, strengthen the body, and has an extremely important promoting effect on the physical and mental health of the human body. However, there are mainly two ways to learn Tai Chi at present. The first is to learn with a group or a teacher indoors or outdoors, and the learning time is limited. The second is that people learn through videos on the computer and network indoors. However, it is difficult to obtain feedback from the videos, and the teaching instruments that can provide feedback are generally too heavy and costly.

[0003] How to achieve low-cost Tai Chi learning for the elderly at any time and place and obtain feedback is of great significance. Summary of the Invention

[0004] The present invention overcomes the deficiencies of the prior art and provides a Tai Chi teaching system based on motion perception. The lightweight and streamlined hardware design and the rule library-assisted OvR strategy CNN fusion fully connected neural network algorithm effectively solve the problems of high cost, heavy weight, and deficiencies in feedback and recognition of the existing Tai Chi teaching system. The prior knowledge introduced by the rule library improves the accuracy of the deep learning model, improves the accuracy of action recognition, and further improves the scalability and interpretability of the program. The flexible algorithm structure design can adapt to different hardware scales and launch different models to meet the needs of different consumer levels by adjusting the number of CNN base learners according to different hardware performances and teaching requirements.

[0005] The Tai Chi teaching system based on motion perception includes a video acquisition module, a central processing module, and an interaction module;

[0006] The video acquisition module and the interaction module are respectively electrically connected to the central processing module;

[0007] The interaction module includes a touch screen, a speaker, and a Tai Chi teaching knowledge material library. The Tai Chi teaching knowledge material library contains Tai Chi teaching videos with action nodes marked and Tai Chi teaching theory knowledge;

[0008] The action node is the key time point when a standard Tai Chi action is made in the Tai Chi teaching video;

[0009] The action set corresponding to the action node includes the standard Tai Chi action state and all error action states corresponding to this standard Tai Chi action state;

[0010] The video acquisition module collects the user's video in real time and transmits it to the central processing module;

[0011] The central processing module uses the OvR strategy CNN fusion fully connected neural network algorithm to recognize the actions of the user video. The central processing module constructs a tai chi teaching rule base according to the tai chi teaching knowledge material library, and uses the tai chi teaching rule base to give judgment guidance opinions and action evaluation opinions on the recognition results of the OvR strategy CNN fusion fully connected neural network algorithm;

[0012] The structure of the OvR strategy CNN fusion fully connected neural network algorithm consists of the OvR strategy CNN part, the second fusion layer, and a three-layer fully connected neural network. The structure of the OvR strategy CNN part consists of N CNN base learners and the first fusion layer;

[0013] The structure of the CNN base learner consists of three two-dimensional convolutional layers and two fully connected layers. The recognition result is the action state probability that the current user makes the action corresponding to the CNN base learner. After being spliced by the first fusion layer, it becomes the total action probability set;

[0014] The tai chi teaching rule base includes the action evaluation opinions corresponding to each action state and the decision condition set for giving judgment guidance opinions according to the total action probability set.

[0015] The process of the central processing module recognizing the actions of the user video specifically includes the following steps:

[0016] Step S1: Train the OvR strategy CNN fusion fully connected neural network algorithm;

[0017] Step S2: Deploy the OvR strategy CNN fusion fully connected neural network algorithm on the central processing module;

[0018] Step S3: The video acquisition module inputs the user video to the central processing module in real time;

[0019] Step S4: The central processing module samples each action node in the user video once to generate a key frame, and the central processing module inputs the key frame into the OvR strategy CNN part;

[0020] Step S5: The CNN base learner in the OvR strategy CNN part independently processes the key frame and splices it in the first fusion layer to obtain the output of the OvR strategy CNN part. Then, the output of the OvR strategy CNN part and 0 are filled and spliced in the second fusion layer to obtain the splicing result;

[0021] Step S6: The splicing result is passed into the three-layer fully connected neural network, and the total action probability set is output to the tai chi teaching rule base;

[0022] Step S7: The Tai Chi teaching rule base searches for the corresponding action set in the action probability total set according to the current action node, and traverses the action state probabilities in the action set;

[0023] Step S8: When at least one of the traversed action state probabilities is higher than the threshold, the Tai Chi teaching rule base selects the action state with the highest probability among them and gives the corresponding action evaluation opinion to the interaction module. Otherwise, the Tai Chi teaching rule base gives a judgment guidance opinion according to the judgment conditions corresponding to the probability total set;

[0024] Step S9: The central processing module performs sentence vectorization on the judgment guidance opinion to generate an implicit vector and sends it to the second fusion layer for a second determination;

[0025] Step S10: When each action state probability in the action set corresponding to the current action node in the action probability total set in the second determination result is still lower than the threshold, set the highest action state probability among them to 1 and give the corresponding action evaluation opinion to the interaction module according to the corresponding action state. Otherwise, the Tai Chi teaching rule base selects the action state with the highest probability in the current action set and gives the corresponding action evaluation opinion to the interaction module.

[0026] Further, in step S1, training the OvR strategy CNN fusion fully connected neural network algorithm includes the following steps:

[0027] Step S11: Extract key frames from the Tai Chi teaching video according to the action nodes, label the action states with one-hot encoding, and make a training picture set;

[0028] Step S12: Use the training picture set to train the OvR strategy CNN part in the OvR strategy CNN fusion fully connected neural network algorithm. After the training is completed, freeze the parameters of the OvR strategy CNN part;

[0029] Step S13: Combine the key frame samples in the training picture set with 0-padding input for combined data augmentation and combine them with the implicit vector corresponding to the action state label for combined data augmentation to form an augmented data set; Combine the OvR strategy CNN part with frozen parameters with the second fusion layer and three fully connected neural networks;

[0030] Step S14: Use the augmented data set to train the OvR strategy CNN fusion fully connected neural network algorithm. Since the parameters are frozen, the OvR strategy CNN part is not affected at this time;

[0031] Step S15: Obtain the trained OvR strategy CNN fusion fully connected neural network algorithm.

[0032] Further, in step S12, the OvR strategy CNN part in the OvR strategy CNN fusion fully connected neural network algorithm is trained, which specifically includes the following steps:

[0033] Step S121: Define N CNN base learners according to N action states. Depending on different hardware performances and teaching requirements, the value range of N is 24 - 240. The task of each CNN base learner is to perform binary classification learning on its corresponding action state, and the output of the CNN base learner is the action state probability of its corresponding action state.

[0034] Step S122: Split the training image set according to the action state labels and input it into each CNN base learner for training.

[0035] Step S123: Concatenate the outputs of each CNN base learner in the first fusion layer to obtain the OvR strategy CNN part that can output the total set of action probabilities.

[0036] The beneficial effects of the present invention are as follows:

[0037] (1) Introduce prior knowledge to improve the accuracy of the deep learning model, improve the action recognition accuracy, further enhance the scalability and interpretability of the program, and the low-cost and low-volume hardware improves the overall economy and portability of the product, effectively solving the problems of high cost and bulkiness of the existing Tai Chi teaching system, enabling the elderly to learn Tai Chi at low cost anytime and anywhere.

[0038] (2) Pioneeringly adopt the sentence vectorization technology to enable the rule base to assist the deep learning algorithm for secondary determination, introduce prior knowledge, and further improve the action recognition accuracy.

[0039] (3) The action evaluation opinions prepared in advance according to the Tai Chi teaching theory knowledge in the rule base provide accurate and easy-to-understand learning feedback for the elderly.

[0040] (4) Different hardware scales can be adapted by adjusting the number of CNN base learners according to different hardware performances and teaching requirements, and different models can be launched to meet the needs of different consumer levels. Description of the Drawings

[0041] Figure 1 It is the action recognition principle diagram of the Tai Chi teaching system based on action perception proposed by the present invention.

[0042] Figure 2 It is the schematic diagram of the CNN base learner of the Tai Chi teaching system based on action perception proposed by the present invention.

[0043] The accompanying drawings are used to provide a further understanding of the present invention and form a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation to the present invention. Detailed implementation manners

[0044] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0045] Embodiment 1, the present invention provides a Taiji teaching system based on action perception, including a video acquisition module, a central processing module, and an interaction module;

[0046] The video acquisition module and the interaction module are respectively electrically connected to the central processing module;

[0047] The interaction module includes a touch screen, a speaker, and a Taiji teaching knowledge material library. The Taiji teaching knowledge material library contains Taiji teaching videos with action nodes marked and Taiji teaching theory knowledge;

[0048] The action node is the key time point when a standard Taiji action is made in the Taiji teaching video;

[0049] The action set corresponding to the action node includes the standard Taiji action state and all error action states corresponding to this standard Taiji action state;

[0050] The video acquisition module real-time collects the user video and transmits it to the central processing module;

[0051] The central processing module uses the OvR strategy CNN fusion fully connected neural network algorithm to perform action recognition on the user video. The central processing module constructs a Taiji teaching rule library according to the Taiji teaching knowledge material library, and uses the Taiji teaching rule library to give judgment guidance opinions and action evaluation opinions on the recognition results of the OvR strategy CNN fusion fully connected neural network algorithm;

[0052] The structure of the OvR strategy CNN fusion fully connected neural network algorithm is composed of an OvR strategy CNN part, a second fusion layer, and a three-layer fully connected neural network. The structure of the OvR strategy CNN part is composed of N CNN base learners and a first fusion layer;

[0053] The structure of the CNN base learner is composed of three layers of two-dimensional convolutional layers and two layers of fully connected layers. The recognition result is the action state probability that the current user makes the action corresponding to the CNN base learner. After being spliced by the first fusion layer, it becomes the total action probability set;

[0054] The Taiji teaching rule library contains the action evaluation opinions corresponding to each action state and the determination condition set for giving judgment guidance opinions according to the total action probability set.

[0055] Example 2. Refer to Figure 2 , this example is based on the above example. The structure of the CNN base learner consists of three two-dimensional convolutional layers and two fully connected layers. The recognition result is the action state probability of the current user making the action corresponding to this CNN base learner. After being spliced by the first fusion layer, it becomes the total set of action probabilities.

[0056] Example 3. Refer to Figure 1 , this example is based on the above example. The process of the central processing module for action recognition of the user video includes the following steps:

[0057] Step S1: Train the OvR strategy CNN fusion fully connected neural network algorithm;

[0058] Step S2: Deploy the OvR strategy CNN fusion fully connected neural network algorithm on the central processing module;

[0059] Step S3: The video acquisition module inputs the user video to the central processing module in real time;

[0060] Step S4: The central processing module samples each action node in the user video once to generate key frames, and the central processing module inputs the key frames into the OvR strategy CNN part;

[0061] Step S5: The CNN base learner in the OvR strategy CNN part independently processes the key frames and splices them in the first fusion layer to obtain the output of the OvR strategy CNN part. Then, the output of the OvR strategy CNN part and 0 are spliced in the second fusion layer to obtain the splicing result;

[0062] Step S6: The splicing result is input into the three-layer fully connected neural network, and the total set of action probabilities is output to the Taiji teaching rule library;

[0063] Step S7: The Taiji teaching rule library searches for the corresponding action set in the total set of action probabilities according to the current action node, and traverses the action state probabilities in the action set;

[0064] Step S8: If at least one of the traversed action state probabilities is higher than the threshold, the Taiji teaching rule library selects the one with the highest probability as the action state, and gives the corresponding action evaluation opinion to the interaction module. Otherwise, the Taiji teaching rule library gives the judgment guidance opinion according to the judgment conditions corresponding to the total set of probabilities;

[0065] Step S9: The central processing module performs sentence vectorization on the judgment guidance opinion to generate an implicit vector and sends it to the second fusion layer for a second determination;

[0066] Step S10: If the action state probabilities in the action set corresponding to the current action node in the total action probability set of the second determination result are all still lower than the threshold, set the highest action state probability to 1 and give the corresponding action evaluation opinion to the interaction module according to the corresponding action state; otherwise, the Tai Chi teaching rule library selects the action state with the highest probability in the current action set and gives the corresponding action evaluation opinion to the interaction module;

[0067] Embodiment 4. Based on the above embodiment, in step S1, training the OvR strategy CNN fusion fully connected neural network algorithm includes the following steps:

[0068] Step S11: Extract key frames from the Tai Chi teaching video according to the action nodes, label the action states with one-hot encoding, and make a training picture set;

[0069] Step S12: Use the training picture set to train the OvR strategy CNN part in the OvR strategy CNN fusion fully connected neural network algorithm. After the training is completed, freeze the parameters of the OvR strategy CNN part;

[0070] Step S13: Combine the samples in the training picture set with 0-padding input for combined data augmentation or combine them with the hint vector corresponding to the action state label to form an augmented data set; combine the OvR strategy CNN part with frozen parameters with the second fusion layer and a three-layer fully connected neural network;

[0071] Step S14: Use the augmented data set to train the OvR strategy CNN fusion fully connected neural network algorithm. Since the parameters are frozen, the OvR strategy CNN part is not affected at this time;

[0072] Step S15: Obtain the trained OvR strategy CNN fusion fully connected neural network algorithm.

[0073] Embodiment 5. Based on the above embodiment, in step S12, the number of CNN base learners is 24, and the OvR strategy CNN part in the OvR strategy CNN fusion fully connected neural network algorithm is trained, specifically including the following steps:

[0074] Step S121: Define 24 CNN base learners according to 24 action states. At least 24 CNN base learners are sufficient to run in most poor hardware environments and can still meet the minimum recognition and feedback of the 24-style simplified Tai Chi. The task of each CNN base learner is to perform binary classification learning on its corresponding action state. The output of the CNN base learner is the action state probability that the user video is the action state corresponding to this CNN base learner;

[0075] Step S122: Split the training image set according to the action state labels and input it into each CNN base learner for training;

[0076] Step S123: Concatenate the outputs of each CNN base learner in the first fusion layer to obtain the OvR strategy CNN part that can output the total set of action probabilities.

[0077] Embodiment Six. Based on Embodiments One to Four, in step S12, the number of CNN base learners is 240, and the OvR strategy CNN part in the OvR strategy CNN fusion fully connected neural network algorithm is trained. The specific steps are as follows:

[0078] Step S121: Define 240 CNN base learners according to 24 * 10 action states. The highest 240 CNN base learners are sufficient to identify any situation actually encountered in Tai Chi teaching, laying a foundation for further processing. The task of each CNN base learner is to perform binary classification learning on its corresponding action state. The output of the CNN base learner is the action state probability of the user video for the action state corresponding to this CNN base learner;

[0079] Step S122: Split the training image set according to the action state labels and input it into each CNN base learner for training;

[0080] Step S123: Concatenate the outputs of each CNN base learner in the first fusion layer to obtain the OvR strategy CNN part that can output the total set of action probabilities.

[0081] Embodiment Seven. This embodiment is based on the above embodiments. This embodiment runs in the Windows operating system environment. The experiment relies on the Thonny environment and uses Scikit and TensorFlow as the algorithm frameworks. For the data processing stage, the Pandas library and the Numpy tool are used to complete the data preprocessing work, converting the original data into a matrix format convenient for analysis, and the optimizer is selected as adam.

[0082] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

[0083] The above description of the present invention and its embodiments is not restrictive. What is shown in the drawings is only one of the embodiments of the present invention, and the actual structure is not limited thereto. In general, if those of ordinary skill in the art are inspired by the present invention and, without departing from the gist of the present invention, design similar structural modes and embodiments to the technical solution without creative efforts, they shall fall within the protection scope of the present invention.

Claims

1. A tai chi teaching system based on motion perception, characterized in that: The action perception-based Tai Chi teaching system includes: a video acquisition module, a central processing module, and an interaction module; The video acquisition module and the interaction module are electrically connected to the central processing module respectively; The interaction module has a built-in Tai Chi teaching knowledge material library, which contains Tai Chi teaching theoretical knowledge and Tai Chi teaching videos with action nodes marked; The action node is the key time point for performing standard Tai Chi actions in the Tai Chi teaching video; The action set corresponding to the action node includes a standard Tai Chi action state and an incorrect action state; The video acquisition module collects the user video in real time and transmits it to the central processing module; The central processing module uses the OvR strategy CNN fusion fully connected neural network algorithm to perform action recognition on the user video. The central processing module constructs a Tai Chi teaching rule library according to the Tai Chi teaching knowledge material library, and uses the Tai Chi teaching rule library to give judgment guidance opinions and action evaluation opinions on the recognition results of the OvR strategy CNN fusion fully connected neural network algorithm; The structure of the OvR strategy CNN fusion fully connected neural network algorithm consists of an OvR strategy CNN part, a second fusion layer, and a three-layer fully connected neural network. The structure of the OvR strategy CNN part consists of N CNN base learners and a first fusion layer; The recognition result of the CNN base learner is the action state probability that the current user makes the action corresponding to the CNN base learner. After being spliced by the first fusion layer, it becomes the total action probability set; The Tai Chi teaching rule library contains action evaluation opinions corresponding to each action state and a set of judgment conditions for giving judgment guidance opinions according to the total action probability set.

2. The Tai Chi teaching system based on motion perception according to claim 1, characterized in that: The process of the central processing module performing action recognition on the user video includes the following steps: Step S1: Train the OvR strategy CNN fusion fully connected neural network algorithm; Step S2: Deploy the OvR strategy CNN fusion fully connected neural network algorithm on the central processing module; Step S3: The video acquisition module inputs the user video to the central processing module in real time; Step S4: The central processing module samples each action node in the user video once to generate a key frame, and the central processing module inputs the key frame into the OvR strategy CNN part; Step S5: The CNN base learners in the OvR strategy CNN part independently process the key frame and splice it in the first fusion layer to obtain the output of the OvR strategy CNN part. Then, the output of the OvR strategy CNN part and 0 are filled and spliced in the second fusion layer to obtain a splicing result; Step S6: The splicing result is input into the three-layer fully connected neural network, and the total action probability set is output to the Tai Chi teaching rule library; Step S7: The Tai Chi teaching rule library searches for the corresponding action set in the total action probability set according to the current action node, and traverses the action state probabilities in the action set; Step S8: When at least one of the traversed action state probabilities is higher than the threshold, the Tai Chi teaching rule library selects the action state with the highest probability among them and gives the corresponding action evaluation opinion to the interaction module. Otherwise, the Tai Chi teaching rule library gives judgment guidance opinions according to the judgment conditions corresponding to the total probability set; Step S9: The central processing module vectorizes the sentences of the judgment guidance opinion to generate an implicit vector and sends it to the second fusion layer for a second determination; Step S10: When the action state probabilities in the action set corresponding to the current action node in the total action probability set of the second determination result are all still lower than the threshold, set the highest action state probability to 1 and give the corresponding action evaluation opinion to the interaction module according to the corresponding action state. Otherwise, the Tai Chi teaching rule base selects the action state with the highest probability in the current action set and gives the corresponding action evaluation opinion to the interaction module.

3. The Tai Chi teaching system based on motion perception according to claim 2, characterized in that: In step S1, training the OvR strategy CNN fusion fully connected neural network algorithm includes the following steps: Step S11: Extract key frames from the Tai Chi teaching video according to the action nodes, label the action states with one-hot encoding, and make a training picture set; Step S12: Use the training picture set to train the OvR strategy CNN part in the OvR strategy CNN fusion fully connected neural network algorithm. After the training is completed, freeze the parameters of the OvR strategy CNN part; Step S13: Combine the key frame samples in the training picture set with 0-padding input for data augmentation and combine them with the implicit vector corresponding to the action state label to form an augmented data set; Combine the OvR strategy CNN part with frozen parameters with the second fusion layer and a three-layer fully connected neural network; Step S14: Use the augmented data set to train the OvR strategy CNN fusion fully connected neural network algorithm. Since the parameters are frozen, the OvR strategy CNN part is not affected at this time; Step S15: Obtain the trained OvR strategy CNN fusion fully connected neural network algorithm.

4. The Tai Chi teaching system based on motion perception according to claim 3, characterized in that: In step S12, training the OvR strategy CNN part in the OvR strategy CNN fusion fully connected neural network algorithm specifically includes the following steps: Step S121: Define N CNN base learners according to N action states. According to different hardware performances and teaching requirements, the value range of N is 24 - 240. The task of each CNN base learner is to perform binary classification learning on its corresponding action state. The output of the CNN base learner is the action state probability of the action state corresponding to this CNN base learner; Step S122: Split the training picture set according to the action state labels and input it into each CNN base learner for training; Step S123: Concatenate the outputs of each CNN base learner in the first fusion layer to obtain the OvR strategy CNN part that can output the total action probability set.

Images