A VR motion training teaching method and system based on motion capture technology

Abstract

The present application relates to a VR motion training teaching method and system based on motion capture technology, which comprises the following steps: when a training instruction is received, identifying its motion type; matching a virtual sub-scene based on the identified motion type, and obtaining teaching and training information; collecting and comparing the motion sensor data of the target object; when abnormal comparison data is identified, identifying its abnormal type; identifying the continuous motion data associated with the abnormal comparison data; providing teaching and training feedback to the user end based on the teaching audio and video data, the abnormal comparison data, and the continuous motion data; the present application distinguishes different motion types, and matches the corresponding virtual sub-scenes and teaching and training information, and at the same time, based on the current abnormal data combined with the continuous motion data for analysis, not only points out the specific errors in the user's motion, but also analyzes and feeds back the causes of the errors in the connection of the continuous motions, which helps the user to fully understand the problems existing in their own motions and make targeted improvements.

Description

Technical Field

[0001] The present application relates to the technical field of virtual reality, and in particular to a VR motion training teaching method and system based on motion capture technology. Background Art

[0002] With the rapid development of virtual reality (VR) technology, its application in the field of education is becoming more and more extensive, especially in action training teaching. VR technology has brought innovation to the traditional teaching mode with its unique immersive experience and high interactivity. The existing VR action training teaching method usually builds a virtual environment to simulate the action training process in real scenes, allowing users to practice repeatedly in the virtual space, thereby effectively improving the mastery of action skills.

[0003] However, although the existing VR action training teaching system has improved the fun and effectiveness of teaching to a certain extent, there are still some significant defects. First, the existing VR action training teaching system often lacks the fine distinction between different action types. Whether it is martial arts, martial arts or lion dance, the same training mode and feedback mechanism are used for various distinctive actions, resulting in the lack of targeted training effect and difficulty in meeting the diverse learning needs of users; secondly, the existing VR action training teaching system often lacks the consideration of coherence, that is, it fails to combine the abnormal data with the continuous action data before and after for analysis, resulting in one-sided feedback information, and it is difficult for users to fully understand the problems of their own actions. At the same time, the teaching feedback form that lacks coherence is relatively single, often simply pointing out the error of the current action, and the abnormal error of a certain action in the coherent action may usually be caused by the incorrect connection between the actions, and there is a lack of feedback on the cause of the error caused by the connection between the previous and subsequent continuous actions. Summary of the invention

[0004] In order to solve the above-mentioned defects, the present application provides a VR motion training teaching method and system based on motion capture technology.

[0005] The above-mentioned invention objective of the present application is achieved through the following technical solutions:

[0006] A VR motion training teaching method based on motion capture technology, comprising the following steps:

[0007] When receiving a training instruction from the user, identifying the action type associated with the training instruction, the action type includes martial arts action, combat action and lion dance;

[0008] Matching a preset corresponding virtual sub-scene based on the recognized action type, and obtaining teaching training information of the virtual sub-scene, wherein the teaching training information includes teaching audio and video data and standard action data;

[0009] Collect the motion sensor data of the target object in the virtual sub-scene, and compare and analyze the collected motion sensor data with the standard motion data in real time;

[0010] When abnormal comparison data is identified, its abnormal type is identified, and the abnormal type includes abnormal posture, abnormal amplitude, abnormal speed and abnormal rhythm;

[0011] Based on the action type, the exception type and the preset identification strategy, identify the continuous action data associated with the abnormal comparison data;

[0012] Based on teaching audio and video data, abnormal comparison data and continuous action data, teaching training feedback is provided to the user.

[0013] By adopting the above technical solution, when a training instruction is received from the user end, the action type corresponding to the instruction is first identified, and the action types include martial arts actions, combat actions and lion awakening. The preset corresponding virtual sub-scene is matched according to the identified action type. Each virtual sub-scene contains teaching audio and video data and standard action data for a specific action type. When the user is training in the virtual sub-scene, the action sensor data of the target object is collected in real time for comparison and analysis with the standard action data to evaluate the action performance of the target object. When the comparison and analysis finds abnormal comparison data, the type of abnormal comparison data is further identified, including abnormal posture, abnormal amplitude, abnormal speed and abnormal rhythm, so as to accurately locate the problems in the user's action. Based on the action type, abnormal type and preset recognition strategy, the continuous action associated with the abnormal comparison data is identified. The application distinguishes different action types in a refined way and matches corresponding virtual sub-scenes and teaching and training information to provide more targeted training content and feedback, thereby effectively improving the pertinence and effectiveness of training. At the same time, based on the current abnormal data combined with the continuous action data for analysis, comprehensive teaching feedback is provided, which not only points out the specific errors in the user's actions, but also analyzes and feeds back the causes of errors in the connection of continuous actions, which helps users to fully understand the problems of their own actions and make targeted improvements, and has the effect of providing users with coherent and comprehensive feedback information, thereby enriching the feedback form and improving the user experience and learning effect.

[0014] In a preferred example, the present application can be further configured as follows: the step of collecting the motion sensing data of the target object in the virtual sub-scene and comparing and analyzing the collected motion sensing data with the standard motion data in real time includes the steps of:

[0015] Modeling standard action data to generate a standard action model;

[0016] Collecting motion sensing data of the target object in the virtual sub-scene in real time, and extracting motion sensing features of the motion sensing data;

[0017] Matching the motion sensing features with the standard motion model and calculating a first similarity score thereof;

[0018] The motion sensing data corresponding to the motion sensing feature having a first similarity score lower than the similarity threshold is set as abnormal comparison data.

[0019] By adopting the above technical solution, the standard action data is deeply modeled to generate an accurate standard action model, and the sensor data of the target object when performing the action in the virtual sub-scene is collected in real time, and the key action sensing features are extracted from it, and the extracted action sensing features are carefully matched with the previously established standard action model, and a first similarity score is calculated to evaluate the degree of fit between the action performance of the target object and the standard action; if the first similarity score is lower than the preset similarity threshold, the corresponding action sensing data is marked as abnormal comparison data, thereby realizing real-time monitoring and identification of abnormal action conditions. Based on the above steps, the present application has the effect of improving the accuracy and real-time performance of action data processing, and enables the system to more accurately identify the abnormal performance of the target object during the action execution process, and through real-time comparison and analysis, it is realized to feedback accurate information about the action execution status to the user, thereby helping the user to promptly discover and correct deficiencies in the action, and effectively improve the user's training effect.

[0020] In a preferred example, the present application can be further configured as follows: the step of identifying continuous action data associated with abnormal comparison data based on action type, abnormal type and preset identification strategy includes the following steps:

[0021] Based on the action type and the abnormal comparison data, identify the abnormal action item corresponding to the abnormal comparison data;

[0022] Based on the abnormal action item, abnormal type and preset identification strategy, identify the continuous action data of the timestamp corresponding to the abnormal comparison data.

[0023] By adopting the above technical solution, based on the action type and abnormal comparison data, the specific action item that causes the abnormality, that is, the abnormal action item, is accurately identified. Based on the identified abnormal action item, combined with the abnormal type and the pre-set identification strategy, the continuous action data with the timestamp corresponding to the abnormal comparison data is further traced and identified, so as to accurately lock the abnormal link in the action execution; at the same time, through association analysis, the potential connection between the abnormal action and the previous and subsequent actions is revealed, so as to provide users with more comprehensive and coherent feedback; based on the above steps, the present application improves the system's recognition accuracy of abnormal actions, making the feedback more accurate and specific. Secondly, through association analysis of continuous action data, it helps users better understand the causes of abnormal actions and how they affect the entire action process, so as to guide users to make targeted improvements.

[0024] In a preferred example, the present application can be further configured as follows: the step of providing teaching training feedback to the user terminal based on the teaching audio and video data, abnormal comparison data and continuous action data includes the following steps:

[0025] Modeling standard action data to generate a standard action model;

[0026] Extracting continuous motion features from continuous motion data and matching the continuous motion features with standard motion models to identify abnormal motion sequence features;

[0027] Inputting abnormal action sequence features and abnormal comparison data into a pre-trained abnormal analysis model, so that the abnormal analysis model outputs abnormal information, wherein the abnormal information includes abnormal degree information and abnormal cause information;

[0028] Provide teaching training feedback to the user based on abnormal information and teaching audio and video data.

[0029] By adopting the above technical solution, by modeling the standard action data, an accurate standard action model is generated, the continuous action features in the continuous action data are extracted, and the extracted features are matched with the standard action model to accurately identify the abnormal action sequence features, and the identified abnormal action sequence features and abnormal comparison data are input into the pre-trained abnormal analysis model, the model will output detailed abnormal information, including the degree of abnormality and the specific cause of the abnormality, based on the abnormal information output by the abnormal analysis model, combined with the teaching audio and video data, comprehensive and targeted teaching and training feedback is provided to the user end; based on the above steps, the present application not only improves the accuracy and comprehensiveness of abnormal identification, but also makes the teaching and training feedback more specific and targeted, through the matching of the standard action model and the in-depth analysis of the abnormal analysis model, so as to accurately point out the problems in the user's action, and provide detailed abnormal information and improvement suggestions, at the same time, combined with the teaching audio and video data, to provide users with intuitive and vivid demonstrations and explanations, so as to help users better understand the essentials of the action, quickly correct errors, and improve the training effect.

[0030] In a preferred example, the present application can be further configured as follows: the abnormal analysis model includes an integration layer, a matching layer and an analysis layer, and the abnormal action sequence features and abnormal comparison data are input into the pre-trained abnormal analysis model so that the abnormal analysis model outputs abnormal information, and the abnormal information includes abnormal degree information and abnormal cause information, including the steps of:

[0031] The integration layer integrates the action sequence features and the abnormal comparison data to obtain the integrated feature vector;

[0032] The matching layer matches the integrated feature vector with the pre-stored abnormal pattern and generates a second similarity score based on the matching result;

[0033] The analysis layer generates abnormality cause information and abnormality degree information based on the integrated feature vector and its second similarity score, thereby outputting abnormality information.

[0034] By adopting the above technical solution, the abnormal analysis model deeply processes the input abnormal action sequence features and abnormal comparison data: the integration layer integrates the features of these two types of data to generate an integrated feature vector containing integrated information; the matching layer matches the integrated feature vector with the pre-stored abnormal pattern, and obtains the second similarity score by calculation, thereby quantifying the similarity between the abnormal action and the known abnormal pattern; the analysis layer deeply analyzes the cause of the abnormality based on the integrated feature vector and its second similarity score, and evaluates the severity of the abnormality, and finally outputs complete abnormal information containing abnormal cause information and abnormal degree information; based on the above steps, the present application not only improves the accuracy and comprehensiveness of abnormal analysis, but also makes the output abnormal information more specific and targeted. Through the collaborative work of the integration layer, the matching layer and the analysis layer, the abnormal analysis model can accurately identify the abnormal links in the action sequence and deeply analyze the root causes of the abnormality; at the same time, by quantifying the degree of abnormality, the abnormal analysis model can also provide users with intuitive feedback on the severity of the abnormality, thereby helping users better understand the problems with their own actions and make targeted improvements.

[0035] In a preferred example, the present application may be further configured as follows: after the step of providing teaching training feedback to the user terminal based on the abnormal information and the teaching audio-visual data, the following steps are performed:

[0036] Preset corresponding tactile feedback mechanism and biofeedback mechanism based on abnormal information and abnormal type matching;

[0037] Based on the matching tactile feedback mechanism and biofeedback mechanism, the user is provided with real-time tactile feedback and biofeedback during the teaching and training process.

[0038] By adopting the above technical solution, according to the abnormal information and its corresponding abnormal type, the preset tactile feedback mechanism and the biofeedback mechanism are intelligently matched, and a more intuitive and immersive training experience is provided to the user through real-time tactile and biofeedback: according to the identified abnormal action and the degree of abnormality, the corresponding tactile feedback, such as vibration, pressure change, etc., is triggered to guide the user to perceive and correct the wrong action. At the same time, the biofeedback mechanism will reflect the user's physical and mental state in real time by monitoring the user's physiological indicators, such as heart rate, muscle tension, etc., to help the user adjust the training rhythm and attention allocation; by combining the teaching and training methods of tactile and biofeedback, the user's perception and understanding of abnormal actions are enhanced, and the user's self-regulation of the physical and mental state is promoted, thereby improving the training effect. Through the real-time feedback mechanism, the user can immediately realize the error of his action, and make timely adjustments under the guidance of the system to form correct action habits.

[0039] The second object of the invention is achieved by the following technical solutions:

[0040] A VR motion training and teaching system based on motion capture technology, comprising:

[0041] The action recognition module is used to identify the action type associated with the training instruction when receiving the training instruction from the user end, and the action type includes martial arts action, combat action and lion dance;

[0042] An information acquisition module, used to match a preset corresponding virtual sub-scene based on the recognized action type, and acquire teaching and training information of the virtual sub-scene, wherein the teaching and training information includes teaching audio and video data and standard action data;

[0043] A comparison and analysis module is used to collect motion sensor data of the target object in the virtual sub-scene, and compare and analyze the collected motion sensor data with standard motion data in real time;

[0044] An abnormal type identification module is used to identify the abnormal type of abnormal comparison data when abnormal comparison data is identified, and the abnormal type includes abnormal posture, abnormal amplitude, abnormal speed and abnormal rhythm;

[0045] A continuous action recognition module, used to identify continuous action data associated with abnormal comparison data based on action type, abnormal type and preset recognition strategy;

[0046] The teaching and training feedback module is used to provide teaching and training feedback to the user end based on the teaching audio and video data, abnormal comparison data and continuous action data.

[0047] By adopting the above technical solution, the action recognition module is used to identify the action type corresponding to the training instruction when receiving the training instruction from the user end, and the action type includes martial arts action, martial arts action and lion awakening; the information acquisition module is used to match the preset corresponding virtual sub-scene based on the recognized action type, and obtain the teaching and training information of the virtual sub-scene, and the teaching and training information includes teaching audio and video data and standard action data; the comparison and analysis module is used to collect the action sensor data of the target object in the virtual sub-scene, and compare and analyze the collected action sensor data with the standard action data in real time; the abnormal type recognition module is used to identify the abnormal type when the abnormal comparison data is identified, and the abnormal type includes abnormal posture, abnormal amplitude, abnormal speed and abnormal rhythm; the continuous action recognition module is used to identify the continuous action data associated with the abnormal comparison data based on the action type, abnormal type and preset recognition strategy; the teaching and training feedback module is used to provide teaching and training feedback to the user end based on the teaching audio and video data, abnormal comparison data and continuous action data.

[0048] In a preferred example, the present application can be further configured as follows: the comparison and analysis module includes:

[0049] The model generation submodule is used to model the standard action data and generate a standard action model;

[0050] A feature extraction submodule is used to collect the motion sensing data of the target object in the virtual subscene in real time and extract the motion sensing features of the motion sensing data;

[0051] A first similarity score calculation submodule, used to match the motion sensing feature with the standard motion model and calculate a first similarity score thereof;

[0052] The abnormal setting submodule is used to set the motion sensing data corresponding to the motion sensing feature whose first similarity score is lower than the similarity threshold as abnormal comparison data.

[0053] By adopting the above technical solution, a model generation submodule is used to model standard action data and generate a standard action model; a feature extraction submodule is used to collect the action sensing data of the target object in the virtual subscene in real time and extract the action sensing features of the action sensing data; a first similarity score calculation submodule is used to match the action sensing features with the standard action model and calculate their first similarity scores; and an abnormal setting submodule is used to set the action sensing data corresponding to the action sensing features whose first similarity scores are lower than the similarity threshold as abnormal comparison data.

[0054] In summary, the present application includes at least one of the following beneficial technical effects:

[0055] 1. This application distinguishes different action types in a refined manner and matches corresponding virtual sub-scenes and teaching training information to provide more targeted training content and feedback, thereby effectively improving the pertinence and effectiveness of training. At the same time, based on the current abnormal data combined with the continuous action data for analysis, it provides comprehensive teaching feedback, which not only points out the specific errors in the user's actions, but also analyzes and feedbacks the causes of errors in the connection of continuous actions, which helps users to fully understand the problems in their own actions and make targeted improvements. It has the effect of providing users with coherent and comprehensive feedback information, thereby enriching the feedback form and improving the user experience and learning effect;

[0056] 2. By combining tactile and biofeedback teaching and training methods, the user's perception and understanding of abnormal movements are enhanced, while the user's self-regulation of physical and mental state is promoted, thereby improving the training effect. Through the real-time feedback mechanism, the user can immediately realize the error of their movements, and make timely adjustments under the guidance of the system to form correct movement habits. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of an embodiment of a VR motion training teaching method based on motion capture technology in the present application;

[0058] Figure 2 This is a flowchart for implementing step S30 in an embodiment of a VR action training teaching method based on motion capture technology in the present application;

[0059] Figure 3 This is a flowchart for implementing step S60 in an embodiment of a VR action training teaching method based on motion capture technology of the present application;

[0060] Figure 4 This is a flowchart for implementing step S63 in an embodiment of a VR action training teaching method based on motion capture technology in the present application. DETAILED DESCRIPTION

[0061] The following is combined with Figure 1-4 This application is described in further detail.

[0062] In one embodiment, if Figure 1 As shown, the present application discloses a VR action training teaching method based on motion capture technology, which specifically includes the following steps:

[0063] S10: When receiving a training instruction from the user, identifying an action type associated with the training instruction, wherein the action type includes martial arts action, combat action and lion dance;

[0064] In this embodiment, the training instruction is an electric signal instruction issued by the user end for performing specific action training; the action type is a specific type of training action including martial arts action, combat action and lion dance;

[0065] Specifically, when a training instruction is received from the user, the action type corresponding to the instruction is first identified, and the action types include martial arts actions, combat actions, and lion dance;

[0066] S20: matching a preset corresponding virtual sub-scene based on the recognized action type, and obtaining teaching training information of the virtual sub-scene, wherein the teaching training information includes teaching audio and video data and standard action data;

[0067] In this embodiment, the virtual sub-scene is a virtual training environment designed for different types of actions; the teaching audio-visual data is multimedia materials such as video and audio used for teaching; the standard action data is reference data describing the correct way to perform an action, including information such as action trajectory, speed, amplitude, etc.;

[0068] Furthermore, when the user terminal sends a martial arts training instruction, virtual sub-scenes related to the martial arts are matched and loaded, such as a boxing practice field, a leg training room, etc.; when the user terminal sends a martial arts training instruction, virtual sub-scenes with artistic exaggeration and plot elements are matched and loaded, such as a fighting scene in a costume martial arts film; when the user terminal sends a lion dance training instruction, virtual sub-scenes such as a traditional lion dance performance stage are matched and loaded;

[0069] Specifically, matching a preset corresponding virtual sub-scene according to the recognized action type, and obtaining the teaching audio-visual data and standard action data for a specific action type contained in each virtual sub-scene;

[0070] S30: collecting motion sensing data of the target object in the virtual sub-scene, and comparing and analyzing the collected motion sensing data with standard motion data in real time;

[0071] In this embodiment, the motion sensing data is the real-time data of the target object when performing the action, including position, speed, acceleration, angle, etc., collected by the sensor; the comparison analysis is to compare the collected data with the standard data to identify the differences and anomalies;

[0072] Specifically, when the user is training in the virtual sub-scene, the motion sensor data of the target object is collected in real time for comparison and analysis with the standard motion data to evaluate the motion performance of the target object;

[0073] S40: When abnormal comparison data is identified, its abnormal type is identified, and the abnormal type includes abnormal posture, abnormal amplitude, abnormal speed and abnormal rhythm;

[0074] In this embodiment, the abnormal comparison data is the collected data that is significantly different from the standard data; the abnormal type is the specific characteristics describing the abnormal data, including abnormal posture, abnormal amplitude, abnormal speed and abnormal rhythm;

[0075] Among them, in the recognition and comparison of motion sensor data based on martial arts movements, due to the high requirements of martial arts movements for accuracy and consistency, the comparison of motion sensor data with standard motion data needs to accurately reflect the start and end points of each movement, force changes, speed control and other details. Abnormal comparison data is mainly reflected in incorrect posture (abnormal posture), insufficient or excessive amplitude (abnormal amplitude), too fast or too slow speed (abnormal speed) and uncoordinated rhythm (abnormal rhythm). In the recognition and comparison of motion sensor data based on martial arts movements, since martial arts movements may not completely follow the norms of traditional martial arts, the comparison standards for motion sensor data and standard motion data will also be relatively Loose, abnormal data is usually more related to the creativity and visual effects of the action. Abnormal comparison data is mainly reflected in the mismatch of the exaggeration of the action (abnormal amplitude) and the mismatch of the smoothness of the action (abnormal rhythm). In the recognition and comparison of motion sensor data based on lion awakening, because lion awakening requires motion sensor data to accurately reflect the charm and momentum of the lion, as well as the tacit cooperation between the team, abnormal comparison data is mainly reflected in the uncoordinated action (abnormal posture), lack of charm (abnormal amplitude, abnormal rhythm and abnormal speed), and also includes the lack of tacit cooperation of the team (abnormal posture, abnormal amplitude, abnormal speed and abnormal rhythm when multiple users perform the same timestamp action);

[0076] Specifically, when the comparison analysis finds abnormal comparison data, the type of the abnormal comparison data is further identified, including abnormal posture, abnormal amplitude, abnormal speed and abnormal rhythm, so as to accurately locate the problem in the user's action;

[0077] S50: Based on the action type, the exception type and the preset identification strategy, identify the continuous action data associated with the exception comparison data;

[0078] In this embodiment, the continuous action data is action data connected to the abnormal data in time or logic; the preset identification strategy is a rule method for identifying and associating the continuous action data, specifically, presetting different continuous action data identification quantities based on different action types and their abnormal types;

[0079] Specifically, based on the action type, the exception type and the preset recognition strategy, the continuous action data associated with the abnormal comparison data is recognized, and the data of the continuous actions before and after the action corresponding to the abnormal comparison data is recognized;

[0080] S60: providing teaching training feedback to the user terminal based on the teaching audio and video data, the abnormal comparison data, and the continuous action data;

[0081] In this embodiment, the teaching and training feedback is feedback information provided to the user terminal about the execution of the action, including problem points and corresponding improvement suggestions, etc.;

[0082] Specifically, based on the teaching audio and video data, abnormal comparison data and continuous action data, comprehensive teaching and training feedback is provided to the user end.

[0083] In one embodiment, if Figure 2 As shown, step S30 includes the steps of:

[0084] S31: Modeling the standard action data to generate a standard action model;

[0085] S32: collecting motion sensing data of the target object in the virtual sub-scene in real time, and extracting motion sensing features of the motion sensing data;

[0086] S33: Matching the motion sensing feature with the standard motion model and calculating a first similarity score thereof;

[0087] S34: setting the motion sensing data corresponding to the motion sensing feature whose first similarity score is lower than the similarity threshold as abnormal comparison data;

[0088] In this embodiment, the standard action model is a mathematical model that can represent the correct action characteristics by processing and modeling the standard action data; the action sensing feature is the key feature information that can represent the action characteristics extracted from the action sensing data; the first similarity score is a value reflecting the similarity between the action sensing feature and the standard action model; the similarity threshold is a preset value used to determine whether the similarity between the action sensing feature and the standard action model reaches an acceptable range; the abnormal comparison data is the action sensing data whose first similarity score is lower than the similarity threshold, representing an abnormality or error in the action execution;

[0089] Specifically, the standard action data is deeply modeled to generate an accurate standard action model, the sensor data of the target object when performing the action in the virtual sub-scene is collected in real time, and the key action sensing features are extracted from it. The extracted action sensing features are carefully matched with the previously established standard action model, and a first similarity score is calculated to evaluate the degree of fit between the target object's action performance and the standard action; if the first similarity score is lower than a preset similarity threshold, the corresponding action sensing data is marked as abnormal comparison data, thereby realizing real-time monitoring and identification of abnormal action situations.

[0090] In one embodiment, step S50 includes the steps of:

[0091] S51: Based on the action type and the abnormal comparison data, identifying the abnormal action item corresponding to the abnormal comparison data;

[0092] S52: Based on the abnormal action item, the abnormal type and the preset identification strategy, identify the continuous action data corresponding to the timestamp of the abnormal comparison data;

[0093] In this embodiment, the abnormal action item is an abnormal or erroneous action part that occurs during the execution of the action, wherein each action type includes a number of preset action items; the corresponding timestamp is a time stamp defined in association with the preset identification strategy, which is used to correspond to the length or quantity of continuous action data;

[0094] Specifically, based on the action type and abnormal comparison data, the specific action item that causes the abnormality, namely the abnormal action item, is accurately identified. Based on the identified abnormal action item, combined with the abnormal type and the pre-set identification strategy, the continuous action data with the timestamp corresponding to the abnormal comparison data is further traced and identified, so as to accurately lock the abnormal link in the action execution; at the same time, through correlation analysis, the potential connection between the abnormal action and the previous and subsequent actions is revealed, so as to provide users with more comprehensive and coherent feedback.

[0095] In one embodiment, if Figure 3 As shown, step S60 includes the steps of:

[0096] S61: Modeling the standard action data to generate a standard action model;

[0097] S62: extracting continuous motion features of the continuous motion data, and matching the continuous motion features with the standard motion model to identify abnormal motion sequence features;

[0098] S63: inputting the abnormal action sequence features and the abnormal comparison data into a pre-trained abnormal analysis model, so that the abnormal analysis model outputs abnormal information, wherein the abnormal information includes abnormal degree information and abnormal cause information;

[0099] S64: Providing teaching training feedback to the user terminal based on the abnormal information and the teaching audio and video data;

[0100] In this embodiment, the continuous action feature is the key feature information extracted from the continuous action data that can represent the action characteristics; the abnormal action sequence feature is the sequence feature of the continuous action that does not match the standard action model, indicating that there is an abnormality or error in the action execution; the abnormal analysis model is a pre-trained model for analyzing the abnormal action sequence features and abnormal comparison data and outputting abnormal information; the abnormal degree information is the information of the value or level describing the severity of the abnormal action; the abnormal cause information is the information explaining the cause or factor that causes the abnormal action;

[0101] Specifically, by modeling the standard action data, an accurate standard action model is generated, the continuous action features in the continuous action data are extracted, and the extracted features are matched with the standard action model to accurately identify the abnormal action sequence features. The identified abnormal action sequence features and abnormal comparison data are input into a pre-trained abnormal analysis model. The model will output detailed abnormal information, including the degree of the abnormality and the specific cause of the abnormality. Based on the abnormal information output by the abnormal analysis model and combined with the teaching audio and video data, comprehensive and targeted teaching and training feedback is provided to the user end.

[0102] In one embodiment, the anomaly analysis model includes an integration layer, a matching layer, and an analysis layer, such as Figure 4 As shown, step S63 includes the steps of:

[0103] S631: The integration layer integrates the action sequence features and the abnormal comparison data to obtain an integrated feature vector;

[0104] S632: The matching layer matches the integrated feature vector with the pre-stored abnormal pattern, and generates a second similarity score based on the matching result;

[0105] S633: the analysis layer generates abnormality cause information and abnormality degree information based on the integrated feature vector and its second similarity score, thereby outputting abnormality information;

[0106] In this embodiment, the integration layer is the first layer of the anomaly analysis model, which is used to integrate the input action sequence features and anomaly comparison data; feature integration is to combine feature data from different sources or different types to form a more comprehensive feature vector, specifically to integrate the action sequence features and anomaly comparison data, and feature integration includes splicing, weighted sum, feature fusion, etc., wherein, before integration, the action sequence features and the anomaly comparison data are preprocessed, including normalization, standardization, and feature removal, etc.; the integrated feature vector is a vector obtained by feature integration, which contains action sequence features and anomaly comparison data information, and at the same time, the action sequence features and the anomaly comparison data are integrated into an integrated feature vector, which can reduce the noise and misjudgment caused by a single feature; the matching layer is the second layer of the anomaly analysis model, which is used to match the integrated feature vector with the pre-stored anomaly pattern; the anomaly pattern is a feature pattern representing a known anomaly type, which is used to compare with the integrated feature vector; the second similarity score is a numerical value reflecting the similarity between the integrated feature vector and the anomaly pattern; the analysis layer is the third layer of the anomaly analysis model, which is used to generate anomaly information according to the integrated feature vector and the second similarity score;

[0107] Specifically, the anomaly analysis model performs in-depth processing on the input abnormal action sequence features and abnormal comparison data: the integration layer integrates the features of these two types of data to generate an integrated feature vector containing integrated information; the matching layer matches the integrated feature vector with the pre-stored abnormal pattern, and calculates the second similarity score to quantify the similarity between the abnormal action and the known abnormal pattern; the analysis layer deeply analyzes the cause of the anomaly based on the integrated feature vector and its second similarity score, and evaluates the severity of the anomaly, and finally outputs complete anomaly information including the cause of the anomaly information and the severity of the anomaly information.

[0108] In one embodiment, after step S64, the following steps are performed:

[0109] S65: Preset corresponding tactile feedback mechanism and biological feedback mechanism based on abnormal information and abnormal type matching;

[0110] S66: Based on the matching tactile feedback mechanism and biofeedback mechanism, the user terminal is provided with tactile feedback and biofeedback in real time during the teaching and training process;

[0111] In this embodiment, the tactile feedback mechanism is a mechanism that provides feedback through tactile stimulation, including vibration, pressure change, etc. After the user wears the smart device, the smart device provides the user with reminder feedback about posture adjustment; the biofeedback mechanism is a mechanism that uses biological signals (such as heart rate, muscle activity, etc.) to provide feedback to help the user understand his or her physical condition;

[0112] Furthermore, for abnormal posture, vibration feedback can be used to remind the user to adjust the posture; for abnormal amplitude, pressure feedback can be used to make the user feel the difference in the amplitude of the movement; for abnormal speed, vibration frequency can be changed to simulate the change in speed; for abnormal rhythm, periodic vibration or pressure change can be used to guide the user to find the correct rhythm;

[0113] Furthermore, the tactile feedback mechanism is used to provide tactile feedback to the user during the teaching and training process, for example:

[0114] Vibration feedback: When abnormalities in user movements are detected (such as incorrect posture, too large or too small amplitude), the vibration motor on the VR device can be used to give the user instant vibration feedback. The intensity, frequency and duration of the vibration can be adjusted according to the severity of the abnormality to attract the user's attention and prompt them to adjust their movements.

[0115] Force feedback: Motion capture technology is combined with force feedback devices in VR gloves or clothing. When the user performs an action, the force sense is simulated according to the standard action data. When the action is abnormal, the force output by the force feedback device is adjusted, such as increasing resistance or providing reverse force, to help the user correct the action.

[0116] Temperature feedback: In some cases, feedback can be provided by changing the temperature of the part of the VR device that contacts the user's skin. For example, when it is detected that the user is moving too fast or too slow, the device temperature can be briefly increased or decreased to remind the user to adjust the rhythm.

[0117] Haptic simulation: For certain specific action training, such as striking in martial arts, the tactile simulator on the VR device can be used to simulate the tactile sensation when striking an object, such as hardness and elasticity. When the action is abnormal, the simulator can change these tactile parameters to guide the user to adjust the action;

[0118] Furthermore, the biofeedback mechanism is used to provide biofeedback to the user during the teaching and training process, for example:

[0119] Heart rate monitoring and feedback: By wearing a smart device equipped with a heart rate monitoring device, the user's heart rate changes are monitored in real time. When the system detects that the user's heart rate increases due to abnormal movements or tension, the system can give the user feedback through sound, images or vibration to help them relax and adjust their movements;

[0120] Muscle tension feedback: Use smart devices equipped with electromyography (EMG) sensors to monitor changes in the tension of key muscles of users. When abnormal muscle tension is detected, visual prompts or tactile feedback on the VR interface are used to guide users to adjust their muscle status to achieve the correct movement requirements.

[0121] Breathing rhythm feedback: The user's breathing rhythm is monitored in real time through a smart device equipped with a breathing monitoring device. When the breathing rhythm is detected to be inconsistent with the movement, the user is prompted to adjust his breathing through sound or images to help him better control the movement;

[0122] Neurofeedback: Using neurofeedback technology, such as electroencephalogram (EEG), to monitor the user's brain wave changes, analyze brain wave data to understand the user's attention, relaxation level and other states, and give corresponding feedback based on this information to help the user better concentrate and relax;

[0123] Specifically, according to the abnormal information and its corresponding abnormal type, the preset tactile feedback mechanism and biofeedback mechanism are intelligently matched to provide users with a more intuitive and immersive training experience through real-time tactile and biofeedback: according to the identified abnormal actions and the degree of abnormality, the corresponding tactile feedback is triggered, such as vibration, pressure change, etc., to guide users to perceive and correct wrong actions. At the same time, the biofeedback mechanism will reflect the user's physical and mental state in real time by monitoring the user's physiological indicators, such as heart rate, muscle tension, etc., to help users adjust the training rhythm and attention allocation.

[0124] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0125] In one embodiment, a VR motion training and teaching system based on motion capture technology is provided, and the VR motion training and teaching system based on motion capture technology corresponds one-to-one to the VR motion training and teaching method based on motion capture technology in the above embodiment.

[0126] A VR motion training and teaching system based on motion capture technology, comprising:

[0127] The action recognition module is used to identify the action type associated with the training instruction when receiving the training instruction from the user end, and the action type includes martial arts action, combat action and lion dance;

[0128] An information acquisition module, used to match a preset corresponding virtual sub-scene based on the recognized action type, and acquire teaching and training information of the virtual sub-scene, wherein the teaching and training information includes teaching audio and video data and standard action data;

[0129] A comparison and analysis module is used to collect motion sensor data of the target object in the virtual sub-scene, and compare and analyze the collected motion sensor data with standard motion data in real time;

[0130] An abnormal type identification module is used to identify the abnormal type of abnormal comparison data when abnormal comparison data is identified, and the abnormal type includes abnormal posture, abnormal amplitude, abnormal speed and abnormal rhythm;

[0131] A continuous action recognition module, used to identify continuous action data associated with abnormal comparison data based on action type, abnormal type and preset recognition strategy;

[0132] The teaching and training feedback module is used to provide teaching and training feedback to the user end based on the teaching audio and video data, abnormal comparison data and continuous action data;

[0133] Optional comparison analysis modules include:

[0134] The model generation submodule is used to model the standard action data and generate a standard action model;

[0135] A feature extraction submodule is used to collect the motion sensing data of the target object in the virtual subscene in real time and extract the motion sensing features of the motion sensing data;

[0136] A first similarity score calculation submodule, used to match the motion sensing feature with the standard motion model and calculate a first similarity score thereof;

[0137] an abnormal setting submodule, used for setting the motion sensing data corresponding to the motion sensing feature whose first similarity score is lower than the similarity threshold as abnormal comparison data;

[0138] Optionally, the continuous action recognition module includes:

[0139] The abnormal action item identification submodule is used to identify the abnormal action item corresponding to the abnormal comparison data based on the action type and the abnormal comparison data;

[0140] The continuous action data identification submodule is used to identify the continuous action data corresponding to the timestamp of the abnormal comparison data based on the abnormal action item, the abnormal type and the preset identification strategy;

[0141] Optional, teaching and training feedback modules include:

[0142] The sequence feature recognition submodule is used to extract the continuous action features of the continuous action data and match the continuous action features with the standard action model to identify abnormal action sequence features;

[0143] An input submodule, used to input abnormal action sequence features and abnormal comparison data into a pre-trained abnormal analysis model, so that the abnormal analysis model outputs abnormal information, wherein the abnormal information includes abnormal degree information and abnormal cause information;

[0144] The teaching and training feedback submodule is specifically used to provide teaching and training feedback to the user terminal based on abnormal information and teaching audio and video data;

[0145] Optionally, also include:

[0146] The integration layer module is used to integrate the action sequence features and the abnormal comparison data to obtain the integrated feature vector;

[0147] A matching layer module, configured to match the integrated feature vector with the pre-stored abnormal pattern and generate a second similarity score based on the matching result;

[0148] An analysis layer module, used for generating abnormality cause information and abnormality degree information based on the integrated feature vector and its second similarity score, thereby outputting abnormality information;

[0149] Optionally, also include:

[0150] A feedback mechanism matching module, used to match preset corresponding tactile feedback mechanisms and biofeedback mechanisms based on abnormal information and abnormal types;

[0151] The feedback mechanism implementation module is used to provide real-time tactile feedback and biofeedback to the user during the teaching and training process based on the matching tactile feedback mechanism and biofeedback mechanism.

[0152] For the specific definition of a VR motion training and teaching system based on motion capture technology, please refer to the definition of a VR motion training and teaching method based on motion capture technology mentioned above, which will not be repeated here. Each module in the above-mentioned VR motion training and teaching system based on motion capture technology can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0153] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A VR action training teaching method based on motion capture technology, characterized by: Includes steps: When receiving a training instruction from the user, identifying the action type associated with the training instruction, the action type includes martial arts action, combat action and lion dance; Matching a preset corresponding virtual sub-scene based on the recognized action type, and obtaining teaching training information of the virtual sub-scene, wherein the teaching training information includes teaching audio and video data and standard action data; Collect the motion sensor data of the target object in the virtual sub-scene, and compare and analyze the collected motion sensor data with the standard motion data in real time; When abnormal comparison data is identified, its abnormal type is identified, and the abnormal type includes abnormal posture, abnormal amplitude, abnormal speed and abnormal rhythm; Based on the action type, the exception type and the preset identification strategy, identify the continuous action data associated with the abnormal comparison data; Provide teaching training feedback to the user based on teaching audio and video data, abnormal comparison data and continuous action data; The step of providing teaching training feedback to the user terminal based on the teaching audio and video data, abnormal comparison data and continuous action data includes the following steps: Modeling standard action data to generate a standard action model; Extracting continuous motion features from continuous motion data and matching the continuous motion features with standard motion models to identify abnormal motion sequence features; Inputting abnormal action sequence features and abnormal comparison data into a pre-trained abnormal analysis model, so that the abnormal analysis model outputs abnormal information, wherein the abnormal information includes abnormal degree information and abnormal cause information; Provide teaching training feedback to the user based on abnormal information and teaching audio and video data; The abnormal analysis model includes an integration layer, a matching layer and an analysis layer. The abnormal action sequence features and abnormal comparison data are input into the pre-trained abnormal analysis model so that the abnormal analysis model outputs abnormal information, and the abnormal information includes abnormal degree information and abnormal cause information, including the steps of: The integration layer integrates the abnormal action sequence features and the abnormal comparison data to obtain an integrated feature vector; The matching layer matches the integrated feature vector with the pre-stored abnormal pattern and generates a second similarity score based on the matching result; The analysis layer generates abnormality cause information and abnormality degree information based on the integrated feature vector and its second similarity score, thereby outputting abnormality information.

2. A VR action training teaching method based on motion capture technology according to claim 1, characterized in that: The step of collecting the motion sensing data of the target object in the virtual sub-scene and comparing and analyzing the collected motion sensing data with the standard motion data in real time comprises the following steps: Modeling standard action data to generate a standard action model; Collecting motion sensing data of the target object in the virtual sub-scene in real time, and extracting motion sensing features of the motion sensing data; Matching the motion sensing features with the standard motion model and calculating a first similarity score thereof; The motion sensing data corresponding to the motion sensing feature having a first similarity score lower than the similarity threshold is set as abnormal comparison data.

3. The VR action training teaching method based on motion capture technology according to claim 1 is characterized by: The step of identifying the continuous action data associated with the abnormal comparison data based on the action type, the abnormality type and the preset identification strategy comprises the steps of: Based on the action type and the abnormal comparison data, identify the abnormal action item corresponding to the abnormal comparison data; Based on the abnormal action item, abnormal type and preset identification strategy, identify the continuous action data of the timestamp corresponding to the abnormal comparison data.

4. The VR action training teaching method based on motion capture technology according to claim 1, characterized in that: After the step of providing teaching training feedback to the user terminal based on the abnormal information and the teaching audio and video data, the following steps are performed: Preset corresponding tactile feedback mechanism and biofeedback mechanism based on abnormal information and abnormal type matching; Based on the matching tactile feedback mechanism and biofeedback mechanism, the user is provided with real-time tactile feedback and biofeedback during the teaching and training process.

5. A VR motion training teaching system based on motion capture technology, used in the steps of a VR motion training teaching method based on motion capture technology as claimed in any one of claims 1 to 4, characterized in that: include: The action recognition module is used to identify the action type associated with the training instruction when receiving the training instruction from the user end, and the action type includes martial arts action, combat action and lion dance; An information acquisition module, used to match a preset corresponding virtual sub-scene based on the recognized action type, and acquire teaching and training information of the virtual sub-scene, wherein the teaching and training information includes teaching audio and video data and standard action data; A comparison and analysis module is used to collect motion sensor data of the target object in the virtual sub-scene, and compare and analyze the collected motion sensor data with standard motion data in real time; An abnormal type identification module is used to identify the abnormal type of abnormal comparison data when abnormal comparison data is identified, and the abnormal type includes abnormal posture, abnormal amplitude, abnormal speed and abnormal rhythm; A continuous action recognition module, used to identify continuous action data associated with abnormal comparison data based on action type, abnormal type and preset recognition strategy; The teaching and training feedback module is used to provide teaching and training feedback to the user end based on the teaching audio and video data, abnormal comparison data and continuous action data.

6. The VR motion training and teaching system based on motion capture technology according to claim 5, characterized in that: The comparison and analysis module comprises: The model generation submodule is used to model the standard action data and generate a standard action model; A feature extraction submodule is used to collect the motion sensing data of the target object in the virtual subscene in real time and extract the motion sensing features of the motion sensing data; A first similarity score calculation submodule, used to match the motion sensing feature with the standard motion model and calculate a first similarity score thereof; The abnormal setting submodule is used to set the motion sensing data corresponding to the motion sensing feature whose first similarity score is lower than the similarity threshold as abnormal comparison data.

Images