VR motion training teaching method and system based on motion capture technology
Through motion capture technology, we can identify and analyze the action types and abnormalities in VR action training, provide highly targeted virtual subscenes and feedback, solve the problems of insufficient fine and incomplete feedback in the existing system, and improve user experience and Learning effect.
Patent Information
- Application Number
- CN202411519751.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-29
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-10-29
AI Technical Summary
The existing VR action training and teaching system lacks a refined distinction between different action types, cannot provide targeted training and feedback, and the feedback information is one-sided, making it difficult to fully understand action problems, and lacks analysis of continuous action connection errors.
Using VR action training teaching methods based on motion capture technology, we can identify user action types, match virtual subscene and teaching training information, collect and compare action sensing data in real time, identify abnormal types and continuous action data, and provide comprehensive teaching feedback. .
It improves the pertinence and effectiveness of training, provides coherent and comprehensive feedback information, helps users understand action problems and make targeted improvements, and improves user experience and learning effects.
Smart Images

Figure CN119048301A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of virtual reality, and in particular to a VR motion training teaching method and system based on motion capture technology. Background Art
[0002] With the rapid development of virtual reality (VR) technology, its application in education is becoming increasingly widespread, particularly in motor training. With its unique immersive experience and high degree of interactivity, VR technology has revolutionized traditional teaching methods. Existing VR motor training methods typically create virtual environments that simulate real-world motor training processes, enabling users to practice repeatedly in virtual spaces, effectively improving their mastery of motor skills.
[0003] However, although the existing VR action training and teaching systems have improved the fun and effectiveness of teaching to a certain extent, they still have some significant defects. First, the existing VR action training and teaching systems often lack the fine-grained distinction between different action types. Whether it is martial arts, martial arts, lion dance, or other distinctive actions, they all use the same training model and feedback mechanism, resulting in a weak targeted training effect and difficulty in meeting the diverse learning needs of users. Second, the existing VR action training and teaching systems often lack continuity considerations, that is, they fail to combine abnormal data with the continuous action data before and after for analysis, resulting in one-sided feedback information, making it difficult for users to fully understand the problems with their own actions. At the same time, the lack of continuity in the form of teaching feedback is relatively simple, often simply pointing out the errors in the current action. The abnormal error of a certain action in the continuous action may also be caused by the incorrect connection between the actions, and there is a lack of feedback on the cause of the error in the connection between the previous and next continuous actions. Summary of the Invention
[0004] In order to solve the above-mentioned defects, the present application provides a VR motion training teaching method and system based on motion capture technology.
[0005] The above-mentioned invention objective of this application is achieved through the following technical solutions: A VR motion training teaching method based on motion capture technology, comprising the following steps: When receiving a training instruction from the user, identifying an action type associated with the training instruction, the action type including martial arts action, combat action, and lion dance; Matching a preset corresponding virtual sub-scene based on the identified action type, and obtaining teaching and training information of the virtual sub-scene, wherein the teaching and training information includes teaching audio and video data and standard action data; Collect the target object's motion sensor data in the virtual sub-scene, and compare and analyze the collected motion sensor data with the standard motion data in real time; When abnormal comparison data is identified, its abnormal type is identified, and the abnormal type includes abnormal posture, abnormal amplitude, abnormal speed and abnormal rhythm; Based on the action type, anomaly type and preset recognition strategy, identify the continuous action data associated with the anomaly comparison data; Based on teaching audio and video data, abnormal comparison data and continuous action data, teaching training feedback is provided to the user end.
[0006] By adopting the above technical solution, when a training instruction is received from the user side, the action type corresponding to the instruction is first identified, and the action types include martial arts actions, combat actions and lion awakening. The preset corresponding virtual sub-scene is matched according to the identified action type. Each virtual sub-scene contains teaching audio and video data and standard action data for a specific action type. When the user is training in the virtual sub-scene, the action sensor data of the target object is collected in real time for comparison and analysis with the standard action data to evaluate the action performance of the target object. When the comparison analysis finds abnormal comparison data, the type of abnormal comparison data is further identified, including abnormal posture, abnormal amplitude, abnormal speed and abnormal rhythm, so as to accurately locate the problems in the user's action. Based on the action type, abnormal type and preset recognition strategy, the continuous motion associated with the abnormal comparison data is identified. The application distinguishes different action types in a refined manner and matches corresponding virtual sub-scenes and teaching training information to provide more targeted training content and feedback, thereby effectively improving the pertinence and effectiveness of training. At the same time, based on the current abnormal data combined with the continuous action data for analysis, comprehensive teaching feedback is provided, which not only points out the specific errors in the user's actions, but also analyzes and feeds back the causes of errors in the connection of continuous actions, which helps users to fully understand the problems with their own actions and make targeted improvements. It has the effect of providing users with coherent and comprehensive feedback information, thereby enriching the feedback form and improving user experience and learning effects.
[0007] In a preferred example, the present application can be further configured as follows: the step of collecting motion sensing data of the target object in the virtual sub-scene and comparing and analyzing the collected motion sensing data with standard motion data in real time includes the following steps: Modeling standard action data to generate a standard action model; Collecting motion sensing data of the target object in the virtual sub-scene in real time and extracting motion sensing features of the motion sensing data; Matching the motion sensing features with the standard motion model and calculating the first similarity score; The motion sensing data corresponding to the motion sensing feature having the first similarity score lower than the similarity threshold is set as abnormal comparison data.
[0008] By adopting the above technical solution, the standard action data is deeply modeled to generate an accurate standard action model, and the sensor data of the target object when performing the action in the virtual sub-scene is collected in real time, and the key action sensing features are extracted from it. The extracted action sensing features are carefully matched with the previously established standard action model, and a first similarity score is calculated to evaluate the degree of fit between the target object's action performance and the standard action; if the first similarity score is lower than the preset similarity threshold, the corresponding action sensing data is marked as abnormal comparison data, thereby realizing real-time monitoring and identification of abnormal action conditions. Based on the above steps, the present application has the effect of improving the accuracy and real-time performance of action data processing, and enables the system to more accurately identify the abnormal performance of the target object during the action execution process. Through real-time comparison and analysis, accurate information about the action execution status is fed back to the user, thereby helping the user to promptly discover and correct deficiencies in the action, effectively improving the user's training effect.
[0009] In a preferred example, the present application can be further configured as follows: the step of identifying continuous action data associated with abnormal comparison data based on the action type, the abnormality type, and the preset identification strategy includes the following steps: Based on the action type and abnormal comparison data, identify the abnormal action item corresponding to the abnormal comparison data; Based on the abnormal action item, abnormal type and preset identification strategy, the continuous action data corresponding to the timestamp of the abnormal comparison data is identified.
[0010] By adopting the above technical solution, based on the action type and abnormal comparison data, the specific action item that causes the abnormality, namely the abnormal action item, is accurately identified. Based on the identified abnormal action item, combined with the abnormal type and the pre-set identification strategy, the continuous action data with the timestamp corresponding to the abnormal comparison data is further traced and identified, thereby accurately locking the abnormal link in the action execution; at the same time, through correlation analysis, the potential connection between the abnormal action and the previous and next actions is revealed, thereby providing users with more comprehensive and coherent feedback; based on the above steps, this application improves the system's recognition accuracy of abnormal actions, making the feedback more accurate and specific. Secondly, through correlation analysis of continuous action data, it helps users better understand the causes of abnormal actions and how they affect the entire action process, thereby guiding users to make targeted improvements.
[0011] In a preferred example, the present application can be further configured as follows: the step of providing teaching training feedback to the user terminal based on the teaching audio and video data, abnormal comparison data, and continuous action data includes the following steps: Modeling standard action data to generate a standard action model; Extract continuous motion features from continuous motion data and match the continuous motion features with standard motion models to identify abnormal motion sequence features; Inputting abnormal action sequence features and abnormal comparison data into a pre-trained abnormality analysis model, so that the abnormality analysis model outputs abnormality information, wherein the abnormality information includes abnormality degree information and abnormality cause information; Provide teaching and training feedback to the user end based on abnormal information and teaching audio and video data.
[0012] By adopting the above technical solution, by modeling the standard action data, an accurate standard action model is generated, the continuous action features in the continuous action data are extracted, and the extracted features are matched with the standard action model to accurately identify the abnormal action sequence features. The identified abnormal action sequence features and the abnormal comparison data are input into the pre-trained abnormal analysis model. The model will output detailed abnormal information, including the degree of abnormality and the specific cause of the abnormality. Based on the abnormal information output by the abnormal analysis model, combined with the teaching audio and video data, comprehensive and targeted teaching and training feedback is provided to the user end; based on the above steps, the present application not only improves the accuracy and comprehensiveness of abnormality recognition, but also makes the teaching and training feedback more specific and targeted. Through the matching of the standard action model and the in-depth analysis of the abnormal analysis model, the problems in the user's action are accurately pointed out, and detailed abnormal information and improvement suggestions are provided. At the same time, combined with the teaching audio and video data, intuitive and vivid demonstrations and explanations are provided to users, thereby helping users better understand the key points of the action, quickly correct errors, and improve training effects.
[0013] In a preferred example, the present application can be further configured as follows: the abnormality analysis model includes an integration layer, a matching layer, and an analysis layer; the step of inputting abnormal action sequence features and abnormal comparison data into a pre-trained abnormality analysis model so that the abnormality analysis model outputs abnormality information, wherein the abnormality information includes abnormality degree information and abnormality cause information, includes the following steps: The integration layer integrates the action sequence features and the abnormal comparison data to obtain the integrated feature vector; The matching layer matches the integrated feature vector with the pre-stored abnormal pattern and generates a second similarity score based on the matching result; The analysis layer generates abnormality cause information and abnormality degree information based on the integrated feature vector and its second similarity score, thereby outputting abnormality information.
[0014] By adopting the above technical solution, the anomaly analysis model deeply processes the input abnormal action sequence features and abnormal comparison data: the integration layer integrates the features of the two types of data to generate an integrated feature vector containing integrated information; the matching layer matches the integrated feature vector with the pre-stored abnormal pattern and calculates a second similarity score to quantify the similarity between the abnormal action and the known abnormal pattern; the analysis layer deeply analyzes the cause of the anomaly based on the integrated feature vector and its second similarity score, and evaluates the severity of the anomaly, and finally outputs complete anomaly information containing abnormal cause information and abnormality degree information; based on the above steps, the present application not only improves the accuracy and comprehensiveness of anomaly analysis, but also makes the output abnormal information more specific and targeted. Through the collaborative work of the integration layer, matching layer and analysis layer, the anomaly analysis model can accurately identify the abnormal links in the action sequence and deeply analyze the root cause of the anomaly; at the same time, by quantifying the degree of anomaly, the anomaly analysis model can also provide users with intuitive feedback on the severity of the anomaly, thereby helping users better understand the problems with their own actions and make targeted improvements.
[0015] In a preferred example, the present application may be further configured as follows: after the step of providing teaching training feedback to the user terminal based on the abnormal information and the teaching video and audio data, the following steps are performed: Preset corresponding tactile feedback mechanism and biofeedback mechanism based on abnormal information and abnormal type matching; Based on the matching tactile feedback mechanism and biofeedback mechanism, the user is provided with real-time tactile feedback and biofeedback during the teaching and training process.
[0016] By adopting the above technical solution, the preset tactile feedback mechanism and biofeedback mechanism are intelligently matched according to the abnormal information and its corresponding abnormal type, and a more intuitive and immersive training experience is provided to the user through real-time tactile and biofeedback: according to the identified abnormal movement and the degree of abnormality, the corresponding tactile feedback, such as vibration, pressure change, etc., is triggered to guide the user to perceive and correct the wrong movement. At the same time, the biofeedback mechanism will reflect the user's physical and mental state in real time by monitoring the user's physiological indicators, such as heart rate, muscle tension, etc., to help the user adjust the training rhythm and attention allocation; by combining tactile and biofeedback teaching and training methods, the user's perception and understanding of abnormal movements is enhanced, and the user's self-regulation of the physical and mental state is promoted, thereby improving the training effect. Through the real-time feedback mechanism, the user can immediately realize his or her movement errors and make timely adjustments under the guidance of the system to form correct movement habits.
[0017] The second object of the present invention is achieved through the following technical solutions: A VR motion training and teaching system based on motion capture technology, including: The action recognition module is used to identify the action type associated with the training instruction when receiving the training instruction from the user terminal, and the action type includes martial arts action, combat action and lion dance; An information acquisition module is used to match a preset corresponding virtual sub-scene based on the recognized action type and obtain teaching and training information of the virtual sub-scene, wherein the teaching and training information includes teaching audio and video data and standard action data; The comparison and analysis module is used to collect the motion sensor data of the target object in the virtual sub-scene and compare and analyze the collected motion sensor data with the standard motion data in real time; An abnormality type identification module is used to identify the abnormality type when abnormal comparison data is identified, and the abnormality type includes abnormal posture, abnormal amplitude, abnormal speed and abnormal rhythm; A continuous action recognition module is used to identify continuous action data associated with abnormal comparison data based on action type, abnormality type and preset recognition strategy; The teaching and training feedback module is used to provide teaching and training feedback to the user end based on teaching audio and video data, abnormal comparison data and continuous action data.
[0018] By adopting the above technical solution, the action recognition module is used to identify the action type corresponding to the training instruction when receiving the training instruction from the user terminal, and the action types include martial arts actions, combat actions and lion dance; the information acquisition module is used to match the preset corresponding virtual sub-scene based on the identified action type, and obtain the teaching and training information of the virtual sub-scene, and the teaching and training information includes teaching audio and video data and standard action data; the comparison and analysis module is used to collect the action sensor data of the target object in the virtual sub-scene, and compare and analyze the collected action sensor data with the standard action data in real time; the abnormal type recognition module is used to identify the abnormal type of abnormal comparison data when abnormal comparison data is identified, and the abnormal type includes abnormal posture, abnormal amplitude, abnormal speed and abnormal rhythm; the continuous action recognition module is used to identify the continuous action data associated with the abnormal comparison data based on the action type, abnormal type and preset recognition strategy; the teaching and training feedback module is used to provide teaching and training feedback to the user terminal based on the teaching audio and video data, abnormal comparison data and continuous action data.
[0019] In a preferred example, the present application can be further configured as follows: the comparison and analysis module includes: The model generation submodule is used to model the standard action data and generate a standard action model; A feature extraction submodule is used to collect the motion sensing data of the target object in the virtual sub-scene in real time and extract the motion sensing features of the motion sensing data; A first similarity score calculation submodule is used to match the motion sensing feature with the standard motion model and calculate a first similarity score thereof; The abnormality setting submodule is configured to set the motion sensing data corresponding to the motion sensing feature whose first similarity score is lower than the similarity threshold as abnormal comparison data.
[0020] By adopting the above technical solution, the model generation submodule is used to model the standard action data and generate a standard action model; the feature extraction submodule is used to collect the action sensing data of the target object in the virtual subscene in real time and extract the action sensing features of the action sensing data; the first similarity score calculation submodule is used to match the action sensing features with the standard action model and calculate their first similarity scores; and the abnormality setting submodule is used to set the action sensing data corresponding to the action sensing features whose first similarity scores are lower than the similarity threshold as abnormal comparison data.
[0021] In summary, this application includes at least one of the following beneficial technical effects: 1. This application provides more targeted training content and feedback by finely distinguishing different action types and matching corresponding virtual sub-scenes with teaching and training information, thereby effectively improving the relevance and effectiveness of training. At the same time, based on the current abnormal data combined with the continuous action data for analysis, it provides comprehensive teaching feedback. It not only points out the specific errors in the user's actions, but also analyzes and provides feedback on the causes of errors in the connection of continuous actions. This helps users fully understand the problems in their own actions and make targeted improvements. It provides users with coherent and comprehensive feedback information, thereby enriching the feedback form and improving user experience and learning effects. 2. By combining tactile and biofeedback teaching and training methods, users' perception and understanding of abnormal movements are enhanced, while promoting self-regulation of their physical and mental states, thereby improving training effectiveness. Through a real-time feedback mechanism, users can immediately realize their movement errors and make timely adjustments under the guidance of the system to form correct movement habits. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 This is a flow chart of an embodiment of a VR action training teaching method based on motion capture technology in the present application; Figure 2 This is a flowchart for implementing step S30 in an embodiment of a VR action training teaching method based on motion capture technology of the present application; Figure 3 This is a flowchart for implementing step S60 in an embodiment of a VR action training teaching method based on motion capture technology of the present application; Figure 4This is a flowchart for implementing step S63 in an embodiment of a VR action training teaching method based on motion capture technology in the present application. DETAILED DESCRIPTION
[0023] The following is combined with Figure 1-4 This application is described in further detail.
[0024] In one embodiment, if Figure 1 As shown, this application discloses a VR action training teaching method based on motion capture technology, which specifically includes the following steps: S10: When a training instruction is received from the user, identifying an action type associated with the training instruction, wherein the action type includes martial arts action, combat action, and lion dance; In this embodiment, the training instruction is an electric signal instruction issued by the user end for performing specific action training; the action type is a specific type of training action including martial arts action, combat action and lion dance; Specifically, when a training instruction is received from the user, the action type corresponding to the instruction is first identified, and the action types include martial arts, combat and lion dance. S20: matching a preset corresponding virtual sub-scene based on the identified action type, and obtaining teaching and training information of the virtual sub-scene, wherein the teaching and training information includes teaching audio and video data and standard action data; In this embodiment, the virtual sub-scenes are virtual training environments designed for different types of movements; the teaching audio-visual data are multimedia materials such as videos and audios used for teaching; and the standard movement data are reference data describing the correct way to perform movements, including movement trajectory, speed, amplitude, and other information. Furthermore, when the user terminal sends a martial arts training instruction, virtual sub-scenes related to the martial arts, such as a boxing practice field and a leg training room, are matched and loaded; when the user terminal sends a martial arts training instruction, virtual sub-scenes with artistic exaggeration and plot elements, such as a fighting scene in a costume martial arts film, are matched and loaded; when the user terminal sends a lion dance training instruction, virtual sub-scenes such as a traditional lion dance performance stage are matched and loaded; Specifically, matching preset corresponding virtual sub-scenes according to the identified action type, and obtaining teaching audio-visual data and standard action data for the specific action type contained in each virtual sub-scene; S30: collecting motion sensor data of the target object in the virtual sub-scene, and comparing and analyzing the collected motion sensor data with standard motion data in real time; In this embodiment, the motion sensing data is the real-time data of the target object when performing the action, including position, speed, acceleration, angle, etc., collected by the sensor; the comparison analysis is to compare the collected data with the standard data to identify differences and anomalies; Specifically, when the user is training in a virtual sub-scene, the target object's motion sensor data is collected in real time and compared with the standard motion data to evaluate the target object's motion performance; S40: When abnormal comparison data is identified, its abnormal type is identified, and the abnormal type includes abnormal posture, abnormal amplitude, abnormal speed and abnormal rhythm; In this embodiment, abnormal comparison data refers to collected data that is significantly different from standard data; abnormal type refers to specific features that describe abnormal data, including abnormal posture, abnormal amplitude, abnormal speed, and abnormal rhythm; Among them, in the recognition and comparison of motion sensor data based on martial arts movements, due to the high requirements of martial arts movements for accuracy and consistency, the comparison of motion sensor data with standard motion data needs to accurately reflect the starting and ending points of each movement, force changes, speed control and other details. Abnormal comparison data is mainly reflected in incorrect posture (abnormal posture), insufficient or excessive amplitude (abnormal amplitude), too fast or too slow speed (abnormal speed) and uncoordinated rhythm (abnormal rhythm). In the recognition and comparison of motion sensor data based on martial arts movements, due to the fact that martial arts movements may not completely follow the norms of traditional martial arts, the comparison standards for motion sensor data and standard motion data will also be relatively Abnormal data is usually more related to the creativity and visual effects of the movements. Abnormal comparison data is mainly reflected in the mismatch of the exaggeration of the movement (abnormal amplitude) and the mismatch of the smoothness of the movement (abnormal rhythm). In the recognition and comparison of motion sensor data based on the lion dance, because the lion dance requires the motion sensor data to accurately reflect the lion's spirit and momentum, as well as the tacit cooperation between the team, abnormal comparison data is mainly reflected in the uncoordinated movement (abnormal posture), lack of spirit (abnormal amplitude, abnormal rhythm, and abnormal speed), and also includes the lack of tacit team cooperation (abnormal posture, abnormal amplitude, abnormal speed, and abnormal rhythm when multiple users perform the same timestamp movement). Specifically, when the comparison analysis finds abnormal comparison data, the type of abnormal comparison data is further identified, including abnormal posture, abnormal amplitude, abnormal speed, and abnormal rhythm, so as to accurately locate the problem in the user's action; S50: Based on the action type, the exception type and the preset identification strategy, identify the continuous action data associated with the exception comparison data; In this embodiment, the continuous action data is action data that is temporally or logically connected to the abnormal data; the preset identification strategy is a rule method for identifying and associating the continuous action data, specifically, presetting different continuous action data identification quantities based on different action types and their abnormal types; Specifically, based on the action type, the abnormality type and the preset recognition strategy, the continuous action data associated with the abnormal comparison data is identified, and the data of the coherent actions before and after the action corresponding to the abnormal comparison data is recognized; S60: Providing teaching training feedback to the user terminal based on the teaching video and audio data, abnormality comparison data, and continuous action data; In this embodiment, the teaching and training feedback is feedback information provided to the user terminal on the execution of the action, including problem points and corresponding improvement suggestions; Specifically, based on teaching audio and video data, abnormal comparison data and continuous action data, comprehensive teaching and training feedback is provided to the user end.
[0025] In one embodiment, if Figure 2 As shown, step S30 includes the steps of: S31: Modeling the standard action data to generate a standard action model; S32: collecting motion sensing data of the target object in the virtual sub-scene in real time, and extracting motion sensing features of the motion sensing data; S33: Matching the motion sensing feature with the standard motion model and calculating a first similarity score thereof; S34: setting the motion sensing data corresponding to the motion sensing feature having the first similarity score lower than the similarity threshold as abnormal comparison data; In this embodiment, the standard motion model is a mathematical model that is generated by processing and modeling the standard motion data and can represent the correct motion characteristics; the motion sensing feature is key feature information that is extracted from the motion sensing data and can represent the characteristics of the motion; the first similarity score is a numerical value reflecting the degree of similarity between the motion sensing feature and the standard motion model; the similarity threshold is a preset numerical value used to determine whether the similarity between the motion sensing feature and the standard motion model reaches an acceptable range; the abnormal comparison data is the motion sensing data whose first similarity score is lower than the similarity threshold, which represents an abnormality or error in the execution of the motion; Specifically, the standard action data is deeply modeled to generate an accurate standard action model, and the sensor data of the target object when performing the action in the virtual sub-scene is collected in real time, and the key action sensing features are extracted from it. The extracted action sensing features are carefully matched with the previously established standard action model, and a first similarity score is calculated to evaluate the degree of fit between the target object's action performance and the standard action; if the first similarity score is lower than the preset similarity threshold, the corresponding action sensing data is marked as abnormal comparison data, thereby realizing real-time monitoring and identification of abnormal action situations.
[0026] In one embodiment, step S50 includes the steps of: S51: Based on the action type and the abnormal comparison data, identifying the abnormal action item corresponding to the abnormal comparison data; S52: Based on the abnormal action item, the abnormal type and the preset identification strategy, identify the continuous action data corresponding to the timestamp of the abnormal comparison data; In this embodiment, the abnormal action item is an abnormal or erroneous action part that occurs during the execution of the action, wherein each action type includes a number of preset action items; the corresponding timestamp is a time stamp defined in association with the preset identification strategy, which is used to correspond to the length or number of continuous action data; Specifically, based on the action type and abnormal comparison data, the specific action item that causes the abnormality, namely the abnormal action item, is accurately identified. Based on the identified abnormal action item, combined with the abnormality type and the pre-set identification strategy, the continuous action data with the corresponding timestamp of the abnormal comparison data is further traced and identified, so as to accurately lock the abnormal link in the action execution; at the same time, through correlation analysis, the potential connection between the abnormal action and the previous and subsequent actions is revealed, thereby providing users with more comprehensive and coherent feedback.
[0027] In one embodiment, if Figure 3 As shown, step S60 includes the steps of: S61: Modeling the standard action data to generate a standard action model; S62: extracting continuous motion features from the continuous motion data, and matching the continuous motion features with a standard motion model to identify abnormal motion sequence features; S63: Inputting the abnormal action sequence features and the abnormal comparison data into a pre-trained abnormality analysis model, so that the abnormality analysis model outputs abnormality information, wherein the abnormality information includes abnormality degree information and abnormality cause information; S64: Providing teaching training feedback to the user terminal based on the abnormal information and the teaching audio and video data; In this embodiment, the continuous action feature is the key feature information extracted from the continuous action data that can represent the characteristics of the action; the abnormal action sequence feature is the sequence feature of the continuous action that does not match the standard action model, indicating that there is an abnormality or error in the action execution; the abnormality analysis model is a pre-trained model for analyzing abnormal action sequence features and abnormal comparison data and outputting abnormal information; the abnormality degree information is the numerical value or level information describing the severity of the abnormal action; the abnormality cause information is the information explaining the cause or factor that caused the abnormal action to occur; Specifically, by modeling the standard action data, an accurate standard action model is generated, the continuous action features in the continuous action data are extracted, and the extracted features are matched with the standard action model to accurately identify the abnormal action sequence features. The identified abnormal action sequence features and abnormal comparison data are input into a pre-trained abnormal analysis model. The model will output detailed abnormal information, including the degree of abnormality and the specific cause of the abnormality. Based on the abnormal information output by the abnormal analysis model and combined with the teaching audio and video data, comprehensive and targeted teaching and training feedback is provided to the user end.
[0028] In one embodiment, the anomaly analysis model includes an integration layer, a matching layer, and an analysis layer, such as Figure 4 As shown, step S63 includes the steps of: S631: The integration layer integrates the action sequence features and the abnormal comparison data to obtain an integrated feature vector; S632: The matching layer matches the integrated feature vector with the pre-stored abnormal pattern and generates a second similarity score based on the matching result; S633: The analysis layer generates abnormality cause information and abnormality degree information based on the integrated feature vector and its second similarity score, thereby outputting abnormality information; In this embodiment, the integration layer is the first layer of the anomaly analysis model, which is used to integrate the input action sequence features and anomaly comparison data; feature integration is to combine feature data from different sources or different types to form a more comprehensive feature vector, specifically to integrate the action sequence features and anomaly comparison data, and feature integration includes splicing, weighted sum, feature fusion, etc., wherein, before integration, the action sequence features and the anomaly comparison data are preprocessed, including normalization, standardization and feature removal, etc.; the integrated feature vector is obtained by feature integration, which contains action sequence features and anomaly comparison data information. At the same time, integrating the action sequence features and the anomaly comparison data into an integrated feature vector can reduce the noise and misjudgment caused by a single feature; the matching layer is the second layer of the anomaly analysis model, which is used to match the integrated feature vector with the pre-stored anomaly pattern; the anomaly pattern is a feature pattern representing a known anomaly type, which is used to compare with the integrated feature vector; the second similarity score is a numerical value reflecting the degree of similarity between the integrated feature vector and the anomaly pattern; the analysis layer is the third layer of the anomaly analysis model, which is used to generate anomaly information based on the integrated feature vector and the second similarity score; Specifically, the anomaly analysis model performs in-depth processing on the input abnormal action sequence features and abnormal comparison data: the integration layer integrates the features of these two types of data to generate an integrated feature vector containing the integrated information; the matching layer matches the integrated feature vector with the pre-stored abnormal pattern and calculates a second similarity score to quantify the similarity between the abnormal action and the known abnormal pattern; the analysis layer deeply analyzes the cause of the anomaly based on the integrated feature vector and its second similarity score, and evaluates the severity of the anomaly, and finally outputs complete anomaly information containing the cause of the anomaly and the severity of the anomaly.
[0029] In one embodiment, after step S64, the following steps are performed: S65: Preset corresponding tactile feedback mechanism and biofeedback mechanism based on abnormal information and abnormal type matching; S66: Based on the matching tactile feedback mechanism and biofeedback mechanism, real-time tactile feedback and biofeedback are provided to the user during the teaching and training process; In this embodiment, the tactile feedback mechanism is a mechanism that provides feedback through tactile stimulation, including vibration, pressure change, etc. After the user wears the smart device, the smart device will provide the user with reminders and feedback on posture adjustment. The biofeedback mechanism is a mechanism that uses biological signals (such as heart rate, muscle activity, etc.) to provide feedback to help the user understand their physical condition. Furthermore, for abnormal posture, vibration feedback can be used to remind users to adjust their posture; for abnormal amplitude, pressure feedback can be used to make users feel the difference in movement amplitude; for abnormal speed, vibration frequency can be changed to simulate speed changes; for abnormal rhythm, periodic vibration or pressure changes can be used to guide users to find the correct rhythm. Furthermore, the tactile feedback mechanism is used to provide tactile feedback to the user during the teaching and training process, for example: Vibration feedback: When abnormal user movements are detected (such as incorrect posture, excessive or insufficient amplitude), the vibration motor on the VR device can provide the user with immediate vibration feedback. The intensity, frequency, and duration of the vibration can be adjusted according to the severity of the abnormality to attract the user's attention and prompt them to adjust their movements. Force feedback: Motion capture technology is combined with force feedback devices in VR gloves or clothing. When the user performs an action, the force sense is simulated according to the standard action data. If the action is abnormal, the force output by the force feedback device is adjusted, such as increasing resistance or providing reverse force, to help the user correct the action. Temperature feedback: In some cases, feedback can be provided by changing the temperature of the part of the VR device that contacts the user's skin. For example, when it is detected that the user is moving too fast or too slow, the device temperature can be briefly increased or decreased to remind the user to adjust the rhythm. Haptic simulation: For certain specific movement training, such as striking in martial arts, a haptic simulator on a VR device can simulate the tactile sensation of striking an object, such as hardness and elasticity. If the movement is abnormal, the simulator can change these tactile parameters to guide the user to adjust the movement. Furthermore, the biofeedback mechanism is used to provide biofeedback to the user during the teaching and training process, for example: Heart rate monitoring and feedback: By wearing a smart device equipped with a heart rate monitor, the user's heart rate changes are monitored in real time. When the system detects that the user's heart rate is elevated due to abnormal movements or tension, it can provide feedback through sound, images, or vibrations to help the user relax and adjust their movements. Muscle tension feedback: Smart devices equipped with electromyography (EMG) sensors monitor changes in the tension of key muscles. When abnormal muscle tension is detected, visual cues or tactile feedback on the VR interface guide the user to adjust their muscle state to achieve the correct movement requirements. Breathing rhythm feedback: The user's breathing rhythm is monitored in real time through a smart device equipped with a respiratory monitoring device. When a discord between the breathing rhythm and the movement is detected, the user is prompted to adjust their breathing through sound or images to help them better control their movements. Neurofeedback: Using neurofeedback technology, such as electroencephalogram (EEG), to monitor changes in the user's brainwaves, the system analyzes brainwave data to understand the user's attention and relaxation levels, and provides feedback based on this information to help the user better focus and relax. Specifically, based on the abnormal information and its corresponding abnormal type, the preset tactile feedback mechanism and biofeedback mechanism are intelligently matched, and a more intuitive and immersive training experience is provided to the user through real-time tactile and biofeedback: according to the identified abnormal movements and the degree of abnormality, corresponding tactile feedback is triggered, such as vibration, pressure changes, etc., to guide the user to perceive and correct the wrong movements. At the same time, the biofeedback mechanism will reflect the user's physical and mental state in real time by monitoring the user's physiological indicators, such as heart rate, muscle tension, etc., to help the user adjust the training rhythm and attention allocation.
[0030] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0031] In one embodiment, a VR motion training and teaching system based on motion capture technology is provided. The VR motion training and teaching system based on motion capture technology corresponds one-to-one to the VR motion training and teaching method based on motion capture technology in the above embodiment.
[0032] A VR motion training and teaching system based on motion capture technology, including: The action recognition module is used to identify the action type associated with the training instruction when receiving the training instruction from the user terminal, and the action type includes martial arts action, combat action and lion dance; An information acquisition module is used to match a preset corresponding virtual sub-scene based on the recognized action type and obtain teaching and training information of the virtual sub-scene, wherein the teaching and training information includes teaching audio and video data and standard action data; The comparison and analysis module is used to collect the motion sensor data of the target object in the virtual sub-scene and compare and analyze the collected motion sensor data with the standard motion data in real time; An abnormality type identification module is used to identify the abnormality type when abnormal comparison data is identified, and the abnormality type includes abnormal posture, abnormal amplitude, abnormal speed and abnormal rhythm; A continuous action recognition module is used to identify continuous action data associated with abnormal comparison data based on action type, abnormality type and preset recognition strategy; The teaching and training feedback module is used to provide teaching and training feedback to the user end based on teaching audio and video data, abnormal comparison data, and continuous action data; Optional comparison analysis modules include: The model generation submodule is used to model the standard action data and generate a standard action model; A feature extraction submodule is used to collect the motion sensing data of the target object in the virtual sub-scene in real time and extract the motion sensing features of the motion sensing data; A first similarity score calculation submodule is used to match the motion sensing feature with the standard motion model and calculate a first similarity score thereof; an abnormality setting submodule, configured to set the motion sensing data corresponding to the motion sensing feature having a first similarity score lower than a similarity threshold as abnormal comparison data; Optionally, the continuous action recognition module includes: An abnormal action item identification submodule is used to identify the abnormal action item corresponding to the abnormal comparison data based on the action type and the abnormal comparison data; The continuous action data identification submodule is used to identify the continuous action data corresponding to the timestamp of the abnormal comparison data based on the abnormal action item, abnormal type and preset identification strategy; Optional teaching and training feedback modules include: The sequence feature recognition submodule is used to extract the continuous motion features of the continuous motion data and match the continuous motion features with the standard motion model to identify abnormal motion sequence features; An input submodule is used to input abnormal action sequence features and abnormal comparison data into a pre-trained abnormal analysis model, so that the abnormal analysis model outputs abnormal information, which includes abnormal degree information and abnormal cause information; The teaching and training feedback submodule is specifically used to provide teaching and training feedback to the user terminal based on abnormal information and teaching audio and video data; Optionally, also include: The integration layer module is used to integrate the action sequence features and the abnormal comparison data to obtain the integrated feature vector; a matching layer module, configured to match the integrated feature vector with pre-stored anomaly patterns and generate a second similarity score based on the matching result; An analysis layer module, configured to generate abnormality cause information and abnormality degree information based on the integrated feature vector and its second similarity score, thereby outputting abnormality information; Optionally, also include: A feedback mechanism matching module is used to match preset corresponding tactile feedback mechanisms and biofeedback mechanisms based on abnormal information and abnormality types; The feedback mechanism implementation module is used to provide real-time tactile feedback and biofeedback to the user during the teaching and training process based on the matching tactile feedback mechanism and biofeedback mechanism.
[0033] For the specific definition of a VR motion training and teaching system based on motion capture technology, please refer to the definition of a VR motion training and teaching method based on motion capture technology above, which will not be repeated here. The various modules in the above-mentioned VR motion training and teaching system based on motion capture technology can be implemented in whole or in part through software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0034] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.
Claims
1. A VR action training teaching method based on motion capture technology, characterized by: Includes steps: When receiving a training instruction from the user, identifying the action type associated with the training instruction, the action type includes martial arts action, combat action and lion dance; Matching a preset corresponding virtual sub-scene based on the recognized action type, and obtaining teaching training information of the virtual sub-scene, wherein the teaching training information includes teaching audio and video data and standard action data; Collect the motion sensor data of the target object in the virtual sub-scene, and compare and analyze the collected motion sensor data with the standard motion data in real time; When abnormal comparison data is identified, its abnormal type is identified, and the abnormal type includes abnormal posture, abnormal amplitude, abnormal speed and abnormal rhythm; Based on the action type, the exception type and the preset identification strategy, identify the continuous action data associated with the abnormal comparison data; Based on teaching audio and video data, abnormal comparison data and continuous action data, teaching training feedback is provided to the user.
2. A VR action training teaching method based on motion capture technology according to claim 1, characterized in that: The step of collecting the motion sensing data of the target object in the virtual sub-scene and comparing and analyzing the collected motion sensing data with the standard motion data in real time comprises the following steps: Modeling standard action data to generate a standard action model; Collecting motion sensing data of the target object in the virtual sub-scene in real time, and extracting motion sensing features of the motion sensing data; Matching the motion sensing features with the standard motion model and calculating a first similarity score thereof; The motion sensing data corresponding to the motion sensing feature having a first similarity score lower than the similarity threshold is set as abnormal comparison data.
3. The VR action training teaching method based on motion capture technology according to claim 1 is characterized by: The step of identifying the continuous action data associated with the abnormal comparison data based on the action type, the abnormality type and the preset identification strategy comprises the steps of: Based on the action type and the abnormal comparison data, identify the abnormal action item corresponding to the abnormal comparison data; Based on the abnormal action item, abnormal type and preset identification strategy, identify the continuous action data of the timestamp corresponding to the abnormal comparison data.
4. The VR action training teaching method based on motion capture technology according to claim 1, characterized in that: The step of providing teaching training feedback to the user terminal based on the teaching audio and video data, abnormal comparison data and continuous action data includes the following steps: Modeling standard action data to generate a standard action model; Extracting continuous motion features from continuous motion data and matching the continuous motion features with standard motion models to identify abnormal motion sequence features; Inputting abnormal action sequence features and abnormal comparison data into a pre-trained abnormal analysis model, so that the abnormal analysis model outputs abnormal information, wherein the abnormal information includes abnormal degree information and abnormal cause information; Provide teaching training feedback to the user based on abnormal information and teaching audio and video data.
5. The VR action training teaching method based on motion capture technology according to claim 4 is characterized by: The abnormal analysis model includes an integration layer, a matching layer and an analysis layer. The abnormal action sequence features and abnormal comparison data are input into the pre-trained abnormal analysis model so that the abnormal analysis model outputs abnormal information, and the abnormal information includes abnormal degree information and abnormal cause information, including the steps of: The integration layer integrates the action sequence features and the abnormal comparison data to obtain the integrated feature vector; The matching layer matches the integrated feature vector with the pre-stored abnormal pattern and generates a second similarity score based on the matching result; The analysis layer generates abnormality cause information and abnormality degree information based on the integrated feature vector and its second similarity score, thereby outputting abnormality information.
6. The VR action training teaching method based on motion capture technology according to claim 4 is characterized by: After the step of providing teaching training feedback to the user terminal based on the abnormal information and the teaching audio and video data, the following steps are performed: Preset corresponding tactile feedback mechanism and biofeedback mechanism based on abnormal information and abnormal type matching; Based on the matching tactile feedback mechanism and biofeedback mechanism, the user is provided with real-time tactile feedback and biofeedback during the teaching and training process.
7. A VR motion training and teaching system based on motion capture technology, characterized by: include: The action recognition module is used to identify the action type associated with the training instruction when receiving the training instruction from the user end, and the action type includes martial arts action, combat action and lion dance; An information acquisition module, used to match a preset corresponding virtual sub-scene based on the recognized action type, and acquire teaching and training information of the virtual sub-scene, wherein the teaching and training information includes teaching audio and video data and standard action data; A comparison and analysis module is used to collect motion sensor data of the target object in the virtual sub-scene, and compare and analyze the collected motion sensor data with standard motion data in real time; An abnormal type identification module is used to identify the abnormal type of abnormal comparison data when abnormal comparison data is identified, and the abnormal type includes abnormal posture, abnormal amplitude, abnormal speed and abnormal rhythm; A continuous action recognition module, used to identify continuous action data associated with abnormal comparison data based on action type, abnormal type and preset recognition strategy; The teaching and training feedback module is used to provide teaching and training feedback to the user end based on the teaching audio and video data, abnormal comparison data and continuous action data.
8. The VR motion training and teaching system based on motion capture technology according to claim 7, characterized in that: The comparison and analysis module comprises: The model generation submodule is used to model the standard action data and generate a standard action model; A feature extraction submodule is used to collect the motion sensing data of the target object in the virtual subscene in real time and extract the motion sensing features of the motion sensing data; A first similarity score calculation submodule, used to match the motion sensing feature with the standard motion model and calculate a first similarity score thereof; The abnormal setting submodule is used to set the motion sensing data corresponding to the motion sensing feature whose first similarity score is lower than the similarity threshold as abnormal comparison data.
Citation Information
Patent Citations
Tactile feedback type training system used for body training
CN109999468A
Intelligent physical education method based on artificial intelligence
CN116611966A
Dance teaching evaluation system and method
CN118366226A
Behavior recognition method and system
CN118839158A
Deep learning system for real time analysis of manufacturing operations
WO2018191555A1
