An interactive teaching method and system based on MR technology

Abstract

The application belongs to the technical field of MR technology, and discloses an interactive teaching method and system based on MR technology, which comprises the following steps: obtaining hand data of a user, generating corresponding virtual hand data, sending a virtual teaching request to the user to guide the user to move the hands to a plurality of target positions corresponding to teaching content in sequence, converting gesture changes generated by the user into corresponding gesture change data according to a preset linear interpolation formula to obtain the virtual hand data after movement, and judging whether the virtual hand data after movement all reach the corresponding target positions; if yes, it is determined that the user completes the virtual teaching request; if no, the user is reminded to move the hands to the plurality of target positions again until the virtual hand data after movement all reach the corresponding target positions; whether the user completes the virtual teaching request is determined according to whether the virtual hand data after movement obtained after each movement all reach the corresponding target positions, and the efficiency of virtual teaching is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of MR technology, in particular to an interactive teaching method and system based on MR technology. BACKGROUND

[0002] With the acceleration of the new round of global technological revolution and industrial change, information technology and education teaching are deeply integrated. MR technology (Mixed Reality, mixed reality technology) is a key field to promote education innovation and build new education productivity. MR technology can make teachers and students obtain a large amount of three-dimensional interactive information related to teaching content in real time on the classroom, innovate teaching methods, and improve learning effect by deeply integrating advanced information technology with virtual scenes. The introduction of MR technology can open up new possibilities for education and teaching, stimulate students' interest in learning, and make abstract knowledge more vivid and intuitive.

[0003] However, although the MR interactive teaching system has broad prospects, it still faces some challenges in actual application. For example, the existing MR technology interactive teaching method may be relatively complex in operation process, and students are difficult to quickly adapt and master, and there is a lack of simple and intuitive guidance method, which not only reduces the teaching efficiency, but also affects the use experience and popularization and application of MR technology in education scene. If students lack clear and effective guidance when performing interactive operations, they are likely to be confused, affecting the continuity and enthusiasm of learning. In addition, the existing technology may have problems such as the generated three-dimensional teaching content of the device being out of synchronization with the display screen of the projection device, which can easily lead to disordered display of knowledge points. These problems limit the further development and application effect of MR technology in the field of education.

[0004] Therefore, in order to solve the technical problem that the insufficient guidance in the existing MR technology teaching method leads to the decline of teaching efficiency and learning experience, it is urgent to provide an interactive teaching method and system based on MR technology. SUMMARY

[0005] The purpose of the present application is to provide an interactive teaching method and system based on MR technology, which calculates the virtual hand data after movement by using a preset linear interpolation formula and user gesture changes, to determine whether the virtual hand data after movement obtained each time reaches the corresponding target position, and to determine whether the user completes the virtual teaching request, thereby solving the problem that the insufficient guidance in the existing MR technology teaching method leads to the decline of teaching efficiency and learning experience, guiding the user to complete the teaching content through a preset path and action, reducing the confusion and errors caused by operation uncertainty, and improving the learning efficiency of virtual teaching.

[0006] In a first aspect, the present application provides an interactive teaching method based on MR technology, comprising:

[0007] obtaining hand data of a user;

[0008] generating virtual hand data corresponding to the hand data;

[0009] dividing teaching content into a plurality of target positions, and sending a virtual teaching request to the user to guide the user to move the hand to the plurality of target positions in sequence;

[0010] obtaining gesture changes generated by the user, and converting the gesture changes into gesture change data corresponding to the virtual hand data according to a preset linear interpolation formula to obtain moved virtual hand data;

[0011] judging whether the moved virtual hand data obtained after each movement reaches the corresponding target position; if yes, it is determined that the user completes the virtual teaching request; if no, the user is reminded to move the hand to the initial position and then move the hand to the plurality of target positions again until the moved virtual hand data obtained after each movement reaches the corresponding target position.

[0012] The interactive teaching method based on MR technology provided by the present application can guide the user to complete the teaching content. The moved virtual hand data is calculated by the preset linear interpolation formula and the gesture changes generated by the user to judge whether the moved virtual hand data obtained after each movement reaches the corresponding target position, determine whether the user completes the virtual teaching request, solve the problem of insufficient guidance in the existing MR technology teaching method, which leads to the decline of teaching efficiency and learning experience, guide the user to complete the teaching content through the preset path and action, reduce the confusion and errors caused by operation uncertainty, and improve the learning efficiency of virtual teaching.

[0013] Optionally, the hand data includes the bone identification number of each bone of the hand and the position of the corresponding bone; the virtual hand data corresponding to the hand data is generated by:

[0014] generating initial virtual hand data based on the identification number of each bone of the hand in the hand data and combining a preset virtual hand template;

[0015] adjusting the posture of the initial virtual hand data according to the position of each bone of the hand in the hand data to obtain the virtual hand data corresponding to the hand data.

[0016] The interactive teaching method based on the MR technology provided in the application can guide a user to complete teaching content, and by combining a template structure and real-time position information, virtual hand data that is correct in structure and accurately reflects a current state of a hand of the user is generated, thereby providing a reliable basis for subsequent virtual teaching requests, gesture change conversion, and target position determination, and improving the accuracy and effectiveness of the interactive teaching of the MR technology.

[0017] Optionally, the gesture change generated by the user is acquired, and the gesture change data corresponding to the virtual hand data is converted according to a preset linear interpolation formula, to obtain the virtual hand data after movement.

[0018] The gesture change generated by the user is acquired, and initial gesture change data corresponding to the virtual hand data is converted according to a preset linear interpolation formula.

[0019] The initial gesture change data is predicted and corrected by using a Kalman filtering algorithm, to obtain filtered gesture change data.

[0020] The virtual hand data is updated according to the filtered gesture change data, to obtain the virtual hand data after movement.

[0021] Optionally, the gesture change generated by the user is acquired, and the gesture change data corresponding to the virtual hand data is converted according to a preset linear interpolation formula, to obtain the virtual hand data after movement.

[0022] The gesture change image generated by the user is recorded.

[0023] The gesture change image is converted into corresponding multiple frames of rendering images.

[0024] Based on the multiple frames of rendering images, initial gesture change data generated by the virtual hand data is calculated by using a preset linear interpolation formula.

[0025] Optionally, the gesture change image is converted into corresponding multiple frames of rendering images, including:

[0026] The gesture change image of each frame is converted into a preliminary rendering image.

[0027] It is judged whether the hand data in the preliminary rendering image of each frame is complete; if yes, the preliminary rendering images of each frame are synthesized to obtain multiple frames of rendering images; if not, the preliminary rendering image that lacks hand data is supplemented to obtain a complete preliminary rendering image, and the complete preliminary rendering image is synthesized to obtain multiple frames of rendering images.

[0028] The interactive teaching method based on the MR technology provided in the application can guide the user to complete the teaching content, check and process the incomplete hand data, ensure that the final obtained multiple frames of rendered images contain complete hand information, guarantee the integrity of the input data, improve the reliability of the whole gesture change conversion process, thereby helping to more accurately update the virtual hand data, and improve the precision and user experience of the interactive teaching.

[0029] Optionally, the preliminary rendered image lacking hand data is supplemented to obtain a complete preliminary rendered image, including:

[0030] It is judged whether the hand data lacking in the preliminary rendered image is key position data.

[0031] If not, the complete hand data of the user is used to supplement the hand data lacking to obtain a complete preliminary rendered image.

[0032] If yes, the user is reminded to pay attention to the occlusion problem, and the step of issuing a virtual teaching request to the user is returned to be executed.

[0033] Optionally, after the virtual hand data is updated according to the filtered gesture change data to obtain the moved virtual hand data, the method further includes:

[0034] The gesture type of the moved virtual hand data is identified, and if the gesture type is a special gesture type, the teaching process is paused, and a corresponding operation is performed according to the semantics corresponding to the special gesture type.

[0035] Optionally, it is judged whether the moved virtual hand data obtained after each movement reaches the corresponding target position, including:

[0036] The distance between the position of each bone in the moved virtual hand data obtained after each movement and the position of the corresponding target bone in the corresponding target position is calculated.

[0037] Based on the size relationship between the distance of all bones obtained after each movement and a preset error threshold, it is sequentially judged whether the moved virtual hand data obtained after each movement reaches the corresponding target position; when all the distances are less than the preset error threshold, it is determined that the moved virtual hand data reaches the corresponding target position; when there is any one distance greater than or equal to the preset error threshold, it is determined that the distance greater than or equal to the preset error threshold corresponds to the moved virtual hand data that has not reached the corresponding target position.

[0038] Optionally, after it is determined that the user completes the virtual teaching request, the method further includes:

[0039] stopping acquiring the gesture change of the user until the next sending of the virtual teaching request to avoid acquiring error information.

[0040] In a second aspect, the present application provides an interactive teaching system based on MR technology, comprising:

[0041] an acquisition module configured to acquire hand data of a user;

[0042] a generation module configured to generate virtual hand data corresponding to the hand data;

[0043] a guidance module configured to divide teaching content into a plurality of target positions, and send a virtual teaching request to the user to guide the user to move the hand to the plurality of target positions in sequence;

[0044] a conversion module configured to acquire a gesture change generated by the user, and convert the gesture change into gesture change data corresponding to the virtual hand data according to a preset linear interpolation formula to obtain moved virtual hand data;

[0045] a judgment module configured to judge whether the moved virtual hand data obtained after each movement reaches the corresponding target position; if yes, it is determined that the user completes the virtual teaching request; if no, the user is reminded to move the hand to the initial position and then move the hand to the plurality of target positions again until the moved virtual hand data obtained after each movement reaches the corresponding target position.

[0046] The interactive teaching system based on MR technology calculates the moved virtual hand data through the preset linear interpolation formula and the gesture change generated by the user to judge whether the moved virtual hand data obtained after each movement reaches the corresponding target position, and determines whether the user completes the virtual teaching request, thereby solving the problem of insufficient guidance in the existing MR technology teaching method, which leads to the decline of teaching efficiency and learning experience, guiding the user to complete the teaching content through the preset path and action, reducing the confusion and errors caused by operation uncertainty, and improving the learning efficiency of virtual teaching.

[0047] Beneficial effects: The interactive teaching method and system based on MR technology provided by the present application calculate the moved virtual hand data through the preset linear interpolation formula and the gesture change generated by the user to judge whether the moved virtual hand data obtained after each movement reaches the corresponding target position, and determine whether the user completes the virtual teaching request, thereby solving the problem of insufficient guidance in the existing MR technology teaching method, which leads to the decline of teaching efficiency and learning experience, guiding the user to complete the teaching content through the preset path and action, reducing the confusion and errors caused by operation uncertainty, and improving the learning efficiency of virtual teaching. Attached Figure Description

[0048] Figure 1 A flowchart of an interactive teaching method based on MR technology provided in an embodiment of this application.

[0049] Figure 2 A schematic diagram of the structure of an interactive teaching system based on MR technology provided in an embodiment of this application.

[0050] Labeling Explanation: 1. Acquisition Module; 2. Generation Module; 3. Guidance Module; 4. Conversion Module; 5. Judgment Module. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0052] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0053] Please refer to Figure 1 , Figure 1 This application presents an interactive teaching method based on MR technology in some embodiments, used to guide users to complete teaching content, including the following steps:

[0054] Step S101: Obtain the user's hand data;

[0055] Step S102: Generate virtual hand data corresponding to the hand data;

[0056] Step S103: Divide the teaching content into multiple target positions and send a virtual teaching request to the user to guide the user to move their hands to multiple target positions in sequence;

[0057] Step S104: Obtain the gesture changes generated by the user, and convert them into gesture change data corresponding to the virtual hand data according to the preset linear interpolation formula to obtain the moved virtual hand data.

[0058] In step S105, it is determined whether the virtual hand data after each movement reaches the corresponding target position; if yes, it is determined that the user completes the virtual teaching request; if no, the user is reminded to move the hand to the initial position and then move the hand to the plurality of target positions again until the virtual hand data after each movement reaches the corresponding target position.

[0059] The interactive teaching method based on the MR technology calculates the virtual hand data after movement through the preset linear interpolation formula and the gesture change generated by the user to determine whether the virtual hand data after each movement reaches the corresponding target position and determine whether the user completes the virtual teaching request, thereby solving the problem of insufficient guidance in the existing MR technology teaching method, which leads to the problem of decline in teaching efficiency and learning experience. The user is guided to complete the teaching content through the preset path and action, which reduces the confusion and errors caused by operation uncertainty and improves the learning efficiency of virtual teaching.

[0060] Specifically, in step S101, hand data of the user is obtained, wherein the hand data includes the bone identification number of each bone of the hand and the position of the corresponding bone, and the hand data can be obtained by a sensor such as a depth camera or an inertial measurement unit, which is used to capture real-time position and posture information of the user's hand.

[0061] Specifically, in step S102, virtual hand data corresponding to the hand data is generated, including:

[0062] Based on the identification number of each bone of the hand in the hand data, an initial virtual hand data is generated in combination with a preset virtual hand template;

[0063] According to the position of each bone of the hand in the hand data, the posture of the initial virtual hand data is adjusted to obtain virtual hand data corresponding to the hand data.

[0064] In step S102, the identification number of each bone in the hand data is used in combination with the preset virtual hand template to construct an initial virtual hand with a basic structure and a bone correspondence relationship. This step provides a basic framework for the virtual hand to ensure that the virtual model matches the structure of the real hand. According to the actual position information of each bone in the hand data, the posture of the initial virtual hand model is adjusted. This adjustment process enables the virtual hand to accurately mimic the current posture and shape of the user's real hand. By combining the template structure and real-time position information, a virtual hand data is generated that is both structurally correct and accurately reflects the current state of the user's hand, providing a virtual representation of the user's hand.

[0065] Specifically, in step S103, the teaching content is decomposed into a series of target positions that need to be reached by the hand in sequence, and a virtual teaching request is issued to guide the user to move the hand to the target positions in sequence, thereby providing the user with clear and step-by-step operation guidance.

[0066] Specifically, in step S104, the gesture change generated by the user is obtained, and is converted into gesture change data corresponding to the virtual hand data according to a preset linear interpolation formula, to obtain the moved virtual hand data, including:

[0067] The gesture change generated by the user is obtained, and is converted into initial gesture change data corresponding to the virtual hand data according to a preset linear interpolation formula;

[0068] The initial gesture change data is predicted and corrected by using a Kalman filtering algorithm to obtain filtered gesture change data;

[0069] The virtual hand data is updated according to the filtered gesture change data to obtain the moved virtual hand data.

[0070] In step S104, after obtaining the gesture change of the user, the gesture change of the user is preliminarily mapped to the change amount of the virtual hand data by linear interpolation to obtain initial gesture change data. This preliminary mapping may contain sensor noise or inaccuracy. In order to improve the smoothness and accuracy of the virtual hand action, a Kalman filtering algorithm is introduced to process the initial gesture change data. Kalman filtering optimally estimates the state by combining the dynamic model of the system and the measurement data, thereby effectively filtering out noise and obtaining more stable and accurate gesture change data. The state of the virtual hand model (virtual hand data) is updated using the filtered gesture change data, so that it more accurately reflects the actual hand action of the user. The Kalman filtering is a prior art, which will not be described in detail here.

[0071] Specifically, in step S104, the gesture change generated by the user is obtained, and is converted into initial gesture change data corresponding to the virtual hand data according to a preset linear interpolation formula, including:

[0072] Record the gesture change image generated by the user;

[0073] Convert the gesture change image into a corresponding plurality of rendered images;

[0074] Based on the plurality of rendered images, the initial gesture change data generated by the virtual hand data is calculated by using a preset linear interpolation formula.

[0075] In step S104, initial gesture change data is converted by capturing a sequence of continuous images of the user's hand movement, i.e., gesture change images. The hand is separated from the background in the gesture change images, and key feature points or skeletal positions of the hand are identified. The processing results of each frame of image are converted into a structured data representation, forming a multi-frame rendering image sequence. This sequence reflects the dynamic changes of the user's hand over a period of time. Then the multi-frame sequence is analyzed, and a preset linear interpolation algorithm is applied. The algorithm calculates a smooth and continuous virtual hand initial change quantity based on the motion trajectory and posture change information provided by the multi-frame data. This initial change quantity represents how the virtual hand should move or change its posture to match the actual hand movement of the user. By using multi-frame data for calculation, the influence of noise or instantaneous errors that may be brought by single-frame images can be effectively reduced, thereby improving the accuracy and stability of the initial change data. Thus, a more reliable basis is provided for subsequent filtering and updating of virtual hand data, ensuring that the virtual hand can more accurately and smoothly follow the actual hand movement of the user, and improving the immersion and effectiveness of the interaction.

[0076] wherein the preset linear interpolation formula is specifically:

[0077] v = a + (b - a) * t;

[0078] wherein v is the displacement or rotation of the target bone in the next frame of image, i.e., the pose of the target bone in the next frame of image; b is the target displacement or target rotation of the target bone, i.e., the pose of the target bone in the target position; a is the displacement or rotation of the target bone in the current frame of image, i.e., the pose of the target bone in the current frame of image; and t is the interval between each frame of image, which is usually a small decimal number.

[0079] Specifically, in step S104, the gesture change images are converted into corresponding multi-frame rendering images, including:

[0080] converting each frame of gesture change image into a preliminary rendering image;

[0081] judging whether the hand data in each frame of preliminary rendering image is complete; if yes, then synthesizing each frame of preliminary rendering image to obtain the multi-frame rendering image; if not, then supplementing the preliminary rendering image lacking hand data to obtain a complete preliminary rendering image, and synthesizing the complete preliminary rendering image to obtain the multi-frame rendering image.

[0082] In step S104, the gesture change image generated by the user at each time is acquired and converted into a preliminary rendering image. During the conversion process, the hand data in the preliminary rendering image of some frames can be incomplete due to occlusion or other factors. Each preliminary rendering image is checked to identify the completeness of the hand data. If the hand data is complete, the preliminary rendering image is marked as available. If the hand data is incomplete, attempts are made to supplement the missing data, such as using valid data from the previous frame or the next frame for interpolation, or estimating according to a preset hand model, to generate a complete hand preliminary rendering image. All preliminary rendering images that have passed the integrity check or have been supplemented are combined into a multi-frame rendering image sequence. In this way, the accuracy and completeness of the input data when calculating the initial gesture change data based on the multi-frame rendering image are ensured, the accuracy of the virtual hand data update is improved, and the smoothness and effectiveness of the interactive teaching are improved.

[0083] Specifically, in step S104, the preliminary rendering image lacking hand data is supplemented to obtain a complete preliminary rendering image, including:

[0084] determining whether the missing hand data in the preliminary rendering image is key position data;

[0085] If not, the complete hand data of the user is used to supplement the missing hand data to obtain a complete preliminary rendering image;

[0086] If yes, the user is reminded to pay attention to the occlusion problem, and the step of issuing a virtual teaching request to the user is returned to perform.

[0087] In step S104, it is judged whether the missing hand data in the preliminary rendering image is key position data, which can be based on a preset key skeleton set. For example, the key position data can include palm center, thumb tip, index finger tip and other skeleton data that are crucial to determining the hand posture and position. The key position data can be set according to actual needs. If the missing skeleton point does not belong to the key position data, for example, the skeleton point of a non-key knuckle is missing, the complete hand skeleton data or model previously acquired by the user is used to calculate and supplement the position of the missing non-key skeleton point by interpolation or model matching, so as to generate a complete preliminary rendering image containing all skeleton points. Thus, even if part of the non-key data is missing, complete image data for subsequent processing can be obtained, and the robustness of data processing is improved. If the missing skeleton point belongs to the key position data, for example, the palm center or main finger tip skeleton point is missing, it is considered that the current acquired hand data is insufficient to accurately represent the user's hand posture, and reliable supplementation cannot be performed. At this time, the current data processing process needs to be interrupted, a prompt is sent to the user, for example, "please make sure that the hand is not blocked, and then try again", and the teaching process is rolled back to the link of guiding the user to move the hand. After receiving the prompt, the user adjusts the hand posture or position and tries again to complete the teaching guide action. Only when the complete preliminary rendering image (whether original complete or complete after non-key data supplementation) is successfully acquired, the subsequent steps will be continued, for example, multiple frames of complete preliminary rendering images are combined into multiple frames of rendering images, and gesture change data is calculated based on this. This processing mechanism ensures that only reliable hand data can be used for subsequent teaching judgment and feedback, and improves the accuracy and effectiveness of the teaching process.

[0088] Specifically, in step S104, after updating the virtual hand data according to the filtered gesture change data to obtain the moved virtual hand data, the following steps are further included:

[0089] The gesture type of the moved virtual hand data is recognized. If the gesture type is a special gesture type, the teaching process is paused, and corresponding operations are performed according to the semantics corresponding to the special gesture type.

[0090] In step S104, after obtaining the moved virtual hand data, gesture type recognition is performed on the moved virtual hand data. The recognition process can include comparing the bone posture of the virtual hand data with preset special gesture templates. If the comparison result shows that the current gesture matches any special gesture template, it is determined that the gesture type is a special gesture type, the current teaching process is paused, and new virtual teaching requests are stopped from being sent to the user or the current virtual scene element is frozen. The preset semantics associated with the identified special gesture type are queried, such as "pause", "repeat", "return", and the like. According to the queried semantics, the system performs corresponding operations, such as pausing the timing, repeating the teaching content segment, or returning to the previous teaching step. In this way, the user can control the teaching process through a specific gesture, improving the flexibility of interaction and user experience.

[0091] Specifically, in step S105, it is determined whether the moved virtual hand data obtained after each movement reaches the corresponding target position, including:

[0092] The distance between the position of each bone in the moved virtual hand data obtained after each movement and the position of the corresponding target bone in the corresponding target position is calculated.

[0093] Based on the size relationship between the distance of all bones obtained after each movement and the preset error threshold, it is determined in turn whether the moved virtual hand data obtained after each movement reaches the corresponding target position. When all distances are less than the preset error threshold, it is determined that the moved virtual hand data reaches the corresponding target position. When there is any distance greater than or equal to the preset error threshold, it is determined that the moved virtual hand data corresponding to the distance greater than or equal to the preset error threshold has not reached the corresponding target position.

[0094] In step S105, after the user moves the hand each time and updates the virtual hand data to obtain the moved virtual hand data, first, the distance between the current three-dimensional coordinates of each skeletal joint in the moved virtual hand data obtained after each movement and the three-dimensional coordinates of the corresponding skeletal joint in the target position set by the teaching content is calculated. For example, the distance between the virtual index finger tip bone and the target index finger tip position, the distance between the virtual thumb root bone and the target thumb root position, and so on can be calculated, and the distances of all key bones are calculated. Then, the calculated distances are compared with a preset error threshold. The error threshold represents the maximum allowed position deviation. If all the calculated skeletal distances are less than the preset error threshold, it is determined that the user's hand has successfully moved to the target position, and the virtual teaching request is completed. Otherwise, as long as the distance of any skeletal joint is greater than or equal to the error threshold, it is determined that the user's hand has not accurately reached the target position. This method avoids ambiguous judgments by performing fine-grained skeletal level checks on the overall posture and position of the virtual hand and setting clear error limits, ensuring the reliability of the judgment result and thus ensuring the smooth progress of the teaching process.

[0095] When the moved virtual hand data obtained after each movement reaches the corresponding target position, it is determined that the user completes the virtual teaching request; when there is any moved virtual hand data obtained after each movement that has not reached the corresponding target position, the user is reminded to move the hand to the initial position and then move the hand to the multiple target positions again until the moved virtual hand data obtained after each movement reaches the corresponding target position.

[0096] Specifically, after it is determined that the user completes the virtual teaching request in step S105, the method further includes:

[0097] The acquisition of the gesture change of the user is stopped until the next virtual teaching request is sent, so as to avoid acquiring incorrect information.

[0098] In step S105, after it is determined that the user completes the virtual teaching request, a stop acquisition signal is triggered to stop receiving or processing the gesture change images recorded by tracking the hand of the user. This stop state is maintained until a new virtual teaching request is sent to the user according to the teaching process. When the new virtual teaching request is sent, a start acquisition signal is activated to start tracking the hand of the user and recording the gesture change images. Thus, the gesture data is actively acquired and utilized only when the user is required to perform interactive guidance, and the data generated by non-guided or random hand movements after the completion of the teaching content is not mistakenly treated as teaching interactive data for processing, thereby ensuring the accuracy of the data processed by the system.

[0099] According to the above, the interactive teaching method based on the MR technology obtains hand data of a user, generates virtual hand data corresponding to the hand data, divides teaching content into a plurality of target positions, and sends a virtual teaching request to the user to guide the user to move the hand to the plurality of target positions in sequence, obtains gesture changes generated by the user, converts the gesture changes into gesture change data corresponding to the virtual hand data according to a preset linear interpolation formula to obtain moved virtual hand data, and determines whether the moved virtual hand data obtained after each movement reaches the corresponding target position. If yes, it is determined that the user completes the virtual teaching request. If no, the user is reminded to move the hand to the initial position and then move the hand to the plurality of target positions again until the moved virtual hand data obtained after each movement reaches the corresponding target position. Thus, the moved virtual hand data is calculated according to the preset linear interpolation formula and the gesture changes generated by the user to determine whether the moved virtual hand data obtained after each movement reaches the corresponding target position and determine whether the user completes the virtual teaching request, thereby solving the problem of low teaching efficiency and learning experience caused by insufficient guidance in the existing MR technology teaching method, guiding the user to complete the teaching content through a preset path and action, reducing confusion and errors caused by operation uncertainty, and improving the learning efficiency of virtual teaching.

[0100] Reference Figure 2 The application provides an interactive teaching system based on MR technology for guiding a user to complete teaching content, comprising:

[0101] An acquisition module 1 is configured to acquire hand data of a user.

[0102] A generation module 2 is configured to generate virtual hand data corresponding to the hand data.

[0103] A guidance module 3 is configured to divide teaching content into a plurality of target positions and send a virtual teaching request to the user to guide the user to move the hand to the plurality of target positions in sequence.

[0104] A conversion module 4 is configured to obtain gesture changes generated by the user and convert the gesture changes into gesture change data corresponding to the virtual hand data according to a preset linear interpolation formula to obtain moved virtual hand data.

[0105] A determination module 5 is configured to determine whether the moved virtual hand data obtained after each movement reaches the corresponding target position. If yes, it is determined that the user completes the virtual teaching request. If no, the user is reminded to move the hand to the initial position and then move the hand to the plurality of target positions again until the moved virtual hand data obtained after each movement reaches the corresponding target position.

[0106] The interactive teaching system based on the MR technology solves the problem of insufficient guidance in the existing MR technology teaching method, which leads to the problem of decline in teaching efficiency and learning experience, guides the user to complete the teaching content through the preset path and action, reduces the confusion and errors caused by operation uncertainty, and improves the learning efficiency of virtual teaching. The interactive teaching system based on the MR technology solves the problem of insufficient guidance in the existing MR technology teaching method, which leads to the problem of decline in teaching efficiency and learning experience, guides the user to complete the teaching content through the preset path and action, reduces the confusion and errors caused by operation uncertainty, and improves the learning efficiency of virtual teaching.

[0107] Specifically, the acquisition module 1, when executed, acquires hand data of the user, wherein the hand data includes a bone identification number of each bone of the hand and a position where the corresponding bone is located, and the hand data can be acquired by a sensor such as a depth camera or an inertial measurement unit, for capturing real-time position and posture information of the user's hand.

[0108] Specifically, the generation module 2, when generating virtual hand data corresponding to the hand data, performs:

[0109] Based on the identification number of each bone of the hand in the hand data, an initial virtual hand data is generated in combination with a preset virtual hand template;

[0110] According to the position of each bone of the hand in the hand data, the posture of the initial virtual hand data is adjusted to obtain virtual hand data corresponding to the hand data.

[0111] The generation module 2, when executed, uses the identification number of each bone in the hand data in combination with the preset virtual hand template to construct an initial virtual hand having a basic structure and a bone correspondence relationship. This step provides a basic framework for the virtual hand, ensuring that the virtual model matches the structure of the real hand. According to the actual position information of each bone in the hand data, the posture of the initial virtual hand model is adjusted. This adjustment process enables the virtual hand to accurately mimic the current posture and shape of the user's real hand. By combining the template structure and real-time position information, a virtual hand data is generated that is both structurally correct and accurately reflects the current state of the user's hand, providing a virtual representation of the user's hand.

[0112] Specifically, the guidance module 3, when executed, decomposes the teaching content into a series of target positions that the hand needs to reach in sequence, and issues a virtual teaching request to guide the user to move the hand to the target positions in sequence, thereby providing the user with clear, step-by-step operation guidance.

[0113] Specifically, the conversion module 4, when acquiring the gesture change generated by the user to convert the gesture change data corresponding to the virtual hand data into virtual hand data according to the preset linear interpolation formula, performs:

[0114] obtain gesture change data corresponding to the virtual hand data according to a preset linear interpolation formula;

[0115] predict and correct the initial gesture change data by using a Kalman filtering algorithm to obtain filtered gesture change data;

[0116] update the virtual hand data according to the filtered gesture change data to obtain moved virtual hand data.

[0117] When the conversion module 4 is executed, after obtaining the gesture change of the user, the gesture change of the user is initially mapped to the change amount of the virtual hand data by linear interpolation to obtain initial gesture change data. This initial mapping may contain sensor noise or inaccuracy. In order to improve the smoothness and accuracy of the virtual hand action, a Kalman filtering algorithm is introduced to process the initial gesture change data. Kalman filtering optimally estimates the state by combining the dynamic model of the system and the measurement data, thereby effectively filtering out noise and obtaining more stable and accurate gesture change data. The state of the virtual hand model (virtual hand data) is updated using the filtered gesture change data, so that it more accurately reflects the actual hand action of the user. Among them, Kalman filtering is prior art, which will not be described in detail here.

[0118] Specifically, when the conversion module 4 obtains the gesture change generated by the user to convert into initial gesture change data corresponding to the virtual hand data according to a preset linear interpolation formula, it performs:

[0119] record the gesture change image generated by the user;

[0120] convert the gesture change image into corresponding multiple frames of rendering images;

[0121] based on the multiple frames of rendering images, use a preset linear interpolation formula to calculate the initial gesture change data generated by the virtual hand data.

[0122] The conversion module 4, when executed, converts to obtain initial gesture change data by capturing a sequence of continuous images of user hand actions, i.e. gesture change images. The hand is separated from the background in the gesture change images, and key feature points or skeletal positions of the hand are identified. The processing result of each frame of image is converted into a structured data representation, forming a multi-frame rendering image sequence. This sequence reflects the dynamic changes of the user's hand over a period of time. Then the multi-frame sequence is analyzed, and a preset linear interpolation algorithm is applied. The algorithm calculates a smooth and continuous virtual hand initial change quantity according to the motion trajectory and posture change information provided by the multi-frame data. This initial change quantity represents how the virtual hand should move or change its posture to match the actual hand action of the user. By using multi-frame data for calculation, the influence of noise or instantaneous error that may be brought by single-frame image can be effectively reduced, thereby improving the accuracy and stability of the initial change data. Thus, a more reliable basis is provided for subsequent filtering and updating of virtual hand data, ensuring that the virtual hand can more accurately and smoothly follow the actual hand action of the user, and improving the immersion and effectiveness of the interaction.

[0123] wherein the preset linear interpolation formula is specifically:

[0124] v = a + (b - a) * t;

[0125] wherein v is the displacement or rotation of the target bone in the next frame of image, i.e. the pose of the target bone in the next frame of image; b is the target displacement or target rotation of the target bone, i.e. the pose of the target bone in the target position; a is the displacement or rotation of the target bone in the current frame of image, i.e. the pose of the target bone in the current frame of image; t is the interval between each frame of image, which is usually a small decimal number.

[0126] Specifically, the conversion module 4, when converting the gesture change images into corresponding multi-frame rendering images, performs:

[0127] converts each frame of gesture change image into a preliminary rendering image;

[0128] judges whether the hand data in each frame of preliminary rendering image is complete; if yes, the multi-frame rendering images are synthesized by using each frame of preliminary rendering image; if not, the preliminary rendering images lacking hand data are supplemented to obtain complete preliminary rendering images, and the multi-frame rendering images are synthesized by using the complete preliminary rendering images.

[0129] The conversion module 4, when executed, acquires the gesture change image generated by the user at each time point and converts it into a preliminary rendering image. During the conversion process, the hand data in the preliminary rendering image of some frames can be incomplete due to occlusion or other factors. Each preliminary rendering image is checked to identify the completeness of the hand data. If the hand data is complete, the preliminary rendering image is marked as available. If the hand data is incomplete, attempts are made to supplement the missing data, such as using valid data from the previous frame or the next frame for interpolation, or estimating based on a pre-set hand model, to generate a complete hand preliminary rendering image. All preliminary rendering images that have passed the integrity check or have been supplemented are combined into a multi-frame rendering image sequence. In this way, the accuracy and completeness of the input data when calculating the initial gesture change data based on the multi-frame rendering image is ensured, the accuracy of the virtual hand data update is improved, and the smoothness and effectiveness of the interactive teaching is improved.

[0130] Specifically, when the conversion module 4 supplements the preliminary rendering image that lacks hand data to obtain a complete preliminary rendering image, it performs:

[0131] determining whether the missing hand data in the preliminary rendering image is key position data;

[0132] If not, the complete hand data of the user is used to supplement the missing hand data to obtain a complete preliminary rendering image;

[0133] If yes, the user is reminded to pay attention to the occlusion problem, and the step of issuing a virtual teaching request to the user is returned to perform.

[0134] The conversion module 4 determines whether the missing hand data in the preliminary rendering image is key position data when executing. The determination can be based on a preset key skeleton set. For example, the key position data can include palm center, thumb tip, index finger tip, and other skeleton data that is crucial to determining the hand posture and position. The key position data can be set according to actual needs. If the missing skeleton point does not belong to the key position data, for example, the skeleton point of a non-key knuckle is missing, the complete hand skeleton data or model previously obtained by the user is used to calculate and supplement the position of the missing non-key skeleton point by interpolation or model matching, thereby generating a complete preliminary rendering image containing all skeleton points. Thus, even if part of the non-key data is missing, complete image data for subsequent processing can be obtained, and the robustness of data processing is improved. If the missing skeleton point belongs to the key position data, for example, the palm center or main finger tip skeleton point is missing, it is considered that the current hand data is insufficient to accurately represent the user's hand posture, and reliable supplementation cannot be performed. At this time, the current data processing process needs to be interrupted, a prompt is sent to the user, for example, “please make sure that the hand is not blocked, and then try again”, and the teaching process is rolled back to the step of guiding the user to move the hand. After receiving the prompt, the user adjusts the hand posture or position and tries again to complete the teaching guide action. Only when the complete preliminary rendering image (whether original complete or complete after non-key data supplementation) is successfully obtained, the subsequent steps, such as combining multiple frames of complete preliminary rendering images into multiple frames of rendering images and calculating gesture change data based on the rendering images, are continued. This processing mechanism ensures that only reliable hand data can be used for subsequent teaching judgment and feedback, and improves the accuracy and effectiveness of the teaching process.

[0135] Specifically, after the conversion module 4 updates the virtual hand data according to the filtered gesture change data to obtain the moved virtual hand data, the conversion module 4 further performs:

[0136] recognizes the gesture type of the moved virtual hand data. If the gesture type is a special gesture type, the teaching process is paused, and a corresponding operation is performed according to the semantics corresponding to the special gesture type.

[0137] The conversion module 4, when executed, after obtaining the moved virtual hand data, performs gesture type recognition on the moved virtual hand data. The recognition process can include comparing the bone posture of the virtual hand data with a preset special gesture template. If the comparison result shows that the current gesture matches any special gesture template, it is determined that the gesture type is a special gesture type, the current teaching process is paused, and new virtual teaching requests are stopped from being sent to the user or the current virtual scene element is frozen. The preset semantics associated with the identified special gesture type are queried, such as "pause", "repeat", "return", etc. According to the queried semantics, the system performs corresponding operations, such as pausing the timing, repeating the teaching content segment, or returning to the previous teaching step. In this way, the user can control the teaching process through a specific gesture, improving the flexibility of interaction and user experience.

[0138] Specifically, the judgment module 5, when judging whether the moved virtual hand data obtained after each movement reaches the corresponding target position, performs:

[0139] The distance between the position of each bone in the moved virtual hand data obtained after each movement and the corresponding target bone position in the corresponding target position is calculated.

[0140] Based on the size relationship between the distance of each bone after each movement and the preset error threshold, it is sequentially judged whether the moved virtual hand data obtained after each movement reaches the corresponding target position. When all distances are less than the preset error threshold, it is determined that the moved virtual hand data reaches the corresponding target position. When there is any distance greater than or equal to the preset error threshold, it is determined that the moved virtual hand data corresponding to the distance greater than or equal to the preset error threshold has not reached the corresponding target position.

[0141] The judgment module 5, when executed, first calculates the distance between the current three-dimensional coordinates of each skeletal joint in the moved virtual hand data obtained after each movement of the user and the three-dimensional coordinates of the corresponding skeletal joint in the target position set by the teaching content after the user moves the hand each time and updates the virtual hand data. For example, the distance between the virtual index finger tip bone and the target index finger tip position, the distance between the virtual thumb root bone and the target thumb root position, and so on can be calculated, and the distances of all key bones are calculated. Then, the calculated distances are compared with a preset error threshold. The error threshold represents the maximum allowed position deviation. If all calculated skeletal distances are less than the preset error threshold, it is determined that the user's hand has successfully moved to the target position, and the virtual teaching request is completed. Otherwise, as long as the distance of any skeletal joint is greater than or equal to the error threshold, it is determined that the user's hand has not accurately reached the target position. This method avoids ambiguous judgments by performing fine-grained skeletal-level checks on the overall posture and position of the virtual hand and setting clear error limits, ensuring the reliability of the judgment result and thus ensuring the smooth progress of the teaching process.

[0142] When the moved virtual hand data obtained after each movement reaches the corresponding target position, it is determined that the user completes the virtual teaching request; when any moved virtual hand data obtained after each movement has not reached the corresponding target position, the user is reminded to move the hand to the initial position and then move the hand to the multiple target positions until the moved virtual hand data obtained after each movement reaches the corresponding target position.

[0143] Specifically, after determining that the user completes the virtual teaching request, the judgment module 5 further performs:

[0144] Stop acquiring the gesture changes of the user until the next virtual teaching request is sent, so as to avoid acquiring incorrect information.

[0145] When executed, the judgment module 5, after determining that the user completes the virtual teaching request, triggers a stop acquiring information signal to stop receiving or processing the gesture change images recorded by tracking the user's hand. This stop state is maintained until a new virtual teaching request is sent to the user according to the teaching process. When a new virtual teaching request is sent, the start acquiring information signal is activated to start tracking the user's hand and recording the gesture change images. Thus, gesture data is actively acquired and utilized only when the user is required to interact with the guide, and data generated by non-guided or random hand movements after the completion of the teaching content will not be mistakenly processed as teaching interaction data, thereby ensuring the accuracy of the system processing data.

[0146] From the above, the interactive teaching system based on the MR technology obtains hand data of a user, generates virtual hand data corresponding to the hand data, divides teaching content into a plurality of target positions, and sends a virtual teaching request to the user to guide the user to move the hand to the plurality of target positions in sequence, obtains gesture changes generated by the user, converts the gesture changes into gesture change data corresponding to the virtual hand data according to a preset linear interpolation formula, obtains the virtual hand data after movement, and judges whether the virtual hand data after movement obtained each time after movement reaches the corresponding target position. If yes, it is determined that the user completes the virtual teaching request. If no, the user is reminded to move the hand to the initial position and then move the hand to the plurality of target positions again until the virtual hand data after movement obtained each time after movement reaches the corresponding target position. Thus, the virtual hand data after movement is calculated through the preset linear interpolation formula and the gesture changes generated by the user to judge whether the virtual hand data after movement obtained each time after movement reaches the corresponding target position, determine whether the user completes the virtual teaching request, solve the problem that the existing MR technology teaching method has insufficient guidance, leading to a decline in teaching efficiency and learning experience, guide the user to complete the teaching content through the preset path and action, reduce confusion and errors caused by operation uncertainty, and improve the learning efficiency of virtual teaching.

[0147] In the embodiments provided in the present application, it should be understood that the disclosed system and method can be implemented in other manners. The embodiment described above is only schematic. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0148] In addition, the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units. They can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments.

[0149] In addition, each functional module in the embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0150] In this document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.

[0151] The above description is merely illustrative of the application and not in limitation of the principles of the application. Numerous modifications and adaptations thereof will be readily apparent to those skilled in the art without departing from the spirit and scope of the application as defined in the following claims.

Claims

1. An interactive teaching method based on MR technology for guiding a user through a teaching content, characterized in that, The method comprises the steps of: obtaining hand data of a user; generating virtual hand data corresponding to the hand data; dividing teaching content into a plurality of target positions and issuing a virtual teaching request to the user to guide the user to move the hand to the plurality of target positions in sequence; obtaining gesture changes generated by the user to convert into gesture change data corresponding to the virtual hand data according to a preset linear interpolation formula to obtain moved virtual hand data; determining whether the moved virtual hand data obtained after each movement reaches the corresponding target position; if yes, it is determined that the user completes the virtual teaching request; if no, the user is reminded to move the hand to the initial position and then move the hand to the plurality of target positions again until the moved virtual hand data obtained after each movement reaches the corresponding target position; obtaining gesture changes generated by the user to convert into gesture change data corresponding to the virtual hand data according to a preset linear interpolation formula to obtain moved virtual hand data, comprising: obtaining gesture changes generated by the user to convert into initial gesture change data corresponding to the virtual hand data according to a preset linear interpolation formula; using a Kalman filtering algorithm to predict and correct the initial gesture change data to obtain filtered gesture change data; updating the virtual hand data according to the filtered gesture change data to obtain moved virtual hand data; obtaining gesture changes generated by the user to convert into initial gesture change data corresponding to the virtual hand data according to a preset linear interpolation formula, comprising: recording gesture change images generated by the user; converting the gesture change images into a plurality of rendered images corresponding thereto; based on the plurality of rendered images, using a preset linear interpolation formula to calculate initial gesture change data generated by the virtual hand data; converting the gesture change images into a plurality of rendered images corresponding thereto, comprising: converting each frame of the gesture change images into a preliminary rendered image; determining whether the hand data in each frame of the preliminary rendered image is complete; if yes, the plurality of rendered images are synthesized from each frame of the preliminary rendered image; if no, the preliminary rendered image lacking hand data is supplemented to obtain a complete preliminary rendered image, and the complete preliminary rendered image is synthesized to obtain the plurality of rendered images; supplementing the preliminary rendered image lacking hand data to obtain a complete preliminary rendered image, comprising: determining whether the hand data lacking in the preliminary rendered image is key position data; if no, the complete hand data of the user is used to supplement the hand data lacking to obtain a complete preliminary rendered image; if yes, the user is reminded to pay attention to the occlusion problem, and the step of issuing a virtual teaching request to the user is executed again.

2. The MR technology based interactive teaching method as claimed in claim 1, wherein, The hand data comprises a bone identification number of each bone of the hand and a position corresponding to the bone; generating virtual hand data corresponding to the hand data, comprising: Generate initial virtual hand data based on the identification number of each bone of the hand in the hand data, in combination with a preset virtual hand template; Adjust the posture of the initial virtual hand data according to the position of each bone of the hand in the hand data, to obtain virtual hand data corresponding to the hand data.

3. The MR technology based interactive teaching method as claimed in claim 1, wherein, After updating the virtual hand data according to the filtered gesture change data to obtain the moved virtual hand data, the method further includes: Identify the gesture type of the moved virtual hand data, and if the gesture type is a special gesture type, pause the teaching process and perform corresponding operations according to the semantics corresponding to the special gesture type.

4. The MR technology based interactive teaching method as claimed in claim 1, wherein, Determine whether the moved virtual hand data obtained after each movement reaches the corresponding target position, including: Calculate the distance between the position of each bone in the moved virtual hand data obtained after each movement and the position of the corresponding target bone in the corresponding target position; Based on the size relationship between each distance and a preset error threshold, sequentially determine whether the moved virtual hand data obtained after each movement reaches the corresponding target position; when all the distances are less than the preset error threshold, it is determined that the moved virtual hand data reaches the corresponding target position; when there is any distance greater than or equal to the preset error threshold, it is determined that the distance corresponding to the moved virtual hand data greater than or equal to the preset error threshold has not reached the corresponding target position.

5. The MR technology based interactive teaching method as claimed in claim 1, wherein, After determining that the user completes the virtual teaching request, the method further includes: Stop obtaining the gesture change of the user until the next virtual teaching request is sent, to avoid obtaining incorrect information.

6. An interactive teaching system based on MR technology for guiding a user through a teaching content, characterized in that, Including: An acquisition module for acquiring hand data of a user; A generation module for generating virtual hand data corresponding to the hand data; A guidance module for dividing teaching content into a plurality of target positions and issuing a virtual teaching request to the user to guide the user to move the hand to the plurality of target positions in sequence; A conversion module for obtaining gesture changes generated by the user, to convert the gesture changes into gesture change data corresponding to the virtual hand data according to a preset linear interpolation formula, to obtain moved virtual hand data; A judgment module for determining whether the moved virtual hand data obtained after each movement reaches the corresponding target position; if so, it is determined that the user completes the virtual teaching request; if not, the user is reminded to move the hand to an initial position and then move the hand to the plurality of target positions again until the moved virtual hand data obtained after each movement reaches the corresponding target position; The conversion module for obtaining gesture changes generated by the user, to convert the gesture changes into gesture change data corresponding to the virtual hand data according to a preset linear interpolation formula, to obtain moved virtual hand data, including: Obtain gesture changes generated by the user, to convert the gesture changes into initial gesture change data corresponding to the virtual hand data according to a preset linear interpolation formula; The initial gesture change data is predicted and corrected by using a Kalman filtering algorithm to obtain filtered gesture change data; The virtual hand data is updated according to the filtered gesture change data to obtain moved virtual hand data; The gesture change generated by the user is obtained, and the initial gesture change data corresponding to the virtual hand data is converted according to a preset linear interpolation formula, including: An image of the gesture change generated by the user is recorded; The image of the gesture change is converted into corresponding multiple frames of rendering images; The initial gesture change data generated by the virtual hand data is calculated by using a preset linear interpolation formula based on the multiple frames of rendering images; Converting the image of the gesture change into corresponding multiple frames of rendering images includes: The image of the gesture change of each frame is converted into a preliminary rendering image; It is judged whether the hand data in each frame of the preliminary rendering image is complete; if yes, multiple frames of rendering images are synthesized from each frame of the preliminary rendering image; if not, the preliminary rendering image lacking hand data is supplemented to obtain a complete preliminary rendering image, and multiple frames of rendering images are synthesized from the complete preliminary rendering image; Supplementing the preliminary rendering image lacking hand data to obtain a complete preliminary rendering image includes: It is judged whether the hand data lacking in the preliminary rendering image is key part data; If not, the complete hand data of the user is used to supplement the hand data lacking to obtain a complete preliminary rendering image; If yes, the user is reminded to pay attention to the occlusion problem, and the step of issuing a virtual teaching request to the user is returned to be executed.

Images