Myopia intervention treatment-oriented AR system for massage skill training

Abstract

The invention belongs to the technical field of intelligent medical systems, and relates to a myopia intervention treatment-oriented massage skill training AR system, which comprises an AR training scene construction module, a massage skill training operation table, a model algorithm service module and an interactive feedback module. According to the invention, the AR technology and massage skill training are combined, through the visual guide path in the AR scene, the trainee can more intuitively understand the position of the massage acupoint and the massage sequence, and the problem that the acupoint is difficult to position in the traditional training is effectively solved. Gestures are recognized and fed back in real time through the gesture recognition technology, students can correct wrong manipulations in time, the degree of mastering the manipulations is improved, and compared with traditional abstract manipulation teaching, the training effect is remarkably improved. The system can score and feed back the operation of the student in real time, generates a training result report, facilitates the self-assessment of the student and the teaching management of a trainer, and improves the training efficiency and quality.

Description

Technical Field

[0001] The present invention relates to the technical field of smart medical systems, to augmented reality and machine vision technologies, and in particular to an AR system for myopia intervention treatment and massage technique training. Background Art

[0002] With the increasing prevalence of electronic products and the pressure of studying, the incidence of myopia has been increasing year by year, with a trend toward younger age groups. Massage, a traditional Chinese medicine treatment, stimulates acupuncture points around the eyes and regulates the flow of qi and blood in the eye meridians, making it effective in the treatment of myopia. However, traditional massage technique training methods have many problems: First, acupuncture point location relies primarily on verbal explanations and demonstrations by the trainer, making it difficult for trainees to accurately grasp the specific locations of the acupuncture points, resulting in poor massage results. Second, the teaching of massage techniques is relatively abstract, making it difficult for trainees to understand and imitate the correct techniques, resulting in low training efficiency. With the development of augmented reality (AR) technology, its application in training can provide a more intuitive and interactive learning experience. However, there is currently no AR system specifically designed for training massage techniques for myopia intervention.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0004] The purpose of the present invention is to solve the technical problems of inaccurate acupoint positioning and abstract teaching techniques in traditional training methods in the background technology. To this end, an AR system for training massage techniques for myopia intervention treatment is provided to improve the training efficiency and quality of massage techniques for myopia intervention treatment.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] An AR system for massage technique training for myopia intervention treatment, comprising: an AR training scenario construction module, a massage technique training console, a model algorithm service module, and an interactive feedback module;

[0007] AR training scene construction module, responsible for massage scene modeling, visual guidance information design, 3D registration, model rendering, interactive script development, and multimodal interaction design;

[0008] The massage technique training console uses the Hololens 2 device to conduct virtual and real massage technique training, and transmits real-time images to the model algorithm service module, which then conducts corresponding training movement corrections based on the interactive feedback visual information guidance;

[0009] The model algorithm service module is responsible for receiving data from the massage skills training console, launching the machine vision algorithm to process the received frames, and identifying the user's gestures and massage acupoints;

[0010] The interactive feedback module is responsible for comparing and mapping the visual information guidance library with the logical rules, and ultimately generating a visual guidance path.

[0011] The following is a technical solution further defined by the present invention: in the AR training scene construction module, augmented reality technology is used to register the virtual massage scene into the real massage environment; in the model algorithm service module, machine vision technology is used to process the images captured by Hololens2 in real time, identify gestures and acupoints, judge the status of the current operation step based on the current image, and provide guidance for the current operation step. If the previous step is correct, the corresponding response of the current operation is given through animation technology and three-dimensional registration technology. If the current step is incorrect, the operation guidance of the current action is given again until all steps are completed. Afterwards, a comprehensive analysis of this training is given.

[0012] The following is a technical solution further defined by the present invention. In the model algorithm service module, for gesture recognition, MediaPipe is used to extract hand key points, the Hololens2 sensor obtains hand bone depth information, and the ST-GCN algorithm is used to model the motion trajectory of the hand key points to identify user operation gestures; for acupoint recognition, a coordinate system is established based on facial feature points, the relative spatial coordinates of the eye acupoints are calculated, and the SLAM algorithm is used to realize dynamic tracking of AR markers.

[0013] The following is a technical solution further defined by the present invention, which analyzes the massage training operation process, determines the objects that need to be virtualized, then models the equipment required for training, renders the training equipment, and sets lighting, mapping, materials and textures to make it conform to the real scene, thereby achieving immersive interaction that integrates virtual and real.

[0014] The following is a technical solution further defined by the present invention: the three-dimensional registration of the massage scene is based on the direct method SLAM, which integrates the direct method with the SLAM features, and uses the spatial anchor technology in the Microsoft Mixed Reality Toolkit to align the virtual objects with the real environment.

[0015] The following is a technical solution further defined by the present invention. The system includes a server, a client, and a development end. The server uses the TCP / IP protocol to connect and transmit data. The client transmits the captured images to the server every two frames, with a size of 640x640. After receiving the images, the server starts a deep learning algorithm for processing and transmits the processed data to the client. The client restores the three-dimensional coordinates of the massage gesture based on the received two-dimensional coordinates and displays them in the form of holographic projection.

[0016] The following is a technical solution further defined by the present invention, which uses 3D Text and gaze interaction scripts in MRTK, combined with machine vision technology, to achieve online guidance and error warnings in the training process through logic and image data annotation methods. For different training processes, the visual path guidance design of specific training is completed. Simple training techniques are implemented using image data annotation methods, while complex training techniques are implemented based on gestures, acupoint locations, and specific training logic. In the massage technique training step, when the difference between the gesture and the acupoint position exceeds a threshold, an error warning is automatically displayed, and correct operation guidance is given.

[0017] The following is a technical solution further defined by the present invention. Before training the deep learning network, the captured data set is subjected to data enhancement methods, and the data set images are translated, flipped, randomly cropped, noise-perturbed, and scaled, and then placed in the deep learning network for training.

[0018] The following is a technical solution further defined by the present invention: During the training of the gesture recognition model, the K-means algorithm is used to re-cluster the anchor box size of the dataset, and the anchor is improved. The activation function is set to: leaky ReLU, the loss adopts cross-entropy loss, and necessary parameters are added. The learning rate is adjusted using the epoch-decay strategy, which decreases with the iteration cycle.

[0019] The following is a technical solution further defined by the present invention: the deep learning model after training is converted into the onnx intermediate format, the Hololens2 application is developed, the onnx model and the Hololens2 main camera are called and loaded, and the application with the target recognition model is deployed on the Hololens2.

[0020] Compared with the prior art, the present invention has the following technical effects:

[0021] This invention combines AR technology with massage technique training. Through the visual guidance path in the AR scene, students can more intuitively understand the location and order of massage points, effectively solving the problem of difficult acupoint positioning in traditional training. By using gesture recognition technology to identify and provide feedback on gestures in real time, students can promptly correct incorrect techniques and improve their mastery of the techniques. Compared with traditional abstract technique teaching, the training effect is significantly improved. The system can provide real-time scoring and feedback on students' operations and generate training results reports, facilitating student self-assessment and trainer teaching management, thereby improving the efficiency and quality of training.

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 It is a system architecture diagram of the present invention;

[0025] Figure 2 It is a system flow chart of the present invention;

[0026] Figure 3 This is a massage technique guidance flow chart based on the system of the present invention. DETAILED DESCRIPTION

[0027] To make the above-mentioned objects, features, and advantages of the present invention more readily apparent, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. The following description sets forth numerous specific details to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0028] like Figure 1 As shown, an AR system for myopia intervention treatment and massage technique training is provided, which mainly includes the following modules:

[0029] The AR training scene construction module is responsible for massage scene modeling, visual guidance information design, 3D registration, model rendering, interactive script development, and multimodal interaction design. Specifically: the modeling software is responsible for modeling massage gestures, and Unity is responsible for setting up the massage scene. The MRTK toolkit provided by Microsoft is used to implement massage gesture interactions. The script is developed in C++ / C# to implement interactive functions such as grabbing, rotating, and scaling of massage gestures; particle systems and fluid simulation plug-ins are used to implement manual guidance animation simulation, and the model is rendered using textures, lights, materials, and other methods to make the model more realistic; guidance animation is produced based on the characteristics of the operation. For online guidance of training operations, the gaze script and 3D Text provided by MRTK are used to display the operation guidance text, combined with machine vision technology to achieve the massage sequence and manual guidance of training operations.

[0030] The massage technique training console uses the Hololens 2 device to conduct virtual and real-world massage technique training, transmitting real-time images to the model algorithm service module. Based on the interactive visual feedback, the module conducts corresponding training movement corrections. Specifically, the Hololens 2-based training operation is responsible for collecting information and sending it to the server side of the model algorithm service module. The server side processes and receives and displays the information on the screen. It also handles voice recognition, gesture recognition, spatial mapping, accessing virtual massage gestures, and interacting with them.

[0031] The model algorithm service module is responsible for receiving data from the massage technique training console, initiating machine vision algorithms to process the received frames, and identifying user gestures and acupressure points. Specifically, on the server side of the model algorithm service module, the module primarily builds a virtual environment and deep learning framework, then trains the deep learning model using a custom dataset. Finally, the trained model is converted to the ONNX intermediate format. A HoloLens 2 application is created in Unity, the ONNX model is imported into the application, and a script is developed to call the ONNX model for target recognition. The application is then deployed to the client, connecting to the server via TCP / IP. During system runtime, the model algorithm service is responsible for receiving information frames, image recognition and classification, and data information transmission.

[0032] The interactive feedback module is responsible for comparing and mapping the visual information guidance library with the logical rules, ultimately generating a visual guidance path. Specifically: Based on the model inference results, the user's actual operation gestures and massage sequence are compared with the preset techniques and sequence, and interactive feedback information is generated in real time in the AR scene to guide and correct the user's operation. For example, if the student's gestures do not meet the requirements, the correct gesture diagram will be displayed in the AR scene; if the massage sequence is incorrect, the current acupoints to be massaged and the correct order will be highlighted.

[0033] The system uses augmented reality technology to register virtual massage gestures within a real-world massage technique training environment. Machine vision technology processes images captured by the Hololens2 in real time, intelligently identifying the current massage gesture category and training steps based on the current image, and providing guidance for each step. Visualized training path guidance is designed for specific training processes. Simple training techniques are implemented using image data annotation, while complex ones are implemented based on specific gestures, acupoint locations, and specific training logic. If the difference between the gesture and acupoint location exceeds a certain threshold during a massage step, an error warning is automatically displayed, providing corrective action instructions.

[0034] For gesture recognition, MediaPipe extracts hand key points, the Hololens2 sensor acquires hand skeletal depth information, and the ST-GCN algorithm is used to model the motion trajectory of these key points to identify user gestures. The AR device's camera captures images of the student's gestures, while a depth camera acquires three-dimensional spatial information of the student's hand. The depth camera captures hand depth data, providing more accurate spatial information for gesture recognition and facilitating subsequent precise analysis of the student's gestures.

[0035] A gesture recognition algorithm is used to identify the student's gestures based on gesture images and three-dimensional spatial information. This algorithm analyzes and determines the shape, angle, and trajectory of the gestures to identify different massage techniques, such as kneading, pressing, and pushing, and to determine whether the student's massage gestures meet the preset requirements.

[0036] Acupoint recognition establishes a coordinate system based on facial feature points, calculates the relative spatial coordinates of eye acupoints, and uses a SLAM algorithm to achieve dynamic tracking of AR markers. An infrared sensor array installed in the training area collects infrared signals around the human eye. Combined with an acupoint recognition algorithm, these acupoints, including Jingming, Zanzhu, Yuyao, Sizhukong, Tongziliao, and Chengqi, are located around the eye. The acupoint recognition algorithm analyzes the intensity and distribution of infrared signals and combines this with information on the anatomical locations of acupoints on the human body to accurately locate these points.

[0037] The massage training process is analyzed to identify objects that require virtualization, typically areas requiring detailed observation. The equipment required for the training is then modeled. The equipment is then rendered, and lighting, mapping, materials, and textures are configured to match the real-world scenario, achieving an immersive interaction that blends virtual and real elements.

[0038] Based on the preset massage sequence for myopia intervention treatment, a visual guidance path for acupoint massage is generated in the AR scene. This visual guidance path uses different colors and line styles to distinguish different massage stages and technique requirements, allowing students to clearly understand the massage process and technique changes.

[0039] Among them, the three-dimensional registration of massage gestures is based on the direct method SLAM, which integrates the direct method with the SLAM features, and uses the spatial anchor technology in the Microsoft Mixed Reality Toolkit to align virtual objects with the real environment.

[0040] The system uses 3D Text and gaze interaction scripts in MRTK, combined with machine vision technology, to implement online guidance and error warnings during the training process through logic and image data annotation methods. Specific visual training path guidance is designed for different training processes. Simple training techniques are implemented using image data annotation, while complex training techniques are implemented based on gestures, acupoint locations, and specific training logic. During massage technique training, if the difference between the gesture and the acupoint position exceeds a threshold, an error warning is automatically displayed, and corrective action instructions are provided. Scripts are bound to the virtual model to implement gaze and manual interaction, such as object manipulation scripts, close-range grasping scripts, and rigid simulation scripts.

[0041] In order to make the massage gesture model more realistic and the simulated massage more immersive, the massage gestures are rendered using MRTK shaders and rendering pipelines, and particle systems, fluid simulation plug-ins, and animation effects are used.

[0042] Before training the deep learning network, the captured data set is enhanced using data augmentation methods, that is, the data set images are translated, flipped, randomly cropped, noise-perturbed, and scaled, so as to make the real-time monitoring process more accurate and more adaptable to environmental changes such as light occlusion. This is achieved using imgaug, and then put into the deep learning network for training.

[0043] During model training, the K-means algorithm was used to re-cluster the dataset's anchor box sizes. Anchors were optimized, with a leaky ReLU activation function and a cross-entropy loss. The necessary parameters were added, and the learning rate was adjusted using an epoch-decay strategy, decreasing over iterations. After training, the deep learning model was converted to the onnx intermediate format. A HoloLens 2 application was developed, which loaded the onnx model and the HoloLens 2 main camera. The application, which included the object recognition model, was then deployed on the HoloLens 2.

[0044] The system consists of a server, client, and development side. The development side is responsible for modeling massage scenes, 3D registration of augmented reality, rendering of model materials, animation of massage reactions, online guidance of the massage process, fluid simulation, and interactive script development. The server side is responsible for receiving data from the client and initiating a deep learning algorithm to process the received frames, obtaining classification information of the detected objects, categories, and real-time scores of massage steps, and finally transmitting them back to the client. The client is responsible for collecting information and transmitting the collected data to the server for processing, then receiving the processed data from the server. It is also responsible for tasks such as environmental understanding, spatial perception, gesture, and voice recognition. The server and client are connected via the TCP / IP protocol, so that the server and client are on the same network. The server's IP address is added to the script to achieve connection. The server uses the TCP / IP protocol to connect and transmit data. The client transmits the captured images to the server every two frames, with a size of 640x640. After receiving the images, the server starts the deep learning algorithm for processing and transmits the processed data to the client. The client restores the three-dimensional coordinates of the massage gesture based on the received two-dimensional coordinates and displays them in the form of holographic projection.

[0045] Development side: Use Microsoft Visual Studio 2019 to write model interaction scripts and deploy applications to Hololens 2. Use SolidWorks 2021 version to model the equipment required for massage skills training. Use Unity 2018.3.7f1 version to build the massage skills training scene. Import the massage gesture model modeled in SolidWorks. Import the MRTKv2.8.8 version mixed reality toolkit provided by Microsoft, and bind the corresponding script to the model.

[0046] On the server side, we used Windows 10 SDK version 10.0.10941.0 for connecting to HoloLens 2, the YOLO Library for Darknet, and the YOLO 5 deep learning algorithm. We used an NVIDIA GTX 1660S GPU with CUDA support for inference acceleration using NVIDIA Tookit v11.6. We used OpenCV 3.4.5 for image processing.

[0047] Client: Using Microsoft's Hololens 2nd generation glasses, its built-in SLAM system can achieve modeling of unfamiliar scenes and spatial perception. In the intelligent massage technique training system, it is responsible for dynamic loading, image transmission, gesture recognition and voice recognition.

[0048] like Figure 2As shown in the figure, after the system starts, the image acquisition module begins working, acquiring the student's gesture images, 3D hand information, infrared signals around the eyes, and scene information. The gesture recognition module processes the gesture images and 3D information, using a deep learning-based gesture recognition algorithm, such as a convolutional neural network (CNN), to analyze the shape, angle, and motion trajectory of the gestures to determine whether the student's current massage technique meets the preset requirements. For example, when a student performs a kneading technique, the system determines whether the rotation angle and motion trajectory of the hand are within a reasonable range.

[0049] The acupoint recognition module analyzes the infrared signals collected by the infrared sensor array, combines them with the pre-stored infrared feature model of human acupoints and anatomical position information, and uses pattern recognition algorithms such as support vector machines (SVM) to accurately locate relevant acupoints such as Jingming acupoint and Zanzhu acupoint.

[0050] The massage technique visualization module generates a visual guidance path in the AR scene based on a preset massage sequence for myopia intervention treatment, such as starting with the Jingming acupoint, then moving on to the Zanzhu acupoint, the Yuyao acupoint, and so on. The guidance path is displayed using different colors and line styles. For example, blue lines indicate gentle massage, while red lines indicate heavier kneading.

[0051] As students perform massage, the feedback module compares their actual movements with the preset techniques and sequence in real time. If the student performs the massage in the wrong order, the acupoints to be massaged will flash in the AR scene, along with arrows indicating the correct massage sequence. If the technique is not standard, a virtual image of the correct technique will appear over the student's hand.

[0052] like Figure 3 As shown, first, the image captured by the Hololens2 main camera is obtained, and the server starts the machine vision algorithm to determine the massage technique category. The current massage steps and precautions are displayed in the form of 3D text. Then, the virtual massage gestures are accurately registered in the real massage environment. For each massage step (Step 1, Step 2...), the correctness of the massage technique is judged. If it is correct, the score of the current massage step and the massage reaction data are displayed. Otherwise, error feedback, error cause analysis and precautions for the current step are given. After the massage is completed, a comprehensive score and the reason for the loss of score are given.

[0053] This invention utilizes an improved deep learning neural network model and Microsoft's HoloLens 2 glasses to combine machine vision with augmented reality, thereby developing an AR system for massage technique training in myopia intervention treatment. This system allows users to experience massage training more immersively and gain a deeper understanding of the relevant techniques and precautions during the training process, which is of great significance for myopia intervention treatment and the cultivation of medical personnel.

[0054] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Any person skilled in the art can utilize the methods and technical contents disclosed above to make many possible variations and modifications to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, or modify them into equivalent embodiments with equivalent variations. Therefore, any equivalent variations made in accordance with the shape, structure, and principles of the present invention without departing from the content of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. An AR system for myopia intervention and treatment for massage technique training, characterized by: include: AR training scenario construction module, massage technique training operation platform, model algorithm service module and interactive feedback module; AR training scene construction module, responsible for massage scene modeling, visual guidance information design, 3D registration, model rendering, interactive script development, and multimodal interaction design; The massage technique training console uses the Hololens 2 device to conduct virtual and real massage technique training, and transmits real-time images to the model algorithm service module, which then conducts corresponding training movement corrections based on the interactive feedback visual information guidance; The model algorithm service module is responsible for receiving data from the massage skills training console, launching the machine vision algorithm to process the received frames, and identifying the user's gestures and massage acupoints; The interactive feedback module is responsible for comparing and mapping the visual information guidance library with the logical rules, and ultimately generating a visual guidance path.

2. The AR system according to claim 1, wherein: In the AR training scene construction module, augmented reality technology is used to register the virtual massage scene into the real massage environment; in the model algorithm service module, machine vision technology is used to process the images captured by Hololens2 in real time, identify gestures and acupoints, judge the status of the current operation step based on the current image, and provide guidance for the current operation step. If the previous step is correct, the corresponding response of the current operation is given through animation technology and three-dimensional registration technology. If the current step is incorrect, the operation guidance of the current action is given again until all steps are completed. Afterwards, a comprehensive analysis of this training is given.

3. The AR system according to claim 1, wherein: In the model algorithm service module, for gesture recognition, MediaPipe is used to extract hand key points, the Hololens2 sensor obtains hand bone depth information, and the ST-GCN algorithm is used to model the motion trajectory of the hand key points to identify user operation gestures; for acupoint recognition, a coordinate system is established based on facial feature points, the relative spatial coordinates of the eye acupoints are calculated, and the SLAM algorithm is used to achieve dynamic tracking of AR markers.

4. The AR system according to claim 1, wherein: Analyze the massage training operation process, determine the objects that need to be virtualized, then model and render the equipment required for training. By setting the lighting, mapping, materials and textures to make it conform to the real scene, an immersive interaction that integrates the virtual and the real is achieved.

5. The AR system according to claim 1, wherein: The 3D registration of the massage scene is based on direct SLAM, which integrates the direct method with SLAM features, and uses the spatial anchor technology in the Microsoft Mixed Reality Toolkit to align virtual objects with the real environment.

6. The AR system according to claim 1, wherein: The system includes a server, a client, and a development side. The server uses the TCP / IP protocol to connect and transmit data. The client transmits the captured images to the server every two frames, with a size of 640x640. After receiving the images, the server starts the deep learning algorithm for processing and transmits the processed data to the client. The client restores the three-dimensional coordinates of the massage gesture based on the received two-dimensional coordinates and displays them in the form of holographic projection.

7. The AR system according to claim 1, wherein: Using 3D Text and gaze interaction scripts in MRTK, combined with machine vision technology, and through logic and image data annotation methods, online guidance and error warnings for the training process are implemented. For different training processes, visual path guidance for specific training is designed. Simple training techniques are implemented using image data annotation methods, while complex training techniques are implemented based on gestures, acupoint locations, and specific training logic. During massage technique training, if the difference between the gesture and acupoint position exceeds a threshold, an error warning is automatically displayed, and correct operation instructions are given.

8. The AR system according to claim 7, wherein: Before training the deep learning network, the captured data set is augmented using data augmentation methods to translate, flip, randomly crop, perturb with noise, and transform the scale of the data set images, and then placed in the deep learning network for training.

9. The AR system according to claim 7, wherein: During the training of the gesture recognition model, the K-means algorithm is used to re-cluster the anchor box sizes of the dataset. The anchor is improved, the activation function is set to: leaky ReLU, the loss adopts cross entropy loss, and the necessary parameters are added. The learning rate is adjusted using the epoch-decay strategy, which decreases with the iteration cycle.

10. The AR system according to claim 9, wherein: After training, the deep learning model is converted to the onnx intermediate format. The Hololens 2 application is developed, the onnx model and the Hololens 2 main camera are loaded, and the application with the object recognition model is deployed on the Hololens 2.

Images