Immersive propaganda and education system and use method

Abstract

The invention belongs to the technical field of propaganda and education, and discloses an immersive propaganda and education system and a use method, and the system comprises an oral data collection device which is used for collecting intraoral three-dimensional geometric data and intraoral bone mineral density data; the MR visual presentation equipment is used for displaying an optical fusion scene of a virtual operation scene and a real environment and collecting user pose data; the tactile feedback interaction equipment is used for simulating the operation tactile sense of the dental implant surgery and collecting hand action data; the core operation and communication device constructs a three-in-one immersive interaction system by fusing MR vision, touch simulation and 3D spatial sound effects, and realizes accurate simulation and immersive experience of the whole process of an operation in combination with personalized oral data of a patient, so that preoperative anxiety of the patient is effectively relieved, and propaganda and education effects are improved.

Description

Technical Field

[0001] This invention belongs to the field of propaganda and education technology, specifically relating to an immersive propaganda and education system and its usage method. Background Technology

[0002] Dental implant patients often experience pre-operative anxiety, stemming from factors such as uncertainty about the procedure, fear of pain, unfamiliarity with the medical environment, and concerns about treatment outcomes. This anxiety not only affects the patient's experience but can also negatively impact the surgical process and post-operative recovery. Currently, preoperative education for dental implant surgery mainly relies on traditional methods such as verbal explanation, two-dimensional image display, or simple video playback, which have significant limitations in clinical application.

[0003] Traditional methods can only present static or two-dimensional information about the surgery, failing to provide a direct understanding of the surgical procedure, instrument usage, and intraoperative tissue interaction. From the perspective of oral medicine student skills training, traditional training models suffer from limited practical scenarios and a lack of tactile perception. From the perspective of public education, traditional lecture-style presentations and brochure distribution are rather dry and fail to capture the attention of the core target audience, such as the middle-aged and elderly. The dissemination efficiency and retention rate of the information are low, hindering the popularization and promotion of dental implant technology.

[0004] Chinese patent publication number CN112114663A, entitled "Implementation Method of Virtual Reality Software Framework Applicable to Visual-Haptic Fusion Feedback," describes a software framework comprising visual components, haptic components, hardware components, and a core management component. Through the coordinated operation of these components, users can select one or more different tool libraries based on the contact type between virtual objects and virtual avatars, thereby meeting the interaction needs of different contact types such as single-point contact, multi-point contact, sliding contact, and flexible body contact. However, this patent application does not consider the specific issues of oral models and educational purposes, and therefore cannot be directly applied to the education of dental implant surgery. Summary of the Invention

[0005] In order to overcome the problems existing in the prior art, the present invention aims to provide an immersive education system and its usage method, which uses MR technology to integrate virtual surgical scenes with the real environment, achieves accurate processing of patient oral data, immersive scene construction and multi-sensory interaction, and ensures the efficiency of the education process. To achieve the above objectives, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides an immersive missionary system, comprising: Oral data acquisition equipment is used to collect intraoral three-dimensional geometric data and intraoral bone density data; MR visual presentation equipment is used to display the optical fusion scene of virtual surgical scenes and real environment and to collect user pose data; Haptic feedback interactive devices are used to simulate the tactile sensations of dental implant surgery and to collect hand movement data. The core computing and communication equipment is used for system computing and multi-device data interaction; the oral data acquisition device, MR visual presentation device and haptic feedback interaction device are all communicatively connected to the core computing and communication equipment. The core computing and communication device includes a computing module, which comprises: The data processing module is used to organize the data collected by the oral data acquisition device into a three-dimensional model; The scene rendering module is used to load the 3D model output by the data processing module; at the same time, it receives pose data from the MR visual presentation device to adjust the scene perspective. The tactile simulation module is used to receive the pose data of the MR visual presentation device and the collision data between the virtual instruments and tissues between the scene rendering module, and generate electrical signals to drive the tactile feedback interaction device to output matching tactile sensations.

[0006] Optionally, the core computing and communication device further includes a user interaction module for parsing the pose data collected by the MR visual presentation device and sending the results to the tactile simulation module.

[0007] Optionally, the core computing and communication device further includes an effect feedback module, which is used to collect and analyze user operation time, step accuracy, and tactile feedback trigger frequency data in the tactile simulation module, and output parameter optimization suggestions to the data processing module and the tactile simulation module in reverse.

[0008] Optionally, the MR visual presentation device is a mixed reality head-mounted display device with six degrees of freedom spatial positioning and eye tracking functions.

[0009] Optionally, the core computing and communication device is a graphics workstation, which has a multi-core central processing unit and a graphics processing unit.

[0010] Optionally, the data processing module embeds a medical image segmentation algorithm and a model refinement algorithm, which are used to segment, reconstruct, and optimize the topology of the data input from the oral data acquisition device, and convert it into a three-dimensional oral model.

[0011] Optionally, the tactile simulation module is configured with a force feedback control system, a tactile-scene linkage plugin, and a force model construction system.

[0012] Optionally, the haptic feedback interaction device includes a force feedback interaction device and a finger motion capture device.

[0013] Optionally, the finger motion capture device includes a glove body, on which a plurality of finger position signal interface points and palm position signal interface points are provided; the force feedback interaction device is disposed in the palm of the glove body, and the force feedback interaction device includes tactile sensing contacts and an independent thumb sensing chamber; the glove body is also provided with a data transmission interface, and the tactile sensing contacts, the independent thumb sensing chamber, and the signal interface points are all electrically connected to the data transmission interface.

[0014] Secondly, the present invention provides a method for using the aforementioned immersive missionary system, comprising the following steps: Three-dimensional geometric data and intraoral bone density data are collected using oral data acquisition equipment; The MR visual presentation device displays the optical fusion of the virtual surgical scene and the real environment, and collects user pose data. The tactile feedback interactive device simulates the tactile sensations of dental implant surgery and collects hand movement data. The data collected by the oral data acquisition device is processed into a three-dimensional model through the data processing module in the core computing and communication equipment. The scene rendering module loads the 3D model output by the data processing module; at the same time, it receives the pose data of the MR visual presentation device and adjusts the scene perspective. The tactile simulation module receives pose data from the MR visual presentation device and collision data between virtual instruments and tissues from the scene rendering module, generates electrical signals, and drives the tactile feedback interaction device to output matching tactile sensations.

[0015] Compared with the prior art, the present invention has the following beneficial effects: This invention integrates virtual surgical scenes with the real environment through MR technology, and combines tactile simulation technology to restore the real tactile sensation of operations such as drilling and implantation. It also constructs an immersive dental implant surgery scene by combining the patient's personalized oral data, allowing the patient to intuitively perceive the surgical process, reducing preoperative anxiety and improving the patient's cooperation during the surgery.

[0016] Compared to traditional education models, the immersive education system of this invention achieves precise education, making information delivery more intuitive and in-depth, and significantly improving patient participation and information retention rates. At the same time, the design combining dual-mode interaction, voice explanation and text description meets the personalized needs of different groups, and significantly improves patients' technical acceptance and overall satisfaction with the education model. This invention constructs a customized force perception model based on bone density parameters, providing oral medicine students with hands-on training that closely resembles real clinical scenarios. It solves the problems of tactile distortion in extracted tooth training and limited opportunities for clinical observation, thus shortening the transformation cycle from theory to clinical practice for medical students.

[0017] This invention supports three scenarios: preoperative patient education, medical student skills training, and public health education. Through standardized hardware deployment and software module collaboration, it enables the unified output of educational content in different scenarios, thereby improving the standardization of educational work.

[0018] The user operation data collected by the effect feedback module of this invention can be used to optimize the model accuracy of the data processing module and the force parameters of the tactile simulation module, forming a closed loop of "experience-evaluation-optimization" to ensure continuous iterative upgrades of system performance.

[0019] This invention combines MR (Magnetic Resonance) technology with tactile simulation technology and applies it to the communication and education of dental implant surgery patients. It breaks through the limitations of existing MR technology in the medical field, providing new technical ideas and solutions for medical and health education. The system and method of this invention can be further extended to various clinical oral treatment scenarios such as pediatrics and geriatrics, and can also empower oral medicine simulation teaching, contributing to the training of oral medicine professionals. Attached Figure Description

[0020] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of the invention in any way. Furthermore, the shapes and proportions of the components in the drawings are merely schematic to aid in understanding the invention and do not specifically limit the shapes and proportions of the components. In the drawings: Figure 1 This is a schematic diagram of the overall architecture of the system according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the usage method of an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the principle of MR and tactile fusion interaction in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the working principle of an embodiment of the present invention; Figure 5 This is a schematic diagram of a haptic feedback interaction device according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the palm of the haptic feedback interaction device according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the back of the hand of the haptic feedback interaction device according to an embodiment of the present invention; Figure 8 This is a side view of the haptic feedback interaction device according to an embodiment of the present invention; Figure 9 This is a schematic diagram of the data transmission interface of the haptic feedback interaction device according to an embodiment of the present invention; The components include: 1. Glove body; 2. Tactile sensing contacts; 21. Independent thumb sensing compartment; 3. Finger base movable sleeve; 4. Finger position signal interface point; 5. Palm position signal interface point; 6. Indicator light; 7. Data transmission interface. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0022] Therefore, the following detailed description of embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention.

[0023] In the description of the embodiments of the present invention, it should be noted that if terms such as "upper", "lower", "horizontal", "inner" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of the invention is usually placed during use, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention.

[0024] When an element is referred to as being "set on" another element, it can be directly on the other element or there may be an intervening element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or there may be an intervening element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only embodiments. The use of the term "horizontal" does not imply that the component is required to be absolutely horizontal, but rather that it may be slightly tilted. "Horizontal" simply means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but rather that it may be slightly tilted.

[0025] It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification and appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0027] The present invention will now be described in detail with reference to the accompanying drawings.

[0028] An immersive missionary system of the present invention includes: Oral data acquisition equipment is used to collect intraoral three-dimensional geometric data and intraoral bone density data; MR visual presentation equipment is used to display the optical fusion scene of virtual surgical scenes and real environment and to collect user pose data; Haptic feedback interactive devices are used to simulate the tactile sensations of dental implant surgery and to collect hand movement data. The core computing and communication equipment is used for system computing and multi-device data interaction; the oral data acquisition device, MR visual presentation device and haptic feedback interaction device are all communicatively connected to the core computing and communication equipment. The core computing and communication device includes a computing module, which comprises: The data processing module is used to organize the data collected by the oral data acquisition device into a three-dimensional model; The scene rendering module is used to load the 3D model output by the data processing module; at the same time, it receives pose data from the MR visual presentation device to adjust the scene perspective. The tactile simulation module is used to receive the pose data of the MR visual presentation device and the collision data between the virtual instruments and tissues between the scene rendering module, and generate electrical signals to drive the tactile feedback interaction device to output matching tactile sensations.

[0029] The present invention provides an immersive missionary system, which, when used, includes the following steps: Three-dimensional geometric data and intraoral bone density data are collected using oral data acquisition equipment; The MR visual presentation device displays the optical fusion of the virtual surgical scene and the real environment, and collects user pose data. The tactile feedback interactive device simulates the tactile sensations of dental implant surgery and collects hand movement data. The data collected by the oral data acquisition device is processed into a three-dimensional model through the data processing module in the core computing and communication equipment. The scene rendering module loads the 3D model output by the data processing module; at the same time, it receives the pose data of the MR visual presentation device and adjusts the scene perspective. The tactile simulation module receives pose data from the MR visual presentation device and collision data between virtual instruments and tissues from the scene rendering module, generates electrical signals, and drives the tactile feedback interaction device to output matching tactile sensations.

[0030] This invention integrates virtual surgical scenes with the real environment through MR technology, and combines tactile simulation technology to restore the real tactile sensation of operations such as drilling and implantation. It also constructs an immersive dental implant surgery scene by combining the patient's personalized oral data, allowing the patient to intuitively perceive the surgical process, reducing preoperative anxiety, stabilizing the patient's physiological indicators such as blood pressure and heart rate, reducing surgical risks, and improving the patient's cooperation during the surgery.

[0031] Example 1 An immersive missionary system comprising a hardware layer, a software layer, and an application layer.

[0032] The hardware layer includes an oral data acquisition device, an MR visual presentation device, a tactile feedback interaction device, a core computing and communication device, and a CT data interface.

[0033] In this embodiment, the oral data acquisition device includes an oral CT scanner with a resolution ≤0.1mm and an intraoral scanner with a scanning accuracy ≤20μm, used to acquire three-dimensional tomographic and surface morphology data of the user's jawbone, alveolar bone and teeth.

[0034] In this embodiment, the MR visual presentation device is a mixed reality head-mounted display device with six degrees of freedom spatial positioning and eye tracking functions. Its built-in 6DoF spatial positioning sensor and eye tracking module are used to realize the optical fusion of virtual surgical scenes and real environment, and to capture the user's head posture and gaze direction data in real time, so as to support the system to achieve millisecond-level synchronous response of visual and tactile feedback.

[0035] Specifically, in this embodiment, the MR visual presentation device is either Microsoft's HoloLens 2 or Meta's Quest Pro.

[0036] In this embodiment, the tactile feedback interaction device includes a force feedback interaction device and a finger motion capture device. The force feedback interaction device requires an end-effector force feedback accuracy of 0.05N to accurately simulate the tactile sensation of instrument operation during dental implant surgery. The finger motion capture device uses a fingertip pressure sensor with a sampling rate of no less than 100Hz to collect real-time data on fine hand movements and pressure.

[0037] Specifically, in this embodiment, the force feedback interaction device uses the Geomagic Touch X haptic device from 3D Systems, and the finger motion capture device uses the VR Gloves Prime haptic glove from Manus.

[0038] The finger motion capture device includes a glove body 1, which is provided with a plurality of finger position signal interface points 4 and palm position signal interface points 5.

[0039] The force feedback interaction device is located in the palm of the glove body 1, and the force feedback interaction device includes a tactile sensing contact point 2 and a thumb independent sensing chamber 3.

[0040] Optionally, in this embodiment, the thumb independent sensing compartment 21 is connected to the glove body 1 via the finger root movable sleeve 3.

[0041] Optionally, in this embodiment, the number of signal interface points 4 is four, which are respectively located at the knuckles of the back of the hand portion of the glove body 1. Optionally, in this embodiment, the number of tactile sensing contacts 2 is four.

[0042] The glove body 1 is provided with a data transmission interface 7 and an indicator light 6. The indicator light 6, the tactile sensing contact 2, the thumb independent sensing compartment 3, and the signal interface point 4 are all electrically connected to the data transmission interface 7.

[0043] In this embodiment, the core computing and communication device is a graphics workstation, which has a multi-core central processing unit, a graphics processor, and wired and wireless data transmission interfaces. It is used to undertake the core 3D graphics rendering, physical calculation and real-time data interaction tasks between multiple devices.

[0044] Specifically, in this embodiment, the central processing unit adopts an Intel Xeon W series or other products with equivalent performance, the graphics processor adopts an NVIDIA GeForce RTX 4090 or other professional graphics cards with equivalent performance, wired data transmission adopts a USB 3.2 or higher standard interface, and wireless data transmission can adopt a Wi-Fi 6 or higher standard protocol.

[0045] The core computing and communication equipment is connected to the CT data interface.

[0046] The CT data interface is used to import user cone-beam computed tomography (CBCT) data and raw oral scan data.

[0047] In this embodiment, the hardware layer and software layer achieve bidirectional data transmission through PCIe 4.0, WiFi 6, or Bluetooth 5.3 protocols.

[0048] The software layer includes a data processing module, a scene rendering module, a user interaction module, a tactile simulation module, and an effect feedback module. Each module achieves data communication based on a shared memory mechanism and the TCP / IP protocol.

[0049] The data processing module is embedded with medical image segmentation algorithms and model refinement algorithms, which are used to segment, reconstruct and optimize the data input from the oral data acquisition device, convert it into a three-dimensional oral model, and extract bone density and alveolar bone height feature parameters.

[0050] In this embodiment, the medical image segmentation algorithm uses the Mimics interactive medical image control system (Mimics).

[0051] In this embodiment, the model refinement algorithm uses 3-Matic software to represent the geometric model using a single triangular element. Using a single triangular element reduces the complexity of calculating geometric relationships between different elements.

[0052] In this embodiment, MeshLab software format conversion software and Python data cleaning library are used to convert the DICOM and STL format data input from the oral data acquisition device and output a standardized 3D model in FBX format.

[0053] The data processing module can perform fine-grained processing of data through 3D reconstruction to restore the user's oral anatomy.

[0054] The scene rendering module is equipped with Unreal Engine 5, featuring the Nanite geometry system and Lumen lighting system, and NVIDIA Nsight Graphics rendering optimization software, used to load the 3D model output by the data processing module. The scene rendering module establishes a communication link with the MR visual presentation device through the MRTK Mixed Reality Toolkit, ensuring that the spatial alignment accuracy of virtual and real fusion is ≤1mm, and the output screen resolution is ≥2K and the refresh rate is ≥90Hz.

[0055] The scene rendering module is equipped with a real-time 3D graphics rendering engine, which is used to load 3D models, construct and generate realistic standardized virtual scenes for dental implant surgery in real time, receive pose data from MR visual presentation devices, dynamically adjust the scene perspective based on the frustum rendering algorithm, and realize the real-time output of virtual and real fusion images.

[0056] Specifically, in this embodiment, the real-time 3D graphics rendering engine uses Epic Games' Unreal Engine 5, which incorporates the Nanite virtual geometry system and the Lumen global illumination system. Furthermore, NVIDIA's Nsight Graphics software is used for graphics rendering optimization.

[0057] The scene rendering module establishes a communication link with the MR visual presentation device through mixed reality development tools.

[0058] Specifically, in this embodiment, the mixed reality development tool used is Microsoft's MRTK (Mixed Reality Toolkit).

[0059] The user interaction module is equipped with a multi-device input management system, a voice recognition tool, and a gesture feature extraction library. It analyzes various types of input data through a multimodal fusion algorithm, and simultaneously analyzes gesture images collected by the MR visual presentation device, voice commands from the microphone, and hand pose data from the haptic interaction device, converting them into a standardized instruction set in JSON format to ensure smooth and synchronous interaction between the user and the system.

[0060] Specifically, in this embodiment, the multi-device input management system adopts the Input System system of the Unity engine, and the gesture feature extraction library adopts the OpenCV library.

[0061] The user interaction module is equipped with professional voice explanations of surgical procedures and safety points, and allows users to pause at any time to view text instructions. The tactile simulation module is equipped with a force feedback control system, a tactile-scene linkage plugin, and a force model construction system. Based on the bone density parameters of the data processing module, it constructs a mathematical mapping model of drilling depth-resistance value and implant insertion torque-bone density. It receives operation commands from the user interaction module and virtual instrument-tissue collision data from the scene rendering module, generates electrical signals to drive the tactile feedback interaction device to output matched damping and pressure tactile sensations, and provides users with force feedback and realistic tactile sensations throughout the dental implant surgery process, such as the slight stinging sensation of anesthesia injection, the cutting sensation of gingival incision, and the vibration sensation of the cavity preparation process, through tactile gloves.

[0062] Specifically, in this embodiment, the force feedback control system uses Geomagic TouchSDK from 3D Systems; the haptic-scene linkage plugin uses the Haptic Plugin from Unity; and the force model building system uses MATLAB software from MathWorks.

[0063] The tactile simulation module uses tactile gloves to provide users with force feedback and realistic tactile sensations throughout the entire dental implant surgery process, including the slight stinging sensation of anesthesia injection, the cutting sensation of gum incision, and the vibration sensation during cavity preparation. The tactile simulation module sets differentiated force parameters for different bone densities: the peak drilling resistance is 8N for low bone density (1200HU) and 12N for conventional bone density (1800HU).

[0064] The effect feedback module is equipped with a Python data analysis library, a Unity UI system, and TTS speech synthesis software. It is used to collect user operation time, step accuracy, and tactile feedback trigger frequency data to build a publicity and education effect evaluation model. For operation deviations, it outputs visual prompts through the MR head-mounted display and voice prompts through the speaker, and generates a visual evaluation report.

[0065] The feedback module is equipped with a data analysis library, a UI system, and a speech synthesis system, which are used to collect and analyze user operation data.

[0066] Specifically, in this embodiment, the data analysis library uses the Pandas or NumPy library of Python tools; the UI system uses the Unity UI system; and the speech synthesis system uses a TTS (Text-to-Speech) engine.

[0067] The effect feedback module can output parameter optimization suggestions to the data processing module and the tactile simulation module to achieve model accuracy adjustment and force parameter optimization, forming a system iterative closed loop.

[0068] The application layer includes a preoperative experience system and a back-end management system. It is a customized education client developed based on control programs written in C++. The preoperative experience system simulates the entire process of dental implant surgery based on personalized user data to meet users' personalized education needs. The back-end management system is used by medical and nursing education personnel, providing functions such as data uploading, scene selection, experience control, and report viewing. It can also manage and analyze user data and the education process.

[0069] The application layer supports personalized scenario customization, including educational scenarios for different dental implant procedures such as immediate implantation and delayed implantation, while also supporting lightweight deployment for group-based science popularization and education.

[0070] The application layer is the actual user end of the system, including the pre-operative experience system and the back-end management system. A customized educational client developed based on C# is deployed, targeting three user groups: dental implant patients, dental students, and medical education personnel. It provides functions such as personalized scenario selection, experience process control, and evaluation report viewing. It supports customized educational scenarios for different surgical procedures, such as immediate implantation and delayed implantation. Simultaneously, it can manage and analyze user data and the educational process, and supports lightweight deployment to meet the public's science popularization needs.

[0071] Example 2 A method for using an immersive missionary system based on Embodiment 1 includes the following steps: S1. Oral cavity three-dimensional data acquisition and preprocessing: A dental CT scanner with a resolution of ≤0.1mm was used to acquire DICOM format tomographic data of the user's jawbone and alveolar bone, and an intraoral scanner with a scanning accuracy of ≤20μm was used to acquire STL format surface topography data of the edentulous area.

[0072] The raw data is transferred to a graphics workstation equipped with an Intel Xeon W series processor and an NVIDIA RTX 4090 graphics card via the PCIe 4.0 protocol.

[0073] The Mimics software was used to perform a threshold segmentation algorithm to extract the three-dimensional contour of the oral tissue. The 3-Matic software was used to perform model smoothing and topology optimization for defect repair. The MeshLab software was used to convert the model into the FBX universal format and extract bone density and alveolar bone height feature parameters.

[0074] 3D reconstruction utilizes a data processing module to refine the raw data and then uses professional 3D reconstruction software to recreate the user's oral anatomy at a 1:1 scale, generating a personalized 3D oral model.

[0075] S2. Personalized Virtual Surgical Scene Construction and Rendering: The FBX format model output in step S1 is read through a shared memory mechanism, and a standardized dental implant surgery scene is loaded into the Unreal Engine 5 engine equipped with the Nanite geometry system to complete the fusion of the personalized oral model and the standard scene. A communication link is established with the MR headset with a built-in 6DoF spatial positioning sensor through the MRTK Mixed Reality Toolkit, the pose data of the MR headset is received, the scene view is dynamically adjusted based on the frustum rendering algorithm, and the screen resolution is set to ≥2K and the refresh rate is set to ≥90Hz. NVIDIA Nsight Graphics software was used to optimize rendering parameters, ensuring that the spatial alignment accuracy of virtual-real fusion is ≤1mm. The scene rendering module simulates key surgical steps of dental implantation, such as anesthesia injection, gingival incision, and implant placement, based on a personalized 3D oral model, to create an immersive MR surgical scene. S3. Multimodal interaction command parsing and haptic feedback generation: A multimodal input fusion algorithm is adopted, using OpenCV software to parse the gesture images of the MR head-mounted display camera, iFlytek Hearing speech recognition SDK to parse the voice commands of the microphone, and Unity Input System to parse the hand pose data of the force feedback handle / haptic glove, and converting all commands into a standardized instruction set in JSON format; The instruction set is synchronized to the scene rendering module via the TCP / IP protocol, driving the virtual surgical instruments to complete the corresponding actions; Based on the bone density parameters extracted in step S1, the mathematical mapping model of "drilling depth-resistance value" constructed by MATLAB software is called to receive the virtual instrument-oral tissue collision data output by the scene rendering module. The Geomagic Touch SDK software is called to generate an electrical signal, which drives the force feedback handle to output matching damping. The peak resistance is 8N for low bone density of 1200HU and 12N for normal bone density of 1800HU. At the same time, the fingertip airbags of the haptic glove output matching pressure. Users wear MR glasses and haptic gloves, and through the gesture recognition function of the MR glasses, they can touch and control the surgical process. At the same time, the haptic simulation module achieves millisecond-level synchronous response of vision and touch through customized haptic sensors, restoring the real touch of surgery.

[0076] S4. Implementation of Personalized Immersive Evangelistic Experiences: For dental implant users, a "observation + simple operation" mode is adopted. Users wear an MR headset and force feedback gloves, select a personalized surgical scenario through the application layer client, and trigger a prompt when the operation deviation is greater than 15°. The user interaction module switches the MR headset, combined with professional voice explanation and text description, to meet the user's personalized interaction needs.

[0077] For dental students, a "full-process hands-on" mode is adopted, in which students wear MR head-mounted displays and haptic handles to experience tactile feedback throughout the entire dental implant surgery process.

[0078] For the target audience of science popularization, a lightweight deployment using portable graphics workstations is adopted, with residents taking turns wearing MR headsets and controlling the scene progress through voice commands.

[0079] S5. Evaluation and Real-time Feedback on the Effectiveness of Education and Propaganda: Data on user operation duration, step accuracy, and haptic feedback trigger frequency are collected through a shared memory mechanism.

[0080] The evaluation model is built using the Scikit-learn library in Python software, and a visual evaluation report is generated.

[0081] When an operational deviation is detected, a visual prompt will pop up in the MR headset through the Unity UI system, and a voice prompt will be generated by the TTS speech synthesis software and played through the speaker.

[0082] S6. Iterative optimization of system parameters: The parameter optimization suggestions in the evaluation report are transmitted to the data processing module and the tactile simulation module via TCP / IP protocol; The data processing module adjusts the model segmentation threshold to improve the accuracy of oral tissue contour extraction; the tactile simulation module corrects the bone density-resistance value mapping curve to optimize the realism of tactile feedback.

[0083] The bone mineral density characteristic parameter mentioned in step S1 is 1200HU-1800HU, the force feedback accuracy of the force feedback handle mentioned in step S3 is 0.05N, and the sampling rate of the fingertip pressure sensor of the tactile glove is ≥100Hz.

[0084] The MR headset mentioned in step S2 is HoloLens 2 or Meta Quest Pro, and the multimodal input fusion algorithm mentioned in step S3 incorporates the operation command sets of gestures, voice, and gamepad / glove into the same control framework.

[0085] The evaluation model described in step S5 includes three core indicators: completion of operation steps, standardization of operation, and matching degree of tactile parameters.

[0086] Steps S1 to S4 are the forward execution process, and steps S5 to S6 are the reverse optimization process, forming a closed loop of experience-evaluation-optimization. Example 3 Based on the method of using the immersive education system in Embodiment 2, this embodiment is a preoperative immersive education scenario for dental implant patients.

[0087] This embodiment addresses the preoperative anxiety relief needs of patients undergoing immediate mandibular anterior tooth implantation surgery by providing personalized and interactive surgical procedure education and verifying the system's effectiveness in reducing preoperative anxiety levels.

[0088] The baseline characteristics of the target group for this embodiment are: users who need to undergo immediate implantation surgery of the left mandibular central incisor, who are completely unaware of the surgical procedure, have concerns such as "intraoperative pain" and "surgical failure", and have no contraindications to the use of MR equipment.

[0089] Dental CT scanner: Siemens SOMATOM Scope with a resolution of 0.08mm is used to acquire sagittal, coronal, and transverse tomographic data of the user's mandible, with the output format being DICOM 3.0.

[0090] Intraoral scanner: iTero Element 5D was used, with a scanning accuracy of 15μm, to collect surface morphology data of the edentulous area and adjacent teeth, and the output format was STL.

[0091] Graphics workstation: Equipped with an Intel Xeon W-3323 processor, an NVIDIA RTX 4090 professional graphics card, 64GB DDR5 memory, 2TB SSD storage, and pre-installed Windows 11 Professional operating system.

[0092] Data transmission: High-speed data transmission between the CT scanner, intraoral scanner and workstation is achieved via PCIe 4.0 protocol, with a transmission rate of ≥16GB / s.

[0093] MR headset: It uses the Meta Quest Pro, with a built-in 6DoF spatial positioning sensor and eye tracking module, outputting a resolution of 2.56K and a refresh rate of 90Hz.

[0094] Force feedback gloves: Utilizes Geomagic Touch X, with a force feedback accuracy of 0.05N and a working space of 160mm×120mm×70mm, matching the handheld size for dental implant handpieces.

[0095] Auxiliary equipment: omnidirectional microphone (sampling rate 48kHz), noise-canceling speaker (frequency response 20Hz-20kHz), connected to the workstation via Bluetooth 5.3 protocol.

[0096] Step S1: Oral cavity 3D data acquisition and preprocessing: Data acquisition: The CT scanner was set with tube voltage of 120kV, tube current of 80mA, and slice thickness of 0.5mm. The scanning range covered the area from the left central incisor to the second premolar of the mandible. The intraoral scanner continuously scanned along the gingival margin to the alveolar crest of the edentulous area to obtain complete soft and hard tissue boundary data.

[0097] Data transmission: CT machine DICOM data and intraoral scanner STL data are written to the workstation's shared memory via the PCIe 4.0 interface, with a transmission time of ≤30s.

[0098] Model processing: Mimics 25.0 software was used to execute the threshold segmentation algorithm, the bone tissue threshold was set to 1800HU, the three-dimensional contours of alveolar bone and jawbone were extracted, and noise points and artifacts were removed.

[0099] The 3-Matic 17.0 software was used to perform topology optimization on the model: "Smoothing" (10 iterations, smoothing factor 0.2), "Defect Repair" (filling small defects ≤0.5mm on the alveolar bone surface), and "Mesh Simplification" (the number of triangle faces was optimized from 1.2 million to 500,000 to ensure rendering efficiency).

[0100] MeshLab 2022.02 software was used to convert the optimized model into FBX format. At the same time, feature parameters such as alveolar bone height (12mm), width (8mm), and bone density distribution in the edentulous area were extracted and stored in the local database of the workstation.

[0101] Step S2: Personalized Virtual Surgical Scene Construction and Rendering: Scene fusion: The generated 3D oral model is imported into the Unity 3D engine, and the "Standardized Scene for Immediate Implant Surgery of Mandibular Anterior Teeth" is loaded into the Unity 3D engine, which includes 6 core steps: disinfection, local anesthesia, gingival incision, drilling and preparation, implant placement, and suturing. Through the engine's built-in "bone skinning" technology, the personalized oral model in FBX format output from step 1 is spatially aligned with the surgical instruments (implant machine, dental drill, implant, healing abutment) of the standard scene, with an alignment error of ≤0.5.

[0102] Communication link establishment: A TCP / IP communication link is established between the workstation and the Meta Quest Pro headset using the MRTK Mixed Reality Toolkit. The headset uploads 6DoF pose data (head rotation angle, translation distance) in real time with a refresh rate of 90Hz.

[0103] Rendering parameter optimization: Based on the frustum rendering algorithm, the viewpoint and field of view of the virtual scene are dynamically adjusted according to the head-mounted display pose data. The field of view is set to 90° to match the natural human eye perspective.

[0104] The rendering parameters were optimized using NVIDIA Nsight Graphics software: "Nanite Virtual Micropolygon Geometry System" was enabled to achieve adaptive rendering of model details; "Lumen Dynamic Global Illumination" was enabled to simulate the lighting effects of oral tissues under surgical lights.

[0105] Calibrate the accuracy of virtual-real fusion: By using the spatial positioning sensor built into the headset, the spatial position deviation between the virtual implant and the real headset is controlled within 0.8mm to ensure the authenticity of the user's viewing perspective.

[0106] Step S3: Multimodal interaction command parsing and haptic feedback generation: Multimodal command acquisition and parsing: MR visual interaction: Users wear Meta Quest Pro MR glasses and customized haptic gloves. Through the gesture recognition function of the MR glasses, users can perform actions such as "touching" the implant and "manipulating" surgical instruments to interact with the virtual surgical scene. Voice commands: Users can issue commands such as "zoom in on the drilling area" or "play the anesthesia procedure" through the microphone, which will call the iFlytek Hearing Speech Recognition SDK for real-time parsing. The recognition accuracy is ≥95%, and the parsing results are converted into a JSON format command set.

[0107] Haptic glove operation commands: The haptic simulation module controls the force feedback unit of the haptic glove through a piezoelectric sensor, accurately reproducing tactile sensations such as the slight stinging sensation of anesthesia injection (pressure value approximately 0.5N), the cutting sensation of gum incision (pressure value approximately 1N), and the vibration sensation during orifice preparation (frequency approximately 50Hz), achieving millisecond-level synchronous response between vision and touch. The user holds the force feedback handle to simulate drilling actions. The position sensor built into the handle collects hand movement trajectory data, which is converted into standardized control commands through Unity InputSystem 1.7.0 software.

[0108] Command fusion: A multimodal input fusion algorithm is adopted to integrate voice commands and handset operation commands into the same control framework, with the priority set as "handset operation > voice command" to avoid command conflicts.

[0109] Haptic feedback model loading: The "drilling depth-resistance value" mathematical mapping model pre-built in MATLAB software is called. The input parameter is the 1800HU bone density data extracted in step 1. The model outputs the resistance value change curve with drilling depth: 0-3mm (cortical bone layer) resistance value 5-8N, 3-10mm (cancellous bone layer) resistance value 8-12N, 10-12mm (cortical basal layer) resistance value 12-10N.

[0110] Haptic feedback trigger: The scene rendering module detects the collision data between the virtual drill bit and the alveolar bone in real time. When the "drill bit contacts bone tissue" signal is detected, the collision position and depth data are transmitted to the haptic simulation module through shared memory.

[0111] The Geomagic Touch SDK software generates an electrical signal that drives the motor of the force feedback handle to produce a damping force. The magnitude of the damping force is matched in real time with the resistance value output by the model. At the same time, it triggers the handle's vibration feedback, and the vibration frequency increases with the increase of the resistance value (5-8N corresponds to a frequency of 50Hz, and 8-12N corresponds to a frequency of 100Hz).

[0112] Step S4: Implementation of a Differentiated Immersive Evangelistic Experience Experience mode settings: For the target user group, a "observation + simple operation" mode is adopted. Users can independently control the drilling depth and angle, but the operation range is limited to the safe area planned before the operation.

[0113] Personalized Interaction: Users can switch between "Automatic Demonstration" and "Simple Interaction" modes via the MR glasses' handle controller. In "Automatic Demonstration" mode, the system automatically plays a full-process simulated video of dental implant surgery, accompanied by professional voice explanations of the surgical steps and safety points. In "Simple Interaction" mode, users can independently control the progress of the surgical procedure, pause the demonstration at any time, and view text descriptions of the surgical steps. The text descriptions support font enlargement, reduction, and color switching, meeting the needs of users of different ages and education levels.

[0114] Voice prompts: The system uses TTS (Text-to-Speech) software to generate voice prompts, which are then played through a speaker. The volume of the prompts is automatically adjusted to the ambient noise.

[0115] Experience duration control: The duration of a single experience is set at 10 minutes, covering the entire surgical procedure, to avoid users experiencing dizziness due to prolonged wearing of the headset.

[0116] Step S5: Evaluation and Real-time Feedback on the Effectiveness of the Evangelism Campaign Data Acquisition: Three types of evaluation data are collected through shared memory: operation time (time spent in each step); step accuracy (such as whether the drilling angle and depth meet the plan); and haptic feedback trigger frequency (the haptic experience of whether the user has completed key operations).

[0117] Evaluation model operation: The Scikit-learn library of Python 3.10 is used to build an evaluation model that includes three core indicators: operation step completion degree, operation standardization, and tactile parameter matching degree. The weights of each indicator are 0.4, 0.4, and 0.2, respectively, and the comprehensive score is calculated.

[0118] Report Generation and Feedback: After the user experience session, the system automatically generates a visual evaluation report, including: an overall score; and the results of a knowledge mastery Q&A session (questions were asked via a pop-up window on the headset, with a 100% correct answer rate). The doctor then uses the report to explain the safety and controllability of the surgery to the user and answers their questions.

[0119] Implementation effect verification: After user experience, the SAS (Self-Rating Anxiety Scale) score was 38 points (no anxiety), a decrease of 17 points from the baseline; the user's awareness of the surgical procedure increased from 0 to 100%, indicating that "I clearly understand each step of the operation and am no longer afraid of surgery"; the nurse's education time was shortened from the traditional 20 minutes to 10 minutes, and the efficiency of doctor-patient communication improved by 50%.

[0120] Example 4 Based on the method of using an immersive education system according to Embodiment 2, this embodiment is a training scenario for dental implant surgery skills for oral medicine students.

[0121] This embodiment addresses the clinical skills training needs of third-year students majoring in oral medicine by implementing full-process surgical simulation training with tactile feedback, and verifying the system's effectiveness in improving the standardization of medical students' operations.

[0122] Baseline characteristics of the target group: 20 third-year students majoring in dentistry who have completed the theoretical courses of dental implantation and have experience in drilling extracted teeth, but lack clinical practice experience; and have insufficient understanding of the "manual feel of different bone densities".

[0123] Hardware deployment and parameter calibration: Manus VR Gloves Prime haptic gloves (120Hz fingertip pressure sensor sampling rate); HD panoramic camera (4K resolution, 30fps) for capturing fine hand movements and operation processes.

[0124] Step 1: Loading the standardized low bone mineral density model: The data processing module calls the "standard model of low bone density in the maxillary posterior teeth region", with the bone density parameter set to 1200HU, matching the bone characteristics of patients with osteoporosis commonly seen in clinical practice; the model includes pathological features such as alveolar bone resorption and uneven bone density, making it closer to complex clinical cases.

[0125] Step 2: Virtual Scene Customization: The scene rendering module loads the "delayed implantation surgery scene for maxillary posterior teeth" and adds advanced operation steps such as "gingival flap" and "bone augmentation"; the "surgical error warning" function is enabled, and the scene will automatically pause and display the reason for the error when the simulation operation causes "bone perforation" or "damage to adjacent teeth".

[0126] Step 3: Multimodal interaction and haptic feedback optimization: New tactile glove interaction: Students wear tactile gloves to simulate fine operations such as "gingival separation" and "suture knotting". Pressure sensors at the fingertips of the gloves collect data on the pinching force of the fingers and convert it into control commands through the Manus SDK. The tactile simulation module drives the air bladders at the fingertips of the gloves to inflate / deflate according to the elastic parameters of the virtual soft tissue, simulating the soft touch of the gums (pressure value 2-5N).

[0127] Tactile parameter adjustment: Based on the 1200HU low bone density parameter, the peak drilling resistance is adjusted to 8N to simulate the feel of soft and easily collapsed bone; when the drilling depth exceeds the safe range, the handle vibration frequency is increased to 200Hz, triggering the "bone perforation" warning.

[0128] Step 4: Full-process hands-on training: Students used a "full-process hands-on" mode to complete seven steps in sequence: disinfection, anesthesia, gingival flap elevation, drilling and cavity preparation, bone augmentation, implant placement, and suturing. The system recorded the operation time, step standardization, and tactile parameter matching degree of each step in real time.

[0129] Step 5: Skills Assessment and Debriefing Assessment and scoring: The system generates personalized assessment reports, and the average comprehensive score of 20 students increased from 65 points before training to 88 points after training; among the operational standardization indicators, "drilling angle deviation" decreased from 20° before training to 8° after training.

[0130] Video debriefing: High-definition camera recordings of students' hand movements are played back simultaneously with virtual scene images. Instructors can mark problems such as "excessive implant insertion torque" and "improper suturing and knotting" and provide targeted guidance.

[0131] Implementation effect verification: The awareness of "the feel of bone with different bone densities" among 20 students increased from 20% before training to 95%; in the subsequent practical assessment of extracted teeth, the operation error rate decreased by 60%, verifying the significant effect of the system on skills training.

[0132] Example 5 Based on the method of using the immersive education system in Embodiment 2, this embodiment is a community implant dentistry popularization education scenario.

[0133] This embodiment addresses the dental implant education needs of the elderly population in the community, achieving lightweight and group-based science popularization and education, and verifying the effectiveness of the system in improving the efficiency of science popularization dissemination.

[0134] Baseline characteristics of the target population: 50 elderly people in the community, aged 55-75, with a knowledge level of ≤30% of dental implant technology, and generally hold misconceptions such as "dental implants are expensive", "surgery is risky" and "short lifespan".

[0135] Hardware deployment and parameter calibration: A lightweight deployment solution is adopted, consisting of a portable graphics workstation (weighing 2.5kg) and a multi-user shared MR headset (3 units). Force feedback handles and haptic gloves are eliminated to simplify the interaction method; a Bluetooth speaker is provided to enable synchronized voice explanation.

[0136] Step 1: Simplify the science popularization scenario: The data processing module loads a general-purpose 3D oral model, removes complex bone tissue details, and highlights the contrast between "healthy teeth - missing teeth - dental implants"; the scene rendering module simplifies the surgical procedure, retaining the three core popular science segments: "harm of missing teeth", "advantages of dental implants" and "postoperative maintenance", with each segment lasting no more than 3 minutes.

[0137] Step 2: Simplify the interaction method: The user interaction module retains only voice command control, with three simple commands: "Next Page," "Previous Page," and "Repeat Playback." The scene rendering module enables "Auto Playback" mode, allowing the science content to loop without user intervention.

[0138] Step 3: Group Interaction Experience The educators controlled the scene playback through the application layer client, and explained practical information such as the lifespan of dental implants (10-20 years), postoperative maintenance methods (brushing teeth, teeth cleaning, regular check-ups), and medical insurance policies.

[0139] Fifty residents took turns wearing the MR headset for a 5-minute experience each. The feedback module collected information on the knowledge points that the group was concerned about, and the statistics showed that "postoperative dietary restrictions", "dental implant price" and "pain level" were the three most frequently asked questions.

[0140] Step 4: Optimize the effect of popular science education: Based on group experience data, we optimized the content presentation order in science popularization scenarios, placing frequently asked questions at the forefront; and added video clips of "real-life cases of dental implant users" to enhance the persuasiveness of science popularization.

[0141] Implementation results verification: The awareness of dental implant technology among 50 residents increased from ≤30% to 85%; the rate of correction of misconceptions reached 90%; and 30 residents expressed their willingness to consult further about dental implant surgery, verifying the significant effect of the system on popular science dissemination.

[0142] The system and method of this invention achieve a closed-loop process of "data acquisition - model building - scene rendering - interactive experience - effect evaluation - iterative optimization" through a three-level collaborative architecture of hardware layer - software layer - application layer. The three embodiments respectively verify the effectiveness of the system in three major scenarios: patient preoperative anxiety relief, medical student skills training, and public science education. The technical solution is scientific and feasible, and has broad prospects for clinical promotion and application.

[0143] Unless otherwise specified, the equipment components involved in the above embodiments are all conventional equipment components, and the structural settings, working methods or control methods involved are all conventional settings, working methods or control methods in the art unless otherwise specified.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Any other modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention, as long as they do not depart from the spirit and scope of the technical solutions of the present invention, should be covered within the scope of the claims of the present invention.

Claims

1. An immersive missionary system, characterized in that, include: Oral data acquisition equipment is used to collect intraoral three-dimensional geometric data and intraoral bone density data; MR visual presentation equipment is used to display the optical fusion scene of virtual surgical scenes and real environment and to collect user pose data; Haptic feedback interactive devices are used to simulate the tactile sensations of dental implant surgery and to collect hand movement data. The core computing and communication equipment is used for system computing and multi-device data interaction; the oral data acquisition device, MR visual presentation device and haptic feedback interaction device are all communicatively connected to the core computing and communication equipment. The core computing and communication device includes a computing module, which comprises: The data processing module is used to organize the data collected by the oral data acquisition device into a three-dimensional model; The scene rendering module is used to load the 3D model output by the data processing module; at the same time, it receives pose data from the MR visual presentation device to adjust the scene perspective. The tactile simulation module is used to receive the pose data of the MR visual presentation device and the collision data between the virtual instruments and tissues between the scene rendering module, and generate electrical signals to drive the tactile feedback interaction device to output matching tactile sensations.

2. The immersive missionary system according to claim 1, characterized in that, The core computing and communication equipment also includes a user interaction module, which is used to parse the pose data collected by the MR visual presentation device and send the results to the tactile simulation module.

3. The immersive missionary system according to claim 1, characterized in that, The core computing and communication device also includes an effect feedback module, which is used to collect and analyze user operation time, step accuracy, and tactile feedback trigger frequency data from the tactile simulation module, and output parameter optimization suggestions to the data processing module and the tactile simulation module.

4. The immersive missionary system according to claim 1, characterized in that, The MR visual presentation device is a mixed reality head-mounted display device with six degrees of freedom spatial positioning and eye tracking functions.

5. An immersive missionary system according to claim 1, characterized in that, The core computing and communication device is a graphics workstation, which has a multi-core central processing unit and a graphics processing unit.

6. The immersive missionary system according to claim 1, characterized in that, The data processing module is embedded with medical image segmentation algorithms and model refinement algorithms, which are used to segment, reconstruct, and optimize the topology of the data input from the oral data acquisition device, and convert it into a three-dimensional oral model.

7. An immersive missionary system according to claim 1, characterized in that, The tactile simulation module is equipped with a force feedback control system, a tactile-scene linkage plugin, and a force model construction system.

8. An immersive missionary system according to claim 1, characterized in that, The haptic feedback interaction device includes a force feedback interaction device and a finger motion capture device.

9. An immersive missionary system according to claim 8, characterized in that, The finger motion capture device includes a glove body (1), on which a plurality of finger position signal interface points (4) and palm position signal interface points (5) are provided; the force feedback interaction device is located in the palm of the glove body (1), and the force feedback interaction device includes a tactile sensing contact point (2) and a thumb independent sensing chamber (3); the glove body (1) is also provided with a data transmission interface (7), and the tactile sensing contact point (2), the thumb independent sensing chamber (3) and the signal interface points (4) are all electrically connected to the data transmission interface (7).

10. A method of using an immersive missionary system as described in any one of claims 1 to 9, characterized in that, Includes the following steps: Three-dimensional geometric data and intraoral bone density data are collected using oral data acquisition equipment; The MR visual presentation device displays the optical fusion of the virtual surgical scene and the real environment, and collects user pose data. The tactile feedback interactive device simulates the tactile sensations of dental implant surgery and collects hand movement data. The data collected by the oral data acquisition device is processed into a three-dimensional model through the data processing module in the core computing and communication equipment. The scene rendering module loads the 3D model output by the data processing module; at the same time, it receives the pose data of the MR visual presentation device and adjusts the scene perspective. The tactile simulation module receives pose data from the MR visual presentation device and collision data between virtual instruments and tissues from the scene rendering module, generates electrical signals, and drives the tactile feedback interaction device to output matching tactile sensations.

Images