Human body teaching aid puncture positioning method based on space positioning point technology

Abstract

The invention belongs to the technical field of virtual reality, and particularly relates to a human body teaching aid puncture positioning method based on a space positioning point technology. According to the device, the mixed reality glasses are combined with the laser cross positioning function, precise positioning is achieved through the binocular stereoscopic vision principle, the puncture path can be adjusted in real time, and data analysis and feedback including error prompt and correction can be provided. The operation process comprises initialization setting, model loading, puncture planning, practical operation and data analysis and feedback. The system has the advantages of improving puncture skill training efficiency and accuracy, reducing teaching cost and the like, and is suitable for the field of medical teaching. The problems that an existing puncture skill teaching model cannot achieve accurate positioning and feedback, and the cost of a special teaching aid is high are solved.

Description

Technical Field

[0001] The invention belongs to the technical field of virtual reality, in particular to a human teaching aid puncture positioning method based on space positioning point technology. Background Art

[0002] Puncture is an important technique in clinical invasive operations. It is widely used in medical practices such as fluid extraction, tissue sampling and catheter insertion, and constitutes an important part of medical skills training. Traditional puncture skills training methods mainly rely on reading operation manuals and watching teaching videos, and then performing simulated operations on rubber models. However, such methods have significant limitations. Since the model cannot feedback the specific position, depth and angle of the puncture needle in space, there is a large gap between the training effect and the actual operation, and it is difficult for trainees to accurately master the puncture skills. In addition, this traditional training method lacks real-time data feedback, and it is difficult for trainees to evaluate their own accuracy and room for improvement during the operation, which affects the overall effect of the training.

[0003] In the current development of the technology field, mixed reality (MR) technology has begun to be applied to medical skills training, and has gradually attracted attention for its realistic immersive experience and risk-free nature. Common spatial positioning technologies in MR devices include inside-out tracking, visual SLAM, and depth sensors and point cloud technology. 1. Inside-out Tracking: This technology relies on the built-in camera of the device to capture the surrounding environment, combined with visual simultaneous localization and mapping (SLAM) technology to locate the user's head and the position of the device in the physical space in real time. Inside-out tracking can achieve autonomous positioning without external sensors or base stations, and is suitable for mobile scenarios. 2. Visual SLAM (Synchronous Localization and Mapping): The SLAM algorithm captures environmental features (such as edges, corners, etc.) through the camera and detects their relative position changes to generate a three-dimensional map and the device's own positioning. Visual SLAM enables mixed reality devices to autonomously understand and locate the surrounding environment without the need for external sensors or pre-marked space, with high flexibility and accuracy, and is particularly suitable for dynamic scenes. 3. Depth sensor and point cloud technology: Supports the interaction between virtual objects and the real environment by capturing depth information in three-dimensional space. The technology emits infrared light or other light signals and calculates the distance of objects based on the return time difference or phase difference. Combined with visual SLAM to generate a more accurate 3D environment model, users can interact with gestures, and virtual objects can respond reasonably (such as occlusion or placement) based on the real layout.

[0004] Although these technologies have made significant progress in spatial positioning of mixed reality, there is still a problem in providing real-time feedback of accurate position, depth and angle in medical puncture skills training. In addition, although the existing sensor-based models can realize puncture action recognition to a certain extent, once the sensor or model is damaged, the replacement cost is also high. When the number of students is large, it directly increases the teaching cost. Its high equipment and maintenance costs limit its scope of application. How to use these advanced spatial positioning technologies to design a solution that can accurately feedback the puncture operation and is economical and efficient is still an urgent problem to be solved. Summary of the invention

[0005] The present invention relates to a puncture positioning method for human teaching aids based on spatial positioning point technology, which aims to achieve accurate guidance and real-time feedback on puncture operations on human teaching aids for medical teaching by integrating the laser cross positioning function of mixed reality glasses. This method not only improves the efficiency and accuracy of puncture skill training in the medical teaching process, but also helps students better understand the angle, depth and path control during the puncture process through a digitized feedback mechanism.

[0006] In order to achieve the above object, the present invention specifically adopts the following technical methods:

[0007] Performing spatial positioning calibration through mixed reality (MR) glasses, wherein the calibration includes activating a sensor module after wearing the mixed reality glasses and establishing a three-dimensional spatial coordinate system;

[0008] Load a virtual model of a medical teaching human body aid into mixed reality (MR) glasses, which can be rotated, scaled, and moved through gestures or voice commands;

[0009] Plan the puncture path and project it onto the puncture model in the form of a virtual laser line. The virtual laser line can dynamically adjust the viewing angle according to the user's head movement.

[0010] Perform puncture operations and provide real-time feedback through mixed reality glasses, including displaying the current position, angle and depth of the puncture path, as well as detecting incorrect operations and providing prompts and correction suggestions;

[0011] Collect and analyze puncture data and generate feedback reports.

[0012] Furthermore, the puncture device and the mixed reality glasses exchange data to record the puncture depth, angle change and movement path in real time, and ensure the security and real-time nature of data transmission through a wireless transmission protocol.

[0013] Furthermore, the feedback report includes the accuracy of the puncture path, the stability of the depth control and the error analysis. The feedback report is stored in the storage unit of the mixed reality glasses, and the user can choose to save, export or print the report.

[0014] Furthermore, the puncture path planning is performed based on binocular stereo vision technology, the spatial depth is determined by calculating the parallax, and the measurement accuracy is ensured by optical calibration.

[0015] Furthermore, the puncture device has a built-in sensor to provide positioning and tracking data of the puncture path to ensure the accuracy and real-time nature of the data.

[0016] Furthermore, the display screen built into the mixed reality glasses adopts a transparent display, so that the user can see the puncture path and necessary information during operation without affecting the line of sight.

[0017] Furthermore, the operation steps include the user controlling the virtual model and selecting the puncture path through gestures or voice commands after initialization.

[0018] Furthermore, the mixed reality glasses operate synchronously with the laser puncture positioning device to ensure real-time data feedback and path guidance, thereby improving the accuracy and safety of the puncture operation.

[0019] Furthermore, the spatial positioning calibration of the mixed reality glasses in the initialization setting step utilizes the high-definition front camera equipped therewith, combined with visual synchronous positioning and mapping technology, to detect feature points in the environment and infer the movement trajectory of the glasses in space based on the relative position changes of these feature points, thereby establishing an accurate three-dimensional spatial coordinate system.

[0020] Furthermore, at the same time, cameras distributed on the left and right sides with a spacing equal to the baseline distance are used to simultaneously capture images of the same spatial object from different perspectives, and two images of the target area are obtained through an image acquisition device and denoised, grayed, and pre-processed;

[0021] Semi-global stereo matching is used to find the same-name point pairs (p, p') in the two images, and the disparity between the same-name point pairs is calculated to infer the depth information of the object;

[0022] The camera is calibrated by using a prefabricated 3D calibration object or calibration template to obtain the internal and external parameters of the camera, including focal length, optical center position, and rotation and translation matrix, which are used to establish the mapping relationship between the camera coordinate system and the world coordinate system;

[0023] According to the stereo matching results and camera calibration parameters, the perspective projection transformation formula is used to convert the two-dimensional image coordinates into three-dimensional space coordinates. The formula is:

[0024]

[0025]

[0026] zz

[0027] Among them, Δx and Δy are the parallax values ​​on the image plane, f is the focal length, and x0 and y0 are the positions of the optical center. Use epipolar constraint-based methods to improve matching accuracy or use time synchronization technology to ensure the synchronization of the two images, and perform error correction and optimization on the system.

[0028] Furthermore, it also includes virtual-real fusion registration, which is used to register the virtual model to the real-world object, including the following steps:

[0029] For the {Tr} optical tracker coordinate system, {V} virtual model coordinate system, and {H} MR glasses coordinate system of the mixed reality surgical navigation system, the virtual model and the real object are aligned by solving the relevant transformation matrix to determine the relationship between the coordinate systems. The transformation matrix T between the virtual model coordinate system {V} and the optical tracker coordinate system {Tr} is obtained. V Tr , the transformation matrix T between the optical tracker coordinate system {Tr} and the MR glasses coordinate system {H} Tr H , and then obtain the transformation matrix T between the virtual model coordinate system {V} and the MR glasses coordinate system {H} V H =T V Tr ×T Tr H ;

[0030] Based on the knowledge of rigid body transformation, the conversion of different coordinate systems is realized by solving the rotation and translation matrices. It is known that the point set P in the tracker coordinate system and the corresponding point set Q in the MR glasses coordinate system have a specific conversion relationship.

[0031] Taking the error of the source point P relative to the target point set Q under the specific transformation matrices R and T as the criterion, solving the optimal transformation matrices R and T includes the following steps:

[0032] First, find the centroid of each point set P and Q;

[0033] Then, the covariance matrix E is constructed, and the singular value decomposition method is used to determine the optimal matrices R and T;

[0034] When the number of points in the point set exceeds 3, the solution obtained is the least squares solution, which ensures the accuracy and effectiveness of the transformation matrix and realizes accurate transformation between different coordinate systems;

[0035] Display the corresponding virtual ball in the virtual coordinate system of the glasses, move the virtual ball to completely overlap with the reflective ball, obtain the coordinates of the center of the virtual ball in the MR glasses coordinate system and the center of the reflective ball in the optical tracker coordinate system, and calculate the transformation matrix through the two sets of corresponding point coordinates.

[0036] Correspondingly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the human teaching aid puncture positioning method based on spatial positioning point technology are implemented.

[0037] Correspondingly, a non-transitory computer-readable storage medium stores a computer program thereon, characterized in that when the processor executes the program, the steps of the human body teaching aid puncture positioning method based on spatial positioning point technology are implemented.

[0038] Compared with the prior art, the present invention has the following beneficial technical effects:

[0039] By combining binocular stereo vision to calculate parallax to determine depth information with the laser puncture positioning device to measure laser time to determine position, the puncture path can be accurately planned, allowing the puncture to be performed according to the predetermined trajectory. Trainees can accurately simulate surgical operations and reduce the risk of injury. At the same time, the real-time adjustment function allows doctors to deal with complex situations during surgery, ensuring safe and effective operations and improving the success rate of surgery.

[0040] The system's rich model resources and convenient operation functions support a variety of model selections and flexible observations, providing diverse scenarios for teaching. During actual operation, the puncture device interacts with the glasses in real time, and the system evaluates and provides feedback, allowing students to improve their skills, and teachers can adjust their teaching accordingly to improve the quality and effectiveness of teaching and learning.

[0041] Mixed reality glasses can be quickly calibrated after being worn and are comfortable to wear, reducing the trouble of traditional calibration and can be quickly put into use in emergency or high-intensity scenarios; its intuitive operating instructions and interactive design facilitate information acquisition, reduce operating difficulty, reduce errors, allow doctors to focus on surgery, and improve operating efficiency.

[0042] The spatial positioning algorithm based on optical positioning method, combined with multi-sensor and data source fusion technology, overcomes the limitations of a single sensor and ensures positioning stability and accuracy; camera calibration and a variety of error correction optimization measures improve system performance, enable it to adapt to complex and changing environments, ensure accurate and reliable puncture positioning, and enhance system practicality.

[0043] Instruction Manual

[0044] Figure 1 is a flow chart of a specific embodiment of the present invention;

[0045] Figure 2This is a schematic diagram of binocular stereo imaging principle in the present invention;

[0046] Figure 3 It is a schematic diagram of the dual-camera pinhole model of the present invention;

[0047] Figure 4 It is a schematic diagram of the structure of the reflective ball and the probe in the present invention;

[0048] Figure 5 It is a schematic diagram of a virtual model when the present invention is in operation;

[0049] Figure 6 It is a schematic diagram of the points of the same name in the present invention;

[0050] Figure 7 It is a schematic diagram of the surgical procedure;

[0051] Figure 8 It is a schematic diagram of the virtual path during the surgical operation;

[0052] Fig. 9 is a logic flow chart of positioning identification of a specific embodiment of the present invention;

[0053] Fig.10 This is a process of positioning and identification of a specific embodiment of the present invention;

[0054] Fig.11 It is a schematic diagram of the conversion principle of different coordinate systems in the surgical navigation system;

[0055] Fig.12 It is a virtual ball schematic diagram of a certain virtual scene design software; DETAILED DESCRIPTION

[0056] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] The present invention provides a human teaching aid puncture positioning device based on spatial positioning point technology. By integrating the laser cross positioning function of the mixed reality glasses, the problem of being unable to accurately position and provide real-time feedback using ordinary models is first solved, while reducing the high teaching cost brought by special teaching aids. It mainly includes the following steps:

[0058] Precise positioning: MR glasses are used in combination with a three-dimensional positioning method based on binocular stereo vision. Two cameras are used to shoot the same scene from different angles, and the depth information is determined by calculating the parallax between the two images. The puncture path is accurately planned to ensure that each operation can be performed according to the predetermined trajectory.

[0059] Real-time adjustment: adjust the angle and depth of the puncture path and avoid sensitive areas in real time as needed to ensure the safety and effectiveness of the operation.

[0060] Data analysis and feedback: Comprehensively analyze various data during the puncture process, including the accuracy of the puncture path, the stability of depth control, and whether there are any erroneous operations, and provide users with a comprehensive feedback report.

[0061] Error prompts and corrections: The system evaluates the puncture operation according to preset standards and provides error prompts and correction suggestions when necessary to ensure the safety and standardization of the operation process.

[0062] Please refer to Figures 2 to 3 In the process of surgical navigation teaching and guidance, the optical positioning device usually uses a binocular vision stereo system to achieve precise positioning. The system uses two left and right cameras to simultaneously capture target images with multiple markers, and obtains the three-dimensional motion information of the surgical instrument by matching and tracking the corresponding markers in each image frame. This stereo vision system imitates the way humans perceive distance with binoculars and can achieve precise positioning of target points in three-dimensional space.

[0063] The optical positioning method is to use the puncture surgery navigation system to collect optical signals generated by optical devices fixed on the patient's body surface, imaging equipment and (or) puncture surgical instruments, so as to obtain the spatial movement data of each target point, and feed these data back to the puncture surgery navigation system, and complete the navigation function by calibrating the navigation image.

[0064] Binocular stereo measurement has many similarities with "binocular navigation and positioning" in terms of principle and characteristics. Its core principle is to simulate the mechanism of human binocular vision perception of distance, observe the target object from different perspectives through two cameras, and then obtain the spatial information of the object.

[0065] During the measurement process, it is not necessary to collect all points of the object being measured, but only to accurately locate the key feature points of the part of the object being measured that is related to the measurement target. This feature allows the measurement process to focus on key information to a certain extent and reduce unnecessary data processing, but it also places high demands on the accuracy of feature point selection, because the positioning of these key feature points will directly affect the accuracy of subsequent measurement results.

[0066] This technology has high requirements for measurement speed, and usually needs to complete the measurement task of multiple products within one second. This feature is mainly due to its real-time requirements in actual application scenarios, and real-time acquisition of environmental information ensures the timeliness and accuracy of operations, and avoids decision-making errors or operational errors caused by measurement delays.

[0067] Binocular stereo measurement is highly sensitive to the lighting environment. During the measurement operation, since there must be relative displacement between the camera and the object being measured, the image captured by the camera may cover all angles. Changes in lighting conditions can easily affect the quality of the image and the accuracy of feature extraction. Uneven lighting may cause large differences in the brightness of the surface of the object in the image, thereby increasing the difficulty of feature point recognition and extraction; too strong or too dark light may cause the loss of image details, thereby affecting the accuracy of stereo matching, and ultimately having a negative impact on the accuracy of the entire measurement result.

[0068] The general implementation steps are as follows:

[0069] 1. Camera parameter calibration. Obtain the camera's intrinsic parameters and lens distortion coefficients, and obtain the stereo parameters of the dual cameras at the current angle. First, calibrate each camera in the binocular system to confirm the camera's distortion coefficients and intrinsic parameter matrix. Because different cameras and lenses have different distortions and parameters when taking images, it is necessary to confirm the parameters of the selected camera through calibration. Generally, the second-order distortion coefficients k1 and k2 intrinsic parameter matrices are used for measurement.

[0070] 2. Dual-target calibration. After obtaining the parameters of a single camera, prepare a known world coordinate system (usually a high-precision dual-target calibration plate) as a reference, and use dual cameras to shoot the calibration plate from different angles. The two cameras shoot the same object, but due to different angles, two different images will inevitably be produced. The image coordinates of the two images are calculated separately and then matched with the coordinates of the actual calibration plate to obtain the current position parameters of the dual cameras. Generally, it is described by translation vectors and rotation matrices.

[0071] At this point, the calibration process of the binocular system has been completed, and this stage can also be regarded as the completion of the initial configuration of the system at the current position. From now on, the binocular system can be used to shoot and perform subsequent processing operations on specific objects.

[0072] 3. Extract feature points within the field of view of the dual cameras. In this case, after image processing (binarization, edge extraction, denoising, etc.) of the corner points of clothing folds through effective lighting, a series of feature points in the plane coordinates X and Y in the left and right camera images are obtained.

[0073] Fourth, according to the stereo matching principle of binocular stereo vision CCAS - limit constraint formula. By bringing in the X and Y coordinates of the feature points extracted in the third step in the left and right camera fields of view, we can achieve matching of same-name points (same-name points: points on the object with the same feature points in the left and right cameras). Based on the previously calibrated translation vector and rotation matrix parameters, we can calculate the X, Y, and Z coordinates of these feature points.

[0074] 5. Model construction. By putting the three-dimensional coordinates of the feature points into the three-dimensional coordinate system, the contour model of the object can be obtained. Of course, the more feature points are extracted, the higher the accuracy of the model, and the higher the resolution requirement of the camera.

[0075] The core architecture of the present invention integrates key components such as mixed reality glasses, laser puncture positioning device, user interface and battery module. The components work together to achieve precise guidance and real-time feedback on puncture operations on human teaching aids for medical teaching, significantly improving the efficiency and accuracy of puncture skill training, and using a digitized feedback mechanism to assist students in deeply understanding the angle, depth and path control factors during the puncture process.

[0076] The mixed reality glasses include the following features:

[0077] Image acquisition: The high-definition front camera equipped with mixed reality glasses is used to capture images of the puncture area in real time. The camera can capture every detail of the puncture process at a high frame rate to ensure clear and coherent images.

[0078] Information display: The built-in display screen of the glasses can display information such as puncture position, angle and depth in real time to help users perform precise operations. The display screen uses transparent display technology, so that users can see all necessary information without shifting their sight during operation.

[0079] Data interaction: Through mixed reality technology, the glasses can interact with the user and provide intuitive visual and audio feedback. For example, when the puncture needle approaches the target location, the system can issue a sound prompt or display a guide arrow in the field of view.

[0080] The laser puncture positioning device includes the following functions:

[0081] Precise positioning: The device measures the time from the laser emission to the laser reflection back to the receiver through the laser emission and receiving modules, thereby calculating the position of the puncture needle in three-dimensional space. The laser emission module is located at the front end of the mixed reality glasses and can emit a stable laser beam, while the receiving module is responsible for capturing the reflected laser signal.

[0082] Data processing: The built-in processor can quickly process laser signals and calculate the angle and depth of the puncture needle. The processor uses a high-performance chip to ensure the real-time and accuracy of data processing.

[0083] Data transmission: Send the calculation results to the mixed reality glasses wirelessly or wired for display in the user interface. The data transmission module supports multiple protocols to ensure the security and stability of data transmission.

[0084] The tracking and positioning tools for surgical navigation mainly include positioning detection and calibration tools. The positioning probe uses an optical navigation probe: a reflective ball probe mainly based on near-infrared recognition, which uses a stereo vision system to identify the center of the marker ball in the binocular image, and then reconstructs the marker center 3D icon based on the disparity map.

[0085] Using the 4 reflective balls on the reference bracket as real landmarks, design 4 virtual balls of the same size as the reflective landmarks as virtual landmarks. Obtain the coordinates of the 4 virtual balls and the 4 real reflective balls to complete the conversion from the virtual model coordinate system to the optical tracker coordinate system. Collect the coordinates of the patient's tooth cusps and the corresponding tooth cusps on the virtual model to complete the conversion from the optical tracker coordinate system to the MR glasses coordinate system.

[0086] The user interface includes the following features:

[0087] Information display: The user interface can display key parameters such as puncture position, angle, depth, etc. in real time. The overall interface adopts an intuitive design, allowing users to see all necessary information at a glance.

[0088] Operation guide: Provide graphical operation guide to help users understand the puncture path more intuitively. For example, a virtual puncture path can be displayed on the interface, and different colors can be used to indicate different stages of operation.

[0089] Data Recording: Record the data of each operation for later analysis and improvement. The user interface can save the data of each training and allow users to export this data for further analysis.

[0090] The battery module includes the following functions:

[0091] Power supply: Provide power support for mixed reality glasses, laser puncture positioning device and user interface. The battery module uses high-efficiency batteries to ensure long-term battery life.

[0092] Portability: The portable design ensures that the device can be used anywhere. The battery module is small and light, making it easy to carry.

[0093] Charging Management: Built-in charging management system supports fast charging and intelligent power monitoring. Users can check the remaining battery power at any time through the user interface and charge when needed.

[0094] See also Figure 6 The spatial calculation principle of the present invention is based on the mechanism of simulating the binocular perception of distance by human eyes, and is realized by means of two front cameras equipped with AR glasses. The distance between the two cameras is defined as the baseline distance (B), and each camera has a specific focal length f.

[0095] At a specific moment, two cameras synchronously capture images of the same spatial object. Due to the difference in camera positions, the two images obtained show different perspectives of the object. The image acquisition device is responsible for acquiring the two images of the target area and then performing necessary image preprocessing operations, such as denoising to eliminate noise interference in the image and grayscale processing for subsequent calculations.

[0096] Then, the semi-global stereo matching (SGBM) method or other equivalent stereo matching algorithms are used to search for corresponding pairs of points (p, p') in the two images. The pairs of points here refer to the pixel positions that can represent the same physical point in the two images. By calculating the disparity between the two points (i.e., the position deviation between them in different images), the depth information of the point in three-dimensional space can be derived.

[0097] Camera calibration is the core link to ensure accurate measurement of binocular vision system. By using prefabricated 3D calibration objects or calibration templates, the internal and external parameters of the camera are obtained, including focal length, optical center position, rotation and translation matrix, etc. These parameters are indispensable for constructing the mapping relationship between the camera coordinate system and the world coordinate system.

[0098] According to the stereo matching results and camera calibration parameters, the perspective projection transformation formula is used to convert the two-dimensional image coordinates into three-dimensional space coordinates. Specifically, the three-dimensional coordinates of the target point can be calculated by the following formula:

[0099]

[0100]

[0101] Where Δx and Δy are the parallax values ​​on the image plane, f is the focal length, and x0 and y0 are the positions of the optical center;

[0102] In order to improve positioning accuracy, it is also necessary to implement error correction and optimization measures on the system. For example, a method based on epipolar constraints can be used to improve matching accuracy, or time synchronization technology can be used to ensure the synchronization of two images, so as to obtain more accurate measurement results and achieve high-precision spatial calculations.

[0103] In the context of augmented reality (AR) technology, imagine that we wear a pair of AR glasses with a special structure. The two cameras are equipped with a binocular layout, placed on the left and right sides. The distance between the two is defined as the "baseline distance" in the professional field.

[0104] When observing a target object with the glasses, the left and right cameras perform shooting actions synchronously, each generating an image. Due to the different spatial positions of the two, the images captured will show subtle differences in details, which is the "parallax" phenomenon mentioned in the professional field.

[0105] Then, it is necessary to accurately identify the pixel points representing the location of the same object in the two images. These points are named "homophones" in the academic context. After successfully locking the homophones, the disparity value between the homophones can be calculated according to the established algorithm. With the help of the disparity value and the relevant geometric distance measurement principles, the distance of the target object from the observer can be deduced, which is the depth information of the object.

[0106] In order to achieve accurate depth calculation, it is necessary to understand some key parameters of the camera, such as the focal length of the lens and the spatial coordinate position of the center of the camera. These parameters can be determined with the help of the professional technical process of "camera calibration". After obtaining the parameters, a specific coordinate mapping model can be used to map the two-dimensional pixels in the image to the three-dimensional coordinate system of the real world.

[0107] Ultimately, in order to improve the accuracy of the measurement, a series of supplementary work needs to be carried out, such as using error correction algorithms to correct possible measurement deviations and ensuring the time synchronization of the two images during the shooting and processing process, so as to reduce the measurement errors caused by time differences and ensure the accuracy and reliability of the measurement results.

[0108] See also Figure 1 The operation process includes initialization settings: wearing mixed reality glasses, starting the system, and performing spatial positioning calibration; model loading: loading the virtual model of the medical teaching human body aids in the MR glasses; puncture planning: using virtual laser lines to plan the puncture path in the model; practical operation: holding the puncture device, performing the puncture operation according to the virtual guidance, and observing the real-time feedback in the MR glasses; data analysis and feedback: after the operation is completed, the system analyzes the puncture data and provides feedback reports, including success rate, error analysis, and improvement suggestions.

[0109] Detailed description of the operation process of the human teaching aid puncture positioning method based on spatial positioning point technology:

[0110] 1. Initialization settings

[0111] Step 1.1: The user first puts on the mixed reality (MR) glasses, ensures that the glasses fit firmly on the head, and adjusts the comfort and field of view of the glasses to obtain the best visual experience.

[0112] Step 1.2: Start the system application on the MR glasses and start the spatial positioning sensor module. The system will automatically perform an initialization self-test, including checking the sensor status, network connection, etc.

[0113] Step 1.3: Perform spatial positioning calibration. The user needs to follow the system prompts and move the head and body at specific positions so that the system can accurately establish a three-dimensional spatial coordinate system and calibrate the relative position of the glasses and the surrounding environment.

[0114] Furthermore, after the user wears the MR glasses, the system can quickly perform spatial positioning calibration to ensure the accuracy of the three-dimensional spatial coordinate system, reducing the cumbersome steps and waiting time in the traditional calibration process. In addition, the design of the glasses fully considers the comfort of the wearer, and by adjusting the field of view and fit, it ensures that there will be no fatigue even after long-term use.

[0115] 2. Model loading

[0116] Step 2.1: In the system interface, the user selects or searches for the corresponding puncture course. The model related to the course can be a standard model pre-installed in the system, or a specific model customized or downloaded by the user.

[0117] Step 2.2: Click Load Model, the system will download and render the model from local storage or remote server to the display of MR glasses. Users can rotate, scale and move the model through gestures or voice control to observe it from different angles.

[0118] Furthermore, the system has built-in a variety of virtual models of human teaching aids for medical teaching. Users can quickly select or search for corresponding course models as needed without the need to prepare additional physical teaching aids. It supports users to customize or download specific models to meet the needs of different teaching scenarios, improving the flexibility and pertinence of teaching.

[0119] 3.Puncture planning

[0120] Step 3.1: Select the starting point and target point of the puncture on the model. The user can select the point by clicking the virtual button on the screen or using gesture control, and view the detailed information of the selected point through the display interface of the MR glasses.

[0121] Step 3.2: The system automatically generates a virtual puncture path based on the selected starting point and target point, and projects a laser line through the MR glasses to indicate the path. The laser line will adjust the viewing angle in real time as the user's head moves, ensuring that the user can clearly see the puncture path.

[0122] Step 3.3: The user can adjust the puncture path as needed, such as changing the angle, depth, or avoiding sensitive areas. The adjusted path will be updated in real time on the display screen of the MR glasses.

[0123] Furthermore, the virtual laser line projected by MR glasses can accurately plan the puncture path to ensure that each operation can be carried out according to the predetermined trajectory, greatly improving the accuracy of puncture. Users can adjust the angle and depth of the puncture path in real time as needed and avoid sensitive areas to ensure the safety and effectiveness of the operation.

[0124] 4. Practical operation

[0125] Step 4.1: The user picks up the puncture device and aligns it with the starting point of the puncture path displayed in the MR glasses. At this point, the physical marking point or laser receiver on the puncture device should be aligned with the virtual laser line projected by the MR glasses.

[0126] Step 4.2: Along the path indicated by the virtual laser line, the user slowly and steadily pushes the puncture device into the model. During the pushing process, the downward pressure feedback mechanism will record and transmit the downward pressure depth, angle change and moving path of the puncture needle to the data processing unit in real time.

[0127] Furthermore, the physical marking points or laser receivers on the puncture device interact with the MR glasses in real time, recording and transmitting the puncture needle's depression depth, angle changes, and movement path to the data processing unit to ensure the accuracy and real-time nature of the data. The built-in display screen of the MR glasses displays key parameters such as puncture position, angle, depth, etc. in real time, helping users to intuitively understand the operation and make adjustments in time. The system evaluates the puncture operation according to preset standards, and provides error prompts and correction suggestions when necessary to ensure the safety and standardization of the operation process.

[0128] 5. Data analysis and feedback

[0129] Step 5.1: After the puncture operation is completed, the user stops pushing the puncture device and waits for the system to complete data processing. The system will conduct a comprehensive analysis of various data during the puncture process, including the accuracy of the puncture path, the stability of depth control, and whether there is any wrong operation.

[0130] Step 5.2: The system generates a feedback report and displays it to the user through the display screen of the MR glasses. The feedback report should include the evaluation of puncture success, error analysis, and specific improvement suggestions. Users can understand their shortcomings based on the feedback report and conduct targeted exercises and improvements.

[0131] Step 5.3 (optional): Users can choose to save the feedback report for future reference or share it with teachers or classmates for discussion and exchange. At the same time, the system also supports exporting the feedback report as an electronic document or printing it out for backup.

[0132] Furthermore, the system conducts a comprehensive analysis of various data during the puncture process, including the accuracy of the puncture path, the stability of the depth control, and whether there are any wrong operations, and provides users with a comprehensive feedback report. The feedback report contains specific improvement suggestions to help users understand their shortcomings and conduct targeted practice and improvement to improve their puncture skills. Users can choose to save the feedback report for future reference or share it with teachers or classmates for discussion and exchange, so as to promote mutual learning.

[0133] Example

[0134] The human body teaching aid puncture positioning device of the present invention is mainly composed of mixed reality glasses, a laser puncture positioning device, a user interface, a battery module, a puncture device, a puncture model and the like. The specific implementation method thereof is described in detail below.

[0135] The shell of the mixed reality glasses is made of lightweight, high-strength composite materials. The high-definition front camera supports high frame rate shooting and can obtain clear and coherent images. The high-resolution display screen with transparent display technology can display information such as puncture position, angle and depth in real time. The built-in data processing module realizes real-time data exchange with the user interface and provides intuitive visual and audio feedback. Its adjustable headband and nose pads can ensure a stable fit on the faces of different users.

[0136] In the laser puncture positioning device, the laser transmitting module located at the front end of the glasses uses a stable light source to emit a laser beam with uniform brightness and good directivity. The receiving module located at the appropriate position in front of the glasses captures the reflected laser signal with a high-sensitivity receiver. The data processing module with a built-in high-performance processor quickly processes these signals to calculate the angle and depth of the puncture needle. The data transmission module supports a variety of wireless or wired transmission protocols to ensure the security and stability of data transmission.

[0137] The user interface is simple and intuitive, with large fonts and eye-catching colors to distinguish and display key parameters such as puncture position, angle, depth, etc. It also provides graphical operation guidance such as virtual puncture path and marks different operation stages with different colors to facilitate users to understand the puncture path. Its built-in data logging function can save training data and support export for further analysis, while supporting gesture control or voice commands so that users can adjust or query information during operation.

[0138] The battery module uses high-efficiency batteries to ensure long battery life and can work continuously for several hours. Its small and lightweight appearance is easy to carry. The built-in charging management system supports fast charging and intelligent power monitoring. Users can check the remaining power at any time through the interface and charge as needed.

[0139] The puncture device adopts an ergonomic design. The handle fits the curve of the palm, making it comfortable to hold and not easy to slip. The built-in sensor provides real-time dynamic feedback of the positioning tracking data and three-coordinate data of the puncture device's moving path to the mixed reality glasses. The puncture needle is made of medical stainless steel with a smooth surface that is easy to disinfect, reducing the risk of infection.

[0140] The puncture model uses soft silicone material to simulate the touch of human tissue and provide a real puncture experience. The internal design has hierarchical structures of different depths and densities to simulate the anatomical structures of different parts of the human body. The surface is equipped with positioning marks that match the virtual laser lines of the mixed reality glasses to assist in locating the puncture points.

[0141] In terms of system calibration, after the user puts on the mixed reality glasses, the system can quickly perform spatial positioning calibration to ensure the accuracy of the three-dimensional spatial coordinate system. At the same time, it fully considers wearing comfort, adjusts the field of view angle and fit to avoid fatigue caused by long-term use.

[0142] When the model is loaded, the system has built-in virtual models of a rich range of medical teaching human teaching aids. Users can quickly select or search for corresponding course models as needed. It also supports customizing or downloading specific models to meet the needs of different teaching scenarios.

[0143] The puncture planning uses virtual laser lines projected by mixed reality glasses to accurately plan the path. Users can adjust the path angle and depth in real time and avoid sensitive areas to ensure safe and effective operations.

[0144] During actual operation, the physical marking points or laser receiver on the puncture device interact with the mixed reality glasses in real time, recording and transmitting the puncture needle's pressure depth, angle change and moving path to the data processing unit. The mixed reality glasses display screen displays key parameters in real time to help users intuitively understand and adjust the operation. The system evaluates the puncture operation based on preset standards, and gives error prompts and correction suggestions when necessary to ensure standardized and safe operation.

[0145] Finally, in the data analysis and feedback phase, the system comprehensively analyzes various data of the puncture process, such as path accuracy, depth control stability, and erroneous operation conditions, etc. The feedback report contains improvement suggestions to help users practice and improve their puncture skills in a targeted manner. Users can also save the feedback report for future reference or sharing. Through the coordinated operation of the above parts, the human body teaching aid puncture positioning device of the present invention can effectively improve the user's puncture skills and ensure the training effect with the help of system guidance and feedback during equipment assembly, calibration, and actual use.

[0146] See also Figures 11 to 12 , Virtual-real registration model based on singular value decomposition (SVD): The entire mixed reality surgical navigation system includes three coordinate systems, namely: {Tr} optical tracker coordinate system, {V} virtual model coordinate system, {H} MR glasses coordinate system. The transformation relationship of the three coordinate systems is shown in Fig.11 .

[0147] The ultimate goal of virtual-reality fusion registration is to register the virtual model to the real-world object by determining the transformation relationship between the three sub-coordinate systems. The specific steps are as follows:

[0148] (1) Solve for T V Tr , that is, the transformation matrix between the virtual model coordinate system {V} and the optical tracker coordinate system {Tr}.

[0149] (2) Solve for T Tr H , that is, the transformation matrix between the optical tracker coordinate system {Tr} and the MR glasses coordinate system {H}.

[0150] (3) Solve T V H , that is, the transformation matrix T between the virtual model coordinate system {V} and the MR glasses coordinate system {H} V H , which can be expressed as:

[0151]

[0152] According to the knowledge of rigid body transformation, the transformation between different coordinate systems can be obtained by solving a set of rotation and translation matrices. If it is known that P is the point set in the tracker coordinate system and Q is the corresponding point set in the MR glasses coordinate system, then the transformation relationship between P and Q is:

[0153] Q=RP+T

[0154] If f represents the error between the source point P and the target point set Q under the action of the transformation matrices R and T, then solving the optimal transformation matrix is ​​to solve the transformation matrices R and T when the error f is minimized, that is:

[0155]

[0156] Obtaining the optimal transformation matrix can be decomposed into the following steps:

[0157] (1) Find the centroids P^ and Q^ of the two point sets

[0158]

[0159] (2) Construct the covariance matrix E and use the singular value decomposition (SVD) method to find the optimal matrices R and T

[0160]

[0161] The final matrices R and T can be expressed as:

[0162]

[0163] If the number of points in the point set is greater than 3, the solution obtained is the least squares solution.

[0164] Design of virtual-real calibration method: The virtual-real calibration method proposed in this paper is used to obtain the conversion relationship between the optical tracker coordinate system and the MR glasses coordinate system. Fig.12 Based on the four reflective balls on the reference bracket, virtual balls of exactly the same size are displayed in the virtual coordinate system of the glasses, and these virtual balls are moved in sequence to completely overlap with the small balls on the reference bracket. At this time, the position coordinates of the center of the virtual ball in the MR glasses coordinate system and the coordinates of the center of the reflective ball on the reference bracket in the optical tracker coordinate system are obtained. Through the coordinates of these two sets of corresponding points, the transformation matrix from the optical tracker coordinate system to the MR glasses coordinate system is calculated.

[0165] Import the four virtual balls into the virtual scene design software Unity, and attach the script of the grab function to the virtual balls so that the virtual balls can be grabbed and moved by hand. Several virtual buttons with different functions are also designed in the Unity scene, as shown in the figure below. The Connect and Disconnect buttons in area 1 are used to establish and disconnect the wireless connection with the server, and the Get_position button is used to send the coordinates of the virtual ball in the MR glasses coordinate system to the server. The Reset button is used to reinitialize the virtual scene. The Hide button is used to hide the four virtual balls after the alignment is completed. The buttons in area 2 are used to control the translation of the virtual ball in 6 directions. The buttons in area 3 are used to switch the controlled virtual ball.

[0166] Furthermore, the complete registration process consists of 4 steps:

[0167] (1) When registration starts, the registration scene is displayed in the MR glasses. Click the Connect button to establish a connection with the server.

[0168] (2) Grab the four virtual balls one by one and move them to roughly overlap with the corresponding reflective balls on the reference bracket.

[0169] (3) Click the button in area 2 to fine-tune the position of the virtual ball until the virtual ball and the real reflective ball completely overlap.

[0170] (4) Click the Get_position button to get the coordinates of the four virtual balls in the MR glasses coordinate system and the coordinates of the four reflective balls in the optical tracker coordinate system. Through the coordinates of these two sets of corresponding points, the transformation matrix between the optical tracker coordinate system and the MR glasses coordinate system is calculated to complete the registration of the two coordinate systems.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A human body teaching aid puncture positioning method based on spatial positioning point technology, characterized in that: The following steps are involved: Performing spatial positioning calibration through mixed reality glasses, wherein the calibration includes activating a sensor module after wearing the mixed reality glasses and establishing a three-dimensional spatial coordinate system; Load a virtual model of the human body for medical teaching into the mixed reality glasses, which can be rotated, scaled and moved through gestures or voice commands; Plan the puncture path and project it onto the puncture model in the form of a virtual laser line. The virtual laser line can dynamically adjust the viewing angle according to the user's head movement. Perform puncture operations and provide real-time feedback through mixed reality glasses, including displaying the current position, angle and depth of the puncture path, as well as detecting incorrect operations and providing prompts and correction suggestions; Collect and analyze puncture data and generate feedback reports.

2. The method for puncture positioning of human teaching aids based on spatial positioning point technology according to claim 1 is characterized in that: The puncture device and the mixed reality glasses exchange data to record the puncture depth, angle change and movement path in real time, and ensure the security and real-time nature of data transmission through a wireless transmission protocol.

3. The method for puncture positioning of human teaching aids based on spatial positioning point technology according to claim 1, characterized in that: The feedback report includes the accuracy of the puncture path, the stability of the depth control and the error analysis. The feedback report is stored in the storage unit of the mixed reality glasses, and the user can choose to save, export or print the report.

4. The method for puncture positioning of human teaching aids based on spatial positioning point technology according to claim 1, characterized in that: The puncture path planning is performed based on binocular stereo vision technology, the spatial depth is determined by calculating the parallax, and the measurement accuracy is ensured by optical calibration.

5. The method for puncture positioning of human teaching aids based on spatial positioning point technology according to claim 1, characterized in that: The puncture device has built-in sensors to provide positioning and tracking data of the puncture path to ensure the accuracy and real-time nature of the data.

6. The method for puncture positioning of human teaching aids based on spatial positioning point technology according to claim 1, characterized in that: The display screen built into the mixed reality glasses uses a transparent display, allowing the user to see the puncture path and necessary information during operation without affecting the line of sight.

7. The method for puncture positioning of human teaching aids based on spatial positioning point technology according to claim 1, characterized in that: The operation steps include the user controlling the virtual model and selecting the puncture path through gestures or voice commands after initialization settings.

8. The method for puncture positioning of human teaching aids based on spatial positioning point technology according to claim 1, characterized in that: The mixed reality glasses operate synchronously with the laser puncture positioning device to ensure real-time data feedback and path guidance, improving the accuracy and safety of the puncture operation.

9. The method for puncture positioning of human teaching aids based on spatial positioning point technology according to claim 1, characterized in that: The spatial positioning calibration of the mixed reality glasses in the initialization setting step utilizes the high-definition front camera equipped therewith, combined with visual synchronous positioning and mapping technology, to detect feature points in the environment and infer the movement trajectory of the glasses in space based on the relative position changes of these feature points, thereby establishing an accurate three-dimensional spatial coordinate system.

10. The method for puncture positioning of human teaching aids based on spatial positioning point technology according to claim 9, characterized in that: At the same time, cameras distributed on the left and right sides with a spacing equal to the baseline distance are used to simultaneously capture images of the same spatial object from different perspectives. Two images of the target area are obtained through an image acquisition device and denoised, grayed, and preprocessed. Semi-global stereo matching is used to find the same-name point pairs (p, p') in the two images, and the disparity between the same-name point pairs is calculated to infer the depth information of the object; The camera is calibrated by using a prefabricated 3D calibration object or calibration template to obtain the internal and external parameters of the camera, including focal length, optical center position, and rotation and translation matrix, which are used to establish the mapping relationship between the camera coordinate system and the world coordinate system; According to the stereo matching results and camera calibration parameters, the perspective projection transformation formula is used to convert the two-dimensional image coordinates into three-dimensional space coordinates. The formula is: zz Among them, Δx and Δy are the parallax values ​​on the image plane, f is the focal length, and x0 and y0 are the positions of the optical center; Use epipolar constraint-based methods to improve matching accuracy or use time synchronization technology to ensure the synchronization of the two images, and perform error correction and optimization on the system.

11. The method for puncture positioning of human teaching aids based on spatial positioning point technology according to claim 1, characterized in that: It also includes virtual-real fusion registration, which is used to register the virtual model to the real-world object, including the following steps: For the {Tr} optical tracker coordinate system, {V} virtual model coordinate system, and {H} MR glasses coordinate system of the mixed reality surgical navigation system, the virtual model and the real object are aligned by solving the relevant transformation matrix to determine the relationship between the coordinate systems. The transformation matrix T between the virtual model coordinate system {V} and the optical tracker coordinate system {Tr} is obtained. V T r, the transformation matrix T between the optical tracker coordinate system {Tr} and the MR glasses coordinate system {H} Tr H , and then obtain the transformation matrix T between the virtual model coordinate system {V} and the MR glasses coordinate system {H} V H =T V Tr ×T Tr H ; Based on the knowledge of rigid body transformation, the conversion of different coordinate systems is realized by solving the rotation and translation matrices. It is known that the point set P in the tracker coordinate system and the corresponding point set Q in the MR glasses coordinate system have a specific conversion relationship. Taking the error of the source point P relative to the target point set Q under the specific transformation matrices R and T as the criterion, solving the optimal transformation matrices R and T includes the following steps: First, find the centroid of each point set P and Q; Then, the covariance matrix E is constructed, and the singular value decomposition method is used to determine the optimal matrices R and T; When the number of points in the point set exceeds 3, the solution obtained is the least squares solution, which ensures the accuracy and effectiveness of the transformation matrix and realizes accurate transformation between different coordinate systems; Display the corresponding virtual ball in the virtual coordinate system of the glasses, move the virtual ball to completely overlap with the reflective ball, obtain the coordinates of the center of the virtual ball in the MR glasses coordinate system and the center of the reflective ball in the optical tracker coordinate system, and calculate the transformation matrix through the two sets of corresponding point coordinates.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the human teaching aid puncture positioning method based on spatial positioning point technology as described in any one of claims 1 to 11 are implemented.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the processor executes the program, the steps of the human teaching aid puncture positioning method based on spatial positioning point technology as described in any one of claims 1 to 11 are implemented.

Images