Mark-free AR-assisted foramen ovale puncture method and device based on face tracking

Abstract

The invention provides an unmarked AR-assisted foramen ovale puncture method and device based on face tracking, and the method comprises the steps: obtaining the medical scanning data of the head of a patient at a preoperative preparation stage, generating a virtual three-dimensional model of a craniocerebral anatomical structure according to the medical scanning data, and carrying out the recognition of the virtual three-dimensional model; planning an optimal puncture path based on the virtual three-dimensional model to enter the foramen ovale; in the operation implementation stage, after a doctor completes face tracking initialization through a worn head-mounted display, face features of a patient are tracked in real time through a camera, and the real-time pose of the camera is generated; the virtual three-dimensional model is rendered in real time according to the real-time pose of the camera, and the foramen ovale, the McLeod's bursa model and the optimal puncture path which are rendered in real time are displayed on an image of a real operation scene in real time so as to be displayed on a head-mounted display in real time. According to the method, automatic registration without marks is achieved, meanwhile, a clear and visual puncture path can be provided for a doctor, and the accuracy and safety of an operation are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of medical technology, and particularly to a markerless AR-assisted foramen ovale puncture method and device based on facial tracking. Background Art

[0002] Traditional foramen ovale puncture surgery relies on doctors' experience and feel, lacking intuitive auxiliary tools.

[0003] In recent years, with the development of augmented reality technology, new possibilities have been provided for medical surgery. However, existing augmented reality-assisted systems mostly rely on markers or special devices, with complex operations and high costs, and all require the recognition and tracking of artificial markers by a camera to complete the registration between the virtual model and the real surgical scene. At the same time, placing artificial markers in the sterile environment during surgery is time-consuming and cumbersome, and secondary imaging scans of patients with fixed artificial markers are also required, increasing the preparation time of the surgery. Summary of the Invention

[0004] In order to solve the above problems of the prior art, the present invention provides a markerless AR-assisted foramen ovale puncture method and device based on facial tracking to achieve markerless automatic registration.

[0005] To achieve the above object, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a markerless AR-assisted foramen ovale puncture method based on facial tracking, including the steps of: In the preoperative preparation stage, obtain the medical scan data of the patient's head, generate a virtual three-dimensional model of the cranial anatomical structure according to the medical scan data, and plan the optimal puncture path based on the virtual three-dimensional model to enter the foramen ovale; In the surgical implementation stage, after the doctor completes the facial tracking initialization through the worn head-mounted display, the facial features of the patient are tracked in real time by the camera to generate the real-time pose of the camera; The virtual three-dimensional model is rendered in real time according to the real-time pose of the camera, and the foramen ovale, Meckel's cave model, and the optimal puncture path in the virtual three-dimensional model after real-time rendering are displayed in real time on the image of the real surgical scene to obtain an AR surgical image, and the AR surgical image is displayed in real time on the head-mounted display.

[0006] The beneficial effects of the present invention are as follows: The present invention estimates the camera pose by real-time tracking of the patient's facial features by the camera, realizes the augmented reality display of the skull base anatomical structure, and thus realizes registration without artificial markers or external positioning devices. At the same time, through the real-time video display on the head-mounted display, the doctor can directly and accurately locate the foramen ovale of the skull base on the patient's head, overcoming the defect that the existing neurosurgical navigation system requires the doctor to continuously switch the field of view, providing a clear and intuitive puncture path for the doctor, and significantly improving the accuracy and safety of the surgery.

[0007] Optionally, the real-time tracking of the patient's facial features by the camera to generate the real-time camera pose includes the steps of: Real-time tracking of the patient's facial features by the camera to obtain the current frame; Matching the head feature points between the current frame and the pre-loaded key frame by a grid-based motion statistics method; Generating the real-time camera pose according to the matching result of the head feature points.

[0008] Optionally, the matching of the head feature points between the current frame and the pre-loaded key frame by the grid-based motion statistics method includes the steps of: Using the FAST detector to detect the feature points of the current frame and the pre-loaded key frame and calculate the ORB feature of each feature point; For each feature point in the current frame, by comparing the Hamming distance of the ORB features of the current frame and the key frame, find the nearest neighbor of each feature point in the current frame in the key frame, and generate the nearest neighbor matching pair; Dividing the current frame and the key frame into multiple grids respectively, and inputting the nearest neighbor matching pair as the original matching pair of the grid-based motion statistics method to complete the matching of the given feature points.

[0009] Optionally, the generating of the virtual three-dimensional model of the cranial anatomical structure according to the medical scan data includes the steps of: Performing image segmentation on the medical scan data to extract four subsets of the cranial anatomical structure, which are the scalp, the skull base, the internal carotid artery, and the Meckel's cave, and the foramen ovale belongs to the skull base subset; Reconstructing the virtual three-dimensional model of the cranial anatomical structure according to the four extracted subsets.

[0010] Optionally, the performing of image segmentation on the medical scan data includes the steps of: According to the different gray values of the cranial anatomical structure in the medical image, adopting a method combining threshold segmentation, region growing, and manual segmentation to perform image segmentation on the medical scan data.

[0011] Optionally, the facial tracking is initialized by the doctor using the front camera of the head-mounted display to move and align the outer contour of the head projected in the field of view with the patient's head to complete the initialization of facial tracking.

[0012] Optionally, it further includes the steps of: Receiving the doctor's human-computer interaction operation instruction and adjusting the AR surgical image according to the human-computer interaction operation instruction.

[0013] Optionally, it further includes the steps of: During the surgical implementation stage, verifying the safety of the real-time puncture path through a collision detection mechanism and feeding back important blood vessels with the possibility of interference on the real-time puncture path through Boolean operations.

[0014] Optionally, it further includes the steps of: During the surgical implementation stage, collecting and recording the data during the surgical process for the doctor to evaluate the surgical effect.

[0015] In a second aspect, the present invention provides a markerless AR-assisted foramen ovale puncture device based on facial tracking, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the markerless AR-assisted foramen ovale puncture method based on facial tracking in the first aspect.

[0016] Among them, the technical effects corresponding to the markerless AR-assisted foramen ovale puncture device provided in the second aspect refer to the relevant descriptions of the markerless AR-assisted foramen ovale puncture method provided in the first aspect. Description of the Drawings

[0017] Figure 1 It is a schematic diagram of the main process of the markerless AR-assisted foramen ovale puncture method based on facial tracking according to an embodiment of the present invention; Figure 2 It is a schematic diagram of the structure of the markerless AR-assisted foramen ovale puncture device based on facial tracking according to an embodiment of the present invention.

[0018] Description of the Reference Numerals: 1. Markerless AR-assisted foramen ovale puncture device based on facial tracking; 2. Processor; 3. Memory. Detailed Embodiment

[0019] To better understand the above technical solution, the exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0020] Embodiment 1 Please refer to Figure 1 , a markerless AR-assisted foramen ovale puncture method based on facial tracking, including the steps of: S1. In the preoperative preparation stage, obtain the medical scan data of the patient's head, generate a virtual three-dimensional model of the cranial anatomical structure according to the medical scan data, and plan the optimal puncture path based on the virtual three-dimensional model to enter the foramen ovale.

[0021] In this embodiment, generating a virtual three-dimensional model of the cranial anatomical structure according to the medical scan data includes the steps of: Perform image segmentation on the medical scan data to extract four subsets of the cranial anatomical structure, which are the scalp, the cranial base, the internal carotid artery, and Meckel's cave. The foramen ovale belongs to the cranial base subset; Reconstruct a virtual three-dimensional model of the cranial anatomical structure according to the four extracted subsets.

[0022] In this embodiment, performing image segmentation on the medical scan data includes the steps of: According to the different gray values of the cranial anatomical structure in the medical image, use a method combining threshold segmentation, region growing, and manual segmentation to perform image segmentation on the medical scan data.

[0023] Specifically, in this embodiment, first use the threshold segmentation method to separate the target area and the background according to the characteristic that the pixel values of different targets or backgrounds have differences in gray scale. Then use the region growing algorithm to gather pixels with similar properties to form a region. This method first manually selects a seed point, and then sequentially merges the similar pixels around the seed point into the region where the seed pixel is located. Finally, for some fuzzy, uneven, and mis-segmented parts of the image, manual segmentation is used.

[0024] Among them, this embodiment is a three-dimensional model reconstruction method based on surface rendering to perform three-dimensional modeling on the extracted area to obtain a virtual three-dimensional model of the cranial anatomical structure.

[0025] Among them, planning the optimal puncture path based on the virtual three-dimensional model to enter the foramen ovale is realized by using puncture path planning, and it is also necessary to consider avoiding important blood vessels during the planning process.

[0026] In step S1, after obtaining the virtual three-dimensional model and the optimal puncture path, the two need to be packaged and exported in a unified format to facilitate subsequent AR display.

[0027] S2. During the surgical implementation stage, after the doctor completes the facial tracking initialization through the worn head-mounted display, the facial features of the patient are tracked in real time by the camera to generate the real-time pose of the camera.

[0028] In this embodiment, the facial tracking initialization is that the doctor uses the front camera of the head-mounted display to move the projected head outline in the field of view to align with the patient's head to complete the facial tracking initialization.

[0029] In this embodiment, the process of tracking the facial features of the patient in real time by the camera to generate the real-time pose of the camera includes the following steps: S21. Track the facial features of the patient in real time by the camera to obtain the current frame.

[0030] S22. Match the head feature points between the current frame and the pre-loaded key frames based on the grid-based motion statistics method.

[0031] In this embodiment, the current frame is the real-time facial features of the patient, reflecting the real-time changes of the patient's head. The steps to obtain the key frames are to pre-screen the representative and information-rich selected images from the continuous image stream. They not only record the visual information of the scene, but also calculate the 3D coordinates corresponding to the 2D feature points on the selected images by extracting the 2D feature points from the selected images and back-projecting them onto the reconstructed three-dimensional model, and then calculate the camera pose according to the 2D-3D coordinate values of the 2D feature points. The key frames refer to these selected images and their corresponding camera poses.

[0032] Therefore, the feature point matching process in step S22 is as follows: S221. Use the FAST detector to detect the feature points of the current frame and the pre-loaded key frames and calculate the ORB features of each feature point.

[0033] Among them, the frame images include the current frame and the pre-loaded key frames.

[0034] Among them, FAST (Features from Accelerated Segment Test) is an algorithm for quickly detecting image features. This algorithm extracts feature points by detecting the corner points in the image and is widely used in real-time computer vision tasks such as video tracking and image matching.

[0035] Among them, ORB (Oriented FAST and Rotated BRIEF) is a lightweight and rotation-invariant feature descriptor, which is used to generate a unique "fingerprint" for the key points in the image, namely the facial feature points in this embodiment. Its rotation invariance enables the feature descriptor to remain consistent even when the patient's head rotates (such as left and right deflection), avoiding matching failures caused by changes in the viewing angle.

[0036] S222. For each feature point in the current frame, by comparing the Hamming distance of the ORB features of the current frame and the key frame, find the nearest neighbor of each feature point in the current frame in the key frame, and generate the nearest neighbor matching pairs.

[0037] Among them, by comparing the Hamming distance of the ORB features between the current frame and the key frame, find the nearest neighbor of each feature point in the current frame in the key frame, and generate the preliminary 2D-2D nearest neighbor matching pairs.

[0038] Among them, the binary characteristic of the ORB feature allows the similarity to be quickly calculated through the Hamming Distance, much faster than the floating-point descriptor, ensuring real-time performance, so as to quickly find the nearest neighbor of each feature point in the key frame.

[0039] S223. Divide the current frame and the key frame into multiple grids respectively, and input the nearest neighbor matching pairs as the original matching pairs of the grid-based motion statistics method to complete the matching of the given feature points; In this embodiment, the number of grids G= 20*20.

[0040] Among them, the grid-based motion statistics method filters through local motion consistency to exclude false matches caused by changes in illumination and local occlusions such as bloodstains and instruments, improving the robustness of pose estimation.

[0041] S23. Generate the real-time pose of the camera according to the matching results of the head feature points.

[0042] Thus, in the present invention, since the 2D feature points of the current frame and the key frame have been matched, using the known 3D points in the key frame and the 2D points of the current frame, a 2D-3D correspondence is constructed, and the input PnP algorithm is used to solve the real-time pose of the camera in the current frame.

[0043] S3. Render the virtual three-dimensional model in real time according to the real-time pose of the camera, and display the foramen ovale, Meckel's cave model and the optimal puncture path in the virtual three-dimensional model after real-time rendering on the image of the real surgical scene in real time to obtain the AR surgical image, and display the AR surgical image on the head-mounted display in real time.

[0044] At this time, the spatial alignment between the virtual three-dimensional model and the real surgical scene is established according to the real-time pose of the camera, so as to adjust the virtual three-dimensional model according to the real-time pose of the camera, so that the foramen ovale, Meckel's cave model and the optimal puncture path in the virtual three-dimensional model can be aligned with the patient's head in the real surgical scene. The doctor can perform surgical operations according to the displayed optimal puncture path, accurately determine the positions of the foramen ovale and the adjacent important blood vessels from the head-mounted display, and avoid important blood vessels during the operation. In this way, no matter from which orientation in the surgical space to observe, the virtual anatomical structure model and the puncture path can be displayed at the correct position in the surgical scene.

[0045] S4. Receive the human-computer interaction operation instruction of the doctor, and adjust the AR surgical image according to the human-computer interaction operation instruction.

[0046] Among them, the doctor can adjust the puncture path or view views from different angles through voice, gesture operations.

[0047] S5. In the surgical implementation stage, verify the safety of the real-time puncture path through the collision detection mechanism, and feedback the important blood vessels with the possibility of interference on the real-time puncture path through Boolean operations.

[0048] In this embodiment, step S5 needs to be realized by the surgical navigation system to track the surgical instrument. Since the puncture path is linear, three-dimensional modeling of the surgical path can be realized after sampling several position points of the surgical instrument, and then the interference between the puncture path and the important blood vessels is detected through model collision detection and Boolean operations.

[0049] Among them, important anatomical structures are identified and avoided through Boolean operations and model collision detection, so as to provide a real-time feedback and verification mechanism to ensure the accuracy and feasibility of the puncture path.

[0050] S6. In the surgical implementation stage, collect and record the data during the operation for the doctor to evaluate the surgical effect.

[0051] Therefore, this embodiment has the following advantages: (1) No artificial markers or external positioning devices are required, the patient does not need to be scanned twice, and there is no need to calibrate and register artificial markers one by one during the operation, and the operation is simple.

[0052] (2) The registration accuracy and automation degree are high by real-time tracking and identifying facial features.

[0053] (3) Provide an intuitive puncture path display and 3D visualization function, provide a clear and intuitive puncture path for the doctor, and significantly improve the accuracy and safety of the operation.

[0054] (4) Have a safety and accuracy verification mechanism.

[0055] (5) It can record and analyze surgical data to provide an evaluation basis for doctors.

[0056] Embodiment 2 Please refer to Figure 2 , the markerless AR-assisted foramen ovale puncture device 1 based on face tracking includes a memory 3, a processor 2, and a computer program stored on the memory 3 and executable on the processor 2. When the processor 2 executes the computer program, the steps in the above-mentioned Embodiment 1 are implemented.

[0057] Since the system / device described in the above embodiments of the present invention is the system / device adopted for implementing the method in the above embodiments of the present invention, based on the method described in the above embodiments of the present invention, those skilled in the art can understand the specific structure and variations of the system / device, so it will not be elaborated here. Any system / device adopted by the method in the above embodiments of the present invention falls within the scope of protection of the present invention.

[0058] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a device, or a computer program product. Therefore, the present invention can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0059] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (apparatuses), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowchart and / or block diagram can also be implemented.

[0060] It should be noted that in the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The present invention can be implemented by means of hardware including several different components and by means of a properly programmed computer. In the claims listing several devices, several of these devices can be embodied by the same piece of hardware. The use of the words first, second, third, etc. is only for convenience of description and does not indicate any order. These words can be understood as part of the component name.

[0061] In addition, it should be noted that in the description of this specification, the descriptions of terms such as "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" mean that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0062] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications after learning the basic creative concepts. Therefore, the claims should be construed to include the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0063] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and its equivalent technologies, the present invention should also include these modifications and variations.

Claims

1. A markerless AR-assisted foramen ovale puncture method based on facial tracking, characterized in that, Including the steps: In the preoperative preparation stage, obtain the medical scan data of the patient's head, generate a virtual three-dimensional model of the cranial anatomical structure based on the medical scan data, and plan the optimal puncture path based on the virtual three-dimensional model to enter the foramen ovale; In the surgical implementation stage, after the doctor completes the facial tracking initialization through the worn head-mounted display, the facial features of the patient are tracked in real time by the camera to generate the real-time pose of the camera; Render the virtual three-dimensional model in real time according to the real-time pose of the camera, and display the foramen ovale, Meckel's cave model and the optimal puncture path in the virtual three-dimensional model after real-time rendering on the image of the real surgical scene in real time to obtain the AR surgical image, and display the AR surgical image on the head-mounted display in real time.

2. The markerless AR-assisted foramen ovale puncture method based on facial tracking according to claim 1, wherein The step of tracking the facial features of the patient in real time by the camera to generate the real-time pose of the camera includes the steps: Track the facial features of the patient in real time by the camera to obtain the current frame; Match the head feature points between the current frame and the key frames pre-loaded based on the grid-based motion statistics method; Generate the real-time pose of the camera according to the matching result of the head feature points.

3. The method for markerless AR-assisted foramen ovale puncture based on facial tracking according to claim 2, wherein The step of matching the head feature points between the current frame and the key frames pre-loaded based on the grid-based motion statistics method includes the steps: Use the FAST detector to detect the feature points of the current frame and the pre-loaded key frames and calculate the ORB features of each feature point; For each feature point in the current frame, find the nearest neighbor of each feature point in the current frame in the key frame by comparing the Hamming distances of the ORB features of the current frame and the key frame, and generate the nearest neighbor matching pairs; Divide the current frame and the key frames into multiple grids respectively, and use the nearest neighbor matching pairs as the original matching pairs of the grid-based motion statistics method to complete the matching of the given feature points.

4. The method for markerless AR-assisted foramen ovale puncture based on facial tracking according to claim 1, wherein The step of generating the virtual three-dimensional model of the cranial anatomical structure based on the medical scan data includes the steps: Perform image segmentation on the medical scan data to extract four subsets of the cranial anatomical structure, which are the scalp, skull base, internal carotid artery and Meckel's cave, and the foramen ovale belongs to the skull base subset; Reconstruct the virtual three-dimensional model of the cranial anatomical structure according to the four subsets extracted.

5. The method for markerless AR-assisted puncture of the foramen ovale based on facial tracking according to claim 4, wherein The step of performing image segmentation on the medical scan data includes the steps: According to the different gray values of the cranial anatomical structure on the medical image, use a method combining threshold segmentation, region growing and manual segmentation to perform image segmentation on the medical scan data.

6. The method for markerless AR-assisted foramen ovale puncture based on facial tracking according to any one of claims 1 to 5, characterized in that, The facial tracking initialization is that the doctor uses the front camera of the head-mounted display to move and align the projected outer contour of the head in the field of view with the patient's head to complete the facial tracking initialization.

7. The method for markerless AR-assisted foramen ovale puncture based on facial tracking according to any one of claims 1 to 5, characterized in that It also includes the steps: Receive the human-computer interaction operation instruction of the doctor, and adjust the AR surgical image according to the human-computer interaction operation instruction.

8. The markerless AR-assisted foramen ovale puncture method based on facial tracking according to any one of claims 1 to 5, characterized in that It also includes the steps: In the surgical implementation stage, verify the safety of the real-time puncture path through the collision detection mechanism, and feedback the important blood vessels with the possibility of interference on the real-time puncture path through Boolean operation.

9. The markerless AR-assisted foramen ovale puncture method based on face tracking according to any one of claims 1 to 5, characterized in that, It also includes the steps: In the surgical implementation stage, collect and record the data during the operation for the doctor to evaluate the surgical effect.

10. A markerless AR-assisted patent foramen ovale puncture device based on facial tracking, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for markerless AR-assisted foramen ovale puncture based on facial tracking according to any one of claims 1 to 9.

Images