Markerless AR-assisted foramen ovale puncture method and device based on facial tracking

Abstract

The present invention provides a markerless AR-assisted foramen ovale puncture method and device based on facial tracking. The method includes the following steps: in the preoperative preparation stage, obtaining medical scan data of the patient's head, generating a virtual three-dimensional model of the cranial anatomical structure based on the medical scan data, and planning the optimal puncture path to enter the foramen ovale based on the virtual three-dimensional model; in the surgical implementation stage, after the doctor completes facial tracking initialization through the head-mounted display worn by the doctor, the patient's facial features are tracked in real time by the camera to generate the real-time camera posture; the virtual three-dimensional model is rendered in real time according to the real-time camera posture, and the real-time rendered foramen ovale, Meckel's capsule model, and optimal puncture path are displayed in real time on an image of the real surgical scene, so as to be displayed in real time on the head-mounted display. The present invention not only realizes markerless automatic alignment, but also provides the doctor with a clear and intuitive puncture path, significantly improving the accuracy and safety of the surgery.

Description

Technical Field

[0001] The present invention relates to the field of medical technology, and in particular to a markerless AR-assisted foramen ovale puncture method and device based on facial tracking. Background Art

[0002] Traditional foramen ovale puncture surgery relies on the doctor's experience and touch and lacks intuitive auxiliary tools.

[0003] In recent years, the development of augmented reality technology has opened up new possibilities for medical surgery. However, existing augmented reality-assisted systems often rely on landmarks or specialized equipment, making them complex and costly to operate. Furthermore, they rely on cameras to identify and track artificial landmarks to align the virtual model with the real surgical scene. Furthermore, placing artificial landmarks in the sterile surgical environment is time-consuming and cumbersome, requiring a second imaging scan of the patient after the landmarks have been fixed, increasing surgical preparation time. Summary of the Invention

[0004] In order to solve the above problems in the prior art, the present invention provides a markerless AR-assisted foramen ovale puncture method and device based on facial tracking to achieve markerless automatic alignment.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] In a first aspect, the present invention provides a markerless AR-assisted foramen ovale puncture method based on facial tracking, comprising the steps of:

[0007] In the preoperative preparation stage, medical scan data of the patient's head is obtained, a virtual three-dimensional model of the brain anatomical structure is generated based on the medical scan data, and an optimal puncture path is planned based on the virtual three-dimensional model to enter the foramen ovale;

[0008] During the surgical procedure, after the doctor completes facial tracking initialization through the head-mounted display, the camera tracks the patient's facial features in real time and generates the camera's real-time pose.

[0009] The virtual three-dimensional model is rendered in real time according to the real-time position of the camera, and the foramen ovale, McElhone's capsule model and optimal puncture path in the real-time rendered virtual three-dimensional model are displayed in real time on the image of the real surgical scene to obtain an AR surgical image, and the AR surgical image is displayed in real time on a head-mounted display.

[0010] The beneficial effects of this invention lie in that it uses a camera to track the patient's facial features in real time to estimate the camera's position, enabling augmented reality display of the skull base anatomy, thus enabling registration without the need for artificial landmarks or external positioning devices. Furthermore, the surgeon can accurately locate the foramen ovale at the skull base directly on the patient's head using the real-time video displayed on the head-mounted display. This overcomes the drawback of existing neurosurgical navigation systems that require the surgeon to constantly switch fields of view, providing the surgeon with a clear and intuitive puncture path and significantly improving surgical accuracy and safety.

[0011] Optionally, the step of tracking the patient's facial features in real time by using a camera to generate a real-time camera pose comprises the following steps:

[0012] Track the patient's facial features in real time through the camera to obtain the current frame;

[0013] A grid-based motion statistics method matches head feature points between the current frame and the pre-loaded key frame;

[0014] Generate the real-time camera pose based on the matching results of head feature points.

[0015] Optionally, the grid-based motion statistics method for matching head feature points between the current frame and the pre-loaded key frame includes the following steps:

[0016] Use the FAST detector to detect the feature points of the current frame and the pre-loaded key frame and calculate the ORB feature of each feature point;

[0017] For each feature point in the current frame, by comparing the Hamming distance of the ORB features of the current frame and the key frame, find the nearest neighbor of each feature point in the current frame in the key frame and generate a nearest neighbor matching pair;

[0018] The current frame and key frame are divided into multiple grids respectively, and the nearest neighbor matching pairs are used as the original matching pair input of the grid-based motion statistics method to complete the matching of given feature points.

[0019] Optionally, generating a virtual three-dimensional model of the brain anatomical structure according to the medical scan data comprises the steps of:

[0020] Performing image segmentation on the medical scan data to extract four subsets of cranial anatomical structures, namely, scalp, skull base, internal carotid artery, and McBurney capsule, wherein the foramen ovale belongs to the skull base subset;

[0021] A virtual three-dimensional model of the brain anatomical structure was reconstructed based on the four extracted subsets.

[0022] Optionally, performing image segmentation on the medical scan data comprises the steps of:

[0023] According to different grayscale values ​​of the brain anatomical structure in the medical image, a method combining threshold segmentation, region growing and manual segmentation is adopted to perform image segmentation on the medical scan data.

[0024] Optionally, the facial tracking initialization is that the doctor moves the head outline projected in the field of view to align with the patient's head through the front camera of the head-mounted display to complete the facial tracking initialization.

[0025] Optionally, the method further comprises the steps of:

[0026] Receive a human-computer interaction operation instruction from a doctor, and adjust the AR surgical image according to the human-computer interaction operation instruction.

[0027] Optionally, the method further comprises the steps of:

[0028] During the surgical implementation phase, the safety of the real-time puncture path is verified through the collision detection mechanism, and important blood vessels with the possibility of interference on the real-time puncture path are fed back through Boolean operations.

[0029] Optionally, the method further comprises the steps of:

[0030] During the operation, data from the operation is collected and recorded for the doctor to evaluate the effectiveness of the operation.

[0031] In a second aspect, the present invention provides a markerless AR-assisted foramen ovale puncture device based on facial tracking, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the markerless AR-assisted foramen ovale puncture method based on facial tracking of the first aspect is implemented.

[0032] Among them, the technical effects corresponding to the markerless AR-assisted foramen ovale puncture device based on facial tracking provided by the second aspect refer to the relevant description of the markerless AR-assisted foramen ovale puncture method based on facial tracking provided by the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagram of the main process of the markerless AR-assisted foramen ovale puncture method based on facial tracking according to an embodiment of the present invention;

[0034] Figure 2 Schematic diagram of the structure of a markerless AR-assisted foramen ovale puncture device based on facial tracking according to an embodiment of the present invention.

[0035] Description of reference numerals:

[0036] 1. Markerless AR-assisted foramen ovale puncture device based on facial tracking;

[0037] 2. Processor;

[0038] 3. Memory. DETAILED DESCRIPTION

[0039] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0040] Example 1

[0041] Please refer to Figure 1 , a markerless AR-assisted foramen ovale puncture method based on facial tracking, comprising the steps of:

[0042] S1. During the preoperative preparation stage, obtain medical scan data of the patient's head, generate a virtual three-dimensional model of the brain anatomical structure based on the medical scan data, and plan the optimal puncture path to enter the foramen ovale based on the virtual three-dimensional model.

[0043] In this embodiment, generating a virtual three-dimensional model of the brain anatomical structure based on medical scan data includes the following steps:

[0044] Perform image segmentation on medical scan data to extract four subsets of cranial anatomical structures, namely the scalp, skull base, internal carotid artery, and McBurney capsule. The foramen ovale belongs to the skull base subset.

[0045] A virtual three-dimensional model of the brain anatomical structure was reconstructed based on the four extracted subsets.

[0046] In this embodiment, performing image segmentation on medical scan data includes the following steps:

[0047] According to the different grayscale values ​​of brain anatomical structures in medical images, a method combining threshold segmentation, region growing and manual segmentation is used to perform image segmentation on medical scan data.

[0048] Specifically, in this embodiment, a threshold segmentation method is first used to separate the target area from the background, based on the grayscale differences between the pixel values ​​of different targets and backgrounds. Next, a region growing algorithm is used to group pixels with similar properties into regions. This method first manually selects a seed point and then sequentially merges similar pixels around the seed point into the region where the seed pixel resides. Finally, manual segmentation is used to remove blurred, uneven, and incorrectly segmented areas of the image.

[0049] In this embodiment, a three-dimensional model reconstruction method based on surface rendering is used to perform three-dimensional modeling on the extracted area to obtain a virtual three-dimensional model of the brain anatomical structure.

[0050] Among them, the optimal puncture path is planned based on the virtual three-dimensional model to enter the foramen ovale, and puncture path planning is used to achieve this. During the planning process, it is also necessary to consider avoiding important blood vessels.

[0051] In step S1, after obtaining the virtual three-dimensional model and the optimal puncture path, they need to be packaged and exported in a unified format to facilitate subsequent AR display.

[0052] S2. During the operation, after the doctor completes the facial tracking initialization through the head-mounted display, he / she uses the camera to track the patient's facial features in real time and generates the real-time camera posture.

[0053] In this embodiment, the facial tracking is initialized by the doctor using the front camera of the head-mounted display to move the head outline projected in the field of view to align with the patient's head to complete the facial tracking initialization.

[0054] In this embodiment, the steps of tracking the patient's facial features in real time by using a camera to generate the camera's real-time posture include:

[0055] S21. Track the patient's facial features in real time through the camera to obtain the current frame.

[0056] S22. Match head feature points between the current frame and the pre-loaded key frame using a grid-based motion statistics method.

[0057] In this embodiment, the current frame represents the patient's real-time facial features, reflecting real-time changes in the patient's head. The keyframe acquisition step involves pre-selecting representative and information-rich images from the continuous image stream. These images not only record the visual information of the scene, but also extract 2D feature points from the selected images, back-project them onto the reconstructed 3D model, calculate the 3D coordinates corresponding to the 2D feature points on the selected images, and then calculate the camera pose based on the 2D-3D coordinate pairs of the 2D feature points. Keyframes refer to these selected images and their corresponding camera poses.

[0058] Therefore, the feature point matching process in step S22 is as follows:

[0059] S221 , using the FAST detector to detect feature points of the current frame and the pre-loaded key frame and calculate the ORB feature of each feature point.

[0060] The frame image includes the current frame and the key frame loaded in advance.

[0061] Among them, FAST (Features from Accelerated Segment Test) is an algorithm for quickly detecting image features. The algorithm extracts feature points by detecting corner points in the image and is widely used in real-time computer vision tasks such as video tracking and image matching.

[0062] The Oriented FAST and Rotated BRIEF (ORB) feature descriptor is a lightweight, rotation-invariant feature descriptor that generates a unique "fingerprint" for key points in an image, specifically, facial landmarks in this example. Its rotational invariance ensures that the feature descriptor remains consistent even if the patient's head rotates (for example, left or right), preventing matching failures due to changes in viewing angle.

[0063] S222 . For each feature point in the current frame, find the nearest neighbor of each feature point in the current frame in the key frame by comparing the Hamming distance of the ORB features of the current frame and the key frame, and generate a nearest neighbor matching pair.

[0064] Among them, the Hamming distance between the current frame and the key frame on the ORB feature is compared to find the nearest neighbor of each feature point in the current frame in the key frame, and generate a preliminary 2D-2D nearest neighbor matching pair.

[0065] Among them, the binary characteristics of the ORB feature allow for rapid calculation of similarity through Hamming distance, which is much faster than floating-point descriptors, ensuring real-time performance, thereby quickly finding the nearest neighbor of each feature point in the keyframe.

[0066] S223, dividing the current frame and the key frame into multiple grids respectively, and using the nearest neighbor matching pairs as original matching pair inputs of the grid-based motion statistics method to complete matching of given feature points;

[0067] In this embodiment, the number of grids G= 20*20.

[0068] Among them, the grid-based motion statistics method eliminates mismatches caused by lighting changes and local occlusions such as bloodstains and instruments through local motion consistency screening, thereby improving the robustness of pose estimation.

[0069] S23. Generate the real-time camera pose based on the matching results of the head feature points.

[0070] Therefore, in the present invention, since the 2D feature points of the current frame and the key frame have been matched, the known 3D points in the key frame and the 2D points of the current frame are used to construct a 2D-3D correspondence, and the PnP algorithm is input to solve the real-time camera pose of the current frame.

[0071] S3. Render the virtual 3D model in real time according to the real-time position of the camera, and display the foramen ovale, McElhone's capsule model, and optimal puncture path in the virtual 3D model after real-time rendering on the image of the real surgical scene to obtain an AR surgical image, and display the AR surgical image in real time on the head-mounted display.

[0072] At this point, the virtual 3D model is spatially aligned with the actual surgical scene based on the camera's real-time pose. This allows the virtual 3D model to be adjusted based on the camera's real-time pose, ensuring that the foramen ovale, McElhone's capsule model, and optimal puncture path in the virtual 3D model are aligned with the patient's head in the actual surgical scene. The surgeon can then perform the surgery based on the displayed optimal puncture path, accurately determining the location of the foramen ovale and adjacent vital blood vessels from the head-mounted display and avoiding them during surgery. This allows the virtual anatomical model and puncture path to be displayed in the correct position within the surgical scene, regardless of the viewing angle within the surgical space.

[0073] S4. Receive the doctor's human-computer interaction operation instructions and adjust the AR surgical image according to the human-computer interaction operation instructions.

[0074] Doctors can use voice and gestures to operate and adjust the puncture path or view views from different angles.

[0075] S5. During the surgical implementation phase, the safety of the real-time puncture path is verified through the collision detection mechanism, and important blood vessels with the possibility of interference on the real-time puncture path are fed back through Boolean operations.

[0076] In this embodiment, step S5 requires tracking of the surgical instrument by the surgical navigation system. Since the puncture path is linear, sampling several positions of the surgical instrument allows for 3D modeling of the surgical path. Model collision detection and Boolean operations are then used to detect interference between the puncture path and vital blood vessels.

[0077] Among them, Boolean operations and model collision detection are used to identify and avoid important anatomical structures, providing real-time feedback and verification mechanisms to ensure the accuracy and feasibility of the puncture path.

[0078] S6. During the operation, data from the operation process is collected and recorded for the doctor to evaluate the effect of the operation.

[0079] Therefore, this embodiment has the following advantages:

[0080] (1) No artificial markers or external positioning equipment are required, the patient does not need a second scan, and there is no need to calibrate and align artificial markers one by one during the operation, making the operation simple and convenient.

[0081] (2) By real-time tracking and recognition of facial features, the registration accuracy and degree of automation are high.

[0082] (3) Provide intuitive puncture path display and 3D visualization functions, providing doctors with a clear and intuitive puncture path, significantly improving the accuracy and safety of the operation.

[0083] (4) It has a security and accuracy verification mechanism.

[0084] (5) Able to record and analyze surgical data to provide doctors with evaluation basis.

[0085] Example 2

[0086] Please refer to Figure 2 The markerless AR-assisted foramen ovale puncture device 1 based on facial tracking includes a memory 3, a processor 2, and a computer program stored in the memory 3 and executable on the processor 2. When the processor 2 executes the computer program, the steps in the above-mentioned embodiment 1 are implemented.

[0087] Since the systems / devices described in the above embodiments of the present invention are systems / devices used to implement the methods of the above embodiments of the present invention, those skilled in the art will be able to understand the specific structures and variations of these systems / devices based on the methods described in the above embodiments of the present invention, and thus will not be described in detail here. All systems / devices used in the methods of the above embodiments of the present invention are within the scope of protection of the present invention.

[0088] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0089] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions.

[0090] It should be noted that, in the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims enumerating several means, several of these means may be embodied by one and the same hardware. The use of the words first, second, third etc. is for convenience only and does not indicate any order. These words may be understood as part of the component name.

[0091] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0092] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments after learning the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0093] Obviously, those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if such modifications and variations fall within the scope of the claims and their equivalents, the present invention shall also include such modifications and variations.

Claims

1. A markerless AR-assisted foramen ovale puncture device based on facial tracking, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the following steps are implemented: In the preoperative preparation stage, medical scan data of the patient's head is obtained, a virtual three-dimensional model of the brain anatomical structure is generated based on the medical scan data, and an optimal puncture path is planned based on the virtual three-dimensional model to enter the foramen ovale; During the surgical procedure, after the doctor completes facial tracking initialization through the head-mounted display, the camera tracks the patient's facial features in real time and generates the camera's real-time pose. Rendering the virtual three-dimensional model in real time according to the real-time position of the camera, and displaying the foramen ovale, McBurney capsule model, and optimal puncture path in the virtual three-dimensional model rendered in real time on an image of the real surgical scene to obtain an AR surgical image, and displaying the AR surgical image in real time on a head-mounted display; Tracking the patient's facial features in real time using a camera to generate the camera's real-time posture includes the following steps: Track the patient's facial features in real time through the camera to obtain the current frame; A grid-based motion statistics method matches head feature points between the current frame and the pre-loaded key frame; Generate the real-time camera pose based on the matching results of head feature points; The grid-based motion statistics method for matching head feature points between the current frame and the pre-loaded key frame comprises the following steps: Use the FAST detector to detect the feature points of the current frame and the pre-loaded key frame and calculate the ORB feature of each feature point; For each feature point in the current frame, by comparing the Hamming distance of the ORB features of the current frame and the key frame, find the nearest neighbor of each feature point in the current frame in the key frame and generate a nearest neighbor matching pair; The current frame and key frame are divided into multiple grids respectively, and the nearest neighbor matching pairs are used as the original matching pair input of the grid-based motion statistics method to complete the matching of given feature points.

2. The markerless AR-assisted foramen ovale puncture device based on facial tracking according to claim 1 is characterized in that: Generating a virtual three-dimensional model of the brain anatomical structure according to the medical scan data comprises the following steps: Performing image segmentation on the medical scan data to extract four subsets of cranial anatomical structures, namely, scalp, skull base, internal carotid artery, and McBurney capsule, wherein the foramen ovale belongs to the skull base subset; A virtual three-dimensional model of the brain anatomical structure was reconstructed based on the four extracted subsets.

3. The markerless AR-assisted foramen ovale puncture device based on facial tracking according to claim 2, characterized in that: The image segmentation of the medical scan data comprises the steps of: According to different grayscale values ​​of the brain anatomical structure in the medical image, a method combining threshold segmentation, region growing and manual segmentation is adopted to perform image segmentation on the medical scan data.

4. The markerless AR-assisted foramen ovale puncture device based on facial tracking according to any one of claims 1 to 3, characterized in that: The facial tracking initialization is that the doctor uses the front camera of the head-mounted display to move the head outline projected in the field of view to align it with the patient's head to complete the facial tracking initialization.

5. The markerless AR-assisted foramen ovale puncture device based on facial tracking according to any one of claims 1 to 3, characterized in that: Also includes the steps: Receive a human-computer interaction operation instruction from a doctor, and adjust the AR surgical image according to the human-computer interaction operation instruction.

6. The markerless AR-assisted foramen ovale puncture device based on facial tracking according to any one of claims 1 to 3, characterized in that: Also includes the steps: During the surgical implementation phase, the safety of the real-time puncture path is verified through the collision detection mechanism, and important blood vessels with the possibility of interference on the real-time puncture path are fed back through Boolean operations.

7. The markerless AR-assisted foramen ovale puncture device based on facial tracking according to any one of claims 1 to 3, characterized in that: Also includes the steps: During the operation, data from the operation is collected and recorded for the doctor to evaluate the effectiveness of the operation.

Images