Augmented reality calibration method and device based on adaptive corner homography estimation

Abstract

An augmented reality calibration method and apparatus based on adaptive corner homography estimation, the method includes: (1) using a line segment morphology constraint algorithm to identify the 2D key corner coordinates of the tracking marker; (2) acquiring CT images and performing segmentation and three-dimensional reconstruction of craniofacial organs; (3) using a three-dimensional scanner to acquire three-dimensional models of the bidirectional tracking marker and organs; (4) extracting the 3D corners of the tracking marker in the above-mentioned scanning model based on the template of the tracking marker; (5) spatially registering the CT model and the scanning model, and aligning the 3D feature points and 2D feature points of the tracking marker based on the particle swarm optimization algorithm; (6) during the tracking process, extracting the 2D corner coordinates of the tracking marker in each frame, and using PNP homography to estimate the motion posture of the camera.

Description

Technical Field

[0001] This invention relates to the field of augmented reality technology, and more particularly to an augmented reality calibration method based on adaptive corner homography estimation, and an augmented reality calibration device based on adaptive corner homography estimation. Background Technology

[0002] Augmented reality (AR) technology is playing an increasingly important supporting role in industries, medicine, and entertainment, primarily focusing on visual expansion. In clinical medicine, AR technology can overlay the patient's three-dimensional anatomical structure onto a video stream of the real environment in real time, thereby expanding the surgeon's intraoperative field of vision, improving surgical efficiency, and ensuring surgical safety. To achieve accurate overlay of virtual and real organs, a complex spatial calibration process needs to be performed preoperatively, such as spatial registration between the AR device and the real environment, and registration between the camera and the AR device. Currently, AR technology used in medicine mainly relies on expensive optical-magnetic tracking equipment to track the patient's pose during surgery. To reduce the cost of hardware consumables, image tracking marker-based AR technology has been proposed and widely used. However, traditional image tracking markers are easily affected by occlusion during surgery, leading to tracking loss. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, the technical problem to be solved by this invention is to provide an augmented reality calibration method based on adaptive corner homography estimation, which can quickly and accurately overlay preoperative plans onto the patient's surgical area to guide the surgeon in performing the operation. It allows the camera to robustly extract key features of image markers in multi-view situations, thereby ensuring that tracking is not lost during the operation, expanding the surgeon's field of vision during the operation, and guiding the surgeon to quickly complete the operation.

[0004] The technical solution of this invention is: an augmented reality calibration method based on adaptive corner homography estimation, which includes the following steps:

[0005] (1) Use the line segment shape constraint algorithm to identify the coordinates of the 2D key corner points of the tracking marker;

[0006] (2) Acquire CT images and perform segmentation and three-dimensional reconstruction of craniofacial organs;

[0007] (3) Use a 3D scanner to obtain a 3D model of the double-sided tracking marker and the organ;

[0008] (4) Extract the 3D corner points of the tracking marks of the above scanning model based on the template of the tracking marks;

[0009] (5) Align the 3D and 2D feature points of the tracking markers based on the particle swarm optimization algorithm, and spatially register the CT model with the scanning model;

[0010] (6) During the tracking process, the 2D corner coordinates of the tracking markers on each frame are extracted, and the motion pose of the camera is estimated by using the perspective n-point algorithm pnp homography.

[0011] This invention identifies the feature point coordinates of tracking markers in a two-sided image and solves the camera pose for each frame through homography estimation to achieve accurate augmented reality overlay. Therefore, it can quickly and accurately overlay the preoperative plan onto the patient's surgical area to guide the surgeon. It allows the camera to robustly extract key features of image markers in multi-view situations, thereby ensuring that tracking is not lost during surgery, expanding the surgeon's field of vision during surgery, and guiding the surgeon to quickly complete the surgical operation.

[0012] An augmented reality calibration device based on adaptive corner homography estimation is also provided, which includes:

[0013] The recognition module is configured to use a line segment shape constraint algorithm to identify the coordinates of 2D key corner points of the tracking marker;

[0014] The reconstruction module is configured to acquire CT images and perform segmentation and three-dimensional reconstruction of craniofacial organs.

[0015] The module is configured to use a 3D scanner to acquire a 3D model of a double-sided tracking marker and an organ.

[0016] The extraction module is configured to extract the 3D corner points of the tracking markers of the above-mentioned scanned model based on the template of the tracking markers;

[0017] The alignment module is configured to align the 3D and 2D feature points of the tracking markers based on the particle swarm optimization algorithm, and to spatially register the CT model with the scan model.

[0018] The estimation module is configured to extract the 2D corner coordinates of the tracking markers in each frame during the tracking process and estimate the camera's motion pose using PNP homography. Attached Figure Description

[0019] Figure 1 This is a flowchart of the augmented reality calibration method based on adaptive corner homography estimation according to the present invention. Detailed Implementation

[0020] For example Figure 1 As shown, this augmented reality calibration method based on adaptive corner homography estimation includes the following steps:

[0021] (1) Use the line segment shape constraint algorithm to identify the coordinates of the 2D key corner points of the tracking marker;

[0022] (2) Acquire CT images and perform segmentation and three-dimensional reconstruction of craniofacial organs;

[0023] (3) Use a 3D scanner to obtain a 3D model of the double-sided tracking marker and the organ;

[0024] (4) Extract the 3D corner points of the tracking marks of the above scanning model based on the template of the tracking marks;

[0025] (5) Align the 3D and 2D feature points of the tracking markers based on the particle swarm optimization algorithm, and spatially register the CT model with the scanning model;

[0026] (6) During the tracking process, the 2D corner coordinates of the tracking markers on each frame are extracted, and the motion pose of the camera is estimated by using the perspective n-point algorithm pnp homography.

[0027] This invention identifies the feature point coordinates of tracking markers in a two-sided image and solves the camera pose for each frame through homography estimation to achieve accurate augmented reality overlay. Therefore, it can quickly and accurately overlay the preoperative plan onto the patient's surgical area to guide the surgeon. It allows the camera to robustly extract key features of image markers in multi-view situations, thereby ensuring that tracking is not lost during surgery, expanding the surgeon's field of vision during surgery, and guiding the surgeon to quickly complete the surgical operation.

[0028] Preferably, before step (1), a 3D scanner is used to scan the patient's head wearing double-sided tracking markers, and a template matching method is used to extract the key 3D corner points of the tracking markers.

[0029] Preferably, before step (1), a three-dimensional model of the head is obtained from the patient's CT images using a threshold segmentation method, and the CT three-dimensional model is registered onto the scanned model.

[0030] Preferably, in step (1), the 2D corner points of the tracking marker are quickly extracted based on the shape constraints of the line segment by using a line segment detection algorithm.

[0031] Preferably, in step (5), the 3D corner points of the tracking markers are projected onto the 2D view acquired by the camera to obtain 2D points, and distance similarity matching is performed with the 2D corner points in the original view. The particle swarm optimization algorithm is used to minimize the distance between the projected 2D corner points and the 2D corner points in the original view. After completing the 3D-2D corner point alignment of the tracking markers and the initial registration of the CT model and the scanned model, the initial augmented reality fusion effect can be obtained.

[0032] Preferably, after step (5), a three-dimensional model of the head obtained from the patient's CT images is registered onto the scanned model using a threshold segmentation method.

[0033] Preferably, in step (6), during the tracking process, for each frame, the 2D corner coordinates of its tracking marker are extracted, the transformation matrix of the current frame tracking marker relative to the camera is estimated based on the 3D-2D pnp homography, and the homography tracking matrix of the frame sequence is solved by calculating the pose matrix of the tracking markers of consecutive frames.

[0034] Preferably, this tracking matrix is ​​applied to the CT model so that the CT model can be accurately tracked.

[0035] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. When executed, the program includes the steps of the methods of the above embodiments. The storage medium can be ROM / RAM, magnetic disk, optical disk, memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes an augmented reality calibration device based on adaptive corner homography estimation. This device includes:

[0036] The recognition module is configured to use a line segment shape constraint algorithm to identify the coordinates of 2D key corner points of the tracking marker;

[0037] The reconstruction module is configured to acquire CT images and perform segmentation and three-dimensional reconstruction of craniofacial organs.

[0038] The module is configured to use a 3D scanner to acquire a 3D model of a double-sided tracking marker and an organ.

[0039] The extraction module is configured to extract the 3D corner points of the tracking markers of the above-mentioned scanned model based on the template of the tracking markers;

[0040] The alignment module is configured to align the 3D and 2D feature points of the tracking markers based on the particle swarm optimization algorithm, and to spatially register the CT model with the scan model.

[0041] The estimation module is configured to extract the 2D corner coordinates of the tracking markers in each frame during the tracking process and estimate the camera's motion pose using PNP homography.

[0042] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An augmented reality calibration method based on adaptive corner homography estimation, characterized in that: It includes the following steps: (1) Use the line segment shape constraint algorithm to identify the coordinates of the 2D key corner points of the tracking marker; (2) Acquire CT images and perform segmentation and three-dimensional reconstruction of craniofacial organs; (3) Use a 3D scanner to obtain a 3D model of the double-sided tracking marker and the organ; (4) Extract the 3D corner points of the tracking markers of the above three-dimensional model based on the template of the tracking markers; (5) Align the 3D and 2D feature points of the tracking markers based on the particle swarm optimization algorithm, and spatially register the CT model with the scanning model; (6) During the tracking process, the 2D corner coordinates of the tracking markers on each frame are extracted, and the motion pose of the camera is estimated by using the perspective n-point algorithm pnp homography. Before step (1), a 3D scanner is used to scan the patient's head wearing double-sided tracking markers, and the key 3D corner points of the tracking markers are extracted using a template matching method. Before step (1), a 3D model of the head is obtained from the patient's CT images using a threshold segmentation method, and the CT 3D model is registered onto the scanned model. In step (1), the 2D corner points of the tracking markers are quickly extracted based on the morphological constraints of line segments using a line segment detection algorithm. In step (6), during the tracking process, for each frame, the 2D corner point coordinates of the tracking markers are extracted, and the transformation matrix of the tracking markers relative to the camera in the current frame is estimated based on the 3D-2D pnp homography. The homography tracking matrix of the frame sequence is solved by calculating the pose matrix of the tracking markers in consecutive frames.

2. The augmented reality calibration method based on adaptive corner homography estimation according to claim 1, characterized in that: In step (5), the 3D corner points of the tracking markers are projected onto the 2D view captured by the camera to obtain 2D points, and distance similarity matching is performed with the 2D corner points in the original view. The particle swarm optimization algorithm is used to minimize the distance between the projected 2D corner points and the 2D corner points in the original view.

3. The augmented reality calibration method based on adaptive corner homography estimation according to claim 2, characterized in that: After step (5), a three-dimensional model of the head obtained from the patient's CT images will be registered onto the scanned model using a threshold segmentation method.

4. The augmented reality calibration method based on adaptive corner homography estimation according to claim 3, characterized in that: Applying this tracking matrix to the CT model enables precise tracking of the CT model.

5. An augmented reality calibration device based on adaptive corner homography estimation, characterized in that: It includes: The recognition module is configured to use a line segment shape constraint algorithm to identify the coordinates of 2D key corner points of the tracking marker; The reconstruction module is configured to acquire CT images and perform segmentation and three-dimensional reconstruction of craniofacial organs. The module is configured to use a 3D scanner to acquire a 3D model of a double-sided tracking marker and an organ. The extraction module is configured to extract the 3D corner points of the tracking markers from the aforementioned 3D model based on the template of the tracking markers; The alignment module is configured to align the 3D and 2D feature points of the tracking markers based on the particle swarm optimization algorithm, and to spatially register the CT model with the scan model. The estimation module is configured to extract the 2D corner coordinates of the tracking markers in each frame during the tracking process and use PNP homography to estimate the camera's motion pose. Before the recognition module, a 3D scanner is used to scan the patient's head wearing double-sided tracking markers, and the key 3D corner points of the tracking markers are extracted using a template matching method. Before the recognition module, a 3D model of the head is obtained from the patient's CT images using a threshold segmentation method, and the CT 3D model is registered onto the scanned model. In the recognition module, the 2D corner points of the tracking markers are quickly extracted based on the morphological constraints of line segments using a line segment detection algorithm. In the estimation module, during the tracking process, for each frame, the 2D corner point coordinates of the tracking markers are extracted, and the transformation matrix of the tracking markers relative to the camera in the current frame is estimated based on the 3D-2D pnp homography. The homography tracking matrix of the frame sequence is solved by calculating the pose matrix of the tracking markers in consecutive frames.

Images