Method, apparatus, and computer program for aligning augmented reality objects on basis of voxel markers

Abstract

The present invention relates to a method, apparatus, and computer program for aligning augmented reality objects on the basis of voxel markers. The method for aligning augmented reality objects on the basis of voxel markers of the present invention comprises: a step of performing a CT scan of a target object to which voxel markers are attached; a segmentation step of extracting only a region of interest of the target object and the voxel markers from the CT scan image; a step of reconstructing the segmented CT scan image into a three-dimensional (3D) model; and a step of aligning voxel markers generated by the CT scan with template voxel markers by matching match a position, size, and rotation thereof.

Description

Method, device and computer program for aligning augmented reality objects based on voxel markers

[0001] The present invention relates to a method, device, and computer program for aligning augmented reality objects based on voxel markers, and more particularly, to a method, device, and computer program for aligning augmented reality objects using three-dimensional voxel markers composed of multiple materials with different densities.

[0002]

[0003] Surgery, a medical procedure involving incisions in the body to treat disease or injury, relies heavily on the surgeon's experience. Recent surgical navigation, coupled with augmented reality technology, visually guides surgical instruments to avoid critical anatomical structures and precisely target the affected area, enhancing surgical safety and precision.

[0004] To effectively utilize this surgical navigation, it is necessary to convert the anatomical structure information obtained from CT scans into a 3D model and precisely align it with the actual target object, the patient, through a registration process.

[0005] Conventional point-based registration methods using IR markers suffer from inaccurate registration due to errors in the center coordinates of the IR marker scan model, which can occur depending on CT slice spacing and imaging direction. Inaccurate registration can lead to errors in the patient's anatomical structure during surgery, increasing the risk of complications.

[0006] Even when performing registration using ArUco markers instead of IR markers, there is a problem that the ArUco markers are not detected in CT scans because they are 2D printed only on the marker surface, making the pattern invisible.

[0007] In conclusion, despite the difficulty in achieving precise alignment between scanned models and actual models due to the limitations of existing IR markers and Arco markers, no appropriate solution or proposal has yet been presented.

[0008]

[0009] The present invention was created to solve the problems of the prior art as described above, and aims to provide a method, device and computer program for aligning augmented reality objects based on voxel markers.

[0010] In addition, the present invention aims to provide a method, device, and computer program for aligning an augmented reality object using a three-dimensional voxel marker composed of multiple materials with different densities.

[0011] In addition, the present invention aims to provide a method, device and computer program for aligning augmented reality objects based on voxel markers that can provide precise alignment between a scan model and an actual model.

[0012] The technical problems to be solved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the contents described in this specification.

[0013]

[0014] In a first aspect of the present invention, a method for aligning an augmented reality object based on a voxel marker may include the steps of: CT scanning a target object to which the voxel marker is attached; segmentation step of extracting only a region of interest of the target object and the voxel marker from the CT scan image; restoring the segmented CT scan image into a 3D model; and aligning a voxel marker generated by the CT scan to match the position, size, and rotation of a template voxel marker.

[0015] Here, the voxel marker may be composed of multiple types of voxels having different densities.

[0016] Here, the region of interest may include the patient's affected area or the surgical target area.

[0017] Here, the template voxel marker may be an ideal 3D model voxel marker that reflects the position, size, and rotation of the voxel marker.

[0018] In addition, the above-described matching step may include a first matching step and a second matching step, and the first matching step may include a step of matching a voxel marker generated by a CT scan to the template voxel marker by applying a rigid body transformation matrix, and the second matching step may include a step of matching such that a brightness difference between each voxel included in the voxel marker generated by the CT scan and each voxel included in the template voxel marker is minimized.

[0019] In a second aspect of the present invention, there may be a computer program stored in a medium for executing a method of aligning an augmented reality object based on a voxel marker, in combination with hardware.

[0020] In a third aspect of the present invention, a device for aligning an augmented reality object based on a voxel marker, including a processor, may be configured to perform the following operations: CT scanning a target object to which the voxel marker is attached; performing segmentation to extract only a region of interest of the target object and the voxel marker from a CT scan image; restoring the segmented CT scan image into a 3D model; and aligning a voxel marker generated by the CT scan to a position, size, and rotation of a template voxel marker.

[0021] Here, the voxel marker may be composed of multiple types of voxels having different densities.

[0022] Here, the region of interest may include the patient's affected area or the surgical target area.

[0023] Here, the template voxel marker may be an ideal 3D model voxel marker that reflects the position, size, and rotation of the voxel marker.

[0024] In addition, the above-described matching may include a first matching and a second matching, wherein the first matching may include matching a voxel marker generated by a CT scan to the template voxel marker by applying a rigid body transformation matrix, and the second matching may include matching such that the brightness difference between each voxel included in the voxel marker generated by the CT scan and each voxel included in the template voxel marker is minimized.

[0025]

[0026] Accordingly, a method, device and computer program for aligning augmented reality objects based on voxel markers according to one embodiment of the present invention provides a method for aligning augmented reality objects using three-dimensional voxel markers composed of multiple materials with different densities.

[0027] In addition, a method, device and computer program for aligning an augmented reality object based on a voxel marker according to one embodiment of the present invention provide a method for aligning an augmented reality object based on a voxel marker that can provide precise alignment between a scan model and an actual model.

[0028] The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention pertains from the contents described in this specification.

[0029]

[0030] The accompanying drawings, which are included as part of the detailed description to aid understanding of the present invention, provide examples of the present invention and, together with the detailed description, explain the technical idea of ​​the present invention.

[0031] Figure 1 illustrates an IR marker and an infrared camera used in augmented reality object alignment based on an IR marker (Infra-Red Marker) according to a conventional technology.

[0032] Figure 2 illustrates an ArUco marker used in augmented reality object alignment based on an ArUco marker according to a conventional technology.

[0033] FIG. 3 is a flowchart illustrating a method for aligning augmented reality objects based on voxel markers according to one embodiment of the present invention.

[0034] FIG. 4 is a drawing illustrating a template voxel marker manufactured according to one embodiment of the present invention.

[0035] FIG. 5 illustrates a 3D printing filament material used to create a voxel-based marker having a density difference through a combination of materials according to one embodiment of the present invention.

[0036] FIG. 6 illustrates a 2D scan image obtained by CT scanning a target object to which a voxel-based 3D marker is attached according to one embodiment of the present invention.

[0037] FIG. 7 illustrates an image that has undergone segmentation processing to extract only the region of interest and voxel markers of a target object from a CT scan image according to one embodiment of the present invention.

[0038] FIG. 8 illustrates a 3D model generated by integrating a plurality of 2D CT scan images that have undergone segmentation processing according to one embodiment of the present invention.

[0039] FIG. 9 illustrates a flowchart for aligning voxel markers generated by a CT scan with template voxel markers according to one embodiment of the present invention.

[0040] FIG. 10 is a flowchart showing each step of configuring primary and secondary alignment for aligning voxel markers generated by a CT scan to template voxel markers according to one embodiment of the present invention.

[0041] FIG. 11 is a diagram illustrating a process of first aligning voxel markers generated by a CT scan with template voxel markers according to one embodiment of the present invention.

[0042] FIG. 12 is a drawing showing the result of first aligning voxel markers generated by a CT scan with template voxel markers according to one embodiment of the present invention.

[0043] FIG. 13 is a drawing showing the result of secondary alignment of voxel markers generated by a CT scan with template voxel markers according to one embodiment of the present invention.

[0044] Figure 14 illustrates a device to which the proposed method of the present invention can be applied.

[0045]

[0046] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. The purpose, specific advantages, and novel features of the present invention will become more apparent from the following detailed description and preferred embodiments in conjunction with the attached drawings.

[0047] Prior to this, the terms and words used in this specification and claims should be interpreted as meanings and concepts that are consistent with the technical idea of ​​the present invention and are appropriately defined by the inventor to explain his own invention in the best possible way, and are only for the purpose of explaining embodiments and should not be construed as limiting the present invention.

[0048] When assigning reference numerals to components, identical or similar components will be assigned the same reference numerals, regardless of the reference numerals, and any duplicate descriptions thereof will be omitted. The suffixes "module" and "part" used in the following descriptions are assigned or used interchangeably for the convenience of writing specifications, and do not have distinct meanings or roles in themselves, and may refer to either software or hardware components.

[0049] When describing components of the present invention, if a component is expressed in the singular, it should be understood that the component also includes the plural, unless otherwise specified. Furthermore, terms such as "first," "second," etc. are used to distinguish one component from another, and the components are not limited by these terms. Furthermore, when a component is connected to another component, it means that another component may be connected between the component and the other component.

[0050] In addition, when describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiments disclosed in this specification, the detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included within the spirit and technical scope of the present invention.

[0051]

[0052] Hereinafter, exemplary embodiments of a method, device, and computer program for aligning augmented reality objects based on voxel markers according to the present invention will be described in detail with reference to the attached drawings.

[0053] Figure 1 illustrates an IR marker (110) and an infrared camera (120) used in augmented reality object alignment based on an IR marker (Infra-Red Marker) according to a conventional technology.

[0054] The alignment using the existing IR marker is point and surface-based, and there are many cases where the alignment is inaccurate due to errors in the center coordinates of the IR marker scan model depending on the slice interval and shooting direction of CT (Computed Tomography).

[0055] When a 3D model of the patient (or part of the patient) including the IR marker is restored through a CT scan after attaching the IR marker to the patient's body, the 3D restored model is scanned based on the center of the IR marker and aligned with the template marker model (Point-based alignment).

[0056] However, since the alignment results are inaccurate with only point-based alignment, surface-based alignment is additionally performed.

[0057] A point cloud for the surface is collected by rubbing the IR marker probe over the skin. Then, additional registration is performed using Iterative Closest Point based on the collected point cloud information for the surface (surface-based registration).

[0058] However, because skin hardness is generally not very high, surface point information may not be collected properly due to deformations such as skin compression during the probe's rubbing process. Furthermore, when collecting only a limited amount of point information, the collected points may actually act as point noise when performing Iterative Closest Point, resulting in inaccurate alignment.

[0059] Figure 2 illustrates an ArUco marker used in augmented reality object alignment based on an ArUco marker according to a conventional technology.

[0060] The ARCO marker (210) is a type of binary square reference marker used in camera pose estimation, object tracking, and other applications. Each marker has a unique identification number or symbol, allowing for differentiation between markers. Unlike IR markers, which track markers within a shared field of view between cameras, ARCO markers can be tracked using just a single camera.

[0061] However, since the Arco marker is used by 2D printing on printable materials such as paper or foam (2D printing only on the surface of the material), there is a problem in that the marker pattern is not visible in the CT scan results. In other words, the Arco marker (220) with the Arco pattern printed on the surface may not have the pattern visible in the CT scan results (230).

[0062] FIG. 3 is a flowchart illustrating a method for aligning augmented reality objects based on voxel markers according to one embodiment of the present invention.

[0063] A voxel marker-based augmented reality object alignment method (300) according to one embodiment of the present invention includes a step (310) of CT scanning a target object (mainly, a patient or a patient's affected area) to which a voxel marker is attached, a segmentation step (320) of extracting only a region of interest and voxel markers of the target object from the CT scan image, a step (330) of restoring the segmented CT scan image (2D image) into a 3D model, and a step (340) of aligning a voxel marker (scan voxel marker) generated by the CT scan to match the position, size, and rotation of a template voxel marker. Here, the template voxel marker is a 3D voxel marker generated to correspond to an actually manufactured voxel marker, and is an ideal 3D model voxel marker that reflects the position, size, and rotation of the actual voxel marker.

[0064] In the step (310) of CT scanning a target object with voxel markers attached, a voxel marker is attached to at least one location of the target object while the patient corresponding to the target object is fixed motionless on the table of the CT scanner. In this state, an X-ray photograph (CT scan) of the target object is performed while the CT scanner rotates in a circular manner. As a result of the CT scan, multiple 2D CT scan images of the target object with voxel markers attached are generated. Voxel is a compound word of Volume + Pixel and is the minimum unit in a three-dimensional space used in computer science. Although it is expressed in the shape of a cube, it does not mean a physical object, but is a unit for handling three-dimensional data. Here, the physical voxel marker attached to the target object may not be divided into an actual number of cubic voxels, but since the 3D voxel marker model restored through the CT scan is composed of voxel units, the physical marker attached to the target object is referred to as a voxel marker in the description of the embodiments of the present invention.

[0065] In the segmentation step (320) of extracting only the region of interest and voxel markers of the target object from the CT scan image, only the region of interest and voxel markers of the target object are extracted from each 2D CT scan image, and the remaining portions are removed. At this time, the region of interest of the target object corresponds to an area that is worth expressing in augmented reality content, such as a patient's affected area or a surgical site. As a result, a 2D CT scan image is generated in which the region of interest of the target object and the voxel markers are extracted.

[0066] In the step (330) of restoring a segmented CT scan image (2D image) into a 3D model, a 3D model is generated by integrating a plurality of 2D CT scan images from which unnecessary or uninteresting regions (parts) have been removed by segmentation. That is, a set of 2D scan voxel markers of a plurality of 2D CT scan image sets that have undergone segmentation processing are stacked in 3D space and restored as 3D scan voxel markers, and a set of 2D target objects of a plurality of 2D CT scan image sets that have undergone segmentation processing are stacked in 3D space and restored as 3D scan target objects. The 3D scan target object can be restored as a polygonal 3D model through an algorithm such as Marching Cube (it is not limited to a polygonal 3D model, and can also be restored as a 3D voxel model). Since the process of generating a 3D image by combining a plurality of 2D CT scan images corresponds to a prior art, a detailed description thereof will be omitted.

[0067] In the step (340) of aligning the voxel marker (scan voxel marker) generated by the CT scan to the position, size, and rotation of the template voxel marker, the alignment of the scan voxel marker and the template voxel marker can be performed in two steps. The first alignment is a rough alignment in which a rigid transformation matrix is ​​applied to the scan voxel marker to align the scan voxel marker to the template voxel marker for the first time so that the positions and rotations of the scan voxel marker and the template voxel marker are aligned as similarly as possible. However, since the accuracy of the alignment is insufficient with the first alignment alone, a second alignment is additionally performed. The second alignment is an additional alignment performed to minimize the alignment error between the scan voxel marker and the template voxel marker after the first alignment. In the second alignment, the alignment is performed such that the brightness difference between each voxel included in the scan voxel marker and each voxel included in the template voxel marker is minimized. The difference in brightness between voxels refers to the difference in luminance between voxels, and the difference in brightness between voxels plays a role in representing the shape recognition or location of the target object. That is, since the brightness value of a voxel can represent the difference in density of the target object (for example, a high brightness value can represent a dense tissue such as bone, and a low brightness value can represent a low-density tissue such as soft tissue), it can be used to identify the relative location of the voxel among the target object. Ultimately, by modifying (registering) the location and angle of the scan voxel marker to the location and angle where the difference in brightness values ​​of numerous voxels included in each of the scan voxel marker and the template voxel marker is minimized, the difference in the coordinate systems between the 3D CT scan image and the template voxel marker is minimized, so that the location, size, and rotation of the 3D CT scan image and the actual object are matched as much as possible.

[0068] FIG. 4 is a drawing illustrating a template voxel marker manufactured according to one embodiment of the present invention.

[0069] The voxel marker of the present invention is configured such that its upper surface (410) and lower surface (420) have the same pattern, and even if the middle part of the marker is cut, the pattern appearing on the cut surface is also identical to the upper or lower surface pattern. That is, the voxel marker of the present invention is manufactured such that the arcuate pattern forms a columnar shape and the arcuate pattern area is composed of a material with a different density from the area excluding the pattern area. In addition, the voxel marker of the present invention can be divided into countless voxels, and the number of voxels is determined according to the resolution.

[0070] Creating 3D markers based on point clouds requires a significant computational load, as each point requires calculations. Furthermore, because point clouds have no volume and are single points, infinite magnification can create distances between points, making it difficult to accurately represent the model.

[0071] On the other hand, in the case of voxel-based 3D markers, it can be thought of as wrapping a set of point clouds of a point cloud-based model into a single voxel and assigning a single brightness value to that voxel (or assigning a separate brightness value to each 3D voxel). Since the calculation is performed per voxel rather than per point, the amount of calculation is reduced, and since the shape of the voxel is similar to that of the marker due to the shape characteristic of the hexahedron, the accuracy of marker-based registration is also higher. For example, when a voxel-based 3D marker is composed of two voxels, a white voxel and a black voxel, the white voxel can be assigned a brightness value of 1 and the black voxel can be assigned a brightness value of 0.

[0072] Taking advantage of the fact that high-density objects are represented as white and low-density objects as black on CT scans, the white portions of voxel-based 3D markers can be manufactured using high-density materials and the black portions using low-density materials (multiple densities can also be distinguished by using two or more different materials). This allows voxel-based 3D markers that reflect density differences to maintain their marker patterns on CT scans.

[0073] FIG. 5 illustrates a 3D printing filament material used to create a voxel-based marker having a density difference through a combination of materials according to one embodiment of the present invention.

[0074] In the example of Figure 5, high-density objects can be made with white filament and low-density objects can be made with black filament.

[0075] FIG. 6 illustrates a 2D scan image obtained by CT scanning a target object to which a voxel-based 3D marker is attached according to one embodiment of the present invention, FIG. 7 illustrates an image obtained by segmentation processing to extract only a region of interest and voxel markers of a target object from a CT scan image according to one embodiment of the present invention, and FIG. 8 illustrates a 3D model generated by integrating a plurality of 2D CT scan images obtained by segmentation processing according to one embodiment of the present invention.

[0076] FIG. 8(a) shows a plurality of 2D CT scan image sets (e.g., DICOM image sets) that have undergone segmentation processing, and FIG. 8(b) shows that 2D scan voxel marker sets of the plurality of 2D CT scan image sets that have undergone segmentation processing are stacked in 3D space and restored as 3D scan voxel markers, and that 2D target object sets of the plurality of 2D CT scan image sets that have undergone segmentation processing are stacked in 3D space and restored as 3D scan target objects. The 3D scan target objects can be restored as polygonal 3D models through algorithms such as Marching Cube (they are not limited to polygonal 3D models and can also be restored as 3D voxel models). Since combining a plurality of 2D CT scan images to generate a 3D image is a prior art, a detailed description thereof will be omitted.

[0077] FIG. 9 is a flowchart (including first and second alignment) for aligning voxel markers generated by a CT scan to template voxel markers according to one embodiment of the present invention, and FIG. 10 is a flowchart showing each step of configuring first and second alignment for aligning voxel markers generated by a CT scan to template voxel markers according to one embodiment of the present invention.

[0078] Scan voxel markers can be aligned with template voxel markers in two steps. The first alignment (1010) is a rough alignment in which a rigid body transformation matrix is ​​applied to the scan voxel markers to first align them with the template voxel markers (so that the positions and rotations of the scan voxel markers and the template voxel markers are expressed as similarly as possible).

[0079] However, since the first rough alignment alone is insufficient for alignment accuracy, a second alignment is additionally performed. The second alignment (1020) is an additional alignment performed to minimize the alignment error between the scan voxel markers and the template voxel markers after the first alignment.

[0080] In the secondary registration, the registration is performed so that the difference in brightness between each voxel included in the scan voxel marker and each voxel included in the template voxel marker is minimized. At this time, the difference in brightness between voxels means the difference in brightness between voxels, and the difference in brightness between voxels plays a role in indicating the shape recognition or location of the target object. In other words, since the brightness value of a voxel can indicate the difference in density of the target object (for example, a high brightness value can indicate a high-density tissue such as bone, and a low brightness value can indicate a low-density tissue such as soft tissue), it can be used to identify the relative location of the corresponding voxel among the target object. Ultimately, by comparing the differences in brightness values ​​of numerous voxels included in each of the scan voxel markers and the template voxel markers and correcting (registering) the positions and angles of the scan voxel markers to the positions and angles where the differences are minimized, the difference in coordinate systems between the scan voxel markers and the template voxel markers is minimized, thereby minimizing the difference in coordinate systems between the 3D CT scan image and the actual object (the actual object based on the template voxel markers), so that the positions and rotations of the 3D CT scan image and the actual object are as consistent as possible.

[0081] The first and second alignment processes are described in more detail with reference to Fig. 10.

[0082] In the first alignment, the average coordinates of the template voxel markers and the scan voxel markers are first calculated.

[0083]

[0084]

[0085] Here, S i (i=1, 2,...,n) are scan voxel coordinates, and T i (i=1, 2,...,n) are template voxel coordinates, n is the total number of voxels, is the scan model average coordinate, is the template marker model average coordinate. At this time, Si Wow T i can be expressed as follows.

[0086]

[0087]

[0088] Next, vector centering of the template voxel markers and scan voxel markers is performed.

[0089]

[0090]

[0091] Here, (i=1, 2,...,n) is the centered vector of scan voxel markers, (i=1, 2,...,n) is the centered vector of template voxel markers.

[0092] Next, singular value decomposition (SVD) of the template voxel markers and scan voxel markers is performed.

[0093]

[0094] Here, H is the covariance matrix and is calculated as follows.

[0095]

[0096] And UΣV T is the result of applying SVD to the covariance matrix.

[0097] At this time, U, Σ, and V can be illustrated as follows.

[0098]

[0099]

[0100]

[0101] Next, the rotation matrices of the template voxel markers and the scan voxel markers are derived.

[0102] (If det(R)<0, flip the sign of the third column of V and recalculate)

[0103] Here, R is a rotation matrix, which can be illustrated as follows.

[0104]

[0105] Next, the movement vectors of the template voxel marker and the scan voxel marker are calculated.

[0106]

[0107] Here, t is the translation vector.

[0108] Next, the rigid body transformation matrix of the template voxel marker and the scan voxel marker is derived.

[0109]

[0110] Here, X is the rigid body transformation matrix.

[0111] Next, a rigid body transformation matrix is ​​applied to the scan voxel markers to produce the first-order registration result.

[0112]

[0113] Here, S old is the scan voxel marker matrix before the first alignment, and S new is the scan voxel marker matrix after the first alignment.

[0114] In the following secondary alignment, the brightness difference between the template voxel marker and the scan voxel marker is first calculated.

[0115]

[0116] Here, E a (p) is an error function representing the brightness difference between models at arbitrary coordinates p, and I c (p) is a constant value because it is a grayscale CT image with brightness at arbitrary coordinates p.

[0117] Next, the mean square error of the brightness of the template voxel marker and the scan voxel marker is calculated.

[0118]

[0119] Here, Δp is the change in coordinate.

[0120] If the mean square error calculated by the above formula is greater than a predetermined reference value (ΔE), the coordinates of the scan voxel marker are updated to produce the secondary registration result. At this time, the predetermined reference value (ΔE) is a value that can be set according to the precision or resolution requirements (if high precision is required, the reference value can be lowered, and if low precision is tolerated, the reference value can be higher).

[0121] FIG. 11 is a diagram illustrating a process of first aligning voxel markers generated by a CT scan with template voxel markers according to one embodiment of the present invention.

[0122] The lower left corner of 1210 is a template voxel marker, and the upper right corner is a scan voxel marker. After the first alignment process described above, the eight corners of the scan voxel markers are aligned, producing a first alignment result similar to 1220.

[0123] FIG. 12 is a drawing showing the result of first-order alignment of voxel markers generated by a CT scan with template voxel markers according to one embodiment of the present invention, and FIG. 13 is a drawing showing the result of second-order alignment of voxel markers generated by a CT scan with template voxel markers according to one embodiment of the present invention.

[0124] Looking at the first registration result illustrated in Fig. 12, it can be seen that each voxel forming the scan voxel marker has its hexahedral shape displayed to some extent by the scan (appears blurry). In addition, the display color is displayed as a white part (high density part) and a black part (low density part) due to the difference in density of the voxels forming the scan voxel marker. However, it is difficult to display the shape of each voxel or the boundary between white and black to the level of the template voxel marker (1210) at the lower left of Fig. 11 due to the performance of the CT device.

[0125] When the second registration is performed based on the first registration result illustrated in Fig. 12, the position, size, and rotation (angle) of the scan voxel marker are adjusted to be closer to the template voxel marker, thereby producing a final registration result as illustrated in Fig. 13. As illustrated in Fig. 13, the second registration result is produced in which the position, size, and rotation (angle) of the ARCO pattern at the top and bottom and the cross-sectional pattern inside the scan voxel marker are adjusted to be closer to the template voxel marker than in the first registration (in which the eight corners of the scan voxel marker are aligned with the template voxel marker).

[0126]

[0127] Device to which the proposed method of the present invention can be applied

[0128] Figure 14 illustrates a device (1500) to which the proposed method of the present invention can be applied. The device (1500) may be a server or terminal that aligns augmented reality objects based on voxel markers.

[0129] Referring to FIG. 14, the device (1500) may be a server device or a terminal device configured to implement a process for aligning an augmented reality object based on a voxel marker.

[0130] For example, the device (1500) to which the proposed method of the present invention can be applied may include network devices such as repeaters, hubs, bridges, switches, routers, gateways, etc., computer devices such as desktop computers, workstations, etc., mobile terminals such as smartphones, portable devices such as laptop computers, etc., home appliances such as digital TVs, etc., and transportation means such as automobiles, etc. As another example, the device (1200) to which the present invention can be applied may be included as a part of an ASIC (Application Specific Integrated Circuit) implemented in the form of a SoC (System On Chip).

[0131] The memory (1520) can be connected to the processor (1510) when it operates, and can store programs and / or commands for processing and controlling the processor (1510), and can store data and information used in the present invention, control information required for data and information processing according to the present invention, temporary data generated during data and information processing, etc. The memory (1520) can be implemented as a storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a flash memory, an SRAM (Static RAM), an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc.

[0132] The processor (1510) may be operatively connected to the memory (1520) and the network interface (1530), and controls the operation of each module within the device (1500). In particular, the processor (1510) may perform various control functions for performing the proposed method of the present invention. The processor (1510) may also be referred to as a controller, a microcontroller, a microprocessor, a microcomputer, etc. The proposed method of the present invention may be implemented by hardware, firmware, software, or a combination thereof. When the present invention is implemented using hardware, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. configured to perform the present invention may be provided in the processor (1510). Meanwhile, when implementing the proposed method of the present invention using firmware or software, the firmware or software may include instructions related to modules, procedures, or functions that perform functions or operations necessary to implement the proposed method of the present invention, and the instructions may be stored in the memory (1520) or stored in a computer-readable recording medium (not shown) separate from the memory (1520) and, when executed by the processor (1510), the device (1500) may be configured to implement the proposed method of the present invention.

[0133] In addition, the device (1500) may include a network interface device (1530). The network interface device (1530) is connected to the processor (1510) when in operation, and the processor (1510) may control the network interface device (1530) to transmit or receive wireless / wired signals carrying information and / or data, signals, messages, etc. through a wireless / wired network. The network interface device (1530) supports various communication standards, such as, for example, IEEE 802 series, 3GPP LTE(-A), 3GPP 5G, etc., and may transmit and receive control information and / or data signals according to the communication standards. The network interface device (1530) may also be implemented outside the device (1500) as needed.

[0134]

[0135] The embodiments and drawings described herein are merely exemplary and do not limit the scope of the present invention in any way. In addition, the lines connecting or connecting members between the components depicted in the drawings are merely exemplary of functional connections and / or physical or circuit connections, and may be replaced or represented as various additional functional connections, physical connections, or circuit connections in an actual device. In addition, if there is no specific mention such as "essential" or "important," the component may not be absolutely necessary for the application of the present invention.

[0136] The use of the term "above" and similar referential terms in the specification of the present invention (especially in the claims) may refer to both singular and plural. In addition, when a range is described in the present invention, it includes inventions that apply individual values ​​belonging to the range (unless stated to the contrary), and it is the same as describing each individual value constituting the range in the detailed description of the invention. In addition, the steps presented in the method inventions of the present invention are not necessarily intended to be bound by the order of their chronological order, and the order may be appropriately changed as needed, unless a certain step must come first depending on the nature of each process. The use of all examples or exemplary terms (e.g., "for example," etc.) in the present invention is merely to describe the present invention in detail, and the scope of the present invention is not limited by the examples or exemplary terms, unless limited by the claims. In addition, those skilled in the art will understand that various modifications, combinations, and variations can be configured according to design conditions and elements within the scope of the appended claims or their equivalents.

Claims

1. A method for aligning augmented reality objects based on voxel markers, Steps for CT scanning a target object with the above voxel markers attached: A segmentation step of extracting only the region of interest and the voxel marker of the target object from the CT scan image; Step of restoring the segmented CT scan image into a 3D model; and A method comprising: a step of aligning voxel markers generated by a CT scan to the position, size, and rotation of template voxel markers.

2. In paragraph 1, A method wherein the above voxel marker is composed of multiple types of voxels having different densities.

3. In paragraph 1, A method wherein the region of interest includes the patient's affected area or the surgical site.

4. In paragraph 1, A method wherein the above template voxel marker is an ideal 3D model voxel marker that reflects the position, size, and rotation of the voxel marker.

5. In paragraph 1, The above matching step includes a first matching step and a second matching step, The above first registration step includes a step of applying a rigid body transformation matrix to the voxel marker generated by the CT scan to align it to the template voxel marker, A method wherein the second alignment step includes a step of aligning each voxel included in the voxel marker generated by the CT scan and each voxel included in the template voxel marker so as to minimize the difference in brightness.

6. A computer program stored on a medium for executing a method of aligning an augmented reality object based on a voxel marker of any one of claims 1 to 5, in combination with hardware.

7. A device including a processor and aligning augmented reality objects based on voxel markers, The above processor, CT scanning of a target object with the above voxel markers attached: Performing segmentation to extract only the region of interest and the voxel marker of the target object from the CT scan image; Reconstructing segmented CT scan images into 3D models; and A device comprising: aligning voxel markers generated by a CT scan to the position, size and rotation of template voxel markers; 8. In paragraph 7, The above voxel marker is a device composed of multiple types of voxels having different densities.

9. In paragraph 7, The device, wherein the region of interest includes the patient's affected area or the surgical site.

10. In paragraph 7, The above template voxel marker is an ideal 3D model voxel marker that reflects the position, size, and rotation of the voxel marker.

11. In paragraph 7, The above alignment includes primary alignment and secondary alignment, The above first alignment includes applying a rigid body transformation matrix to the voxel marker generated by the CT scan to align it to the template voxel marker, A device wherein the secondary alignment comprises alignment such that the brightness difference between each voxel included in the voxel marker generated by the CT scan and each voxel included in the template voxel marker is minimized.

Images