Method and apparatus for registering in vivo medical images with an anatomical model

Abstract

This document describes a method for registering a medical image of a subject with a 3D model of the subject, which includes globally calibrating the 3D model by aligning markers on the subject with corresponding markers on the 3D model; and locally calibrating the 3D model by aligning a scanned image of the internal structure of the subject with the corresponding internal structure of the 3D model. This document also describes an apparatus for performing this method.

Description

Background Art

[0001] Medical imaging involves techniques and processes for creating a visual representation of the interior of a living body, such as a patient. The visual representation, commonly referred to as a "medical image", reveals the operation or function of the organs, tissues, or structures of the living body that cannot be observed from outside the living body. Medical practitioners, such as doctors or veterinarians, can use the visual representation as part of medical diagnosis or clinical analysis and then determine whether or how to apply medical intervention or treatment to the living body. Brief Description of the Drawings

[0002] Aspects of the present disclosure are best understood from the following detailed description when read in conjunction with the accompanying drawings. It should be noted that, in accordance with standard practice in the industry, the various features are not drawn to scale. In fact, for clarity of discussion, the dimensions of the various features may be arbitrarily increased or decreased:

[0003] Figure 1A is a flowchart and schematic diagram of a medical image registration method according to some embodiments;

[0004] Figure 1B is a medical image of a subject with a 3D model of the subject according to some embodiments;

[0005] Figure 2 is a flowchart of a method for selecting a 3D model representing the subject from a 3D model database according to some embodiments;

[0006] Figure 3A is a flowchart of a method for globally calibrating a 3D model by aligning markers on a subject according to some embodiments;

[0007] Figure 3B is a medical image of a subject with markers and corresponding markers on the 3D model according to some embodiments;

[0008] Figure 4A is a flowchart of a method for locally calibrating a 3D model by aligning a scanned image of an internal structure of a subject with a corresponding internal structure of the 3D model according to some embodiments;

[0009] Figure 4B is a medical image, a scanned image, and a standard plane of a subject according to some embodiments;

[0010] Figure 4C is a flowchart of a method for locally calibrating a 3D model by aligning a scanned image of an internal structure of a subject with a corresponding internal structure of the 3D model according to some embodiments;

[0011] Figure 4D is a medical image and a 3D model of a subject according to some embodiments;

[0012] Figure 5 It is a diagram of a device for registering a medical image of a subject with a 3D model of the subject according to some embodiments. Detailed implementation manners

[0013] The detailed description of the present disclosure is mainly presented in terms of processes, steps, logical blocks, processing, or other symbolic representations that are directly or indirectly analogous to the operations of a device or system contemplated in the present disclosure. Those skilled in the art typically use these descriptions and statements to most effectively convey the substance of their work to other technicians in the art.

[0014] The mention of "one embodiment" or "some embodiments" herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present disclosure. The phrase "in one embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. In addition, the order of the blocks in the process flow chart or diagram or the use of the serial numbers representing one or more embodiments of the present disclosure does not inherently indicate any particular order, nor does it imply any limitation to the present disclosure.

[0015] In some embodiments, the present specification is directed to methods and devices for registering a medical image of a subject with an anatomical model of a patient. In some embodiments, the methods and devices are implemented in various clinical medicine or diagnostic applications including ultrasound scans.

[0016] In some embodiments, the subject is a human. In some embodiments, the subject is an animal. For simplicity, the present specification will describe the methods of the device with reference to human subjects. However, those of ordinary skill in the art will understand that the methods or devices are applicable to animal subjects or other subjects.

[0017] One of the main challenges in medical imaging lies in the non-intuitiveness of the visual representation. This non-intuitive nature makes it more difficult to correctly interpret medical images. In some cases, extensive medical training over many years is required before a practitioner can interpret or otherwise understand a medical image with satisfactory accuracy and detail.

[0018] In some applications, a medical image constitutes a two-dimensional (2D) cross-section of a patient's body anatomy, rather than a three-dimensional (3D) replica of the actual body object being examined, which is an organ, tissue, or structure. Therefore, it is not easy to establish a correlation between the 2D medical image and the anatomical structure of the body object. In other words, it is not easy to identify which anatomical cross-section of the 3D body object the 2D medical image represents and which internal structures are shown in the anatomical cross-section.

[0019] Therefore, finding a more intuitive way to relate 2D medical images to the 3D body object being examined will contribute to accurate medical assessment.

[0020] Method for registering a medical image of a subject with a 3D model

[0021] Reference Figure 1A and Figure 1B In some embodiments, this specification is directed to a method for registering a medical image of a subject with a 3D model of the subject.

[0022] In some embodiments, the method includes selecting a 3D model representing the subject from a 3D model database (step 100), calibrating the 3D model by aligning markers on the subject with corresponding markers on the 3D model (step 200); and calibrating the 3D model by aligning a scanned image of the internal structure of the subject with the corresponding internal structure of the 3D model (step 300). As used herein, the phrase "internal structure" refers to the body structure of the subject that is not exposed on the outer surface of the subject. Internal structures include organs, subcutaneous tissues, and the like.

[0023] In some embodiments, step 100 can be omitted, and the 3D model of the subject used in step 200 or 300 is a general or universal model that is equally applicable to various subjects of different genders, heights, weights, races, ages, etc. According to these embodiments, entries of subject-specific information are not used to register the medical image with the 3D model. However, due to the variations in the external and internal dimensions of human individuals, selecting a 3D model from the 3D model database based on subject-specific information will obtain a 3D model with dimensions closer to the subject, thereby making the calibration steps 200 and 300 (described in detail below) more accurate and less computationally intensive.

[0024] A large number of 3D human models are available for the gaming and animation industries. Although many 3D human models in the gaming and animation industries do not meet medical-grade requirements, some models are capable of meeting these requirements. Additionally, as detailed below, 3D models can be calibrated to better represent the external and internal dimensions of the subject. The calibrated 3D model can be stored and used as the initial uncalibrated 3D model for a second subject, and calibration is performed for the second subject. Therefore, in some embodiments, the 3D model database includes 3D human models from the gaming or animation industries.

[0025] 3D models for anatomical research generally have better quality than 3D human models available in the gaming and animation industries. Additionally, these 3D models typically include the internal structures of the human body, such as organs. Thus, in some embodiments, the 3D model database includes 3D models for anatomical research. In some embodiments, the 3D model database includes 3D models from the Visible Human Project, which are reconstructed from pictures of cadaver slices. Since the 3D models from the Visible Human Project models are constructed from real humans, the quality of these 3D models is good, at least because of the high resolution and realistic anatomical features.

[0026] Reference Figure 2 , in some embodiments, selecting a 3D model representative of a subject from the 3D model database (step 100) includes selecting the 3D model based on information about the subject.

[0027] In some embodiments, the 3D model database includes 3D model entries representative of a subject's body. In some embodiments, the entries of the 3D model include not only the external shape of the subject but also the internal structure of the subject. In some embodiments, each entry of the 3D model is labeled with information related to the entry, such as external dimensions or dimensions of the internal structure.

[0028] In some embodiments, selecting a 3D model representative of a subject from the 3D model database (step 100) includes inputting information about the subject (step 101).

[0029] In some embodiments, the information about the subject is input into the memory of a computer. In some embodiments, the information includes demographic information of the subject, such as the subject's gender, height, weight, race, age, etc. In some embodiments, the information includes personal information of the subject, such as body measurements, body fat percentage (BFP), body mass index (BMI), etc. In some embodiments, it includes both demographic information and personal information.

[0030] If the demographic information or personal information of the subject is sufficient, the external dimensions of the subject's body can be estimated, and even the dimensions of internal structures such as organs of the subject can be estimated.

[0031] Thus, in some embodiments, selecting a 3D model representative of a subject from the 3D model database (step 100) further includes determining whether the input information is sufficient (step 103). In some embodiments, this determination is made by a processor of the computer. In some embodiments, the processor assigns a value to each item of the input and calculates the total value. If the total value is greater than or equal to a predetermined value, the processor determines that the information input in step 101 is sufficient to select a 3D model; if the total value is less than the predetermined value, the processor determines that the information input in step 101 is insufficient.

[0032] In some embodiments, when step 103 determines that the information input in step 101 is insufficient, selecting a 3D model representative of the subject from the 3D model database (step 100) further includes scanning the subject with a 3D scanner (step 105).

[0033] In some embodiments, the 3D scanner includes a smartphone having a camera and installed with a 3D scanning application, a professional 3D scanner such as a medical 3D scanner, or a computed tomography scanner (also known as a "computerized axial tomography scanner", "CT scanner" or "CAT scanner", hereinafter referred to as "CT scanner"). A smartphone is an inexpensive and easily accessible option, but only allows a rough estimate of the external dimensions of the subject. Professional 3D scanners are more expensive, but allow a fairly accurate estimate of the external dimensions of the subject. CT scanners are generally the most expensive and least accessible, and result in radiation to the subject, but can very accurately estimate the external dimensions of the subject and the dimensions of internal organs. Therefore, the cost and benefits of a particular type of scanner should be considered when selecting a 3D scanner. After the 3D scanning step 105, continue to select the model that best represents the subject from the database (step 107).

[0034] In some embodiments, when step 103 determines that the information input in step 101 is sufficient, continue to select the model that best represents the subject from the database (step 107) without performing step 105.

[0035] In step 107, the dimensions of the subject (including external dimensions or dimensions of internal organs) can be estimated based on the demographic / personal information of the subject input in step 101, or the dimensions of the subject can be estimated based on the results of the 3D scan from step 105. In some embodiments, the estimation of the dimensions is performed by a processor. In some embodiments, the estimated dimensions match the dimensions of the 3D model stored in the database. In some embodiments, the 3D models stored in the database are labeled with relevant demographic / personal information, and the matching is based directly on the demographic / personal information; according to these embodiments, the estimation of the subject's dimensions can be omitted or not. In some embodiments, the matching is performed by a processor. In some embodiments, the processor selects from the database a 3D model having dimensions that best match the estimated dimensions.

[0036] Reference Figure 3A and Figure 3B , in some embodiments, the 3D model is calibrated (step 200) by aligning markers on the subject with corresponding markers on the 3D model. The use of fiducial markers is performed using fiducial markers. The use of fiducial markers is described in U.S. Application No. 15 / 610,127, the entire content of which is incorporated herein by reference.

[0037] In some embodiments, calibrating the 3D model (step 200) by aligning markers on a subject with corresponding markers on the 3D model includes calibrating between a physical coordinate system and a virtual coordinate system (step 201). See Figure 3B , as used herein, the term "physical coordinate system" refers to a coordinate system that represents the real-world space in which the subject is located (such as Figure 3B represented by a coordinate system having axes X, Y, and Z), and the term "virtual coordinate system" refers to a coordinate system of the virtual space in which the 3D model is located (such as Figure 3B represented by a coordinate system having axes X′, Y′, and Z′).

[0038] In some embodiments, calibrating between a physical coordinate system and a virtual coordinate system (step 201) includes moving a probe attached to or including a position sensor to one or more real-world space positions corresponding to positions in the virtual space, and determining the real-world coordinates of the position sensor via a navigation sensor. In some embodiments, the position sensor and the navigation sensor are part of a tracking system. In some embodiments, the tracking system is a GPS tracking system, an optical tracking system, or an electromagnetic tracking system. In some embodiments, the tracking system is a three-degree-of-freedom (3DOF) tracking system that tracks rotation (θX, θY, and θZ) about the X, Y, and Z axes. In some embodiments, the tracking system is a six-degree-of-freedom (6DOF) tracking system that tracks position and rotation about the X, Y, and Z axes. One of ordinary skill in the art will appreciate that 3DOF tracking systems are relatively inexpensive but less accurate; 6DOF tracking systems are more expensive but allow tracking of the position and orientation of the position sensor and the probe.

[0039] In some embodiments, calibrating the 3D model (step 200) by aligning markers on a subject with corresponding markers on the 3D model includes selecting markers on the 3D model that correspond to fiducial markers on the subject (step 203).

[0040] In some embodiments, globally calibrating the 3D model (step 200) by aligning markers on a subject with corresponding markers on the 3D model further includes moving the probe to the fiducial marker to record the position of the probe, thereby obtaining the real-world position of the fiducial marker (step 205). In some embodiments, recording the position of the probe can be performed in a manner similar to those described above using a tracking system that includes a position sensor or a navigation sensor.

[0041] Although those of ordinary skill in the art would expect that the more fiducial markers used in calibration, the more accurate the calibration result would be, the present inventors have found that a larger number of fiducial markers generally do not necessarily translate into better calibration results. Thus, in some embodiments, the number of fiducial markers ranges from 1 to 10, such as 2 to 8, such as 2 to 6, such as 3 to 5. The choice of the number of fiducial markers depends on the body part to be calibrated. For example, in some embodiments, when performing an ultrasound scan only on the abdomen of a subject, only the fiducial markers in or near the abdomen are used, because precise calibration is only required in the abdominal region. Similarly, in some embodiments, if an ultrasound scan is to be performed only on the brain of a subject, only the fiducial markers on or near the head are used.

[0042] Since it may take some time to move the probe to the fiducial marker to record the position of the probe, and during this period the subject may not be able to maintain the position, in some embodiments, step 205 includes deriving the relative position of the fiducial markers on the subject. According to these embodiments, a reference sensor is attached to the subject, and the position of the fiducial markers relative to the reference sensor is derived by the processor using the inputs of the position sensor, the navigation sensor, and the reference sensor. The reference sensor works well when calibrating the torso region of the subject because the exterior of the torso can be considered rigid. Those of ordinary skill in the art will understand that the reference sensor can be used in a similar manner as described in the local calibration step 300 herein.

[0043] In some embodiments, the global calibration 3D model (step 200) further includes calculating a transformation matrix between the outer surface of the subject and the 3D model based on the correlation between the real-world positions of the fiducial markers and the virtual positions of the corresponding markers (step 207). In some embodiments, the transformation matrix is a rigid transformation matrix or an affine transformation matrix. In some embodiments, the transformation matrix is a matrix capable of transforming any position back and forth between two coordinate systems. In some embodiments, the transformation matrix is a matrix capable of handling translation, rotation, and scaling between two sets of marker coordinates. In some embodiments, calculating the transformation matrix (step 207) can be performed using an algorithm capable of calculating uniform scaling across all three directions or an algorithm capable of giving different scale factors in different directions. The present inventors have found that an algorithm capable of giving different scale factors in different directions can better address the size differences between the 3D model and the subject to a certain extent.

[0044] In some embodiments, the global calibration 3D model further includes globally calibrating the 3D model by applying the transformation matrix to the 3D model (step 209).

[0045] Reference Figure 4A 、 Figure 4B 、 Figure 4C and Figure 4D, in some embodiments, a method of registering a medical image of a subject with a 3D model includes locally calibrating the 3D model by aligning a scanned image of the internal structure of the subject with the corresponding internal structure of the 3D model (step 300).

[0046] The present inventors have found that while global calibration or local calibration sometimes produces useful results, the combination of both global calibration and local calibration has a synergistic effect. Taking the torso as an example, global calibration addresses the differences in the orientation, position, and scale of the torso between the subject and the 3D model. The relative positions and orientations of structures within the torso, such as the liver, may still vary significantly between different subjects. Local calibration addresses such relative differences within the torso. In other words, global calibration and local calibration, when both are performed, produce results that are far more satisfactory than either global calibration alone or local calibration alone.

[0047] Although the global calibration step 200 can calibrate the position, orientation, and scale of the 3D model to address the differences between the 3D model and the subject, step 200 only calibrates the external dimensions of the 3D model. However, those of ordinary skill in the art can understand that for different subjects, the positions and orientations of the internal structures often vary. Since the internal structures are not exposed, it is not feasible to place sensors on the internal structures. This makes it impossible to use fiducial markers as described in step 200. To address such differences in the internal structures, a local calibration step 300 is performed to align specific internal structures between the 3D model and the subject.

[0048] Although the internal structures of different subjects may have different positions and orientations, the overall shapes of the internal structures are similar. Thus, the local calibration step 300 can be performed as an association between the scanned image of the internal structure and a standard plane (step 310) (see Figure 4A and Figure 4B ), or as an association between the reconstructed 3D model of the internal structure of the subject and the internal structure of the 3D model (step 320) (see Figure 4C and Figure 4D ).

[0049] Refer to Figure 4A and Figure 4B, in some embodiments, performing step 300 (step 310) by correlating a scanned image of the internal structure with a standard plane includes: selecting a standard plane including an identifiable internal structure in the 3D model (step 311); obtaining a scanned image of the subject showing an internal structure similar to the standard plane (step 313); calculating a transformation matrix between the internal structure of the subject and the internal structure of the 3D model based on the correlation between the standard plane and the scanned image (step 315); and locally calibrating the 3D model by applying the transformation matrix to the internal structure of the 3D model (step 317).

[0050] In some embodiments, selecting a standard plane including an identifiable internal structure in the 3D model (step 311) includes selecting a standard plane showing a unique cross-section of the internal structure. The inventors have found that for many organs, a unique cross-sectional standard plane is sufficient for local calibration of the organ.

[0051] In some embodiments, obtaining a scanned image of the subject showing an internal structure similar to the standard plane (step 313) includes displaying a virtual probe in a virtual coordinate system and indicating on which scan plane the scanned image was taken. According to these embodiments, the placement of the probe by the operator can be simplified when obtaining the scanned image.

[0052] In some embodiments, obtaining a scanned image of the subject showing an internal structure similar to the standard plane (step 313) includes moving the probe until a scanned image as close as possible to the standard plane is obtained. In some embodiments, step 313 includes the processor calculating a similarity index between the scanned image and the image of the standard plane in the 3D model; when the similarity index is equal to or greater than a predetermined value, the processor selects the scanned image. In some embodiments, the processor selects the scanned image with the maximum similarity index among a plurality of scanned images taken by the probe.

[0053] In some embodiments, calculating the similarity index includes the processor converting the image of the standard plane and the scanned image into templates using a boundary detection algorithm; and the processor calculating the similarity index by performing affine invariant template matching of the templates.

[0054] In some embodiments, the calculation of the transformation matrix in step 315 is the same as or similar to those in step 207 above.

[0055] The above steps 200 and 310 can address the differences in position, orientation, and scale between the internal structure of the subject and the internal structure of the 3D model. In addition to position, orientation, and scale, the shape of the same organ can also be different between two different subjects. To address the shape differences, the matrix can be calculated based on the correlation between the reconstructed 3D model of the internal structure of the subject and the internal structure of the 3D model (step 320).

[0056] Referring to Figure 4C and Figure 4D , in some embodiments, performing step 300 (step 320) by correlating a reconstructed 3D model of the internal structure of a subject with the internal structure of the 3D model includes: constructing, by a processor, a 3D volume of a scanned image of the internal structure of the subject (step 321); extracting, by the processor, a 3D model of the internal structure from the 3D volume of the scanned image (step 323); calculating, by the processor, a transformation matrix between the internal structure of the subject and the internal structure of the 3D model based on a correlation between the shape of the extracted 3D model of the internal structure and the shape of the internal structure of the 3D model (step 325); and locally calibrating, by the processor, the 3D model by applying the transformation matrix to the internal structure of the 3D model (step 327).

[0057] In some embodiments, extracting the 3D model of the internal structure (step 321) includes extracting the 3D model of the internal structure from the 3D volume of the scanned image by a segmentation algorithm.

[0058] In some embodiments, calculating the transformation matrix (step 325) includes: identifying, by the processor, surface points of the extracted 3D model of the internal structure; and calculating, by the processor, the transformation matrix based on the surface points using an Iterative Closest Point (ICP) algorithm. Since the number of surface points obtained from the 3D model of the internal structure may be very large, in some embodiments, the processor culls the surface points, thereby reducing the amount of computation and improving performance.

[0059] In some embodiments, the transformation is performed according to a deformation model. In some embodiments, during deformation, a list of control points on the model is moved. In some embodiments, during deformation, the center of gravity of the two models is used as a reference and kept stationary, and the surface points of the 3D model are adjusted to match the positions of the surface points on the subject.

[0060] In some embodiments, the correlation between the subject and the 3D model, as well as the transformation matrices obtained in steps 200 and 300, are used to train an artificial neural network system. Due to the development of GPUs, FPGAs, ASICs, etc. and the increasing amount of available data, computing power has increased rapidly, and artificial neural networks such as convolutional neural networks have been widely experimented with in many different fields. Generally speaking, the working principle of an artificial neural network is as follows: The artificial neural network is supplied with a large amount of training data, where the expected results are known. The weight factors of the artificial network are adjusted according to the error between the results produced by the network and the expected results. When a sufficient amount of training data is fed into the artificial neural system, the neural network will produce results in the expected manner. When new data that does not belong to the training data is fed into the trained artificial neural network, the network should produce a result that can be used to make a decision. This process is called inference. An artificial neural network can produce two types of results: discrete results or continuous results. Discrete results are often used to solve classification problems, while continuous results are often used to solve regression problems.

[0061] The artificial neural network system as described above can be used for the calculation of the transformation matrices in steps 200, 310, and 320. Here, the use of the artificial neural network system will be described with reference to step 310.

[0062] In some embodiments, the method of registering the medical image of a subject with the 3D model of the subject further includes displaying a virtual image slice corresponding to the image slice generated by the probe on the 3D model (step 400). In some embodiments, the virtual image slice is displayed in a virtual plane on the 3D model and includes a cross-sectional image of the internal structure. The virtual plane corresponds to the scanning plane of the probe, and the cross-sectional image corresponds to the cross-section of the internal structure being scanned.

[0063] In some embodiments, the registration problem is regarded as a classification problem. According to these embodiments, a set of virtual slices corresponding to the standard plane of a specific internal structure is generated. Then, a plurality of scanned images corresponding to the standard plane are obtained from a plurality of different subjects. The relationship between the scanned images and the slices from the 3D model is established and used to train the artificial neural network. After training, when a new scanned image corresponding to the standard plane of the internal structure from a subject is given, the neural network should be able to classify the image as corresponding to one of the virtual slices from the 3D model. Although ultrasound is an unconstrained exploratory scan, when looking for the standard plane, the scan usually follows some protocols of an approved institution. Therefore, it is sufficient to be able to classify the ultrasound image as corresponding to one of the standard planes.

[0064] In some embodiments, the registration problem is treated as a regression problem. According to these embodiments, the result of the artificial neural network is used to generate the parameters for matching slices. These parameters include the translation angle and the rotation angle. When a sufficient number of scanned images corresponding to the standard plane are obtained, the corresponding slices are obtained from the 3D model, and the parameters used to obtain the slices are recorded. During the training process of the artificial neural network, the scanned images are used as inputs, and the known parameters of the slice images corresponding to the scanned images are used as outputs. Once the training is complete, the trained neural network is input with a new scanned image and outputs a set of parameters. Then, the set of parameters is used to generate matching slices in the 3D model. Since the parameter space is usually large, a much larger amount of training data is required to obtain similar results compared to the solution when the problem is treated as a classification problem.

[0065] Device for registering a subject's medical image with a 3D model

[0066] In some embodiments, the present specification is directed to a device 600 for registering a subject's medical image with a 3D model of the subject.

[0067] In some embodiments, the device 600 registers the medical image with the 3D model by methods similar to those described above.

[0068] In some embodiments, the device 600 includes a tracking system 620; a processor 640; and a memory 650.

[0069] In some embodiments, the tracking system 620 includes: a position sensor 621 attached to the probe 610; and a navigation sensor 623. In some embodiments, the navigation sensor is configured to detect the position and orientation of the probe 610 via the position sensor. In some embodiments, the tracking system 620 further includes a reference sensor 625, and the reference sensor 625 is configured to be attached to the subject and detect the position of the subject. In some embodiments, the tracking system 620, including the components of the tracking system 620, operates in a manner similar to those described in detail above.

[0070] In some embodiments, the device 600 includes a probe 610. In some embodiments, the probe 610 is an ultrasound probe, such as an ultrasound sensor.

[0071] In some embodiments, the device 600 further includes a 3D scanner 630. In some embodiments, the 3D scanner 630 is configured to operate in a manner similar to those described in detail above.

[0072] In some embodiments, the memory 650 stores a 3D model of the subject. In some embodiments, the 3D model includes a 3D model of the internal structure of the subject. In some embodiments, the memory includes a 3D model database 651 that includes a plurality of 3D model entries. According to these embodiments, each of the plurality of entries is tagged with information about the 3D model entry, and the 3D model that best represents the subject can be selected from the entries. In some embodiments, the 3D model database is similar to those detailed above.

[0073] In some embodiments, the processor 640 is configured to: globally calibrate the 3D model by aligning markers on the subject with corresponding markers on the 3D model; and locally calibrate the 3D model by aligning a scanned image of the internal structure of the subject with the internal structure of the 3D model. In some embodiments, the processor performs global and local calibration in a manner similar to that detailed above.

[0074] In some embodiments, the processor 640 is further configured to select an initial 3D model representing the subject from the 3D model database by: comparing the information of the subject with the information tagged to the entries of the 3D model database 651; and selecting the initial 3D model tagged with information that matches the information of the subject. In some embodiments, the information includes demographic information such as gender, height, weight, race, or age, personal information such as body measurements, body fat percentage (BFP), or body mass index (BMI), or external dimensions.

[0075] In some embodiments, the processor 640 is configured to globally calibrate the 3D model by: receiving the position of a probe representing the position of a fiducial marker to record the real-world position of the fiducial marker, where the fiducial marker is a marker located on the outer surface of the subject corresponding to a marker on the 3D model; calculating a transformation matrix between the outer surface of the subject and the 3D model based on the correlation between the real-world position of the fiducial marker and the virtual position of the corresponding marker; and globally calibrating the 3D model by applying the transformation matrix to the 3D model. In some embodiments, the transformation matrix is a rigid transformation matrix or an affine transformation matrix.

[0076] In some embodiments, the processor 640 is configured to locally calibrate the 3D model by: selecting a standard plane of the 3D model, where the standard plane includes a cross-section of the internal structure in the 3D model; selecting a scanned image of the subject, where the scanned image includes a cross-section of the internal structure similar to the cross-section in the standard plane; calculating a transformation matrix between the two cross-sections; and locally calibrating the 3D model by applying the transformation matrix to the internal structure of the 3D model.

[0077] In some embodiments, the processor is configured to select a scanned image of a subject that includes a cross-section of an internal structure similar to the cross-section included in a standard plane by: calculating a similarity index between the scanned image and the image of the standard plane in the 3D model; and selecting the scanned image when the similarity index is equal to or greater than a predetermined value.

[0078] In some embodiments, the processor is configured to calculate the similarity index by: using a boundary detection algorithm to convert the image of the standard plane and the scanned image into templates; and calculating the similarity index by performing affine invariant template matching of the templates.

[0079] In some embodiments, the processor 640 is configured to locally calibrate the 3D model by: constructing a 3D volume of a scanned image of an internal structure of a subject; extracting a 3D model of the internal structure from the 3D volume of the scanned image; calculating a transformation matrix between the internal structure of the subject and the internal structure in the 3D model based on the correlation between the shape of the extracted 3D model of the internal structure and the shape of the internal structure in the 3D model; and locally calibrating the 3D model by applying the transformation matrix to the internal structure of the 3D model.

[0080] In some embodiments, the processor 640 is configured to extract a 3D model of the internal structure by: extracting a 3D model of the internal structure from the 3D volume of the scanned image by a segmentation algorithm.

[0081] In some embodiments, the processor 640 is configured to calculate the transformation matrix by: identifying surface points of the extracted 3D model of the internal structure; and calculating the transformation matrix based on the surface points using an iterative closest point (ICP) algorithm.

[0082] In some embodiments, the processor 640 is configured to calculate the transformation matrix by: calculating the center of gravity of the extracted 3D model of the internal structure; and calculating the transformation matrix based on the center of gravity.

[0083] In some embodiments, the apparatus 600 further includes a display configured to display a virtual image slice corresponding to an image slice generated by the probe 610 on the 3D model.

[0084] In some embodiments, the processor 640 is configured to locally calibrate the 3D model by: training an artificial neural network system by feeding training data to the artificial neural network system, the training data including an image of an internal structure and a corresponding 3D model of the internal structure with a known expected result; and calibrating the internal structure of the 3D model by applying the trained artificial neural network.

[0085] In some embodiments, the processor is a single processor. In some embodiments, the processor is multiple processors. In some embodiments, the processor includes a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA) processor, an application specific integrated circuit (ASIC) processor, or a combination thereof.

[0086] From the foregoing, it will be understood that various embodiments of the present disclosure have been described herein for purposes of illustration, and that various modifications can be made without departing from the scope and spirit of the present disclosure. Accordingly, the various embodiments disclosed herein are not intended to be limiting, and the true scope and spirit are indicated by the appended claims.

Claims

1. A method for registering a medical image of a person with a three-dimensional (3D) model, comprising: Globally calibrating the 3D model by a processor to create a globally calibrated 3D model by aligning fiducial markers on the person with corresponding markers on the 3D model; And Locally calibrating the globally calibrated 3D model by the processor by aligning a scanned image of the internal structure of the person with the corresponding internal structure of the globally calibrated 3D model; Wherein locally calibrating the globally calibrated 3D model includes: Constructing, by the processor, a 3D volume of the scanned image of the internal structure of the person; Extracting, by the processor, a 3D model of the internal structure from the 3D volume of the scanned image; Calculating, by the processor, a transformation matrix between the internal structure of the person and the internal structure of the globally calibrated 3D model based on the correlation between the shape of the extracted 3D model of the internal structure and the shape of the internal structure in the globally calibrated 3D model; and Locally calibrating the 3D model by the processor by applying the transformation matrix to the internal structure of the globally calibrated 3D model; Wherein calculating the transformation matrix includes: Calculating, by the processor, the centroid of the extracted 3D model of the internal structure; and Calculating, by the processor, the transformation matrix based on the centroid.

2. The method according to claim 1, further comprising selecting an initial 3D model representing the person from a 3D model database, wherein selecting the initial 3D model includes: Comparing, by the processor, the information of the person with the information of the entries marked in the 3D model database; And Selecting, by the processor, the initial 3D model marked with information matching the information of the person.

3. The method according to claim 2, wherein the information includes demographic information, which includes gender, height, weight, race, age, body measurements, body fat percentage (BFP) or body mass index (BMI) or external dimensions.

4. The method according to claim 1, wherein globally calibrating the 3D model includes: Selecting a marker on the 3D model corresponding to the fiducial marker on the person; Moving a probe to the fiducial marker to record the position of the probe, thereby obtaining the real-world position of the fiducial marker; Calculating, by the processor, a transformation matrix between the outer surface of the person and the 3D model based on the correlation between the real-world position of the fiducial marker and the virtual position of the corresponding marker; And Globally calibrating the 3D model by the processor by applying the transformation matrix to the 3D model.

5. The method according to claim 4, wherein the transformation matrix is a rigid transformation matrix or an affine transformation matrix.

6. The method according to claim 1, wherein extracting the 3D model of the internal structure includes: Extracting the 3D model of the internal structure from the 3D volume of the scanned image by a segmentation algorithm.

7. The method according to claim 1, further comprising displaying a virtual image slice corresponding to an image slice generated by a probe on the 3D model.

8. An apparatus for registering a medical image of a person with a 3D model, comprising: A tracking system, which includes: A position sensor, wherein the position sensor is attachable to a probe; and A navigation sensor, wherein the navigation sensor is configured to detect the position and orientation of the probe via the position sensor; A memory for storing a 3D model of a person, wherein the 3D model includes a 3D model of the internal structure of the person; and A processor connected to the tracking system, wherein the processor is configured to: Globally calibrate the 3D model by aligning fiducial markers on the person with corresponding markers on the 3D model to create a globally calibrated 3D model; and Locally calibrate the globally calibrated 3D model by aligning a scanned image of the internal structure of the person with the internal structure of the globally calibrated 3D model; Wherein the processor is configured to locally calibrate the globally calibrated 3D model: Construct a 3D volume of the scanned image of the internal structure of the person; Extract a 3D model of the internal structure from the 3D volume of the scanned image; Calculate a transformation matrix between the internal structure of the person and the internal structure of the globally calibrated 3D model based on the correlation between the shape of the extracted 3D model of the internal structure and the shape of the internal structure in the globally calibrated 3D model; and Locally calibrate the 3D model by applying the transformation matrix to the internal structure of the globally calibrated 3D model; Wherein the processor is configured to calculate the transformation matrix: Calculate the center of gravity of the extracted 3D model of the internal structure; and Calculate the transformation matrix based on the center of gravity.

9. The apparatus according to claim 8, wherein the memory stores a 3D model database, and the processor is further configured to select an initial 3D model representing a person from the 3D model database by the following steps: Compare the information of the person with the information of the entries marked in the 3D model database; and Select the initial 3D model marked with information matching the information of the person.

10. The apparatus according to claim 8, wherein the processor is configured to globally calibrate the globally calibrated 3D model by the following steps: Receive the probe position representing the position of the fiducial marker to record the real-world position of the fiducial marker, wherein the fiducial marker is a marker located on the outer surface of the person corresponding to the marker on the 3D model; Calculate a transformation matrix between the outer surface of the person and the 3D model based on the correlation between the real-world position of the fiducial marker and the virtual position of the corresponding marker; And Globally calibrate the 3D model by applying the transformation matrix to the 3D model.

Images