Medical image registration method, medical imaging system and ultrasound imaging system

Abstract

The application provides a medical image registration method, a medical imaging system and an ultrasonic imaging system. The method comprises the following steps: acquiring a first medical image of a first modality related to a liver, wherein the first medical image of the first modality is two-dimensional ultrasonic image data; identifying a section type of the two-dimensional ultrasonic image data; acquiring volume data of a second medical image of a second modality related to the liver, and extracting first volume data of an anatomic structure corresponding to the section type in the volume data of the second medical image, wherein the first modality is different from the second modality; based on the two-dimensional ultrasonic image data, acquiring second volume data of the anatomic structure; and performing registration on the first volume data and the second volume data to acquire a registration mapping relationship, so as to realize automatic registration of different images of two modalities.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical equipment, and more particularly to a medical image registration method, a medical imaging system and an ultrasonic imaging system. BACKGROUND

[0002] The ultrasound and computed tomography (CT) / magnetic resonance imaging (MRI) fusion technology gives full play to the technical advantages of ultrasound imaging, such as convenience, real-time, no radiation, high resolution and no blind area of CT / MRI lesions, and plays an important role in the positioning diagnosis and treatment of liver focal lesions, and is particularly suitable for lesions that are not clearly displayed under conventional ultrasound, such as isoechoic lesions and lesions in the background of advanced liver cirrhosis. The commonly used fusion imaging alignment method is manual alignment, such as face-to-face registration and point-to-point registration. The method matches or matches points by using intrahepatic vascular structures (such as the portal vein and the hepatic vein) or lesions (such as cysts) as alignment markers, so as to realize the registration and fusion of three-dimensional volumes. However, the conventional manual method is time-consuming and tedious, and requires a certain medical imaging professional basis and a long application training, which becomes a bottleneck for the clinical promotion of the fusion technology. For some automatic registration technologies, the probe needs to be placed at a predetermined position on the patient to initialize the registration matrix, and then fine-tuning is performed in other parts. The workflow is more cumbersome, and if the fine-tuning area is far away from the lesion area, the registration error is large.

[0003] In view of the above problems, the present application provides a new medical image registration method, a medical imaging system and an ultrasonic imaging system. SUMMARY

[0004] The present application is proposed to solve at least one of the above problems. Specifically, in one aspect, the present application provides a medical image registration method, which includes: acquiring a first medical image of a first modality related to a liver, wherein the first medical image of the first modality is two-dimensional ultrasonic image data; identifying a section type of the two-dimensional ultrasonic image data; acquiring volume data of a second medical image of a second modality related to the liver, and extracting first volume data of an anatomical structure corresponding to the section type in the volume data of the second medical image, wherein the first modality and the second modality are different modalities; based on the two-dimensional ultrasonic image data, acquiring second volume data of the anatomical structure; and registering the first volume data and the second volume data to obtain a registration mapping relationship.

[0005] Another aspect of the present invention provides a medical image registration method, the method comprising: acquiring a first medical image of a first modality related to the liver, wherein the first medical image is two-dimensional ultrasound image data; identifying the section type of the two-dimensional ultrasound image data; acquiring volume data of a second medical image of a second modality related to the liver, and extracting volume data of anatomical structures corresponding to the section type from the volume data of the second medical image; extracting two-dimensional ultrasound data of the anatomical structures based on the two-dimensional ultrasound image data; and registering the volume data of the anatomical structures and the two-dimensional ultrasound data of the anatomical structures to obtain a registration mapping relationship.

[0006] In another aspect, the present invention provides a medical image registration method, the method comprising: acquiring a first medical image of a first modality related to the liver, wherein the first medical image of the first modality is two-dimensional ultrasound image data; identifying the section type of the two-dimensional ultrasound image data; acquiring volume data of a second medical image of a second modality of the liver, and extracting a two-dimensional section image corresponding to the section type from the volume data of the second medical image, wherein the first modality is a different modality from the second modality; and registering the two-dimensional ultrasound image data and the two-dimensional section image to obtain a registration mapping relationship.

[0007] In another aspect, the present invention provides a medical image registration method, the method comprising: acquiring a first medical image of a first modality related to the liver, wherein the first medical image of the first modality is two-dimensional ultrasound image data; identifying the structural type of an anatomical structure at the location of the liver in the two-dimensional ultrasound image data; acquiring volume data of a second medical image of a second modality related to the liver, and extracting volume data of an anatomical structure corresponding to the structural type of the anatomical structure from the volume data of the second medical image; acquiring ultrasound image data of the anatomical structure based on the two-dimensional ultrasound image data; and registering the volume data of the anatomical structure and the ultrasound image data of the anatomical structure to obtain a registration mapping relationship.

[0008] The present invention also provides a medical imaging system, the medical imaging system comprising: a memory for storing executable program instructions; a processor for executing the program instructions stored in the memory, causing the processor to perform the aforementioned medical image registration method; and a display device for displaying visualization information.

[0009] The present invention also provides an ultrasound imaging system, comprising: an ultrasound probe; a transmit / receive sequence controller for controlling the ultrasound probe to transmit ultrasound waves to a scanning location on the liver, and receiving ultrasound echoes based on the ultrasound waves returned from the liver to obtain an ultrasound echo signal; a processor for obtaining multiple frames of two-dimensional ultrasound images related to the liver based on the ultrasound echo signal; a memory for storing executable program instructions; a processor for executing the program instructions stored in the memory, causing the processor to perform the medical image registration method described above; and a display device for displaying visual information.

[0010] According to the medical image registration method, medical imaging system, and ultrasound imaging system of this application, the user does not need to place the probe in a pre-defined posture on a pre-defined position on the patient. Therefore, it increases the doctor's operational space, reduces the cumbersomeness of the doctor's operation, simplifies the workflow, reduces the requirements for the doctor's basic medical imaging knowledge and application training, and improves the registration accuracy between two different modalities of medical images. This is beneficial for doctors to identify and recognize patient lesions through medical images before, during, and after surgery. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart of a medical image registration method according to one embodiment of the present invention is shown;

[0013] Figure 2 A flowchart of a medical image registration method according to another embodiment of the present invention is shown;

[0014] Figure 3 A flowchart of a medical image registration method according to another embodiment of the present invention is shown;

[0015] Figure 4 A flowchart of a medical image registration method according to yet another embodiment of the present invention is shown;

[0016] Figure 5 A schematic block diagram of an ultrasound imaging system according to one embodiment of the present invention is shown;

[0017] Figure 6 A schematic block diagram of a medical imaging system according to one embodiment of the present invention is shown. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of the present invention.

[0019] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.

[0020] It should be understood that the invention can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein. Rather, providing these embodiments will make the disclosure thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0021] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. When used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising” and / or “including,” when used in this specification, identify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups. When used herein, the term “and / or” includes any and all combinations of the associated listed items.

[0022] To fully understand this invention, detailed methods and structures will be presented in the following description to illustrate the technical solutions proposed by this invention. Optional embodiments of the invention are described in detail below; however, in addition to these detailed descriptions, the invention may have other embodiments.

[0023] The medical image registration method, medical imaging system, and ultrasound imaging system provided in this application can be applied to the liver of humans, the liver of various animals, and other tissues and organs.

[0024] Specifically, the medical image registration method, medical imaging system, and ultrasound imaging system of this application will be described in detail below with reference to the accompanying drawings. Unless otherwise specified, the features in the following embodiments and implementations can be combined with each other.

[0025] First, refer to the appendix Figure 1 The medical image registration method in the first embodiment of the present invention will be described.

[0026] As an example, such as Figure 1 As shown, the medical image registration method of this application includes the following steps: In step S110, a first medical image of a first modality related to the liver is acquired, wherein the first medical image of the first modality is two-dimensional ultrasound image data; In step S120, the section type of the two-dimensional ultrasound image data is identified; In step S130, volume data of a second medical image of a second modality related to the liver is acquired, and the first volume data of the anatomical structure corresponding to the section type is extracted from the volume data of the second medical image, wherein the first modality is a different modality from the second modality; In step S140, the second volume data of the anatomical structure is acquired based on the two-dimensional ultrasound image data; In step S150, the first volume data and the second volume data are registered to obtain a registration mapping relationship.

[0027] According to the method of this application, by automatically identifying the section type of the two-dimensional ultrasound image data, the anatomical structure (i.e., the liver region) scanned by the current two-dimensional ultrasound image is obtained. Then, the volume data corresponding to this anatomical structure is found in the second medical image of the second modality. Automatic registration of the first medical image of the first modality and the medical image of the second modality can then be performed, eliminating the need for the user to place the probe in a pre-defined posture on a pre-defined position on the patient for registration initialization. This increases the doctor's operational space, reduces the cumbersome nature of the operation, simplifies the workflow, lowers the requirements for doctors' basic medical imaging knowledge and application training, and improves the registration accuracy between two different modalities of medical images. This is beneficial for doctors to identify and recognize patient lesions through medical images before, during, and after surgery.

[0028] Specifically, prior to step S110, the method further includes: upon entering fusion, accepting and storing imported data of the second modality of the second medical image, such as volume data of the second medical image, wherein the second modality of the second medical image includes computed tomography (CT) images and / or magnetic resonance imaging (MRI) images, or other suitable medical image data. The data of the second modality of the second medical image is CT or MRI data of the liver acquired by the patient before surgery. Optionally, the CT sequence phase can be plain scan, arterial phase, venous phase, portal venous phase, etc., and the MRI sequence phase can be T1, T2, arterial phase, venous phase, hepatobiliary phase, delayed phase, etc. Optionally, the CT sequence phase imported into the system is a sequence with clear liver and blood vessels. In one example, the respiratory phase of the second modality of the second medical image (e.g., CT, MRI sequence) is consistent with the respiratory phase during ultrasound imaging system scanning.

[0029] Optionally, the execution subject of the method of this application can be an ultrasound imaging system or any other medical imaging system. For example, taking an ultrasound imaging system as an example, the method further includes: in response to the operation of the fusion function button, controlling the display device to display a fusion interface on the display interface, wherein the fusion interface has a data import interface for importing data information of the second medical image of the second modality of the patient (e.g., a patient about to undergo surgery) through the data import interface, which can be imported via, for example, a USB flash drive, optical disc, network transmission, etc. Optionally, the second medical image of the second modality can be three-dimensional image data or continuous multi-frame two-dimensional image data, and when it is two-dimensional image data, it includes multiple cross-sectional images.

[0030] Specifically, in step S110, the first medical image of the first modality is generated based on an ultrasound scan of a probe placed in a predetermined area of ​​the patient's body (e.g., a predetermined area of ​​the liver). The user can place the probe in the predetermined area of ​​the liver (i.e., the scanning site), including but not limited to the xiphoid process transverse section, xiphoid process sagittal section, portal vein sagittal section, inguinal plexus sagittal section, right intercostal space, liver and kidney sections, the convergence of the three branches of the portal vein, the area around a lesion (e.g., a tumor), or several pre-defined predetermined areas. The probe scanning method can be sector scan or plain scan, etc. The ultrasound imaging system records magnetic navigation data, images, and image information for each frame for subsequent three-dimensional (3D) reconstruction. The image refers to a two-dimensional ultrasound image, and the image information includes, but is not limited to, at least one of the following: image size, resolution, pixel pitch, lesion size, etc. In this application, the probe can be placed in any of the multiple predetermined areas without affecting the implementation of subsequent steps. The user does not need to place the probe in a pre-agreed posture on a pre-agreed position on the patient, thus increasing the doctor's operational space and reducing the cumbersomeness of the operation.

[0031] Optionally, the first medical image of the first modality can be a real-time two-dimensional ultrasound image acquired by scanning the patient in real time using an ultrasound imaging system. Optionally, the first medical image of the first modality related to the liver includes consecutive multi-frame two-dimensional ultrasound image data.

[0032] Optionally, the first medical image of the first modality is a liver-related medical image, that is, the image includes image information of at least part of the patient's liver location.

[0033] Furthermore, in step S120, the section type of the two-dimensional ultrasound image data includes at least one of the following types: xiphoid process transverse section, xiphoid process sagittal section, portal vein sagittal section, second hepatic hilum section, inguinal nerve plexus sagittal section, right intercostal section, liver and kidney section, convergence of the three branches of the portal vein, or other possible section types.

[0034] The cross-sectional type of the acquired two-dimensional ultrasound image data can be identified using any suitable method known to those skilled in the art. For example, the cross-sectional type of the two-dimensional ultrasound image data can be automatically identified using intelligent recognition methods, including deep learning methods.

[0035] Common classification and recognition methods combine a database with machine learning to learn features or patterns that distinguish different cross-section categories. The learned features or patterns are then used to classify and recognize the input image. This process generally includes database construction and recognition steps. In the database construction step, the database consists of a large number of samples. In practice, for fully supervised learning, each sample consists of a cross-section image and its corresponding cross-section category. For semi-supervised learning, some samples may consist of a cross-section image and its corresponding cross-section category, while others may only contain the image without a label indicating its cross-section category. Alternatively, the database can store samples suitable for both fully supervised and semi-supervised learning methods. In the recognition step, a machine learning algorithm is designed to learn the features or patterns in the database that distinguish different cross-section types, thereby achieving image recognition.

[0036] One embodiment of the step for identifying the section type of two-dimensional ultrasound image data may include a feature extraction sub-step and a classification sub-step. In the feature extraction sub-step, the feature extraction method can be a conventional digital image processing feature extraction method such as PCA (Principal Cause Analysis), LDA (Linear Discriminant Analysis), Haar, texture features, etc. The feature extraction method can also employ deep learning methods, learning features from the constructed database by stacking convolutional layers, pooling layers, activation layers, and fully connected layers. Commonly used deep learning networks include CNN (Convolutional Neural Networks), ResNet, MobileNet, VGG, Inception, DenseNet, two-stage Faster RCNN networks, one-stage YOLO series, SSD series, etc. In the classification sub-step, combined with the features in the database, discriminators such as KNN, SVM, Random Forest, neural networks, and Adaboost are used to classify the extracted features, determining which standard section the currently processed ultrasound image belongs to or the probability of belonging to which standard section. Typically, the classification sub-step can output the probability of the current ultrasound image belonging to each section category.

[0037] Further, in step S130, volume data of a second medical image of a second modality related to the liver is acquired, and first volume data of the anatomical structure corresponding to the section type is extracted from the volume data of the second medical image, wherein the first modality is a different modality from the second modality.

[0038] The volume data of the second medical image in the second modality related to the liver refers to the image information of the liver included in the second medical image, and may also include image information of other tissues and organs besides the liver.

[0039] Optionally, the anatomical structures include at least one of the following structures: blood vessels, middle hepatic vein, kidney, gallbladder, inferior vena cava, diaphragm, and liver capsule. The ultrasound imaging system can extract first volume data of the anatomical structures corresponding to the section type from the volume data of the second medical image based on the section type of the two-dimensional ultrasound image obtained in step S120. For example, when the section type is a xiphoid process transverse section or a xiphoid process sagittal section, the extracted anatomical structures include at least one of the following structures: the xiphoid process, the blood vessels surrounding the xiphoid process, and the diaphragm (e.g., the diaphragm surrounding the xiphoid process); when the section type is a portal vein sagittal section (also known as a portal vein sagittal I-shaped structure), the extracted anatomical structures include at least one of the following structures: the portal vein sagittal portion, the blood vessels surrounding the portal vein sagittal portion, and the liver capsule (e.g., the liver capsule surrounding the portal vein sagittal portion); when the section type is a right hepatic section, the extracted anatomical structures include at least one of the following structures: the blood vessels of the right hepatomegaly, the liver capsule of the right hepatomegaly, and the diaphragm of the right hepatomegaly; when the section type is a femoral plexus sagittal section, the anatomical structures include at least one of the following structures: the femoral plexus, the blood vessels surrounding the femoral plexus, and the liver capsule (e.g., the liver capsule surrounding the femoral plexus). The extracted anatomical structures may include one or more of the above-mentioned anatomical structures, or are not limited to the types listed above; no restrictions are imposed here.

[0040] The following explanation uses blood vessels and the liver as examples to illustrate methods for extracting anatomical structures. Extraction can include segmentation of the anatomical structure. Taking blood vessel segmentation and liver segmentation as examples, methods for segmenting blood vessels include template matching, region growing, machine learning, Hessian matrix, tracking, and deep learning. Different methods can be selected based on the diameter and contrast of the blood vessels. CT and MRI imaging principles differ; CT can also use threshold-based methods, but the blood vessel segmentation method should support the previously imported CT / MRI phase types. Taking deep learning methods as an example, a network is trained using raw data and annotated blood vessel data. The network type can be SSD, U-net, V-net, Mobile net, nn-Unet, etc. Segmentation and reconstruction can be performed at each 2D level, directly on the entire 3D data, or by dividing the 3D data into smaller blocks for segmentation and reconstruction. Using multi-scale Hessian matrix as an example, the Hessian response at different scales is calculated for each point in the image using Gaussian-Laplacian convolution kernel functions at different scales. The maximum value of the Hessian response at each point at different scales is used as the blood vessel probability value, and then blood vessel segmentation is performed using a fixed threshold or an adaptive threshold method. Segmenting the diaphragm can be done by first segmenting the liver, then extracting the liver edge in the head direction as the diaphragm; segmenting the liver capsule can be done by first segmenting the liver, then extracting the liver edge as the liver capsule. Methods for liver segmentation include deep learning, thresholding, template matching, and histogram methods. Histogram methods are based on the fact that the liver is the largest organ in the human body; therefore, the gray-level histogram of CT / MRI images can be statistically analyzed, and a sliding window of predetermined width can be taken. The gray value with the most pixels within the sliding window is used as the representative intensity value of the liver. Template matching involves statistically analyzing the liver shapes of different individuals, aligning them to obtain a probability template for the liver's distribution, using machine learning to locate the approximate location of the liver in CT / MRI, and then placing the template center at the center point of the liver in the CT / MRI image. Regions with high probability template values ​​are further identified. Deep learning methods include three steps: database analysis, feature extraction, and classification. First, feature extraction methods are used to extract liver features from CT / MRI images. Then, the features and liver labels are combined to train a classifier. The trained classifier is then used to test experimental data to obtain the liver segmentation result. No restrictions are placed on the segmentation method, network type, or data processing method here.

[0041] Furthermore, in step S140, based on the two-dimensional ultrasound image data, the second volume data of the anatomical structure is obtained, which refers to the structure corresponding to the anatomical structure extracted from the volume data of the second medical image.

[0042] Second-body data of anatomical structures can be obtained from two-dimensional ultrasound image data using any suitable method known to those skilled in the art. For example, the two-dimensional ultrasound image data can be reconstructed to obtain three-dimensional ultrasound volume data; and the second-body data of the anatomical structures can be extracted from the three-dimensional ultrasound volume data. Alternatively, for example, two-dimensional ultrasound data of anatomical structures can be extracted from two-dimensional ultrasound image data; and three-dimensional reconstruction can be performed on the two-dimensional ultrasound data of the anatomical structures to obtain the second-body data of the anatomical structures.

[0043] Three-dimensional reconstruction of two-dimensional ultrasound image data can be performed using any suitable method known to those skilled in the art, such as freehand or free-arm technology, thereby enabling the reconstruction of three-dimensional ultrasound volume data from multiple two-dimensional ultrasound image data based on position information provided by magnetic navigation or optical navigation.

[0044] The method for extracting second-body data of anatomical structures from three-dimensional ultrasound volume data can be any suitable method well known to those skilled in the art. For example, second-body data of anatomical structures can be extracted from three-dimensional ultrasound volume data based on a trained segmentation network model. For example, taking blood vessels as an example, blood vessel segmentation can be performed based on a three-dimensional blood vessel segmentation network.

[0045] Optionally, the method for extracting anatomical structures from two-dimensional ultrasound image data can be any suitable segmentation method well known to those skilled in the art. For example, taking the segmentation and extraction of blood vessels as an example, methods for segmenting blood vessels include template matching, region growing, machine learning, Hessian matrix, tracking, deep learning, etc. Different methods can be selected according to the different diameters and contrasts of blood vessels. Please refer to the relevant descriptions of blood vessel segmentation in the previous text, which will not be repeated here.

[0046] It is worth mentioning that, in this application, the two-dimensional ultrasound image can be a tissue image, a contrast-enhanced image, or a combination of both. Contrast-enhanced ultrasound imaging can acquire a fundamental wave image based on a single transmission, i.e., a tissue image, simultaneously with the contrast-enhanced ultrasound image. The contrast-enhanced image and the tissue image have a one-to-one pixel correspondence, eliminating the need for separate registration operations to establish a mapping relationship between them.

[0047] It is worth mentioning that the order of the aforementioned steps can be changed without conflict. For example, step S140 can be performed before step S120, or step S140 can be performed after step S120 and before step S130.

[0048] Further, in step S150, the first body data and the second body data are registered to obtain a registration mapping relationship. For example, the first body data and the second body data can be registered based on a configuration algorithm. Alternatively, the two-dimensional ultrasound image and the second medical image, such as a CT / MRI image, can be registered. Alternatively, the first body data and the second body data of the segmented anatomical structure can be registered, and the results of the registration of the two-dimensional ultrasound image and the second medical image, such as a CT / MRI image, can be combined to obtain the final registration mapping relationship. Optionally, the registration mapping relationship can refer to the registration matrix T between the ultrasound image data and the second medical image, such as a CT / MRI image.

[0049] During registration, a spatial transformation needs to be found to map ultrasound image data (such as volume data of extracted anatomical structures, two-dimensional ultrasound image data, etc.) to the second medical image data, so that points corresponding to the same spatial location in two sets of image data from different sources can be matched one-to-one, thereby achieving the purpose of correctly registering and fusing information.

[0050] Optionally, the automatic registration algorithm can include two parts: a similarity measurement method between images and a mapping method between images. Depending on the mapping method, it can be divided into rigid transformation (rotation and translation), affine transformation (scaling, rotation, and translation), and nonlinear transformation (establishing different mappings for different local images). Considering the characteristics of ultrasound and CT / MRI imaging, two implementation schemes for rigid registration and affine registration are given below. This application does not limit itself to a single registration algorithm. If the distance between the two sets of data points is the same, i.e., the acquired pixels are at the same scale, the registration algorithm can be limited to a rigid transformation, including rotation and translation. If the depth design of the two sets of data is inconsistent, an interpolation algorithm is used to scale the two sets of data to the same scale, and then rigid registration calculation is performed. Assume that the pixel point in the CT / MRI data is X. i The image brightness is f(X) i ), Ultrasonic data pixel points Y i The image brightness is g(Y) i The mapping between the two sets of data can be represented as:

[0051]

[0052] When registering data using CT, MRI, and ultrasound images, a similarity metric function can be defined between the two sets of data. The aforementioned similarity measure is typically the sum of least absolute differences (SAD). Similar definitions include the sum of least squared errors (SSD) and maximum cross-correlation (CC). Meanwhile, the measure function f(X)... i ), g(Y if(X) can also be defined as the gray level, local gradient, local gray entropy, feature vector of image feature points, etc. i ), g(Y i The registration can be based on image intensity metrics (such as grayscale, gradient, LC2, MIND, LSOD, SSC, GOA, NGF, MILBP, MCC, NCC, a-MI, MI, SeSaMI, etc.) or image feature metrics (such as SIFT, SURF, HOG, etc.). When using vascular segmentation results from CT, MRI, and ultrasound images for registration, point cloud registration methods can be used. Commonly used point cloud registration methods include ICP, NDT, PFH, and GMM. Taking ICP as an example, firstly, based on the coordinates of points in point set A, the nearest points are searched in point set B; the centroid positions of the two point sets are calculated, the point sets are centered, and a new point set is generated; a positive definite matrix N is calculated from the new point set, and the largest eigenvalue and its largest eigenvector of the positive definite matrix N are calculated. The eigenvector is a quaternion. The quaternion is converted into a rotation matrix, and the translation matrix T is calculated; the rotation matrix and translation matrix are applied to point set A to generate a new point set, and the next iteration is performed until the positions of the two point sets are within an acceptable range. When using vessel segmentation results from CT, MRI, and ultrasound images for registration, and then using the content from CT, MRI, and ultrasound images for further registration, a point cloud registration method can be used to initialize the registration matrix, followed by further registration based on features such as MIND extracted from the image content. Alternatively, both methods can be used as loss functions to optimize the registration matrix. The above configuration algorithm is only an example; other suitable registration methods can also be applied to this application.

[0053] It is worth mentioning that when the ultrasound imaging device is used to provide doctors with real-time ultrasound images of the patient, such as the liver, during surgery, the above-mentioned registration-related steps can be performed before surgery. Thus, during surgery, the registration mapping relationship can be directly obtained based on the registration to perform registration and fusion of the real-time ultrasound images and the second modality of the second medical images acquired during surgery. Alternatively, registration can be performed in real-time during surgery based on the data of the second modality of the second medical images imported before surgery, and subsequent fusion processing can be performed based on the registration mapping relationship.

[0054] In one example, the method of this application further includes the following steps: determining a two-dimensional cross-sectional image of a second medical image of a second modality corresponding to the location of the real-time acquired ultrasound image data based on the registration mapping relationship; fusing the real-time acquired ultrasound image data and the two-dimensional cross-sectional image to obtain a fused image; displaying the fused image, or simultaneously displaying the real-time acquired ultrasound image data and the two-dimensional cross-sectional image of the second medical image, or superimposing the real-time acquired ultrasound image data and the two-dimensional cross-sectional image of the second medical image, thereby combining the information in the second medical image into the ultrasound image to more clearly display the detailed information of the ultrasound scanned area.

[0055] In one example, based on the registration mapping relationship, determining the two-dimensional cross-sectional image of the second medical image of the second modality corresponding to the position of the first medical image of the first modality acquired in real time further includes the following process: The ultrasound imaging system calculates the matrix P of the second medical image, such as CT / MRI image data, in the magnetic field coordinate system. The registration matrix between the ultrasound image data and the second medical image, such as CT / MRI image data, obtained in the aforementioned steps is T. The positional relationship between the magnetic sensor and the ultrasound probe is A. The positional relationship between the magnetic sensor and the magnetic field transmitter is R. Then T = P * R * A, and P = T * A. -1 *R -1 Once the matrix P of the ultrasound image data and the second medical image, such as CT / MRI image data, is obtained in the magnetic field coordinate system, the R matrix changes accordingly as the ultrasound probe moves. The MPR cross-sectional coordinates of the second medical image, such as CT / MRI image, corresponding to the ultrasound image (i.e., the first medical image of the first mode acquired in real time) are MPR = P*R*A, which means determining the two-dimensional cross-sectional image of the second medical image of the second mode corresponding to the position of the first medical image of the first mode acquired in real time.

[0056] In one example, two-dimensional cross-sectional images of a first medical image and a second medical image acquired in real time are fused to obtain a fused image, which is then displayed. This allows the first medical image, such as an ultrasound image, to present the image information of the two-dimensional cross-sectional image of the second medical image, such as a CT / MRI image. This makes it easier to present the image information of lesions or liver tissue to doctors more clearly, thereby playing an important role in the localization, diagnosis, and treatment of focal liver lesions.

[0057] In one example, the method of this application further includes: simultaneously displaying two-dimensional cross-sectional images of a first medical image and a second medical image acquired in real time, so that when the first medical image, such as ultrasound, does not clearly show lesions or some detailed features, the doctor can observe the lesions, etc., through the displayed two-dimensional cross-sectional images of the second medical image. Among these, lesions that are not clearly visible under conventional ultrasound include, for example, isoechoic lesions, lesions in the background of advanced cirrhosis, etc.

[0058] For example, display technologies such as pseudo-color can be used to fuse or display CT / MRI cross-sectional images side by side to achieve a fusion effect.

[0059] In one example, the method of this application further includes: superimposing two-dimensional cross-sectional images of the first medical image and the second medical image acquired in real time, for example, superimposing two-dimensional cross-sectional images of the first medical image and the second medical image acquired in real time with different colors, thereby providing doctors with a clearer view of the condition of the liver or liver-related lesions, and better assisting doctors in diagnosis and treatment of lesions.

[0060] In another embodiment of this application, such as Figure 2 As shown, a medical image registration method is also provided, including the following steps S210 to S250: In step S210, a first medical image of a first modality related to the liver is acquired, wherein the first medical image is two-dimensional ultrasound image data; in step S220, the section type of the two-dimensional ultrasound image data is identified; in step S230, volume data of a second medical image of a second modality related to the liver is acquired, and volume data of anatomical structures corresponding to the section type are extracted from the volume data of the second medical image; in step S240, two-dimensional ultrasound data of anatomical structures are extracted based on the two-dimensional ultrasound image data; in step S250, the volume data of anatomical structures and the two-dimensional ultrasound data of anatomical structures are registered to obtain a registration mapping relationship.

[0061] Some details of steps S210, S220, S230, and S240 in this embodiment can be found in the descriptions of the corresponding steps S110, S120, S130, and S140 above, and will not be repeated here. The following text describes the main differences between this embodiment and the previous embodiment.

[0062] In this embodiment of the application, in step S250, it is not necessary to perform three-dimensional reconstruction of the ultrasound image. For example, the two-dimensional image data of the anatomical structure can be mapped to a reference coordinate system to obtain the mapped two-dimensional image data; registration is performed based on the mapped two-dimensional image data and the volume data of the anatomical structure to obtain the registration mapping relationship.

[0063] In one example, the first medical image of the first modality related to the liver includes a series of consecutive two-dimensional ultrasound image data. Mapping the two-dimensional image data of the anatomical structure to a reference coordinate system to obtain the mapped two-dimensional image data includes: selecting any one frame of the two-dimensional ultrasound image data as a keyframe, and mapping the two-dimensional image data of the anatomical structure in the other frames of the two-dimensional ultrasound image data (excluding the keyframe) to the reference coordinate system to obtain the position of each point of the anatomical structure in the other frames of the image in the reference coordinate system. The mapping can be performed in any suitable way, such as obtaining the position information corresponding to each of the two-dimensional ultrasound image data, which is provided by a magnetic navigation system or an optical navigation system. Based on the position information, the two-dimensional image data of the anatomical structure in the other frames of the two-dimensional ultrasound image data (excluding the keyframe) to the reference coordinate system to obtain the position of each point segmentation result (i.e., anatomical structure, such as blood vessels, liver capsule, diaphragm, etc.) in the other frames of the two-dimensional ultrasound image in the keyframe coordinate system. This method allows us to obtain the coordinates of each point in the point cloud, thus yielding the registration matrix T. This provides the registration mapping relationship between the ultrasound image and a second medical image, such as a CT / MRI image. Before mapping, it is necessary to segment and extract the anatomical structures from each frame of the 2D ultrasound image, excluding the keyframes. The segmentation method is described above and will not be repeated here.

[0064] Subsequently, based on the registration mapping relationship, a two-dimensional cross-sectional image of the second medical image in the second modality corresponding to the position of the first medical image in the first modality acquired in real time is determined. For example, based on the obtained registration matrix T, and then combined with the position information R of the keyframe, the P matrix is ​​calculated, P = T * A. -1 *R -1 When the matrix P of the second medical image, such as CT / MRI image data, is obtained in the magnetic field coordinate system, the R matrix changes accordingly as the ultrasound probe moves. The MPR cross-sectional coordinates of the second medical image, such as CT / MRI image, corresponding to the ultrasound image (i.e., the first medical image of the first mode acquired in real time) are MPR = P*R*A, which means determining the two-dimensional cross-sectional image of the second medical image of the second mode corresponding to the position of the first medical image of the first mode acquired in real time.

[0065] In one example, the method of this application embodiment further includes: fusing two-dimensional cross-sectional images of a first medical image and a second medical image acquired in real time to obtain a fused image, and displaying the fused image, thereby enabling the first medical image acquired in real time, such as an ultrasound image, to present image information of the two-dimensional cross-sectional image of a second medical image, such as a CT / MRI image, so as to present the image information of lesions or liver tissue, etc. to the doctor more clearly, and thus play an important role in the localization diagnosis and treatment of focal liver lesions.

[0066] In one example, the method of this application further includes: simultaneously displaying two-dimensional cross-sectional images of a first medical image and a second medical image acquired in real time, so that when the first medical image, such as ultrasound, does not clearly show lesions or some detailed features, the doctor can observe the lesions, etc., through the displayed two-dimensional cross-sectional images of the second medical image. Among these, lesions that are not clearly visible under conventional ultrasound include, for example, isoechoic lesions, lesions in the background of advanced cirrhosis, etc.

[0067] For example, display technologies such as pseudo-color can be used to fuse or display CT / MRI cross-sectional images side by side to achieve a fusion effect.

[0068] In one example, the method of this application further includes: superimposing two-dimensional cross-sectional images of the first medical image and the second medical image acquired in real time, for example, superimposing two-dimensional cross-sectional images of the first medical image and the second medical image acquired in real time with different colors, thereby providing doctors with a clearer view of the condition of the liver or liver-related lesions, and better assisting doctors in diagnosis and treatment of lesions.

[0069] In yet another embodiment of this application, such as Figure 3 As shown, the medical image registration method in this embodiment includes the following steps S310 to S340: In step S310, a first medical image of a first modality related to the liver is acquired, wherein the first medical image of the first modality is two-dimensional ultrasound image data; In step S320, the section type of the two-dimensional ultrasound image data is identified; In step S330, volume data of a medical image of a second modality related to the liver is acquired, and two-dimensional section images corresponding to the section type are extracted from the volume data of the second medical image, wherein the first modality is a different modality from the second modality; In step S340, the two-dimensional ultrasound image data and the two-dimensional section images are registered to obtain a registration mapping relationship.

[0070] Detailed descriptions of the steps in this embodiment can be found above. The main difference between this embodiment and the previous embodiments is that in this embodiment, the two-dimensional ultrasound image data and the two-dimensional cross-sectional image extracted from the volume data of the second medical image in the second modality are directly registered. The cross-sectional type of the two-dimensional cross-sectional image corresponds to that of the two-dimensional ultrasound image data. The registration of the two-dimensional ultrasound image data and the two-dimensional cross-sectional image can be achieved based on any suitable registration method well known to those skilled in the art, thereby obtaining the configuration matrix T, that is, obtaining the registration mapping relationship.

[0071] The method of directly registering two-dimensional ultrasound image data and two-dimensional cross-sectional images can make points corresponding to the same spatial location in two sets of image data from different sources correspond one-to-one, thereby achieving the purpose of correctly registering and fusing information.

[0072] In one example, the method of this application embodiment further includes: determining a two-dimensional cross-sectional image of a second medical image of a second modality corresponding to the location of a first medical image of a first modality acquired in real time, based on a registration mapping relationship; fusing the two-dimensional cross-sectional images of the first medical image of the first modality and the second medical image acquired in real time to obtain a fused image, and displaying the fused image, thereby enabling the first medical image acquired in real time, such as an ultrasound image, to present the image information of the two-dimensional cross-sectional image of the second medical image, such as a CT / MRI image, so as to present the image information of lesions or liver tissue, etc. to the doctor more clearly, and thus play an important role in the localization diagnosis and treatment of focal liver lesions.

[0073] In one example, the method of this application further includes: simultaneously displaying two-dimensional cross-sectional images of a first medical image and a second medical image acquired in real time, so that when the first medical image, such as ultrasound, does not clearly show lesions or some detailed features, the doctor can observe the lesions, etc., through the displayed two-dimensional cross-sectional images of the second medical image. Among these, lesions that are not clearly visible under conventional ultrasound include, for example, isoechoic lesions, lesions in the background of advanced cirrhosis, etc.

[0074] For example, display technologies such as pseudo-color can be used to fuse or display CT / MRI cross-sectional images side by side to achieve a fusion effect.

[0075] In one example, the method of this application further includes: superimposing two-dimensional cross-sectional images of the first medical image and the second medical image acquired in real time, for example, superimposing two-dimensional cross-sectional images of the first medical image and the second medical image acquired in real time with different colors, thereby providing doctors with a clearer view of the condition of the liver or liver-related lesions, and better assisting doctors in diagnosis and treatment of lesions.

[0076] In yet another embodiment of this application, such asFigure 4 As shown, the medical image registration method in this application embodiment includes the following steps S410 to S450: In step S410, a first medical image of a first modality related to the liver is acquired, wherein the first medical image of the first modality is two-dimensional ultrasound image data; in step S420, the structural type of the anatomical structure at the liver location in the two-dimensional ultrasound image data is identified; in step S430, volume data of a second medical image of a second modality related to the liver is acquired, and volume data of the anatomical structure corresponding to the structural type of the anatomical structure is extracted from the volume data of the second medical image; in step S440, ultrasound image data of the anatomical structure is acquired based on the two-dimensional ultrasound image data; in step S450, the volume data of the anatomical structure and the ultrasound image data of the anatomical structure are registered to obtain a registration mapping relationship. The method of this application achieves automatic registration by first identifying the structural type of the anatomical structure at the liver location in the two-dimensional ultrasound image data, then segmenting the volume data of the corresponding type of anatomical structure from the volume data of the second medical image, and registering the ultrasound image data of the anatomical structure and the volume data of the anatomical structure.

[0077] For details of some steps in the embodiments of this application, please refer to the description of the relevant steps above. For example, steps S410 and S430 in the embodiments of this application can refer to the relevant description of steps S110 and S130 above.

[0078] In step S420, optionally, the anatomical structures include at least one of the following structures: blood vessels, middle hepatic vein, kidney, gallbladder, inferior vena cava, diaphragm, liver capsule, etc.

[0079] The structural type of the anatomical structure at the location of the liver in two-dimensional ultrasound image data can be identified based on any suitable method known to those skilled in the art. For example, the structural type of the anatomical structure at the location of the liver in two-dimensional ultrasound image data can be automatically identified based on intelligent recognition methods. The intelligent algorithm includes, but is not limited to, traditional image processing methods, machine learning methods, or deep learning methods. Without conflict, the methods for anatomical structure extraction and segmentation mentioned above can also be applied to this application.

[0080] In the embodiments of this application, the ultrasound image data of the anatomical structure includes two-dimensional ultrasound image data of the anatomical structure or three-dimensional ultrasound volume data of the anatomical structure. When the ultrasound image data of the anatomical structure is three-dimensional ultrasound volume data, it can be obtained by the following methods. For example, the two-dimensional ultrasound image can be reconstructed in three dimensions first, and then the three-dimensional ultrasound volume data of the anatomical structure can be segmented and extracted. Alternatively, the two-dimensional image data of the anatomical structure can be extracted from the two-dimensional ultrasound image first, and then the two-dimensional image data of the anatomical structure can be reconstructed in three dimensions.

[0081] Furthermore, the method in this application embodiment further includes the following steps: based on the registration mapping relationship, determining a two-dimensional cross-sectional image of a second medical image of a second modality corresponding to the position of a first medical image of a first modality acquired in real time; fusing the two-dimensional cross-sectional images of the first medical image and the second medical image of the first modality acquired in real time to obtain a fused image; displaying the fused image, or simultaneously displaying the two-dimensional cross-sectional images of the first medical image and the second medical image of the first modality acquired in real time, or superimposing the two-dimensional cross-sectional images of the first medical image and the second medical image of the first modality acquired in real time, for example, superimposing the two-dimensional cross-sectional images of the first medical image and the second medical image of the first modality acquired in real time with different colors.

[0082] It is worth mentioning that the order of steps in the methods of the various embodiments of this application can be interchanged, performed simultaneously, or interleaved without conflict.

[0083] In summary, the medical image registration method of this application, along with the medical imaging system and ultrasound imaging system, eliminates the need for the user to place the probe in a pre-defined posture on a pre-defined position on the patient. This increases the doctor's operational space, reduces the complexity of the operation, simplifies the workflow, lowers the requirements for the doctor's basic medical imaging knowledge and application training, and improves the registration accuracy between two different modalities of medical images. This facilitates the doctor's identification and recognition of patient lesions through medical images before, during, and after surgery.

[0084] This application also provides an ultrasound imaging system, referenced in the embodiments. Figure 6The ultrasound imaging system 10 may include: an ultrasound probe 100, a transmit / receive selection switch 101, a transmit / receive sequence controller 102, a processor 103, an output device 104, and a memory 105. The transmit / receive sequence controller 102 controls the ultrasound probe 100 to transmit ultrasound waves to the scanning location of the liver and receives ultrasound-based echoes returning from the liver to obtain ultrasound echo signals. The ultrasound probe 100 is placed in a predetermined area of ​​the patient's body (e.g., a predetermined area of ​​the liver, i.e., the scanning site). The scanning site includes, but is not limited to, the xiphoid process transverse section, the xiphoid process sagittal section, the portal vein sagittal section, the inguinal plexus sagittal section, the right intercostal space, the liver and kidney sections, the convergence of the three branches of the portal vein, the area around a lesion (e.g., a tumor), or several pre-defined predetermined areas. The probe scanning method may be sector scanning or plain scanning, etc. The ultrasound imaging system records magnetic navigation data, images, and image information for each frame for subsequent three-dimensional (3D) reconstruction. The image refers to a two-dimensional ultrasound image, and the image information includes, but is not limited to, at least one of the following: image size, resolution, pixel pitch, lesion size, etc. In this application, the probe can be placed in any of multiple predetermined areas without affecting subsequent steps. The user does not need to place the probe in a pre-defined posture at a pre-defined position on the patient, thus increasing the doctor's operational space and reducing the complexity of the procedure.

[0085] An ultrasonic probe 100 typically comprises an array of multiple elements. During each ultrasonic wave transmission, all or a portion of the elements of the ultrasonic probe 100 participate in the transmission. Each element or portion of these elements is excited by a transmission pulse and transmits an ultrasonic wave. The ultrasonic waves emitted by these elements superimpose during propagation, forming a composite ultrasonic beam that is transmitted to the scanning target. The direction of this composite ultrasonic beam is the ultrasonic propagation direction.

[0086] The ultrasound imaging system also includes a navigation device, such as one mounted on the ultrasound probe. This navigation device can acquire the spatial position information of the ultrasound probe in real time during the ultrasound scan of the liver. The spatial position information includes the probe's position and angle. Navigation devices include, but are not limited to, magnetic navigation and inertial navigation. Magnetic navigation includes a magnetic positioning controller and positioning sensors. The positioning sensors are fixed to the ultrasound probe and continuously provide position information as the probe moves. The six-degree-of-freedom spatial orientation of the ultrasound probe is obtained through the magnetic positioning controller. Inertial navigation includes gyroscopes and accelerometers. An inertial navigation system is an autonomous navigation device directly mounted on the ultrasound probe. When the carrier rotates, the accelerometer and gyroscope also rotate, thus providing continuous, real-time information on the ultrasound probe's characteristics, attitude, and velocity.

[0087] The processor 103 is used to obtain multiple frames of two-dimensional ultrasound images related to the liver based on the ultrasound echo signal. In this embodiment, the two-dimensional ultrasound images include, but are not limited to, images of any mode such as B-mode ultrasound images.

[0088] In one example, processor 103 processes ultrasound echo signals / data to obtain continuous multi-frame two-dimensional ultrasound images related to the liver. These ultrasound images can be B-images (also referred to herein as B-mode ultrasound images), C-images, or other types of ultrasound images. Specifically, processor 103 performs different processing on the ultrasound echo signals according to the user's desired imaging mode, obtaining image data of different modes. These data are then processed through logarithmic compression, dynamic range adjustment, digital scan transformation, etc., to form ultrasound images of different modes, such as B-images, C-images, etc.

[0089] In one example, the memory 105 of the ultrasound imaging system may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 103 may execute the program instructions to implement the functions (implemented by the processor 103) in the embodiments of this application and / or other desired functions. Various application programs and various data, such as various data used and / or generated by the application programs, may also be stored in the computer-readable storage medium.

[0090] The memory 105 is also used to store various image data, such as various data of a second medical image of a second modality imported into an ultrasound imaging system.

[0091] In one example, the processor 103 of the ultrasound imaging system can be implemented by software, hardware, firmware, or a combination thereof. It can use circuits, one or more application-specific integrated circuits (ASICs), one or more general-purpose integrated circuits, one or more microprocessors, one or more programmable logic devices, or combinations of the foregoing circuits or devices, or other suitable circuits or devices, thereby enabling the processor 103 to perform the functions required by it and / or other desired functions. The processor 103 of the ultrasound imaging system is used, for example, for automatic or semi-automatic image processing, such as image segmentation, coordinate space mapping, image MPR (Multi-Planar Reconstruction) reconstruction, image registration, etc.

[0092] In one example, the ultrasound imaging system may also include an input device (not shown) that can be a device used by a user to input instructions, and may include one or more of the following: a keyboard, a trackball mouse, a microphone, and a touch screen.

[0093] In one example, the ultrasound imaging system may also include a human-computer interaction module for responding to the manipulation of displayed image data by hardware devices such as keyboards, mice, and trackballs, such as tracing, target selection, etc.

[0094] In one embodiment of this application, when the program instructions stored in the memory 105 are executed by the processor 103, the processor 103 is used to execute the various related steps of the medical image registration method described above. The specific description of each step can be found above and will not be repeated here.

[0095] The ultrasound imaging system of this application also includes an output device (not shown) that can output various information (e.g., images or sounds) to an external source (e.g., a user), and may include one or more of a display device 104, a printer, a speaker, etc. Ultrasound images acquired by the processor 103 can be stored in a memory 105, and these ultrasound images can be displayed, for example, on the display device 104.

[0096] The display device 104 is used to display various visual information, such as the collected data and the operation results of the human-computer interaction module.

[0097] In one example, the display of this application includes, but is not limited to, displaying a fused image of two-dimensional cross-sectional images of a first medical image and a second medical image acquired in real time, or simultaneously displaying two-dimensional cross-sectional images of a first medical image and a second medical image acquired in real time, or superimposing two-dimensional cross-sectional images of a first medical image and a second medical image acquired in real time, for example, superimposing two-dimensional cross-sectional images of a first medical image and a second medical image acquired in real time with different colors.

[0098] In this embodiment, the display device 104 of the ultrasound imaging system can be a touch screen, a liquid crystal display, or an independent display device such as a liquid crystal monitor or television set, separate from the ultrasound imaging system. It can also be a display screen on an electronic device such as a mobile phone or tablet computer. The display device 104 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the ultrasound imaging equipment. These graphical user interfaces can be composed of graphics, text, icons, videos, and any combination thereof.

[0099] Since the ultrasound imaging system of this application can realize the medical image registration method described above, it also has the advantages of the aforementioned method.

[0100] In one embodiment of this application, such as Figure 6 As shown, a medical imaging system 600 is also provided, including: one or more processors 601, a memory 602, a display device 603, and a communication interface, etc. These components are interconnected via a bus system and / or other forms of connection mechanisms (not shown). It should be noted that... Figure 6 The components and structure of the medical imaging system 600 shown are exemplary and not limiting. The medical imaging system 600 may also have other components and structures as needed.

[0101] The memory 602 is used to store various data (e.g., image data) and executable program instructions during the medical image registration process, such as system programs for the medical imaging system 600, various application programs, or algorithms that implement various specific functions. It may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory.

[0102] Processor 601 may be a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other processing units with data processing and / or instruction execution capabilities, and may control other components in the medical imaging system 600 to perform desired functions. For example, processor 601 may include one or more embedded processors, processor cores, microprocessors, logic circuits, hardware finite state machines (FSMs), digital signal processors (DSPs), graphics processing units (GPUs), or combinations thereof. Furthermore, processor 601 is used to execute program instructions stored in memory, causing processor 601 to perform the various steps of the medical image registration method described above. The specific steps of the medical image registration method can be referred to the preceding description and will not be repeated here.

[0103] In one example, the medical imaging system 600 also includes a communication interface (not shown) for communication between the various components of the medical imaging system 600 and between the various components of the medical imaging system 600 and other devices outside the system (such as an ultrasound imaging system, a server, etc.).

[0104] The communication interface can be any known communication protocol interface, such as a wired or wireless interface. It may include one or more serial ports, USB interfaces, Ethernet ports, WiFi, wired networks, DVI interfaces, device interconnect modules, or other suitable ports, interfaces, or connections. The medical imaging system 600 can also access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G, 5G, or combinations thereof. In one exemplary embodiment, the communication interface receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication interface also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0105] In this embodiment, the display device 603 of the medical imaging system 600 can be a touch screen, a liquid crystal display, or an independent display such as a liquid crystal monitor or television set, separate from the medical imaging system 600. It can also be a display screen on an electronic device such as a mobile phone or tablet. The display can be used to show information input by the user or information provided to the user, as well as various graphical user interfaces of the medical imaging system 600. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof.

[0106] The medical imaging system 600 also includes a user interface through which users can control the operation of the medical imaging system 600. The user interface may include a display device 603, which may include a touchscreen allowing users to input operating commands to the medical imaging system 600 via the display device 603, and / or include one or more control panels, through which users can control the operation of the medical imaging system 600.

[0107] It is worth mentioning that the medical imaging system 600 of this application can communicate with an ultrasound imaging system to obtain ultrasound image information acquired in real time by the ultrasound imaging system.

[0108] Since the medical imaging system 600 of this application can implement the medical image registration method described above, it also has the advantages of the aforementioned method.

[0109] In addition, embodiments of the present invention also provide a computer storage medium on which a computer program is stored. One or more computer program instructions may be stored on the computer-readable storage medium, and a processor may execute the program instructions stored in the storage device to implement the functions (implemented by the processor) in the embodiments of the present invention and / or other desired functions, such as performing corresponding steps of the medical image registration method according to embodiments of the present invention. Various applications and various data may also be stored in the computer-readable storage medium, such as various data used and / or generated by the applications.

[0110] For example, computer storage media may include a memory card for a smartphone, a storage component for a tablet computer, a hard disk for a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media.

[0111] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of the invention. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as claimed in the appended claims.

[0112] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0114] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0115] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of the invention. However, this approach should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with fewer features than all of those in a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0116] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or elements of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature serving the same, equivalent, or similar purpose.

[0117] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0118] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules according to embodiments of the present invention. The present invention can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing some or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0119] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

Claims

1. A medical image registration method, characterized in that, The method includes: Acquire a first medical image of a first modality related to the liver, wherein the first medical image of the first modality is two-dimensional ultrasound image data, and the first medical image of the first modality is generated based on an ultrasound scan of a probe placed in any of a plurality of predetermined regions of the patient's liver; Identify the section type of the two-dimensional ultrasound image data; Acquire volume data of a second medical image of a second modality related to the liver, and extract first volume data of the anatomical structure corresponding to the section type from the volume data of the second medical image, wherein the first modality is a different modality from the second modality; Based on the two-dimensional ultrasound image data, the second volume data of the anatomical structure are obtained; The first body data and the second body data are registered to obtain a registration mapping relationship. Wherein, when the section type is a xiphoid process transverse section or a xiphoid process sagittal section, the anatomical structure includes at least one of the following structures: xiphoid process, blood vessels around the xiphoid process, and diaphragm around the xiphoid process; When the section type is a cross-section of the portal vein sagittal part, the anatomical structure includes at least one of the following structures: the portal vein sagittal part, the blood vessels surrounding the portal vein sagittal part, and the liver capsule surrounding the portal vein sagittal part; When the section type is a right liver section, the anatomical structure includes at least one of the following structures: blood vessels of the right liver, liver capsule of the right liver, and diaphragm of the right liver. When the section type is the sagittal plane of the inguinal plexus, the anatomical structure includes at least one of the following structures: the inguinal plexus, the blood vessels surrounding the inguinal plexus, and the liver capsule surrounding the inguinal plexus.

2. A medical image registration method, characterized in that, The method includes: Acquire a first medical image of a first modality related to the liver, wherein the first medical image is two-dimensional ultrasound image data, and the first medical image of the first modality is generated based on an ultrasound scan of a probe placed in any of a plurality of predetermined regions of the patient's liver; Identify the section type of the two-dimensional ultrasound image data; Obtain volume data of a second medical image of a second modality related to the liver, and extract volume data of anatomical structures corresponding to the section type from the volume data of the second medical image; Based on the two-dimensional ultrasound image data, extract the two-dimensional ultrasound data of the anatomical structure; The volumetric data and two-dimensional ultrasound data of the anatomical structure are registered to obtain a registration mapping relationship. Wherein, when the section type is a xiphoid process transverse section or a xiphoid process sagittal section, the anatomical structure includes at least one of the following structures: xiphoid process, blood vessels around the xiphoid process, and diaphragm around the xiphoid process; When the section type is a cross-section of the portal vein sagittal part, the anatomical structure includes at least one of the following structures: the portal vein sagittal part, the blood vessels surrounding the portal vein sagittal part, and the liver capsule surrounding the portal vein sagittal part; When the section type is a right liver section, the anatomical structure includes at least one of the following structures: blood vessels of the right liver, liver capsule of the right liver, and diaphragm of the right liver. When the section type is the sagittal plane of the inguinal plexus, the anatomical structure includes at least one of the following structures: the inguinal plexus, the blood vessels surrounding the inguinal plexus, and the liver capsule surrounding the inguinal plexus.

3. The method as described in claim 1 or 2, characterized in that, The first medical image of the first modality related to the liver includes continuous multi-frame two-dimensional ultrasound image data.

4. The method as described in claim 1 or 2, characterized in that, The cut surface type includes at least one of the following types: xiphoid process transverse section, xiphoid process sagittal section, portal vein sagittal section, second hepatic hilum section, inguinal nerve plexus sagittal section, right intercostal section, liver and kidney section, and convergence of the three branches of the portal vein.

5. The method as described in claim 1 or 2, characterized in that, The anatomical structures include at least one of the following: blood vessels, middle hepatic vein, kidney, gallbladder, inferior vena cava, diaphragm, and liver capsule.

6. The method as described in claim 1 or 2, characterized in that, The identification of the section type of the two-dimensional ultrasound image data includes: The cross-sectional type of the two-dimensional ultrasound image data is automatically identified based on an intelligent recognition method, wherein the intelligent recognition method includes a deep learning method.

7. The method as described in claim 1, characterized in that, The process of obtaining second-body data of the anatomical structure based on the two-dimensional ultrasound image data includes: The two-dimensional ultrasound image data is reconstructed to obtain three-dimensional ultrasound volume data; Second body data of the anatomical structure are extracted from the three-dimensional ultrasound body data.

8. The method as described in claim 1, characterized in that, The process of obtaining second-body data of the anatomical structure based on the two-dimensional ultrasound image data includes: Extract the two-dimensional ultrasound data of the anatomical structure from the two-dimensional ultrasound image data; Three-dimensional reconstruction is performed on the two-dimensional ultrasound data of the anatomical structure to obtain the second-body data of the anatomical structure.

9. The method as described in claim 2, characterized in that, The registration of the volume data and two-dimensional image data of the anatomical structure to obtain a registration mapping relationship includes: The two-dimensional image data of the anatomical structure is mapped to a reference coordinate system to obtain the mapped two-dimensional image data. Registration is performed based on the mapped two-dimensional image data and the volume data of the anatomical structure to obtain the registration mapping relationship.

10. The method as described in claim 9, characterized in that, The first medical image of the first modality related to the liver includes continuous multi-frame two-dimensional ultrasound image data. Mapping the two-dimensional image data of the anatomical structure to a reference coordinate system to obtain the mapped two-dimensional image data includes: Select any one frame of two-dimensional ultrasound image data from multiple frames of two-dimensional ultrasound image data as a key frame, and use the coordinate system of the key frame as the reference coordinate system; The anatomical structure 2D image data of the 2D ultrasound image data of the other frames (excluding the key frame) in the multi-frame 2D ultrasound image data are respectively mapped to the reference coordinate system.

11. The method as described in claim 10, characterized in that, The step of mapping the two-dimensional image data of anatomical structures in the two-dimensional ultrasound image data of frames other than the keyframe in the multi-frame two-dimensional ultrasound image data to the reference coordinate system includes: The location information corresponding to each of multiple frames of two-dimensional ultrasound image data is acquired, and the location information is obtained by a magnetic navigation system or an optical navigation system. Based on the location information, the two-dimensional image data of anatomical structures in multiple frames of two-dimensional ultrasound image data other than the keyframe are mapped to the reference coordinate system.

12. A medical image registration method, characterized in that, The method includes: Acquire a first medical image of a first modality related to the liver, wherein the first medical image of the first modality is two-dimensional ultrasound image data, and the first medical image of the first modality is generated based on an ultrasound scan of a probe placed in any of a plurality of predetermined regions of the patient's liver; Identify the structural types of anatomical structures at the location of the liver in the two-dimensional ultrasound image data, wherein the structural types of anatomical structures include at least one of the following types: portal sagittal part, vessels surrounding the portal sagittal part, liver capsule surrounding the portal sagittal part, middle hepatic vein, kidney, gallbladder, inferior vena cava, inguinal plexus, vessels surrounding the inguinal plexus, liver capsule surrounding the inguinal plexus, xiphoid process, vessels surrounding the xiphoid process, diaphragm surrounding the xiphoid process, vessels of the right liver, liver capsule of the right liver, and diaphragm of the right liver; Acquire volume data of a second medical image of a second modality related to the liver, and extract volume data of the anatomical structure corresponding to the structural type of the anatomical structure from the volume data of the second medical image; Based on the two-dimensional ultrasound image data, ultrasound image data of the anatomical structure is obtained; The volume data of the anatomical structure and the ultrasound image data of the anatomical structure are registered to obtain a registration mapping relationship.

13. The medical image registration method as described in claim 12, characterized in that, The ultrasound image data of the anatomical structure includes two-dimensional ultrasound image data of the anatomical structure or three-dimensional ultrasound body data of the anatomical structure.

14. The method according to any one of claims 1, 2, and 12, characterized in that, The method further includes: Based on the registration mapping relationship, a two-dimensional cross-sectional image of the second medical image of the second modality corresponding to the position of the first medical image of the first modality acquired in real time is determined; The two-dimensional cross-sectional images of the first medical image and the second medical image acquired in real time are fused to obtain a fused image; The image can be displayed as a fused image, or a two-dimensional cross-sectional image of the first medical image and the second medical image acquired in real time can be displayed simultaneously, or a two-dimensional cross-sectional image of the first medical image and the second medical image acquired in real time can be displayed overlaid.

15. The method as described in claim 14, characterized in that, The superimposed display of the two-dimensional cross-sectional image of the first medical image and the second medical image acquired in real time includes: The two-dimensional cross-sectional images of the first medical image and the second medical image acquired in real time are superimposed and displayed in different colors.

16. The medical image registration method according to any one of claims 1, 2, and 12, characterized in that, The second medical image of the second modality includes computed tomography (CT) images and / or magnetic resonance imaging (MRI) images.

17. A medical imaging system, characterized in that, The medical imaging system includes: Memory is used to store executable program instructions; A processor for executing the program instructions stored in the memory, causing the processor to perform the medical image registration method as described in any one of claims 1 to 16; Display device for displaying visual information.

18. An ultrasound imaging system, characterized in that, The ultrasound imaging system includes: Ultrasonic probe; A transmit / receive sequence controller is used to control the ultrasound probe to transmit ultrasound waves to the scanning location of the liver, and to receive ultrasound echoes based on the ultrasound waves returned from the liver to obtain ultrasound echo signals. A processor is configured to obtain multiple frames of two-dimensional ultrasound images related to the liver based on the ultrasound echo signals; Memory is used to store executable program instructions; The processor is configured to execute the program instructions stored in the memory, causing the processor to perform the medical image registration method as described in any one of claims 1 to 16; Display device for displaying visual information.

Images