Multimodal image registration method, registration device and computer readable storage medium

Abstract

The present invention provides a multimodal image registration method and a registration device and a computer-readable storage medium, wherein the registration method comprises the following steps: obtaining a reference image and an image to be registered; preprocessing the reference image and the image to be registered; obtaining a rotation and translation matrix of the first step based on mutual information as a registration measure, and using the rotation and translation matrix of the first step for preliminary registration; extracting key point feature vectors of target local objects of the reference image and the image to be registered respectively; and registering the extracted key point feature vectors to obtain a rotation and translation matrix of the second step, and using the rotation and translation matrix of the second step for target local object registration. The present invention solves the problem of local tissue deformation better through preliminary registration based on mutual information as a registration measure and key point feature vector registration, helps to improve registration accuracy, and assists doctors in improving the accuracy of diagnosis.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image registration, and in particular to a multimodal image registration method and a registration device and a computer-readable storage medium thereof. Background Art

[0002] Image registration is of great significance in medical image processing. Images of different modalities can provide information on different levels of the same tissue, which helps doctors improve the diagnosis rate of diseases. Common modalities for medical image registration include but are not limited to: magnetic resonance imaging (MRI), computed tomography (CT), and ultrasound imaging (US).

[0003] For example, in the detection of lung lesions, CT images are currently the most ideal functional molecular imaging method for lung diseases. They can be used for the evaluation of primary lung cancer lesions, the design of treatment plans, prognosis, efficacy evaluation, and prediction of recurrence and metastasis; while endobronchial US images can be used to guide transbronchial lung biopsy of peripheral lung lesions, or to guide intrabronchial ultrasound-guided needle aspiration biopsy of hilar mediastinal lymph node masses. Combining two different modalities of medical imaging, CT images and endobronchial US images, can more accurately diagnose patients. For example, in deep brain stimulation surgery (DBS for short), electrodes are implanted in specific areas deep in the brain, and an electric pulse generator is used to electrically stimulate the relevant neural nuclei in the brain, thereby controlling abnormal brain activity, which can significantly improve hereditary dystonia (Parkinson's disease). However, incorrect electrode implantation can lead to many side effects. Therefore, doctors often perform MRI scans on patients before surgery to obtain high-definition soft tissue images, and also perform density-sensitive CT scans of the skull and electrodes after surgery. Finally, the two different modalities of medical imaging, MRI images and CT images, are combined to obtain complete information on structures such as brain soft tissue, skull, and electrodes.

[0004] Therefore, it is necessary to use image registration technology to realize the combination of medical images of different modalities. At present, there are many image registration technologies, but the registration accuracy of multimodal images has not been well solved. At present, the main methods of multimodal image registration are semi-automatic registration and automatic registration; semi-automatic registration is mainly based on artificial marking points for registration, which is not very practical; automatic registration is completely based on image processing for registration, which has high requirements for image processing technology. The existing ones are mainly based on automatic registration. In the process of automatic registration, changes in the patient's physiological state, surgical posture, breathing, heartbeat, etc. will cause local deformation of organs or tissues, resulting in insufficient registration accuracy. For example, an image registration method disclosed in the prior art establishes a registration model with multiple non-coplanar straight lines, determines the mapping data of the first point between the reference image and the image to be registered according to the shortest distance between each two non-coplanar straight lines in the multiple non-coplanar straight lines, and uses the two sets of mapping data to calculate the rotation and translation matrix, and finally realizes the registration between multimodal images, but does not consider the problem of local deformation of organs or tissues, resulting in errors in registration. For another example, the prior art also discloses an image registration method, which uses a B-spline surface as a deformation model to realize the registration of multimodal images, and continuously increases the sampling ratio for cyclic iteration to improve the accuracy, but this method is relatively time-consuming, and only considers the overall deformation of the tissue, and does not consider the local deformation of the tissue. For another example, the prior art also discloses a multimodal deformable registration, which calculates the deformation field of different modal images and performs registration based on the deformation field vector, but for different tissues, it is necessary to prepare high-definition template images of different deformation states of the tissue in advance, which is not very practical.

[0005] Therefore, most of the existing multimodal image registration methods have the problem of registration deviation caused by not considering local tissue deformation, and the registration accuracy is low. Summary of the invention

[0006] In order to solve the above technical problems, the purpose of the present invention is to provide a multimodal image registration method and a registration device and a computer-readable storage medium to solve the problem of local deformation of tissues in medical multimodal images and improve the registration accuracy.

[0007] In order to achieve the above object, a multimodal image registration method is provided according to a first aspect of the present invention. The registration method comprises the following steps:

[0008] Acquire a reference image and an image to be registered;

[0009] Preprocessing the reference image and the image to be registered;

[0010] Based on the mutual information as the registration measure, a rotation and translation matrix of the first step is obtained, and the rotation and translation matrix of the first step is used to perform preliminary registration on the preprocessed reference image and the image to be registered;

[0011] respectively extracting key point feature vectors of target local objects of the reference image and the image to be registered; and

[0012] The key point feature vectors of the extracted reference image and the target local object of the image to be registered are registered to obtain the rotation and translation matrix of the second step, and the rotation and translation matrix of the second step is used to perform target local object registration on the reference image and the image to be registered after preliminary registration.

[0013] Optionally, the step of preprocessing the reference image and / or the image to be registered includes:

[0014] A Gaussian smoothing filter is used to smooth the reference image and / or the image to be registered.

[0015] Optionally, the step of obtaining the rotation and translation matrix of the first step based on mutual information as the registration measure, and using the rotation and translation matrix of the first step to perform preliminary registration on the preprocessed reference image and the image to be registered includes:

[0016] Respectively transforming the pixel coordinate systems of the reference image and the image to be registered into a physical coordinate system, where the physical coordinate system is the current physical coordinate system;

[0017] Based on the image grayscale, respectively calculating the centroids of the reference image and the image to be registered, and calculating the translation matrix between the centroids of the reference image and the image to be registered, and using the translation matrix as the initial matrix;

[0018] Based on the initial matrix, transforming the image to be registered from the current physical coordinate system to a new physical coordinate system through interpolation operation;

[0019] Counting the frequencies of occurrence of different grayscale combinations of corresponding pixel pairs in the overlapping area of ​​the reference image in the current physical coordinate system and the image to be registered in the new physical coordinate system, obtaining a joint histogram, and obtaining a joint entropy of the reference image and the image to be registered; and

[0020] The mutual information metric is calculated according to the joint entropy, and random gradient iteration is performed to obtain the rotation and translation matrix of the first step. The rotation and translation matrix of the first step is used to perform preliminary registration on the reference image and the image to be registered.

[0021] Optionally, the joint entropy is normalized before calculating the mutual information metric.

[0022] Optionally, in the step of transforming the pixel coordinate system of the reference image and the image to be registered into a physical coordinate system, the pixel coordinate system of the reference image and the image to be registered is transformed into a physical coordinate system using label information.

[0023] Optionally, the tag information includes: a spacing value of a single pixel converted to millimeters, an offset value between the origin of the pixel coordinate system and the origin of the physical coordinate system, and a direction value of the pixel coordinate system.

[0024] Optionally, the step of respectively extracting key point feature vectors of the target local object of the reference image and the image to be registered includes:

[0025] Selecting a region of interest for the target local object of the reference image and the image to be registered, to obtain a local reference image and a local image to be registered respectively;

[0026] Finding discrete extreme points in the local reference image and the local image to be registered;

[0027] Placing each discrete extreme point in a multi-quadrant coordinate system, dividing a corresponding image block in the local reference image and the local image to be registered with each discrete extreme point as the center, and dividing the image block into a plurality of image units;

[0028] Calculating the gradient of each pixel point in each image unit on each coordinate axis of the physical coordinate system, and accumulating the directions and gradient amplitudes of all pixel points in each image unit to each quadrant of the multi-quadrant coordinate system to form a gradient histogram; and

[0029] According to the gradient histogram, the feature vector corresponding to each discrete extreme point is obtained as the key point feature vector.

[0030] Optionally, the step of finding discrete extreme points in the local reference image and the local image to be registered includes:

[0031] Selecting Gaussian distributions with different standard deviations as different scales, performing Gaussian smoothing on the local reference image and the local image to be registered, and calculating a Gaussian difference pyramid; and

[0032] In the Gaussian difference pyramid, the specific neighborhood of all pixels is traversed, and the point with the largest gray value in the specific neighborhood is found as the discrete extreme point.

[0033] Optionally, the registration method is used for registration of two-dimensional images, the specific neighborhoods are the six neighborhoods of up, down, left, right, front and back, or all twenty-six neighborhoods of the pixel point, the multi-quadrant coordinate system is a plane coordinate system in which 360 degrees of the two-dimensional plane are equally divided into 8 to 32 quadrants, the image block is a circle centered on the discrete extreme point and formed with a radius of 3 to 6 times the standard deviation of the corresponding Gaussian distribution, the image unit is the remaining M-1 squares of the M squares into which the largest inscribed square of the image block is equally divided, excluding the middle square where the discrete extreme point is located, where M is the square of a positive integer greater than one.

[0034] Optionally, the registration method is used for three-dimensional image registration, the specific neighborhoods are the six neighborhoods of up, down, left, right, front, and back, or all twenty-six neighborhoods of the pixel point, the multi-quadrant coordinate system is a spherical coordinate system of 32 spheres, the image block is a sphere centered on the discrete extreme point and formed with a radius of 3-6 times the standard deviation of the corresponding Gaussian distribution, and the image unit is the remaining M-1 cubes of the M cubes divided by the largest inscribed cube in the image block, excluding the middle cube where the discrete extreme point is located, where M is the cube of a positive integer greater than one.

[0035] Optionally, the key point feature vectors of the extracted reference image and the target local object of the image to be registered are registered to obtain the rotation and translation matrix of the second step, and the step of using the rotation and translation matrix of the second step to register the target local object of the reference image and the image to be registered after preliminary registration includes:

[0036] Randomly sample multiple pairs of key point feature vectors from all key point feature vectors of the local reference image and the local image to be registered;

[0037] Calculate the affine transformation matrix between the local reference image and the local image to be registered based on the sampled multiple pairs of key point feature vectors;

[0038] Apply the affine transformation matrix to all key point feature vectors of the local image to be registered and calculate the error value;

[0039] When the error value meets the predetermined condition, the current affine transformation matrix is ​​used as a valid affine transformation matrix and as the rotation and translation matrix of the second step;

[0040] When the error value does not meet the predetermined conditions, continue random sampling, recalculate the affine transformation matrix, and calculate the error value again until the error value meets the predetermined conditions and use the recalculated affine transformation matrix as the rotation and translation matrix of the second step.

[0041] According to a second aspect of the present invention, a multimodal image registration device is provided, the registration device comprising:

[0042] An image acquisition unit, used for acquiring a reference image and an image to be registered;

[0043] An image preprocessing unit, used for preprocessing the reference image and the image to be registered;

[0044] A preliminary registration unit, which obtains a rotation and translation matrix of the first step based on mutual information as a registration measure, and uses the rotation and translation matrix of the first step to perform preliminary registration on the preprocessed reference image and the image to be registered; and

[0045] The target local object registration unit is used to extract the key point feature vectors of the target local objects of the reference image and the image to be registered respectively, and to register the extracted key point feature vectors of the target local objects of the reference image and the image to be registered to obtain the rotation and translation matrix of the second step, and use the rotation and translation matrix of the second step to perform target local object registration on the reference image and the image to be registered after the preliminary registration.

[0046] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multimodal image registration method as described in any technical solution of the present invention are performed.

[0047] The beneficial effects of the present invention are:

[0048] After local tissue deformation, the tissue on the original coordinates shifts and becomes other tissue. The existing image registration technology does not consider the problem of local tissue deformation. When doctors combine images for diagnosis, it will cause interference or even diagnostic errors. Compared with the existing technology, the multimodal image registration method and its registration device and computer-readable storage medium of the present invention use preliminary registration and key point feature vector registration based on mutual information as the registration measure. In the multimodal medical image registration, the local tissue of interest can be selected as the target local object, and the key point feature vector of the target local object can be extracted for registration, which better solves the problem of local tissue deformation, helps to improve the registration accuracy, and assists doctors to improve the accuracy of diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart of a multimodal image registration method according to an embodiment of the present invention;

[0050] Figure 2 is a flow chart of step S3;

[0051] Figure 3 is a flow chart of step S4;

[0052] Figure 4 is a flowchart of step S42;

[0053] Figure 5 is a flow chart of step S5;

[0054] Figure 6 This is the MRI image of the axial plane of the human brain before surgery;

[0055] Figure 7 This is a coronal MRI image of the human brain before surgery;

[0056] Figure 8 This is a sagittal MRI image of the human brain before surgery;

[0057] Fig. 9 This is a CT image of the axial plane of the human brain after surgery;

[0058] Fig.10 This is a CT image of the coronal plane of the human brain after surgery;

[0059] Fig.11 This is a CT image of the sagittal plane of the human brain after surgery;

[0060] Fig.12 The axial plane MRI image and CT image of the human brain in the same physical coordinate system before registration;

[0061] Fig.13 It is the coronal MRI image and CT image of the human brain in the same physical coordinate system before registration;

[0062] Fig.14 The sagittal MRI image and CT image of the human brain in the same physical coordinate system before registration;

[0063] Fig.15 The axial plane MRI image and CT image of the human brain in the same physical coordinate system after registration;

[0064] Fig.16 It is the coronal MRI image and CT image of the human brain in the same physical coordinate system after registration;

[0065] Fig.17 The sagittal MRI image and CT image of the human brain in the same physical coordinate system after registration. DETAILED DESCRIPTION

[0066] In order to make the content of the present invention clearer and easier to understand, the following is a description of a multimodal image registration method and a registration device and a computer-readable storage medium according to an embodiment of the present invention in conjunction with the accompanying drawings. However, it can be understood that the present invention is not limited to the specific embodiments described below, and general replacements known to those skilled in the art are also covered within the scope of protection of the present invention. It should be noted that the drawings are all in a very simplified form and in an inaccurate scale, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0067] like Figure 1 As shown, an embodiment of the present invention provides a multimodal image registration method, the registration method comprising the following steps:

[0068] Step S1: Acquire a reference image and an image to be registered;

[0069] In step S1, this embodiment takes the human brain as an example, and takes an MRI image before surgery as a reference image (such as Figure 6-Figure 8 CT images were taken after surgery (as shown in Figure 9-11As shown in the figure, as the image to be registered, the registration method provided in this embodiment is to register the CT image to the MRI image, but the present invention is not limited thereto. The image may be a medical image obtained by magnetic resonance imaging (MRI), computed tomography (CT) or ultrasound imaging (US), such as an MRI human brain volume data image, a CTA human brain volume data image, etc. Among them, the CTA image is a special CT image obtained by computed tomography (CT), which can obtain the three-dimensional data structure of the tissue.

[0070] Step S2: preprocessing the reference image and the image to be registered; preprocessing the reference image and the image to be registered refers to image preprocessing operations that eliminate irrelevant information in the image and enhance the detectability of relevant information, thereby improving the reliability of subsequent image processing steps, including removing image noise and enhancing the image.

[0071] Step S3: Based on mutual information as a registration measure, a rotation and translation matrix of the first step is obtained, and the rotation and translation matrix of the first step is used to perform preliminary registration on the preprocessed reference image and the image to be registered;

[0072] Step S4: extracting key point feature vectors of the target local object of the reference image and the image to be registered respectively;

[0073] In step S4, taking the human brain as an example, the human brain includes two parts: the skull and the neural nuclei in the skull. In step S3, in which preliminary registration is performed based on mutual information, it is easy for most areas of the skull to be aligned, while the neural nuclei are not aligned. In DBS surgery, it is necessary to obtain information about the soft tissue of the brain through MRI images before surgery, and to obtain information about the skull and electrodes through CT images after surgery. The electrodes are usually implanted near the deep neural nuclei of the brain. Therefore, in this embodiment, the area where the neural nuclei are located is selected as the target local object of the reference image and the image to be registered, and the target local object is registered to improve the registration accuracy and assist doctors in making diagnoses during DBS surgery.

[0074] Step S5: align the extracted key point feature vectors of the target local object of the reference image and the image to be registered to obtain the rotation and translation matrix of the second step, and use the rotation and translation matrix of the second step to align the target local object of the reference image and the image to be registered after preliminary registration.

[0075] In step S2, the step of preprocessing the reference image and / or the image to be registered includes processing the reference image and / or the image to be registered using a Gaussian smoothing filter. The Gaussian smoothing filter can filter out noise information in the image and blur the image to a certain extent to avoid falling into a local optimum when the mutual information is used as the registration measure in step S3 (the mutual information is a non-smooth unimodal function with many local optimum values).

[0076] like Figure 2 As shown, in step S3, based on the mutual information as the registration measure, the rotation and translation matrix of the first step is obtained, and the steps of using the rotation and translation matrix of the first step to perform preliminary registration on the preprocessed reference image and the image to be registered include:

[0077] Step S31: transform the pixel coordinate systems of the reference image and the image to be registered into a physical coordinate system respectively. The physical coordinate system refers to the final output space after the image transformation, and the position of each pixel is usually measured in actual length units.

[0078] For example, by utilizing the tag information (Tag information) of the reference image and the image to be registered, wherein the tag information includes the spacing value (spacing value) of the conversion between a single pixel of each image and millimeters, the origin offset value (origin value) between the origin of the pixel coordinate system and the physical coordinate system, and the direction value (direction value) of the pixel coordinate system, etc., the pixel coordinate system of the reference image and the image to be registered is transformed into a physical coordinate system to achieve the unification of the size dimension information of the reference image and the image to be registered.

[0079] like Figure 12-14 As shown, the MRI images and CT images of the axial plane, coronal plane and sagittal plane of the human brain in the same physical coordinate system before registration are respectively shown. The MRI image and the CT image cannot be completely overlapped before registration. The human brain tissue has offsets in both rotation angle and translation distance, and registration is required. It cannot be used as a basis for doctors' diagnosis.

[0080] Step S32: Based on the image grayscale, the centroids of the reference image and the image to be registered are calculated respectively, and the translation matrix between the centroids of the reference image and the image to be registered is calculated, and the translation matrix is ​​used as the initial matrix. Here, by calculating the initial matrix, unnecessary iterations in the subsequent step calculations can be reduced.

[0081] Step S33: Based on the initial matrix, the image to be registered is transformed from the existing physical coordinate system to a new physical coordinate system through interpolation operation.

[0082] The interpolation operation is used to solve the problem of assigning grayscale values ​​to pixel values ​​at the new physical coordinate system after the image to be registered is transformed into the new coordinate system. In this embodiment, the interpolation operation can adopt the nearest neighbor method, bilinear method, partial volume distribution interpolation method (Partial Volume Distribution) and other methods. Compared with some other high-order interpolation methods, the amount of calculation is small. The partial volume distribution interpolation method is preferably used. This method uses the weight distribution principle of linear interpolation to distribute contributions. It has better smoothness than the nearest neighbor method and bilinear method, which is conducive to optimizing the search for discrete extreme points in subsequent steps.

[0083] Step 34: Count the frequencies of occurrence of different grayscale combinations of corresponding pixel pairs in the overlapping area of ​​the reference image in the current physical coordinate system and the image to be registered in the new physical coordinate system, obtain a joint histogram, and obtain the joint entropy of the reference image and the image to be registered.

[0084] Specifically, the number of occurrences of different grayscale combinations (represented by (i, j), where i and j represent the grayscale of the reference image and the image to be registered, respectively) of the corresponding pixel pairs in the overlapping area of ​​the reference image and the image to be registered is counted to obtain a joint histogram between the two, and H is obtained. AB (i, j) value. Where A and B represent the reference image and the image to be registered, respectively, and H AB (i, j) is the joint entropy of the reference image and the image to be registered.

[0085] Step 35: Calculate the mutual information measure based on the joint entropy, perform random gradient iteration, obtain the rotation and translation matrix of the first step, and use the rotation and translation matrix of the first step to perform preliminary registration on the reference image and the image to be registered. The joint entropy is preferably normalized, and then the mutual information measure is calculated using the normalized joint entropy, that is, the normalized mutual information is used as the registration measure. The essence of using mutual information as the registration measure for preliminary registration is that when the spatial positions of the reference image and the image to be registered are consistent, their mutual information measure value should reach the maximum value. However, if the registration is performed based on the maximum mutual information method alone, the accuracy and robustness are insufficient. The reason is that the method based on the maximum mutual information treats all pixels in the image equally, which is prone to mismatching, and the normalized mutual information method can better solve this problem.

[0086] The calculation process of calculating the mutual information metric based on the joint entropy or the normalized joint entropy is as follows: the obtained joint entropy H AB The (i, j) value or the normalized value is substituted into the following equations 1 to 4 in turn to calculate the mutual information metric I(A, B), and random gradient iteration is performed to finally obtain the rotation and translation matrix of the first step.

[0087]

[0088]

[0089] P A (i)=∑ j H AB (i, j) (Formula 3)

[0090] P B (j) = ∑ i H AB (i, j) (Formula 4)

[0091] Where i and j represent the grayscale of the reference image and the image to be registered, respectively; A and B represent the reference image and the image to be registered, respectively; I(A, B) is the mutual information measure; P AB (i, j) is the joint probability distribution, P A (i) is the marginal probability distribution of the reference image, P B (j) is the edge probability distribution of the image to be registered.

[0092] like Figure 3 As shown, in step S4, the steps of respectively extracting key point feature vectors of the target local objects of the reference image and the image to be registered include:

[0093] Step S41: selecting a region of interest for the target local object of the reference image and the image to be registered, and obtaining a local reference image and a local image to be registered respectively.

[0094] In this embodiment, the area where the neural nucleus is located is selected as the region of interest. The region of interest can be selected based on prior knowledge, for example, it can be selected based on the relative position of the neural nucleus and the skull. Preferably, the region of interest can be made into a registration file and read directly to reduce the cost of manual intervention.

[0095] Step S42: finding discrete extreme points in the local reference image and the local image to be registered.

[0096] Step S43: placing each discrete extreme point in a multi-quadrant coordinate system, dividing corresponding image blocks in the local reference image and the local image to be registered with each discrete extreme point as the center, and dividing the image blocks into multiple image units.

[0097] Step S44: Calculate the gradient of each pixel point in each image unit on each coordinate axis of the physical coordinate system, and accumulate the directions and gradient amplitudes of all pixel points in each image unit to each quadrant of the multi-quadrant coordinate system to form a gradient histogram.

[0098] Step S45: According to the gradient histogram, a feature vector corresponding to each discrete extreme point is obtained as a key point feature vector.

[0099] The method of extracting key point feature vectors of the target local objects of the reference image and the image to be registered respectively in this embodiment combines the actual image and constructs different gradient histograms to better describe the key point information.

[0100] like Figure 4 As shown, in step S42, the step of finding discrete extreme points in the local reference image and the local image to be registered includes:

[0101] Step S421: Select Gaussian distributions with different standard deviations as different scales, perform Gaussian smoothing on the local reference image and the local image to be registered, and calculate a Gaussian difference pyramid. For example, the Gaussian difference pyramid can be obtained by the following steps: select 5 kinds of σ (σ is the standard deviation of the Gaussian distribution) of Gaussian distributions as different scales, perform Gaussian smoothing on the local reference image and the local image to be registered, and obtain 5 groups of images respectively, calculate the difference between them two by two, and obtain 4 groups of images to form a Gaussian difference pyramid. Of course, other Gaussian smoothing methods with other scale settings can be selected, and the present invention is not limited to this.

[0102] Step S422: Traverse the specific neighborhoods of all pixels in the Gaussian difference pyramid, and find the point with the largest grayscale value in the specific neighborhood as the discrete extreme point. The specific neighborhood is the six neighborhoods of the pixel point, the top, the bottom, the left, the right, the front, and the back, or all the twenty-six neighborhoods. The twenty-six neighborhoods here refer to dividing a cube into 3*3*3 small cubes, and placing a pixel point in the middle of a small cube. Then, there are 26 small cubes around the pixel point, that is, twenty-six neighborhoods.

[0103] When the registration method is used for the registration of two-dimensional graphics, the specific neighborhood is the six neighborhoods of the top, bottom, left, right, front and back of the pixel point or all twenty-six neighborhoods. The multi-quadrant coordinate system is a plane coordinate system in which the 360 ​​degrees of the two-dimensional plane are equally divided into 8 to 32 quadrants, preferably 32 quadrants. The image block is a circle formed with the discrete extreme point as the center and a radius of 3-6 times the standard deviation of the corresponding Gaussian distribution, preferably with a radius of 4.5 times the standard deviation of the corresponding Gaussian distribution (i.e. r = 4.5*σ, σ is the standard deviation of the Gaussian distribution corresponding to the discrete extreme point). The image unit is the remaining M-1 squares of the M squares into which the inscribed largest square of the image block is equally divided, excluding the middle square where the discrete extreme point is located, where M is the square of a positive integer greater than one, such as 4, 8, 9 or 16.

[0104] The following uses the example that the multi-quadrant coordinate system uses a 32-quadrant plane coordinate system, the image block uses a value of 4.5 times the standard deviation of the corresponding Gaussian distribution as the radius, and the image unit uses a segmentation method of 8 squares (that is, the M value is 9) to illustrate the specific steps of the registration method for the registration of two-dimensional graphics:

[0105] The step S43 is: placing each discrete extreme point in a plane coordinate system of 32 quadrants divided equally by 360 degrees of the two-dimensional plane, and forming a circle with the discrete extreme point as the center of the feature point and 4.5 times the standard deviation of the corresponding Gaussian distribution as the radius (i.e. r=4.5*σ, σ is the standard deviation of the Gaussian distribution corresponding to the discrete extreme point), and dividing the largest inscribed square of the circle into 9 squares, excluding the middle square where the discrete extreme point is located, so as to obtain a total of 8 squares.

[0106] The step S44 is: respectively calculating the gradient of each pixel point in the 8 squares in the x-axis and y-axis directions of the physical coordinate system to obtain the gradient of the pixel point, placing the gradient of each pixel point in the 32-quadrant plane coordinate system, calculating the direction and gradient amplitude of the pixel point, and accumulating the directions and gradient amplitudes of all the pixel points in the 8 squares to the 32 quadrants of the plane coordinate system to form a two-dimensional gradient histogram.

[0107] The step S45 is: according to the two-dimensional gradient histogram, 8*32 feature vectors corresponding to each discrete extreme point are obtained. If there are N discrete extreme points, a total of N*8*32 feature vectors are obtained as key point feature vectors; N is a positive integer greater than 1.

[0108] By adopting the above steps, a method for obtaining discrete extreme points can be used to better obtain the key point feature vector of the target local object of the two-dimensional image with high accuracy, which can be generally applicable to the image alignment of various two-dimensional medical images to assist doctors in improving the accuracy of diagnosis.

[0109] Referring to the above-mentioned method for extracting key point feature vectors from two-dimensional images, this embodiment further provides a method for key point feature vectors that can be used for three-dimensional images. When the registration method is used for the registration of three-dimensional images, the specific neighborhood is the six neighborhoods of the top, bottom, left, right, front and back of the pixel point or all twenty-six neighborhoods, and the multi-quadrant coordinate system is a spherical coordinate system of 32 spheres. The image block is a sphere formed with a discrete extreme point as the center and a value of 3-6 times the standard deviation of the corresponding Gaussian distribution as the radius, preferably with a value of 4.5 times the standard deviation of the corresponding Gaussian distribution as the radius (i.e. r=4.5*σ, σ is the standard deviation of the Gaussian distribution corresponding to the discrete extreme point). The image unit is the remaining M-1 cubes of the M cubes that the inscribed maximum cube of the image block is evenly divided into, excluding the middle cube where the discrete extreme point is located, where M is the cube number of a positive integer greater than one, for example, 27 cubes.

[0110] The following uses the example that the multi-quadrant coordinate system uses a 32-quadrant spherical coordinate system, the image block uses a value of 4.5 times the standard deviation of the corresponding Gaussian distribution as the radius, and the image unit uses a segmentation method of 26 cubes (that is, the M value is 27) to illustrate the specific steps of the registration method for the registration of three-dimensional graphics:

[0111] The step S43 is: placing each discrete extreme point in a spherical coordinate system of a 32-sphere (i.e., a spherical coordinate system of a 32-quadrant), and forming a sphere with the discrete extreme point as the feature point center and 4.5 times the standard deviation of the corresponding Gaussian distribution as the radius, and dividing the largest cube inscribed in the sphere into 27 cubes, excluding the middle cube where the discrete extreme point is located, to obtain a total of 26 cubes.

[0112] The step S44 is: respectively calculating the gradient of each pixel point in the 26 cubes in the x-axis, y-axis and z-axis directions of the physical coordinate system to obtain the gradient of the pixel point, placing the gradient of each pixel point in the spherical coordinate system of 32 spheres, calculating the direction and gradient amplitude of the pixel point, and accumulating the directions and gradient amplitudes of all pixel points in the 26 cubes on the 32 spheres of the spherical coordinate system to form a three-dimensional gradient histogram.

[0113] Among them, the gradient amplitude of the pixel point is the modulus of the main gradient, and the main direction of the pixel point in the spherical coordinate system can be calculated as follows (Formula 5):

[0114]

[0115] Among them, θ is the main direction of the pixel, is the main gradient of the pixel, is the unit normal vector of the sphere on which the main gradient of the pixel point falls.

[0116] The step S45 is: according to the three-dimensional gradient histogram, 26*32 feature vectors corresponding to each discrete extreme point are obtained. If there are N discrete extreme points, a total of N*26*32 feature vectors are obtained as key point feature vectors; N is a positive integer greater than 1.

[0117] By adopting the above steps and referring to the method for extracting key point feature vectors of target local objects in two-dimensional images, the method for extracting key point feature vectors of target local objects in three-dimensional images provided in this embodiment also uses discrete extreme points as key point feature vectors, and can better find discrete extreme points with high accuracy. It can be generally applicable to image alignment of various three-dimensional medical images to assist doctors in improving the accuracy of diagnosis.

[0118] like Figure 5 As shown, in step S5, the key point feature vectors of the extracted reference image and the target local object of the image to be registered are registered to obtain the rotation and translation matrix of the second step, and the step of using the rotation and translation matrix of the second step to register the target local object of the reference image and the image to be registered after preliminary registration includes:

[0119] Step S51: randomly sampling multiple pairs of key point feature vectors from all key point feature vectors of the local reference image and the local image to be registered.

[0120] Step S52: Calculate the affine transformation matrix between the local reference image and the local image to be registered based on the sampled pairs of key point feature vectors.

[0121] Step S53: applying the affine transformation matrix to all key point feature vectors of the local image to be registered, and calculating the error value.

[0122] Step S54: When the error value meets the predetermined condition, the current affine transformation matrix is ​​used as a valid affine transformation matrix and as the rotation and translation matrix of the second step. The predetermined condition is set, for example, when the error value is less than a threshold or reaches a set number of iterations.

[0123] Step S55: When the error value does not meet the predetermined conditions, continue random sampling, recalculate the affine transformation matrix, and calculate the error value again until the error value meets the predetermined conditions and use the recalculated affine transformation matrix as the rotation and translation matrix of the second step.

[0124] like Figure 15-17 As shown in the figure, the CT images and MRI images of the axial plane, coronal plane, and sagittal plane of the human brain are respectively registered in the same physical coordinate system. The three views of the human brain can be well aligned after the registration, especially the alignment effect of the target local object (neural nucleus) is good, which effectively avoids the mismatch caused by local tissue deformation, and the registration accuracy is high.

[0125] This embodiment further provides a multimodal image registration device, which is used to execute the multimodal image registration method described in any of the above technical solutions, and the registration device includes:

[0126] An image acquisition unit, used for acquiring a reference image and an image to be registered;

[0127] An image preprocessing unit, used for preprocessing the reference image and the image to be registered;

[0128] A preliminary registration unit, which obtains a rotation and translation matrix of the first step based on mutual information as a registration measure, and uses the rotation and translation matrix of the first step to perform preliminary registration on the preprocessed reference image and the image to be registered; and

[0129] The target local object registration unit is used to extract the key point feature vectors of the target local objects of the reference image and the image to be registered respectively, and to register the extracted key point feature vectors of the target local objects of the reference image and the image to be registered to obtain the rotation and translation matrix of the second step, and use the rotation and translation matrix of the second step to perform target local object registration on the reference image and the image to be registered after the preliminary registration.

[0130] This embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multimodal image registration method described in any of the above technical solutions are performed.

[0131] In summary, the present invention provides a multimodal image registration method and registration device and computer-readable storage medium. After preliminary registration based on mutual information as a registration measure, the key point feature vector of the target local object is extracted to perform target local object registration, thereby improving the registration accuracy of the target local object. In particular, in medical image registration, images of different modalities may undergo local tissue deformation due to changes in the patient's physiological state, and the tissue on the original coordinates may shift and become other tissues, which may interfere with the doctor's diagnosis. The present invention can select the local tissue of interest as the target local object, extract its key point feature vector for registration, thereby better solving the problem of local tissue deformation and improving the registration accuracy and the accuracy of the doctor's diagnosis.

[0132] The above description is only a description of some embodiments of the present invention, and is not intended to limit the scope of the present invention. Any changes or modifications made by a person skilled in the art in the field of the present invention based on the above disclosure shall fall within the protection scope of the present invention.

Claims

1. A multimodal image registration method, characterized in that: include: Acquire a reference image and an image to be registered; Preprocessing the reference image and the image to be registered; Based on the mutual information as the registration measure, a rotation and translation matrix of the first step is obtained, and the rotation and translation matrix of the first step is used to perform preliminary registration on the preprocessed reference image and the image to be registered; Extracting key point feature vectors of target local objects of the reference image and the image to be registered respectively; as well as Registering the extracted key point feature vectors of the target local object of the reference image and the image to be registered to obtain the rotation and translation matrix of the second step, and registering the target local object of the reference image and the image to be registered after the preliminary registration using the rotation and translation matrix of the second step; The steps of respectively extracting key point feature vectors of the target local objects of the reference image and the image to be registered include: Selecting a region of interest for the target local object of the reference image and the image to be registered, to obtain a local reference image and a local image to be registered respectively; Finding discrete extreme points in the local reference image and the local image to be registered; Placing each discrete extreme point in a multi-quadrant coordinate system, dividing a corresponding image block in the local reference image and the local image to be registered with each discrete extreme point as the center, and dividing the image block into a plurality of image units; Calculating the gradient of each pixel point in each image unit on each coordinate axis of the physical coordinate system, and accumulating the directions and gradient amplitudes of all pixel points in each image unit to each quadrant of the multi-quadrant coordinate system to form a gradient histogram; and According to the gradient histogram, the feature vector corresponding to each discrete extreme point is obtained as the key point feature vector.

2. The multimodal image registration method according to claim 1, wherein: The step of preprocessing the reference image and / or the image to be registered comprises: A Gaussian smoothing filter is used to smooth the reference image and / or the image to be registered.

3. The multimodal image registration method according to claim 1, wherein: Based on the mutual information as the registration measure, the rotation and translation matrix of the first step is obtained, and the steps of using the rotation and translation matrix of the first step to perform preliminary registration on the preprocessed reference image and the image to be registered include: Respectively transforming the pixel coordinate systems of the reference image and the image to be registered into a physical coordinate system, where the physical coordinate system is the current physical coordinate system; Based on the image grayscale, respectively calculating the centroids of the reference image and the image to be registered, and calculating the translation matrix between the centroids of the reference image and the image to be registered, and using the translation matrix as the initial matrix; Based on the initial matrix, transforming the image to be registered from the current physical coordinate system to a new physical coordinate system through interpolation operation; Counting the frequencies of occurrence of different grayscale combinations of corresponding pixel pairs in the overlapping area of ​​the reference image in the current physical coordinate system and the image to be registered in the new physical coordinate system, obtaining a joint histogram, and obtaining a joint entropy of the reference image and the image to be registered; and The mutual information metric is calculated according to the joint entropy, and random gradient iteration is performed to obtain the rotation and translation matrix of the first step. The rotation and translation matrix of the first step is used to perform preliminary registration on the reference image and the image to be registered.

4. The multimodal image registration method according to claim 3, characterized in that: Before calculating the mutual information metric, the joint entropy is normalized.

5. The multimodal image registration method according to claim 3 or 4, characterized in that: In the step of transforming the pixel coordinate system of the reference image and the image to be registered into a physical coordinate system, the pixel coordinate system of the reference image and the image to be registered is transformed into a physical coordinate system using label information.

6. The multimodal image registration method according to claim 5, characterized in that: The tag information includes: a spacing value of a single pixel converted to millimeters, an offset value between the origin of the pixel coordinate system and the origin of the physical coordinate system, and a direction value of the pixel coordinate system.

7. The multimodal image registration method according to claim 1, characterized in that: The step of finding discrete extreme points in the local reference image and the local image to be registered comprises: Selecting Gaussian distributions with different standard deviations as different scales, performing Gaussian smoothing on the local reference image and the local image to be registered, and calculating a Gaussian difference pyramid; and In the Gaussian difference pyramid, the specific neighborhood of all pixels is traversed, and the point with the largest gray value in the specific neighborhood is found as the discrete extreme point.

8. The multimodal image registration method according to claim 7, characterized in that: The registration method is used for registration of two-dimensional images, the specific neighborhoods are six neighborhoods of the top, bottom, left, right, front and back or all twenty-six neighborhoods of the pixel point, the multi-quadrant coordinate system is a plane coordinate system in which 360 degrees of the two-dimensional plane are equally divided into 8 to 32 quadrants, the image block is a circle formed with a discrete extreme point as the center and a value of 3 to 6 times the standard deviation of the corresponding Gaussian distribution as the radius, and the image unit is the remaining M-1 squares of the M squares into which the largest inscribed square of the image block is equally divided, excluding the middle square where the discrete extreme point is located, wherein M is the square of a positive integer greater than one.

9. The multimodal image registration method according to claim 7, characterized in that: The registration method is used for registration of three-dimensional images, the specific neighborhoods are six neighborhoods of the top, bottom, left, right, front and back or all twenty-six neighborhoods of the pixel point, the multi-quadrant coordinate system is a spherical coordinate system of 32 spheres, the image block is a sphere formed with a discrete extreme point as the center and a value of 3-6 times the standard deviation of the corresponding Gaussian distribution as the radius, and the image unit is the remaining M-1 cubes of the M cubes that the largest inscribed cube of the image block is evenly divided into, excluding the middle cube where the discrete extreme point is located, wherein M is the cube number of a positive integer greater than one.

10. The multimodal image registration method according to any one of claims 1, 8 and 9, characterized in that: The key point feature vectors of the extracted reference image and the target local object of the image to be registered are registered to obtain the rotation and translation matrix of the second step, and the step of using the rotation and translation matrix of the second step to register the target local object of the reference image after preliminary registration and the image to be registered includes: Randomly sample multiple pairs of key point feature vectors from all key point feature vectors of the local reference image and the local image to be registered; Calculate the affine transformation matrix between the local reference image and the local image to be registered based on the sampled multiple pairs of key point feature vectors; Apply the affine transformation matrix to all key point feature vectors of the local image to be registered and calculate the error value; When the error value meets the predetermined condition, the current affine transformation matrix is ​​used as a valid affine transformation matrix and as the rotation and translation matrix of the second step; When the error value does not meet the predetermined conditions, continue random sampling, recalculate the affine transformation matrix, and calculate the error value again until the error value meets the predetermined conditions and use the recalculated affine transformation matrix as the rotation and translation matrix of the second step.

11. A multimodal image registration device, characterized in that: include: An image acquisition unit, used for acquiring a reference image and an image to be registered; An image preprocessing unit, used for preprocessing the reference image and the image to be registered; A preliminary registration unit, which obtains a rotation and translation matrix of the first step based on mutual information as a registration measure, and uses the rotation and translation matrix of the first step to perform preliminary registration on the preprocessed reference image and the image to be registered; and A target local object registration unit is used to extract key point feature vectors of the target local object of the reference image and the image to be registered, respectively, and to register the extracted key point feature vectors of the target local object of the reference image and the image to be registered to obtain a rotation and translation matrix of the second step, and to perform target local object registration on the reference image and the image to be registered after preliminary registration using the rotation and translation matrix of the second step; The target local object registration unit is used to extract key point feature vectors of the target local objects of the reference image and the image to be registered respectively through the following process: Selecting a region of interest for the target local object of the reference image and the image to be registered, to obtain a local reference image and a local image to be registered respectively; Finding discrete extreme points in the local reference image and the local image to be registered; Placing each discrete extreme point in a multi-quadrant coordinate system, dividing a corresponding image block in the local reference image and the local image to be registered with each discrete extreme point as the center, and dividing the image block into a plurality of image units; Calculate the gradient of each pixel point in each image unit on each coordinate axis of the physical coordinate system, and accumulate the directions and gradient amplitudes of all pixel points in each image unit to each quadrant of the multi-quadrant coordinate system to form a gradient histogram; as well as According to the gradient histogram, the feature vector corresponding to each discrete extreme point is obtained as the key point feature vector.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimodal image registration method according to any one of claims 1 to 10 are performed.

Images