Image registration method and related apparatus

Abstract

An image registration method and related device are provided. The method includes: obtaining an ultrasound contour of a target organ and a computed tomography (CT) contour; the ultrasound contour is obtained according to an ultrasound image collected by an ultrasound device, and the CT contour is obtained according to a CT image collected by a CT device; rotating the CT contour to obtain a rotated CT contour; an included angle between a direction vector of a reference direction in a space of the ultrasound contour and a direction vector of the reference direction in a space of the rotated CT contour is less than or equal to a first threshold value, the reference direction being common to the space of the ultrasound contour and the space of the CT contour; and performing registration based on the rotated CT contour and the ultrasound contour. The registration time can be shortened.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer, in particular to an image registration method and related apparatus. BACKGROUND

[0002] With the continuous development of computer science and technology, image registration technology is also becoming more and more mature. Image registration technology has a wide range of applications in the fields of computer vision, material mechanics and medical image processing.

[0003] For example, in a kidney-related puncture surgery, it is necessary to register a kidney ultrasound contour obtained from an ultrasound image and a kidney CT contour obtained from a computed tomography (CT) image, so that a surgeon or other surgical operator can obtain internal information of the kidney during the operation, thereby reducing surgical errors.

[0004] However, for example, the kidney and other organs are three-dimensional structures, and the three-dimensional structure corresponds to a large amount of three-dimensional image data. The registration time is long by using the existing image space features for registration. SUMMARY

[0005] Embodiments of the present application provide an image registration method and related apparatus, which can shorten the registration time.

[0006] In a first aspect, embodiments of the present application provide an image registration method, comprising:

[0007] Obtaining an ultrasound contour of a target organ and a computed tomography (CT) contour; the ultrasound contour is obtained from an ultrasound image collected by an ultrasound device, and the CT contour is obtained from a CT image collected by a CT device;

[0008] Rotating the CT contour to obtain a rotated CT contour; the included angle between a direction vector of a reference direction in a space of the ultrasound contour and a direction vector of the reference direction in a space of the rotated CT contour is less than or equal to a first threshold value, and the reference direction is common to the space of the ultrasound contour and the space of the CT contour;

[0009] Registering the rotated CT contour and the ultrasound contour.

[0010] In combination with the first aspect, in a possible implementation manner, the rotating the CT contour to obtain the rotated CT contour comprises:

[0011] Determining a first direction vector of the reference direction in the space of the CT contour, and determining a second direction vector of the reference direction in the space of the ultrasound contour;

[0012] determining a rotation matrix according to an angle between the first direction vector and the second direction vector, and an angle between the second direction vector and a coordinate axis;

[0013] rotating the CT contour based on the rotation matrix to obtain a rotated CT contour.

[0014] With reference to the first aspect, in a possible implementation, the reference direction includes a first sub-reference direction, a second sub-reference direction, and a third sub-reference direction; a direction vector of the first sub-reference direction, the second sub-reference direction, and the third sub-reference direction in the space of the ultrasound contour is less than or equal to the first threshold value respectively.

[0015] With reference to the first aspect, in a possible implementation, the first sub-reference direction is a direction from a foot of the subject to a head of the subject, the second sub-reference direction is a direction perpendicular outward of a back of the subject, and the third sub-reference direction is a direction from a body of the subject to a left hand of the subject.

[0016] With reference to the first aspect, in a possible implementation, the registration based on the rotated CT contour and the ultrasound contour includes:

[0017] In a case where a distance between a center coordinate of the rotated CT contour and a center coordinate of the ultrasound contour is less than a second threshold value, the registration based on the rotated CT contour and the ultrasound contour is performed.

[0018] With reference to the first aspect, in a possible implementation, the method further includes:

[0019] In a case where the distance between the center coordinate of the rotated CT contour and the center coordinate of the ultrasound contour is greater than the second threshold value, the rotated CT contour is translated to a position where a distance between the rotated CT contour and the ultrasound contour is less than or equal to the second threshold value to obtain a rotated and translated CT contour.

[0020] The rotated and translated CT contour is registered with the ultrasound contour.

[0021] With reference to the first aspect, in a possible implementation, a similarity between the ultrasound contour and the CT contour is greater than or equal to a third threshold value.

[0022] In a second aspect, an embodiment of the present application provides an image registration method, including:

[0023] acquire an ultrasound profile of a target organ and a computed tomography (CT) profile, the ultrasound profile being obtained from ultrasound images acquired by an ultrasound device, and the CT profile being obtained from CT images acquired by a CT device;

[0024] rotate the ultrasound profile to obtain a rotated ultrasound profile, wherein an angle between a direction vector of a reference direction in a space of the ultrasound profile and a direction vector of the reference direction in a space of the rotated CT profile is less than or equal to a first threshold, the reference direction being common to the space of the ultrasound profile and the space of the CT profile;

[0025] perform registration based on the rotated ultrasound profile and the CT profile.

[0026] With reference to the second aspect, in a possible implementation manner, the rotating the ultrasound profile to obtain a rotated ultrasound profile comprises:

[0027] determining a first direction vector of the reference direction in the space of the CT profile, and determining a second direction vector of the reference direction in the space of the ultrasound profile;

[0028] determining a rotation matrix according to an angle between the first direction vector and the second direction vector, and an angle between the first direction vector and a coordinate axis;

[0029] rotating the ultrasound profile based on the rotation matrix to obtain the rotated ultrasound profile.

[0030] With reference to the second aspect, in a possible implementation manner, the reference direction comprises a first sub-reference direction, a second sub-reference direction, and a third sub-reference direction, and an angle between a direction vector of the first sub-reference direction in the space of the ultrasound profile and a direction vector of the first sub-reference direction in the space of the rotated ultrasound profile is less than or equal to the first threshold, an angle between a direction vector of the second sub-reference direction in the space of the ultrasound profile and a direction vector of the second sub-reference direction in the space of the rotated ultrasound profile is less than or equal to the first threshold, and an angle between a direction vector of the third sub-reference direction in the space of the ultrasound profile and a direction vector of the third sub-reference direction in the space of the rotated ultrasound profile is less than or equal to the first threshold.

[0031] With reference to the second aspect, in a possible implementation manner, the first sub-reference direction is a direction from a foot of a subject to a head of the subject, the second sub-reference direction is a direction perpendicular outward of a back of the subject, and the third sub-reference direction is a direction from a body of the subject to a left hand of the subject.

[0032] With reference to the second aspect, in a possible implementation manner, the performing registration based on the rotated ultrasound profile and the CT profile comprises:

[0033] In a case where a distance between the center coordinate of the rotated ultrasound profile and the center coordinate of the CT profile is less than a second threshold, performing the registration based on the rotated ultrasound profile and the CT profile.

[0034] With reference to the second aspect, in a possible implementation, the method further includes:

[0035] In a case where a distance between the center coordinate of the rotated ultrasound profile and the center coordinate of the CT profile is greater than the second threshold, translating the rotated ultrasound profile to a position where a distance between the rotated ultrasound profile and the CT profile is less than or equal to the second threshold, to obtain a rotated and translated ultrasound profile.

[0036] Registering the rotated and translated ultrasound profile and the CT profile.

[0037] With reference to the second aspect, in a possible implementation, the similarity between the ultrasound profile and the CT profile is greater than or equal to a third threshold.

[0038] The third aspect provides an image registration apparatus, including:

[0039] An obtaining unit is configured to obtain an ultrasound profile of a target organ and a computed tomography (CT) profile; the ultrasound profile is obtained according to an ultrasound image collected by an ultrasound device, and the CT profile is obtained according to a CT image collected by a CT device;

[0040] A rotating unit is configured to rotate the CT profile to obtain a rotated CT profile; an included angle between a direction vector of a reference direction in a space of the ultrasound profile and a direction vector of the reference direction in a space of the rotated CT profile is less than or equal to a first threshold, and the reference direction is common to the space of the ultrasound profile and the space of the CT profile.

[0041] A registration unit is configured to register based on the rotated CT profile and the ultrasound profile.

[0042] The fourth aspect provides an image registration apparatus, including:

[0043] An obtaining unit is configured to obtain an ultrasound profile of a target organ and a computed tomography (CT) profile; the ultrasound profile is obtained according to an ultrasound image collected by an ultrasound device, and the CT profile is obtained according to a CT image collected by a CT device;

[0044] a rotation unit configured to rotate the ultrasound profile to obtain a rotated ultrasound profile, wherein a reference direction is a direction vector common to a space of the ultrasound profile and a space of the CT profile, and an angle between the direction vector of the reference direction in the space of the ultrasound profile and a direction vector of the reference direction in the space of the rotated CT profile is less than or equal to a first threshold value;

[0045] a registration unit configured to perform registration based on the rotated ultrasound profile and the CT profile.

[0046] In a fifth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, the memory is configured to store a computer program, the computer program includes program instructions, the processor is configured to invoke the program instructions, so that the method in the first aspect or any possible implementation manner of the first aspect is executed, or so that the method in the second aspect or any possible implementation manner of the second aspect is executed.

[0047] In a sixth aspect, an embodiment of the present application provides a chip, including a logic circuit and an interface, the logic circuit and the interface are coupled; the interface is configured to input and / or output code instructions, and the logic circuit is configured to execute the code instructions, so that the method in the first aspect or any possible implementation manner of the first aspect is executed, or so that the method in the second aspect or any possible implementation manner of the second aspect is executed.

[0048] In a seventh aspect, an embodiment of the present application discloses a computer program product, the computer program product includes program instructions, when the program instructions are executed by a processor, the method in the first aspect or any possible implementation manner of the first aspect is executed, or the method in the second aspect or any possible implementation manner of the second aspect is executed.

[0049] In an eighth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, when the computer program is executed on a processor, the method in the first aspect or any possible implementation manner of the first aspect is executed, or the method in the second aspect or any possible implementation manner of the second aspect is executed. Exemplarily, the computer program product can be a software installation package.

[0050] Since the pose differences between the original CT profile and the ultrasound profile are large, that is, the angle between the direction vector of the reference direction in the space of the CT profile and the direction vector in the space of the ultrasound profile is greater than the first threshold, the image registration method provided in this application rotates the acquired original CT profile to obtain the rotated CT profile. The angle between the direction vector of the reference direction in the space of the rotated CT profile and the direction vector in the space of the ultrasound profile is less than or equal to the first threshold, so that the poses of the rotated CT profile and the ultrasound profile are closer, the difference between the two is reduced, thereby improving the calculation efficiency, shortening the registration time, and improving the registration efficiency. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments or background art of this application, the accompanying drawings used in the embodiments or background art of this application will be briefly introduced below.

[0052] Figure 1 This is a schematic diagram of a scene for acquiring ultrasound images provided in an embodiment of this application;

[0053] Figure 2 This is a schematic diagram of a CT image provided in an embodiment of this application;

[0054] Figure 3 This is a schematic diagram of another scenario for acquiring ultrasound images provided in an embodiment of this application;

[0055] Figure 4 This is a schematic diagram illustrating the relative positional relationship between CT-based kidney contour and ultrasound-based kidney contour, provided in an embodiment of this application.

[0056] Figure 5 This is a schematic diagram of an ultrasound-based renal contour orientation transformation provided in an embodiment of this application;

[0057] Figure 6 This is a schematic diagram of the registration result between CT-kidney contour and ultrasound-kidney contour provided in an embodiment of this application;

[0058] Figure 7 This is a schematic flowchart of an image registration method provided in an embodiment of this application;

[0059] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0060] Figure 9 This is a schematic diagram of the structure of another electronic device provided in an embodiment of this application. Detailed Implementation

[0061] The terminology used in the following description of the embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the description of the embodiments and the appended claims herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It also will be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. The terms "first", "second", "third", etc. as used in the description and the claims herein and the preceding drawings should not be construed as limiting the present application to a sequence or order unless such is explicitly stated under the circumstances.

[0062] Currently, a surgeon or other surgical operator can perform a procedure in conjunction with medical images. For example, in percutaneous nephrolithotomy, a surgeon can view a needle in real time in conjunction with an ultrasound image of the kidney.

[0063] In embodiments of the present application, an ultrasound image can be understood as an image acquired by an ultrasound device. It can be understood that a wave capable of causing a sound sensation in the auditory organ can be referred to as a sound wave, and a sound wave that cannot be sensed by the human sensory organ can be referred to as an ultrasound wave. In embodiments of the present application, the ultrasound device can be understood as a device that scans a to-be-scanned object using an ultrasound beam, then receives a reflection signal of the ultrasound beam, and processes the reflection signal to obtain an image of an organ in the to-be-scanned object.

[0064] For ease of understanding, for example, refer to Figure 1 , Figure 1 is a schematic diagram of a scene for acquiring an ultrasound image. As Figure 1 indicated, the 101 part therein can be understood as a to-be-scanned object. In embodiments of the present application, the ultrasound device can include an ultrasound probe, such as the ultrasound probe 102 in Figure 1 , which includes a transmitting ultrasound unit and a receiving ultrasound unit, and can be used to transmit an ultrasound wave and receive an ultrasound wave reflected by the to-be-scanned object. The ultrasound wave transmitted by the ultrasound probe is reflected to the probe after attenuation in the tissue of the to-be-scanned object. It can be understood that the ultrasound probe can also be referred to as a probe, an ultrasonic probe, etc.

[0065] The ultrasound device can further include a data processing unit, such as the data processing unit 104 in Figure 1 . The data processing unit 104 can be connected to the ultrasound probe 102, and processes the ultrasound wave reflected by the to-be-scanned object received by the ultrasound probe 102 to obtain an ultrasound image. Optionally, the ultrasound device can further include a display unit, such as the display unit 106 in Figure 1The display unit 103 in the display unit 103 can be used to display the ultrasound image obtained by the data processing unit 104.

[0066] It can be understood that the ultrasound image collected by the ultrasound device is a two-dimensional image corresponding to a section. Since the kidney and other organs or other tissues are three-dimensional spatial structures, the image collector can relatively completely collect the ultrasound image information of the entire part by rotating or moving the ultrasound probe, and the like. Specifically, as shown in the 105 part in the ultrasound image. Figure 1

[0067] The ultrasound image has the advantages of strong real-time performance and low examination cost. However, the low spatial resolution and small field of view of the ultrasound image result in that the ultrasound image provides too little information to the operator during the operation, thereby increasing the difficulty of the operation. Therefore, in some embodiments, CT image data, i.e., a CT image, can be collected by preoperative computed tomography (CT) of the implementation object. Then, the CT image and the ultrasound image collected during the operation are registered. Since the CT image has the advantages of high resolution and can retain more details, the operator can obtain more information during the operation, thereby reducing the difficulty of the operation. For example, in percutaneous nephrolithotomy, the registration of the ultrasound image and the CT image can enable the operator to obtain more image information of the kidney, thereby improving the puncture precision of the kidney operation and reducing the probability of puncture failure.

[0068] In the embodiments of the present application, the implementation object can be understood as any object collected for registration of the ultrasound image and the CT image. For example, the implementation object can be a person, such as a patient, a volunteer, or the like.

[0069] In the embodiments of the present application, the organ capable of collecting the CT image and the ultrasound image can be understood as the target organ. For example, the target organ can be the kidney (including the left kidney and / or the right kidney), the gallbladder, the spleen, or the like. For example, after obtaining the permission of the implementation object or the guardian of the implementation object, the operator such as a doctor can collect the ultrasound image and the CT image of the kidney of the implementation object and perform related processing.

[0070] In the embodiments of the present application, the target organ segmented from the ultrasound image can be understood as the ultrasound contour of the target organ, and the target organ segmented from the CT image can be understood as the CT contour of the target organ. It can be understood that since the target organ itself is three-dimensional, the ultrasound contour and the CT contour can be considered to include multiple images, for example, as shown in the ultrasound contour and the CT contour. Figure 2 It can be understood that the registration of the ultrasound contour and the CT contour can be understood as the registration between the ultrasound image and the CT image. ​

[0071] For the registration of the above ultrasound image and CT image, an exemplary iterative closest point (ICP) algorithm can be employed. The ICP algorithm can transform point sets in different coordinate systems into a common coordinate system by minimizing registration errors. Exemplarily, the ICP algorithm can be represented by equation (1):

[0072] P t =R·P s +T (1)

[0073] Wherein, P t can be understood as a target point cloud, P s can be understood as a source point cloud, and the ICP algorithm process is to solve the rotation matrix R and the translation matrix T, so that the point cloud P s coincides with the point cloud P t after the transformation (rotation and translation), and the mean square error is minimized through continuous iteration.

[0074] Exemplarily, the rotation matrix R and the translation matrix T between the source point cloud and the target point cloud can be represented by a transformation matrix H, as shown in equation (2):

[0075]

[0076] Wherein, the matrix R can be understood as a rotation amount, the matrix T can be understood as a translation amount, the matrix V can be understood as a perspective transformation amount, and the matrix S can be understood as a scale factor.

[0077] Exemplarily, the process of image registration can be divided into two stages of coarse registration and fine registration. The coarse registration stage can be understood as a relatively rough registration performed under the condition that the transformation between the source point cloud and the target point cloud is almost unknown, and the purpose is mainly to provide a better initial transformation matrix for fine registration. Fine registration can be understood as further optimizing a given initial transformation matrix to obtain a more accurate transformation matrix, i.e. the final registration matrix.

[0078] Generally, in the coarse registration stage, the initial adjustment of the image orientation and the image approximation can be performed according to the spatial features of the image, and the spatial features can be normal vectors corresponding to the image data. However, since the target organ inside the implementation object is a three-dimensional structure, the image data is large, and the calculation efficiency of the coarse registration by the spatial features of the image is slow, resulting in a long image registration time.

[0079] Based on the above problems, the embodiment of the present application provides an image registration method and related device. After obtaining the ultrasound contour of the target organ through the ultrasound image and the CT contour of the target organ through the CT image, the ultrasound contour and the CT contour to be registered are subjected to azimuth transformation, so that the poses of the ultrasound contour and the CT contour are similar, and the difference between the ultrasound contour and the CT contour is small, thereby reducing the time required for image registration and improving the registration efficiency.

[0080] It can be understood that the above image registration method can be executed by an image registration device. The image registration device can be any device capable of implementing the method provided by the present application. For example, the image registration device can be a notebook computer, a desktop computer or other electronic equipment. It should be understood that the method embodiment of the present application can also be implemented by a processor executing computer program code.

[0081] In order to visually understand the image registration method provided by the present application, the above embodiment object is exemplarily taken as a patient, the above target organ is taken as a kidney, and the above image registration method is introduced in the scene of simulating the percutaneous nephroscopic surgery of the patient.

[0082] It can be understood that the main object of the percutaneous nephroscopic surgery is the kidney of the patient. Therefore, the CT image and the ultrasound image collected should include the kidney information, such as the kidney contour or the blood vessels around the kidney.

[0083] 1. CT image collection preparation

[0084] The doctor judges whether the enhanced CT image needs to be collected according to the specific condition of the patient. If the doctor judges that the enhanced CT image does not need to be collected, the patient is laid on the CT scanning bed and then pushed into the CT scanning window to collect the normal CT image. If the doctor judges that the patient needs to collect the enhanced CT image, the patient is pushed into the CT scanning window to collect the enhanced CT image after the contrast agent is injected into the vein of the patient.

[0085] It can be understood that the contrast agent can be developed in the parts with rich blood supply through the blood circulation system. The development of these parts can be visually understood as strengthening. After strengthening, the CT image obtained by CT scanning can obtain more information about the kidney, such as kidney stones, space-occupying, kidney cysts and the like. It can be understood that compared with the normal CT image, the doctor can also obtain more kidney information after the enhanced CT image is registered to the ultrasound image.

[0086] 2. CT data collection

[0087] The patient is pushed into the CT scanning window, and the CT data is collected according to the judgment result in step 1, wherein the CT data includes the CT image. After scanning, the CT image data of the kidney of the patient is taken.

[0088] It can be understood that if an enhanced CT scan is performed on the patient, the enhanced CT scan includes a plain scan period, an arterial phase, a venous phase, and a delay period scan. Generally, the plain scan period data can be used for subsequent steps.

[0089] 3. CT data processing

[0090] The CT image collected in step 2 is input into the image registration device (also referred to as a kidney surgery navigation system) provided in the present application. In the kidney surgery navigation system, a three-dimensional kidney contour is segmented and reconstructed from the CT data collected in the above step 2 by an artificial intelligence algorithm. For the sake of understanding and description, the above three-dimensional kidney contour is referred to as CT-kidney contour.

[0091] Exemplarily, the above artificial intelligence algorithm can be a deep learning algorithm, stochastic gradient descent (SGD), backward propagation (backward), and forward propagation (forward), etc. It can be understood that the above CT-kidney contour can include the entire kidney information, such as the external contour of the kidney, the internal parenchyma of the kidney, etc.

[0092] 4. Ultrasound image acquisition

[0093] During the operation, the patient is placed on the operating table, and the ultrasound device is used to scan and observe the patient's kidney along the direction of the patient's back spine, and the real-time spatial position of the ultrasound image collected by the ultrasound device is positioned in combination with the optical tracking device. It can be understood that the patient's breathing rate can be reminded to be as slow and stable as possible during the ultrasound image acquisition process to reduce the spatial position fluctuation error caused by breathing.

[0094] For the sake of understanding, exemplarily, please refer to Figure 3 , Figure 3 is another scene diagram provided by the embodiment of the present application for collecting ultrasound images. As Figure 3 shown, the ultrasound probe 302 is connected with the support 303. The support 303 includes markers, exemplarily, the number of the above markers is greater than or equal to 3, Figure 3 exemplarily, 4 markers are taken as an example, 5 or 6 markers can also be set according to the actual situation, etc., which are not limited by the present application.

[0095] It can be understood that the connection between the ultrasound probe 302 and the support 303 can be fixed connection or movable connection, for example, it can be connected through a threaded structure, which is not limited by the present application.

[0096] Firstly, it can be understood that the ultrasound image collected by the ultrasound device is only a two-dimensional image without three-dimensional spatial information in the absence of the optical tracking device for positioning. After the optical tracking device is turned on, the three-dimensional spatial position information of the ultrasound image can be obtained through the marker coordinates on the ultrasound probe.

[0097] In the embodiment of the present application, the markers on the support 303 are within the tracking range of the optical tracking device 305. In actual cases, the optical tracking device 305 or the position of the ultrasound probe 302 can be moved to make the markers on the support 303 within the tracking range of the optical tracking device 305, so that the optical tracking device 305 can track the positions of the markers.

[0098] It can be understood that the optical tracking device 305 can determine the three-dimensional spatial coordinates of each marker on the support 303 through the reflective coating on the surface of the marker. Then, the optical tracking device 305 can obtain the conversion matrix H us Other objects having a determined positional relationship with the markers can pass through the conversion matrix H us to determine the spatial position information.

[0099] In the embodiment of the present application, the conversion matrix H us can be understood as the conversion matrix between the reference coordinates (for example, the coordinates of the above-mentioned markers) of the reference object collected by the optical tracking device and the origin coordinates. Since it is a three-dimensional spatial coordinate system, the above-mentioned conversion matrix H us includes a rotation component and a translation component. Exemplarily, the above-mentioned conversion matrix can be expressed by formula (3):

[0100]

[0101] Since the structure of the support 303 is determined, the real-time spatial position information in the scanning range below the ultrasound probe 302 can be determined through the structural relationship between the markers and the support 303 and the above-mentioned conversion matrix H us .

[0102] Optionally, after each two-dimensional ultrasound image is collected, the kidney surgery navigation system can segment the kidney contour from the ultrasound image and display the segmentation result on the display unit 304, that is, the segmented kidney contour. It can be understood that the above-mentioned kidney contour is a two-dimensional kidney contour.

[0103] It can be understood that the kidney is a three-dimensional spatial structure, and the position of the ultrasound probe can be changed by the doctor to collect multiple ultrasound images in the kidney and the surrounding area to avoid missing kidney information.

[0104] 5. Ultrasound image processing

[0105] In the kidney surgery navigation system, a three-dimensional kidney contour is reconstructed by an artificial intelligence algorithm according to the plurality of ultrasound images collected in step 4 and the corresponding spatial position information, which is referred to as an ultrasound-kidney contour for the sake of understanding and description.

[0106] In some embodiments, the kidney surgery navigation system can also determine the similarity between the ultrasound-kidney contour and the CT-kidney contour, and consider that the three-dimensional reconstruction of the ultrasound image is successful when the similarity between the two is greater than a threshold A, otherwise, the ultrasound image is re-collected for three-dimensional reconstruction to obtain a new ultrasound-kidney contour, until the similarity between the obtained ultrasound-kidney contour and the CT-kidney contour is greater than the threshold A.

[0107] It can be understood that the threshold A can be set according to actual conditions, for example, it can be any value greater than 90%, generally, the greater the similarity, the higher the subsequent registration efficiency. Exemplarily, in a possible implementation, the similarity between the ultrasound-kidney contour and the CT-kidney contour can be obtained based on the Euclidean distance between the two. In another possible implementation, the similarity can be a Dice similarity coefficient (DSC), generally, the greater the Dice similarity coefficient, the higher the registration accuracy.

[0108] It can be understood that the above steps 1-5 can be considered as a data preparation stage, and the subsequent steps 6-7 can be considered as an image registration stage.

[0109] 6, based on the orientation transformation matrix R ct-us Coarse registration

[0110] In the embodiment of the present application, in the image registration stage, an orientation transformation matrix is determined for the ultrasound image and the CT image to be registered for orientation transformation, wherein the orientation transformation matrix can include a rotation matrix and / or a translation matrix.

[0111] It can be understood that the ultrasound image is collected by an ultrasound device, and the CT image is collected by a CT device, and finally the two images need to be registered in the world coordinate system. Among them, the CT device automatically converts to the world coordinate system after collecting the image, and the spatial position direction of the CT image converted to the world coordinate system is fixed. However, the ultrasound device collects a two-dimensional ultrasound image, which needs to be converted to the world coordinate system by optical tracking device recognition, so the three-dimensional spatial position of the converted ultrasound image is related to the position of the optical tracking device. However, in actual situations, the position of the optical tracking device is generally random, such as being placed according to the operating room environment or according to the user's habit, which will cause the ultrasound-kidney contour and the CT-kidney contour in the world coordinate system to have a large difference in orientation (such as position and angle).

[0112] Reference can be made to the accompanying drawings Figure 4 , Figure 4 is a schematic diagram of the relative position relationship between the CT-kidney contour and the ultrasound-kidney contour provided by an embodiment of the present application.

[0113] As Figure 4 indicated, the coordinate system can be understood as a world coordinate system. As an example, the CT-kidney contour described above can be located in the first quadrant and the fourth quadrant in the world coordinate system, as Figure 4 indicated, the kidney contour 401 and the kidney contour 402 can be understood as CT-kidney contours, and the kidney contour 401 can be understood as a left kidney contour and the kidney contour 402 can be understood as a right kidney contour as an example. Similarly, the ultrasound-kidney contour described above can be located in the fifth quadrant in the world coordinate system, as Figure 4 indicated, the kidney contour 403 and the kidney contour 404 can be understood as ultrasound-kidney contours, and the kidney contour 403 can be understood as a left kidney contour and the kidney contour 404 can be understood as a right kidney contour as an example.

[0114] It can be understood that, in the registration of the CT-kidney contour and the ultrasound-kidney contour, the left kidney in the CT-kidney contour is registered with the left kidney in the ultrasound-kidney contour, and the right kidney in the CT-kidney contour is registered with the right kidney in the ultrasound-kidney contour. In addition, since the registration between the left kidneys and the registration between the right kidneys are similar, for the sake of brevity, the registration between the left kidneys (i.e., the left kidney contour 401 and the left kidney contour 403) is taken as an example for introduction here.

[0115] As Figure 4 indicated, the positions (which can also be understood as center positions) of the CT-kidney contour and the ultrasound-kidney contour are different, and the angles are also different. Before registration, the present application first performs orientation transformation between the kidney contours, so that the CT-kidney contour and the ultrasound-kidney contour are as close as possible in the relative position and the angle position to reduce the difference between them and achieve the purpose of improving the registration efficiency.

[0116] Reference can be made to the accompanying drawings Figure 5 , Figure 5 is a schematic diagram of the orientation transformation of the ultrasound-kidney contour provided by an embodiment of the present application. It can be understood that, for the sake of distinguishing between the two kidney contours, Figure 5 one of the kidney contours is filled with a pattern, and as Figure 5 indicated, the kidney contour 501 in Figure 4 can be understood as the CT-kidney contour 401 in Figure 5 , and as Figure 4 indicated, the kidney contour 502 in Figure 5 can be understood as the ultrasound-kidney contour 403 in

[0117] In order to distinguish the rotation matrix and the translation matrix in the prior registration algorithm, for example, the rotation matrix R and the translation matrix T, the rotation matrix provided by the embodiments of the present application is referred to as a rotation orientation matrix, and the translation matrix provided by the embodiments of the present application is referred to as a translation orientation matrix. Next, the implementation of the rotation orientation matrix and the translation orientation matrix is introduced respectively.

[0118] 6.1, determining the rotation orientation matrix.

[0119] In the embodiments of the present application, the rotation orientation matrix between the ultrasound-kidney contour and the CT-kidney contour is determined based on the direction of the patient as the reference when the position of the patient does not change.

[0120] 6.1.1, determining the relationship between the coordinate system of the patient and the ultrasound device

[0121] As described in step 4, when the ultrasound device is used to scan the kidney of the patient along the direction of the spine of the back of the patient, the X direction of the coordinate system of the ultrasound device is the same as the direction of the foot of the patient pointing to the head (referred to as the head direction), and the Y direction of the coordinate system is perpendicular to the back of the patient (referred to as the back perpendicular direction). Therefore, the X direction of the ultrasound image obtained is the head direction, and the Y direction is the back perpendicular direction.

[0122] It can be understood that although the two-dimensional ultrasound image collected by the ultrasound device does not have three-dimensional space information, that is, Z axis information, according to the existing X axis, Y axis and the right-hand rule that must be met by the coordinate system, it can be considered that the Z direction of the ultrasound image is the direction of the body of the patient pointing to the left of the patient (referred to as the left direction).

[0123] Since the spatial position of the ultrasound-kidney contour is based on the transformation matrix H us of the optical tracking device, the original coordinates of the ultrasound image will also be transformed by the transformation matrix H us . Exemplarily, the original head direction is the X direction of the ultrasound-kidney contour, and after being transformed by the transformation matrix H us into the world coordinate system, the direction vector of the head direction is (A 11 ,A 21 ,A 31 ). Similarly, the direction vector of the back perpendicular direction is (A 12 ,A 22 ,A 32 ), and the direction vector of the left direction is (A 13 ,A 23 ,A 33 ).

[0124] 6.1.2 Determine the relationship between the coordinate systems of the patient and the CT equipment.

[0125] It is understandable that CT images acquired by a CT scanner consist of multiple two-dimensional images, which can be understood as a sequence of two-dimensional images with spatial information. Since the CT scanner automatically transforms its coordinate system to the world coordinate system, the coordinate information of the CT images acquired by the scanner is in the world coordinate system.

[0126] Because of the patient's special position when acquiring CT images, such as the patient's head entering the CT equipment first, the X direction of the CT image obtained by performing a CT scan on the patient is the left direction, the Y direction of the CT image is the vertical direction of the back, and the Z direction of the CT image is the head direction.

[0127] Therefore, the direction vector of the head direction can be obtained. The value is (0,0,1). Similarly, the direction vector in the vertical direction of the back can be obtained. The direction vector to the left is (0,1,0). The value is (1,0,0).

[0128] The above analysis shows that, assuming the patient's posture remains unchanged, the direction of the patient's head, the vertical direction of the back, and the leftward direction are inherently fixed in the real world. However, due to the different coordinate systems used by the devices, when the ultrasound-kidney contour and the CT-kidney contour are placed in the same world coordinate system, the direction vectors corresponding to the patient's head, vertical direction of the back, and leftward direction differ, thus presenting the following... Figure 4 The position and angle of the kidney outline are different in different cases.

[0129] 6.1.3 Calculate and correct the angle between the CT and ultrasound directions.

[0130] a. Correcting head direction

[0131] Calculate head direction and The included angle θ z .

[0132] Depend on The vector projected onto the X direction For (A) 11 A 21 ,0), calculate The angle θ between the vector and the positive X-axis unit vector (1, 0, 0) x .

[0133] Therefore, the head direction (i.e., the Z direction) of the CT image is rotated counterclockwise by θ around the Y axis. z Then rotate θ around the Z-axis xThis allows the head-direction of the CT-based kidney contour to be aligned with the head-direction of the ultrasound-based kidney contour. It can be understood that in A... 21 When A > 0, rotate counterclockwise around the direction of rotation. 21 When the value is ≤0, it rotates clockwise around the direction of rotation.

[0134] b. Correcting the vertical and leftward alignment of the back.

[0135] Based on the aforementioned included angle θ z and the aforementioned included angle θ x The rotation matrix R for head-direction correction of the CT-kidney contour can be obtained. ct-h The above transformation matrix R ct-h It can be expressed by equation (4):

[0136]

[0137] Therefore, after transformation matrix R ct-h After transformation, the leftward direction of the CT-kidney contour For (B) 11 B 21 B 31 ), vertical direction of the back For (B) 12 B 22 B 32 ), head direction For (B) 13 B 23 B 33 ).

[0138] Calculate the dorsal vertical direction of the CT-renal contour (B 12 B 22 B 32 (perpendicular to the dorsal direction of the ultrasound-guided kidney contour) (A 12 A 22 A 32 The included angle α1 is ).

[0139] Calculate the dorsal vertical direction of the CT-renal contour (B 12 B 22 B 32 (and the left direction of the ultrasound-kidney contour) For (A) 13 A 23 A 33 Angle α2 is used to determine the rotation direction of the CT-kidney contour rotation α1. For example, when α2 ≤ 90°, the CT-kidney contour is rotated around the head direction. (B13 23 33 ) anticlockwise rotation of α1; in the case of α2 > 90°, the CT-kidney profile is rotated around the head direction 13 23 33 ) clockwise rotation of α1.

[0140] Since the rotation is in accordance with the coordinate axis (such as rotation around the X axis, Y axis or Z axis), the above correction of the back vertical direction and the left direction can be performed before the correction of the head direction. That is, in the case of α2 < 90°, the CT-kidney profile is rotated around the head direction (0, 0, 1) anticlockwise rotation of α1; in the case of α2 > 90°, the CT-kidney profile is rotated around the head direction (0, 0, 1) clockwise rotation of α1. Then, the head direction calibration in step a is performed, and the calibration of the head direction, the left direction and the back vertical direction is completed.

[0141] Finally, according to the above included angle θ z , the above included angle θ x , the above included angle α1 and the included angle α2, the final rotation matrix R ct-us is obtained. The above rotation matrix R ct-us can be expressed by formula (5):

[0142]

[0143] It can be understood that when calculating the included angle relationship, any one of the head direction, the back vertical direction and the left direction can be taken as the first direction for calculating the included angle. The correction of the head direction is only an exemplary introduction, and the left direction or the back vertical direction can be corrected first, and the correction method is similar to the above.

[0144] 6.2, determining the translation matrix

[0145] Suppose the center coordinates of the CT-kidney profile are (C x , C y , C z ), and the center coordinates of the ultrasound-kidney profile are (U x , U y , U z ). Therefore, in the case of the ultrasound-kidney profile as the source point cloud and the CT-kidney profile as the target point cloud, the translation parameters for moving the CT-kidney profile close to the ultrasound-kidney profile can be expressed by formula (6):

[0146]

[0147] wherein, the above T​​​​​x represents the translation amount of the ultrasound- kidney profile in the X direction, and the T y represents the translation amount of the ultrasound- kidney profile in the Y direction, and the T z represents the translation amount of the ultrasound- kidney profile in the Z direction. It can be understood that the ultrasound- kidney profile can move according to the positive and negative of the T x , T y and T z , if positive, move in the positive direction of the coordinate axis, and if negative, move in the negative direction of the coordinate axis.

[0148] In some embodiments, the kidney profile can be rotated and transformed based on the rotation orientation matrix, that is, the above formula (5); in another embodiment, the kidney profile can be translated and transformed based on the translation orientation matrix, that is, the above formula (6). In still some embodiments, the kidney profile can be rotated and translated and transformed based on the above rotation orientation matrix and the above translation orientation matrix. Exemplarily, the transformation matrix of rotation and translation can be the transformation matrix H ct-us represents:

[0149]

[0150] After the transformation of the above rotation orientation matrix and / or the above translation orientation matrix, the CT- kidney profile and the ultrasound- kidney profile after the orientation transformation are coarsely registered. Since the two are closer in position, the efficiency of coarse registration is improved. Generally, the matching efficiency of the kidney profile after rotation and translation is higher than that of the kidney profile after rotation or translation alone.

[0151] For ease of understanding, please refer to Figure 6 . Figure 6 is a schematic diagram of the registration result of the CT- kidney profile and the ultrasound- kidney profile provided by an embodiment of the present application. As Figure 6 indicated, the result can be understood as registration after rotation and translation transformation.

[0152] 7. Image fine registration

[0153] In an embodiment of the present application, the kidney surgery navigation system can use the above ICP algorithm to realize fine registration of the CT- kidney profile and the ultrasound- kidney profile. Exemplarily, fine registration can include the following steps:

[0154] (1) Find the nearest corresponding point

[0155] In this step, the initial rotation matrix and the initial translation matrix, or the rotation matrix and the translation matrix obtained in the last iteration, are used to transform the initial point cloud, and the CT-kidney contour and the ultrasound-kidney contour are converted into temporary transformed point clouds. Then, the source point cloud corresponding to the ultrasound-kidney contour and the target point cloud corresponding to the CT-kidney contour are compared to find the nearest neighbor point of each point in the source point cloud in the target point cloud.

[0156] (2) Solving the optimal transformation

[0157] In this step, the optimal transformation has a closed-form solution, which can be calculated with the help of matrix singular value decomposition (SVD) decomposition to obtain the optimal translation matrix and the optimal rotation matrix.

[0158] (3) Kidney image iterative transformation

[0159] It can be understood that each iteration can obtain the current optimal transformation parameters, i.e., the current optimal translation matrix T k and the rotation matrix R k (wherein the subscript k is used to represent the kth iteration, and k can be an integer greater than or equal to 1), after each iteration, the translation matrix T k and the rotation matrix R k are applied to the current source point cloud.

[0160] After completing the last iteration and transformation, continue to iterate according to the above "(1) finding the nearest corresponding point" and "(2) solving the optimal transformation" until the iteration termination condition is met. In the embodiments of the present application, the termination condition can be that the change amount of the translation matrix and the change amount of the rotation matrix in the adjacent two iterations are both less than or equal to a threshold, or the loss value of the iteration is less than or equal to a threshold, or the number of iterations reaches a preset number. Among them, the threshold and the preset number can be set according to the actual situation, and the present application does not limit this.

[0161] 8. Kidney image iterative completion processing

[0162] It can be understood that in the above fine registration stage, the ultrasound-kidney contour is the source point cloud, and the CT-kidney contour is the target point cloud, so the registration matrix obtained after step 7 is the registration matrix for registering the ultrasound-kidney contour to the CT-kidney contour. In order to facilitate understanding, the registration matrix for registering the ultrasound-kidney contour to the CT-kidney contour in the fine registration stage is denoted as F us-ct .

[0163] It can be understood that the accuracy of the ultrasound image is lower than that of the CT image, that is, the image information of the CT image is more comprehensive and accurate, and the ultrasound-renal profile as the source point cloud and the CT-renal profile as the target point cloud are registered in the registration stage, so that a more accurate registration result can be obtained. However, since the ultrasound image is obtained in the actual operation space, and the CT image is an image in the CT image acquisition room space (which can be regarded as a virtual space), the CT image needs to be applied to the actual operation scene during the operation process, so finally the CT-renal profile needs to be registered to the ultrasound-renal profile. Therefore, the registration matrix F of the CT-renal profile registered to the ultrasound-renal profile needs to be obtained ct-us .

[0164] In some embodiments, the registration matrix obtained each time in the above step 7 is multiplied (the matrix before iteration is multiplied after iteration) to obtain the final registration matrix F us-ct . The inverse of the above registration matrix F us-ct is obtained, that is, the registration matrix F ct-us , so as to realize the registration of the CT-renal profile to the ultrasound-renal profile.

[0165] It can be understood that if the registration effect of the above registration matrix F ct-us is not ideal, the position can be adjusted manually on the basis of the current registration and the fine registration is performed again until the satisfactory accuracy is achieved.

[0166] The above describes the image registration method provided by the embodiments of the present application in the scenario of simulating a patient to perform a percutaneous nephrectomy, taking the patient as the implementation object and the kidney as the target organ. Next, the image registration method provided by the embodiments of the present application is introduced in the whole in combination with other scenarios applicable to the embodiments of the present application.

[0167] Exemplarily, refer to Figure 7 , Figure 7 is a flowchart of an image registration method provided by the embodiments of the present application. As shown in Figure 7 , the above method comprises the following steps.

[0168] 701: Obtain an ultrasound profile of a target organ and a computed tomography (CT) profile. The ultrasound profile is obtained according to an ultrasound image collected by an ultrasound device, and the CT profile is obtained according to a CT image collected by a CT device.

[0169] It can be understood that the target organ in the embodiments of the present application is an organ capable of collecting ultrasound images and CT images, such as a kidney, a gallbladder, a spleen, etc. A plurality of ultrasound images can be obtained by performing ultrasound scanning on the target organ by using the ultrasound device, and then the ultrasound profile of the target organ is segmented from the plurality of ultrasound images. A plurality of CT images can be obtained by performing CT scanning on the target organ by using the CT device, and then the CT profile of the target organ is segmented from the plurality of CT images.

[0170] When the target organ is the kidney, the ultrasound contour described above can be understood as the ultrasound-kidney contour mentioned earlier, and the CT contour described above can be understood as the CT-kidney contour mentioned earlier. For details, please refer to the descriptions of steps 1-5 above. It is understood that when the target organ is an organ other than the kidney, the processing method is similar to that for the kidney, and will not be repeated here.

[0171] In some embodiments, the similarity between the ultrasound profile and the CT profile is greater than or equal to a third threshold.

[0172] Understandably, in the process of acquiring the ultrasound profile, it is necessary to use optical tracking equipment to determine the space of the ultrasound scanning space (e.g., Figure 3 (As shown in the scenario), the location information of multiple ultrasound images scanned by the ultrasound equipment is determined, and then segmentation and reconstruction are performed to obtain the ultrasound contour. In the above process, the location information of some ultrasound images may be deviated, resulting in an inaccurate ultrasound contour. CT images, on the other hand, are images acquired by a CT scanner. When acquiring CT images, the subject needs to hold their breath, and the image information acquired by the CT scanner itself is more comprehensive and accurate. Therefore, the obtained CT contour can be considered the accurate CT contour of the target organ. Therefore, this application improves the registration efficiency by controlling the similarity between the ultrasound contour and the CT contour, that is, by registering the ultrasound contour with a similarity greater than or equal to the third threshold mentioned above.

[0173] In this embodiment, the aforementioned similarity can be the Dice similarity coefficient or other reasonable indicators, and the aforementioned third threshold can be set according to the actual situation, such as any value greater than or equal to 80%. Optionally, the description of this embodiment can also refer to step 5 above, and the aforementioned third threshold can also be understood as the aforementioned threshold A.

[0174] 702: Rotate the CT profile to obtain the rotated CT profile; the angle between the direction vector of the reference direction in the space of the ultrasound profile and the direction vector of the reference direction in the space of the rotated CT profile is less than or equal to a first threshold, and the reference direction is shared by the space of the ultrasound profile and the space of the CT profile.

[0175] In this embodiment, the reference direction is common to both the space of the ultrasound profile and the space of the CT profile. It can be understood that the reference direction can be determined as a direction vector in the space of the ultrasound profile or in the space of the CT profile.

[0176] Exemplarily, the reference direction can be determined according to the subject. It can be understood that both the ultrasound image and the CT image are acquired based on the subject, and even though the time, spatial information, etc. of acquiring the ultrasound image is different from that of acquiring the CT image, the relative relationship between the target organ and the subject is determined. For example, when the CT image is acquired, the relative relationship between the spatial coordinate information of the CT image and the orientation of the human body is determined; when the ultrasound image is acquired, the relative relationship between the spatial coordinate information of the ultrasound image and the orientation of the human body is also determined. Therefore, the reference direction can be a direction determined according to the subject, which can be the head direction, the back vertical direction, the left direction, a direction in which the head of the subject points to the foot (referred to as the foot direction), a direction in which the body of the subject points to the right of the patient (referred to as the right direction), etc.

[0177] It can be understood that, before the CT contour is rotated, the angle between the direction vector of the reference direction in the space of the ultrasound contour and the direction vector of the reference direction in the space of the CT contour is greater than the first threshold. In this embodiment, the CT contour is rotated to obtain a rotated CT contour, and the angle between the direction vector of the reference direction in the space of the ultrasound contour and the direction vector of the reference direction in the space of the rotated CT contour is less than or equal to the first threshold, so that the difference between the rotated CT contour and the ultrasound contour before rotation is reduced, thereby saving the registration time and improving the registration efficiency in the registration process.

[0178] The image registration device can keep the ultrasound contour unchanged, and rotate the CT contour according to the relationship between the direction vector of the reference direction in the space of the ultrasound contour (referred to as the ultrasound-reference direction vector) and the direction vector of the reference direction in the space of the CT contour (referred to as the CT-reference direction vector).

[0179] In one aspect, when the CT contour is rotated, the CT-reference direction vector can be the same as the ultrasound-reference direction vector (it can be understood that the angle is 0), or the angle between the CT-reference direction vector and the ultrasound-reference direction vector can be other values, as long as it is less than or equal to the first threshold. In another aspect, the reference direction can be one or multiple. For the convenience of understanding, the rotation process in this step can be referred to as correction.

[0180] It can be understood that the first threshold can be set according to the actual situation, which can be any value less than 5°, such as 2° or 3°, etc., and the present application does not limit it. Generally, the smaller the angle, the higher the registration efficiency.

[0181] 703: registering based on the rotated CT contour and the ultrasound contour.

[0182] In this step, the image registration device registers the rotated CT contour and the ultrasound contour, wherein the registration can include coarse registration and fine registration, and the specific implementation can refer to the related description of steps 6 and 7 in the foregoing.

[0183] Since the pose difference between the original CT contour and the ultrasound contour is large, that is, the included angle between the direction vector of the reference direction in the space of the CT contour and the direction vector in the space of the ultrasound contour is greater than the first threshold value, the image registration method provided in the embodiment of the application rotates the obtained original CT contour to obtain a rotated CT contour, and the included angle between the direction vector of the reference direction in the space of the rotated CT contour and the direction vector in the space of the ultrasound contour is less than or equal to the first threshold value, so that the pose between the rotated CT contour and the ultrasound contour is closer, and the difference between the two is reduced, thereby improving the calculation efficiency, shortening the registration time, and improving the registration efficiency.

[0184] In some embodiments, the image processing device can directly rotate the CT contour using the space vectors of the reference direction in the two contour spaces, so that the above-mentioned included angle is 0. Exemplarily, in a possible implementation manner, Figure 7 In the method shown, step 702 includes:

[0185] 7021: determining a first direction vector of the reference direction in the space of the CT contour, and determining a second direction vector of the reference direction in the space of the ultrasound contour.

[0186] 7022: determining a rotation matrix according to the included angle between the first direction vector and the second direction vector, and the included angle between the second direction vector and the coordinate axis.

[0187] 7023: rotating the CT contour based on the rotation matrix to obtain the rotated CT contour.

[0188] It can be understood that, through the method of the embodiment, the included angle between the direction vector of the reference direction in the space of the ultrasound contour and the direction vector of the reference direction in the space of the rotated CT contour is 0, and therefore, it can be considered that the direction vector of the reference direction in the space of the ultrasound contour is the same as the direction vector of the reference direction in the space of the rotated CT contour. Through the above-mentioned manner of rotating the CT contour, the registration efficiency is higher than other manners.

[0189] It can be understood that the reference direction is not the same as the direction vector of the CT contour and the ultrasound contour in space, and the first direction vector and the second direction vector can be determined according to the relative position relationship between the reference direction and the CT contour and the ultrasound contour. Then, according to the formula of the angle between vectors, the angle between the first direction vector and the second direction vector, and the angle between the second direction vector and the coordinate axis can be obtained; finally, the rotation direction of the second direction vector is determined, and the rotation matrix of rotating the first direction vector to the second direction vector is determined, and the CT contour is rotated based on the rotation matrix.

[0190] Exemplarily, in the case that the target organ is a kidney and the reference direction is a head direction, the first direction vector can be understood as the direction vector The second direction vector can be understood as the direction vector The angle between the first direction vector and the second direction vector can be understood as the angle θ z The angle between the second direction vector and the coordinate axis can be understood as the angle θ x The rotation matrix can be understood as the transformation matrix R ct-h For details, please refer to the related description of step 6.1.3 above, which will not be repeated here.

[0191] Alternatively, the reference direction can be multiple, and the image registration device can achieve the effect of step 702 in each reference direction. Exemplarily, it can be two references, referred to as a first sub-reference direction and a second sub-reference direction. In the above case, the image registration device can first rotate the CT contour for the first time to obtain a first rotated CT contour, so that the angle between the CT-reference direction vector and the ultrasound-reference direction vector in the first sub-reference direction is less than or equal to the first threshold; then the first rotated CT contour is rotated for the second time, so that the angle between the CT-reference direction vector and the ultrasound-reference direction vector in the second sub-reference direction is less than or equal to the first threshold.

[0192] Since the CT contour and the ultrasound contour themselves include multiple images, which can be understood as three-dimensional, the difference between the rotated CT contour and the ultrasound contour can be further reduced by correcting in three directions, thereby further improving the calculation efficiency, shortening the registration time, and improving the registration efficiency.

[0193] Therefore, in some embodiments, the reference direction includes a first sub-reference direction, a second sub-reference direction, and a third sub-reference direction; the direction vectors of the first sub-reference direction, the second sub-reference direction, and the third sub-reference direction in the space of the ultrasound profile and in the space of the rotated CT profile are respectively less than or equal to the first threshold value.

[0194] In the embodiment, the reference direction can be three different directions, i.e., the first sub-reference direction, the second sub-reference direction, and the third sub-reference direction. It can be understood that any sub-reference direction can be understood as a reference direction, and therefore, any sub-reference direction can be determined according to the manner of determining the reference direction, which can be understood with reference to the related description of step 702.

[0195] For example, the image registration device can first correct the first sub-reference direction based on the manner of steps 7021-7024, and then further correct the second sub-reference direction and the third sub-reference direction based on the angle relationship between the second sub-reference direction and the third sub-reference direction.

[0196] In some embodiments, the first sub-reference direction is the head direction, the second sub-reference direction is the back vertical direction, and the third sub-reference direction is the left direction, and the three directions are perpendicular to each other, which can effectively correct the CT profile. Based on the setting of the three directions, in the case that the target organ is the kidney, the rotation orientation matrix R ct-us can be calculated according to the related description of step 6.

[0197] In addition to the above correction process, further translation can be performed according to the center positions of the CT profile and the ultrasound profile, and in some embodiments, Figure 7 the method shown in the figure, step 703 includes:

[0198] 7031: In the case that the distance between the center coordinates of the rotated CT profile and the center coordinates of the ultrasound profile is less than a second threshold value, performing the registration based on the rotated CT profile and the ultrasound profile.

[0199] In the embodiment, after obtaining the rotated CT profile, the distance between the center coordinates of the CT profile and the center coordinates of the ultrasound profile can be first determined, and in the case that the distance is less than a second threshold value, registration is performed. Through the above manner, the difference between the CT profile and the ultrasound profile can be within a controllable range, and the registration efficiency is higher than other registration manners.

[0200] It can be understood that the second threshold value can be set according to actual conditions, for example, can be a value less than the minimum distance between the center of the target organ contour and the edge of the target organ contour, or can be an empirical value, which is not limited in the present application.

[0201] 7032: In the case that the distance between the center coordinates of the rotated CT contour and the center coordinates of the ultrasound contour is greater than the second threshold value, the rotated CT contour is translated to a position where the distance between the rotated CT contour and the ultrasound contour is less than or equal to the second threshold value, to obtain a rotated and translated CT contour.

[0202] 7033: The rotated and translated CT contour and the ultrasound contour are registered.

[0203] In the embodiment, after obtaining the rotated CT contour, the distance between the center coordinates of the CT contour and the center coordinates of the ultrasound contour can be determined first, and in the case that the distance is greater than the second threshold value, the rotated CT contour is translated. When translating, as long as the distance between the center coordinates of the rotated and translated CT contour and the center coordinates of the ultrasound contour is less than or equal to the second threshold value, it is acceptable. Through the embodiment, the difference between the two contours to be registered can be further reduced, thereby improving the calculation efficiency, shortening the registration time, and improving the registration efficiency.

[0204] Optionally, the rotated CT contour can be translated so that the center coordinates of the rotated and translated CT contour coincide with the center coordinates of the ultrasound contour. For details, refer to the description of determining the translation matrix in step 6.2, which is not repeated here.

[0205] The embodiment of the present application also provides an image registration method, comprising:

[0206] An ultrasound contour of a target organ and a computed tomography (CT) contour are obtained, the ultrasound contour is obtained according to an ultrasound image collected by an ultrasound device, and the CT contour is obtained according to a CT image collected by a CT device;

[0207] The ultrasound contour is rotated to obtain a rotated ultrasound contour, a reference direction is a direction vector in the space of the ultrasound contour and a direction vector in the space of the CT contour, and an included angle between the direction vector of the reference direction in the space of the ultrasound contour and the direction vector of the reference direction in the space of the rotated CT contour is less than or equal to a first threshold value;

[0208] The rotated ultrasound contour and the CT contour are registered.

[0209] It can be understood that in the case that the distance between the center coordinates of the rotated CT contour and the center coordinates of the ultrasound contour is greater than the second threshold value, the rotated CT contour is translated to a position where the distance between the rotated CT contour and the ultrasound contour is less than or equal to the second threshold value, to obtain a rotated and translated CT contour. Figure 7In the shown method, after the original CT contour and the ultrasound contour are acquired, the CT contour is rotated. In this embodiment, after the original CT contour and the ultrasound contour are acquired, the ultrasound contour is rotated, and the specific implementation is similar to that of the first embodiment, which will not be repeated here. Figure 7 The calculation manner and the rotation principle in the corresponding method are similar, and it can be understood that the object is changed from the CT contour to the ultrasound contour, which will not be repeated here.

[0210] The method provided by the embodiments of the present application is described in detail above, and the device provided by the embodiments of the present application is introduced below.

[0211] Please refer to Figure 8 , Figure 8 is a structural schematic diagram of an electronic device provided by the embodiments of the present application. The electronic device 80 is used to execute the image registration method provided by the embodiments of the present application. Exemplarily, the electronic device 80 can be a desktop computer, a tablet computer, a portable notebook computer, and the like, which are not limited by the embodiments of the present application.

[0212] As Figure 8 shown, the electronic device 80 includes an acquisition unit 801, a rotation unit 802, and a registration unit 803. Optionally, the electronic device 80 can further include a determination unit 804 and a translation unit 805.

[0213] In the first implementation mode, the descriptions of the various units are as follows:

[0214] The acquisition unit 801 is configured to acquire an ultrasound contour of a target organ and a computed tomography (CT) contour. The ultrasound contour is obtained according to an ultrasound image collected by an ultrasound device, and the CT contour is obtained according to a CT image collected by a CT device.

[0215] The rotation unit 802 is configured to rotate the CT contour to obtain a rotated CT contour. An included angle between a direction vector of a reference direction in a space of the ultrasound contour and a direction vector of the reference direction in a space of the rotated CT contour is less than or equal to a first threshold value. The reference direction is common to the space of the ultrasound contour and the space of the CT contour.

[0216] The registration unit 803 is configured to perform registration based on the rotated CT contour and the ultrasound contour.

[0217] In a possible implementation mode, the determination unit 804 is configured to determine a first direction vector of the reference direction in the space of the CT contour, and determine a second direction vector of the reference direction in the space of the ultrasound contour.

[0218] The determining unit 804 is further configured to determine a rotation matrix according to an included angle between the first direction vector and the second direction vector, and an included angle between the second direction vector and a coordinate axis.

[0219] The rotating unit 802 is further configured to rotate the CT contour based on the rotation matrix to obtain a rotated CT contour.

[0220] In a possible implementation, the reference direction includes a first sub-reference direction, a second sub-reference direction, and a third sub-reference direction; a direction vector of the first sub-reference direction, the second sub-reference direction, and the third sub-reference direction in the space of the ultrasound contour is less than or equal to the first threshold value respectively.

[0221] In a possible implementation, the first sub-reference direction is a direction from a foot of the subject to a head, the second sub-reference direction is a direction perpendicular outward of a back of the subject, and the third sub-reference direction is a direction from a body of the subject to a left hand.

[0222] In a possible implementation, the registering unit 803 is specifically configured to perform the registration based on the rotated CT contour and the ultrasound contour in a case where a distance between a center coordinate of the rotated CT contour and a center coordinate of the ultrasound contour is less than a second threshold value.

[0223] In a possible implementation, the translating unit 805 is configured to, in a case where the distance between the center coordinate of the rotated CT contour and the center coordinate of the ultrasound contour is greater than the second threshold value, translate the rotated CT contour to a position where a distance between the rotated CT contour and the ultrasound contour is less than or equal to the second threshold value to obtain a rotated and translated CT contour.

[0224] The registering unit 803 is specifically configured to register the rotated and translated CT contour and the ultrasound contour.

[0225] In a possible implementation, a similarity between the ultrasound contour and the CT contour is greater than or equal to a third threshold value.

[0226] In the second implementation, the descriptions of the units are as follows:

[0227] The obtaining unit 801 is configured to obtain an ultrasound contour of a target organ and a computed tomography (CT) contour; the ultrasound contour is obtained according to an ultrasound image collected by an ultrasound device, and the CT contour is obtained according to a CT image collected by a CT device.

[0228] The rotating unit 802 is configured to rotate the ultrasound profile to obtain a rotated ultrasound profile, and an included angle between a direction vector of a reference direction in a space of the ultrasound profile and a direction vector of the reference direction in a space of the rotated CT profile is less than or equal to a first threshold value, the reference direction being common to the space of the ultrasound profile and the space of the CT profile;

[0229] The registration unit 803 is configured to register the rotated ultrasound profile and the CT profile.

[0230] In a possible implementation, the determining unit 804 is configured to determine a first direction vector of the reference direction in the space of the CT profile, and determine a second direction vector of the reference direction in the space of the ultrasound profile.

[0231] The determining unit 804 is further configured to determine a rotation matrix according to an included angle between the first direction vector and the second direction vector, and an included angle between the first direction vector and a coordinate axis.

[0232] The rotating unit 802 is further configured to rotate the ultrasound profile based on the rotation matrix to obtain the rotated ultrasound profile.

[0233] In a possible implementation, the reference direction includes a first sub-reference direction, a second sub-reference direction and a third sub-reference direction, and an included angle between a direction vector of the first sub-reference direction in the space of the ultrasound profile and a direction vector in the space of the rotated ultrasound profile is less than or equal to the first threshold value.

[0234] In a possible implementation, the first sub-reference direction is a direction from a foot of the subject to a head, the second sub-reference direction is a direction of a back of the subject vertically outward, and the third sub-reference direction is a direction from a body of the subject to a left hand.

[0235] In a possible implementation, the registration unit 803 is specifically configured to perform the registration of the rotated ultrasound profile and the CT profile when a distance between a center coordinate of the rotated ultrasound profile and a center coordinate of the CT profile is less than a second threshold value.

[0236] In a possible implementation, the translating unit 805 is configured to, when the distance between the center coordinate of the rotated ultrasound profile and the center coordinate of the CT profile is greater than the second threshold value, translate the rotated ultrasound profile to a position where a distance between the rotated ultrasound profile and the CT profile is less than or equal to the second threshold value to obtain a rotated and translated ultrasound profile.

[0237] The registration unit 803 is specifically configured to register the rotation-translation ultrasonic profile and the CT profile.

[0238] In a possible implementation, the similarity between the ultrasonic profile and the CT profile is greater than or equal to a third threshold.

[0239] Please refer to Figure 9 , Figure 9 is another structural schematic diagram of an electronic device provided by the embodiment of the present application. As shown in Figure 9 , the electronic device 90 includes a memory 901 and a processor 902. Further optionally, it can also include a communication interface 903 and a bus 904, wherein the memory 901, the processor 902 and the communication interface 903 are communicatively connected to each other through the bus 904.

[0240] The memory 901 is configured to provide a storage space, and the storage space can store data such as an operating system and a computer program. The memory 901 includes but is not limited to a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or a compact disc read-only memory (CD-ROM).

[0241] The processor 902 is a module for arithmetic operation and logical operation, and can be one or a combination of multiple kinds of processing modules such as a central processing unit (CPU), a graphics processing unit (GPU) or a microprocessor unit (MPU).

[0242] The memory 901 stores a computer program, and the processor 902 invokes the computer program stored in the memory 901 to execute the image registration method described above. Exemplarily, in the case of the electronic device 90 being the electronic device 80, the content acquired by the acquisition unit 801 can be implemented by the communication interface 903, and the steps performed by the rotation unit 802, the registration unit 803, the determination unit 804 and the translation unit 805 can be implemented by the processor 902.

[0243] The present application also provides a computer readable storage medium, which stores computer codes, and when the computer codes are run on a computer, the method of the above embodiment is executed.

[0244] The application also provides a computer program product comprising computer code or a computer program which, when run on a computer, causes the method in the above embodiments to be performed.

[0245] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the above claims.

Claims

1. An image registration method, characterized in that, The method includes: The ultrasound profile and computed tomography (CT) profile of the target organ are obtained; the ultrasound profile is obtained from ultrasound images acquired by an ultrasound device, and the CT profile is obtained from CT images acquired by a CT device. The CT contour is rotated to obtain the rotated CT contour; the angle between the direction vector of the reference direction in the space of the ultrasound contour and the direction vector of the reference direction in the space of the rotated CT contour is less than or equal to a first threshold, and the reference direction is common to the space of the ultrasound contour and the space of the CT contour. Registration is performed based on the rotated CT profile and the ultrasound profile; The step of rotating the CT contour to obtain the rotated CT contour includes: Determine a first direction vector of the reference direction in the space of the CT profile, and determine a second direction vector of the reference direction in the space of the ultrasound profile; The rotation matrix is ​​determined based on the angle between the first direction vector and the second direction vector, and the angle between the second direction vector and the coordinate axis. The CT contour is rotated based on the rotation matrix to obtain the rotated CT contour.

2. The method according to claim 1, characterized in that, The reference direction includes a first sub-reference direction, a second sub-reference direction, and a third sub-reference direction; the direction vectors of the first sub-reference direction, the second sub-reference direction, and the third sub-reference direction in the space of the ultrasound profile and the direction vectors in the space of the rotated CT profile are respectively less than or equal to the first threshold.

3. The method according to claim 2, characterized in that, The first sub-reference direction is the direction from the feet of the subject to the head, the second sub-reference direction is the direction from the back of the subject perpendicularly outward, and the third sub-reference direction is the direction from the body of the subject to the left hand.

4. The method according to any one of claims 1-3, characterized in that, The registration based on the rotated CT profile and the ultrasound profile includes: If the distance between the center coordinates of the rotated CT profile and the center coordinates of the ultrasound profile is less than a second threshold, the registration step based on the rotated CT profile and the ultrasound profile is performed.

5. The method according to claim 4, characterized in that, The method further includes: If the distance between the center coordinates of the rotated CT profile and the center coordinates of the ultrasound profile is greater than the second threshold, the rotated CT profile is translated to a position where the distance between it and the ultrasound profile is less than or equal to the second threshold, thus obtaining the rotated and translated CT profile. The rotated and translated CT profile and the ultrasound profile are registered.

6. The method according to any one of claims 1-3, characterized in that, The similarity between the ultrasound profile and the CT profile is greater than or equal to a third threshold.

7. An image registration method, characterized in that, The method includes: The ultrasound profile and computed tomography (CT) profile of the target organ are obtained; the ultrasound profile is obtained from ultrasound images acquired by an ultrasound device, and the CT profile is obtained from CT images acquired by a CT device. The ultrasound profile is rotated to obtain a rotated ultrasound profile; the angle between the direction vector of the reference direction in the space of the ultrasound profile and the direction vector of the reference direction in the space of the rotated CT profile is less than or equal to a first threshold, and the reference direction is common to the space of the ultrasound profile and the space of the CT profile. Registration is performed based on the rotated ultrasound profile and the CT profile; The step of rotating the ultrasonic profile to obtain the rotated ultrasonic profile includes: Determine a first direction vector of the reference direction in the space of the CT profile, and determine a second direction vector of the reference direction in the space of the ultrasound profile. The rotation matrix is ​​determined based on the angle between the first direction vector and the second direction vector, and the angle between the first direction vector and the coordinate axis. The ultrasonic profile is rotated based on the aforementioned rotation matrix to obtain the rotated ultrasonic profile.

8. An electronic device, characterized in that, The method includes a processor and a memory for storing a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions such that the method described in any one of claims 1-7 is executed.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the method as described in any one of claims 1-7 to be performed.

Images