Image registration method and device, electronic device and storage medium

Abstract

The invention relates to an image registration method and device, an electronic device and a storage medium, and is applied to the field of medical image.The image registration method comprises the steps that first image data and second image data are acquired; registering the first image data and the second image data according to the first registration target to obtain an initial registration result; adjusting the initial registration result based on the second registration target to obtain a target registration result; the first registration target and the second registration target are different registration targets based on the same registration task. Through the image registration method and device, the problem that an existing image registration method has a large number of local optimum conditions, and consequently the algorithm precision of image registration is poor is solved.

Description

Technical Field

[0001] The present application relates to the field of medical imaging, and in particular to an image registration method, device, electronic device, and storage medium. Background Art

[0002] Medical image registration involves transforming medical images from the same or different imaging modalities to match their spatial positions and coordinates. The result of registration is that all anatomical points on the two images, at least all points of diagnostic significance and surgical interest, are matched. With the advancement of medical imaging engineering and computer technology, medical imaging has become an indispensable part of modern medicine. Its application permeates the entire clinical work, not only being widely used in disease diagnosis but also playing a vital role in the planning, implementation, and efficacy evaluation of surgical and radiotherapy procedures.

[0003] In the existing technology, the blood vessels of pre-operative 3D data and intra-operative 2D data are segmented separately, and then aligned according to the center lines of the segmented blood vessels; however, due to the countless rotational and translational movements of 3D geometric bodies, they achieve roughly the same 2D projection effect in the 2D camera plane, resulting in a large number of local optimal situations in the alignment process, which in turn leads to poor image alignment algorithm accuracy.

[0004] With regard to the problem of poor accuracy of image registration algorithms in related technologies, no effective solution has been proposed so far. Summary of the Invention

[0005] In this embodiment, an image registration method, device, electronic device, and storage medium are provided to solve the problem of poor accuracy of image registration algorithms in related technologies.

[0006] In a first aspect, an image registration method is provided in this embodiment, the method comprising:

[0007] acquiring first image data and second image data;

[0008] Registering the first image data and the second image data according to the first registration target to obtain an initial registration result;

[0009] The initial registration result is adjusted based on the second registration target to obtain a target registration result; the first registration target and the second registration target are different registration targets based on the same registration task.

[0010] In some embodiments, adjusting the initial registration result based on the second registration target to obtain the target registration result includes:

[0011] Based on the second registration target, registering the first image data and the second image data to obtain an optimized registration result;

[0012] The initial registration result is adjusted according to the optimized registration result to obtain a target registration result.

[0013] In some embodiments, adjusting the initial registration result according to the optimized registration result to obtain a target registration result includes:

[0014] Initialize the first registration target according to the optimized registration result to obtain an initialized first registration target;

[0015] The state variable value corresponding to the initial registration result is adjusted according to the initialized first registration target to obtain a target registration result.

[0016] In some embodiments, the method further comprises:

[0017] According to preset constraints, the matching point weights of the first registration target are adjusted to obtain the second registration target.

[0018] In some embodiments, the method further comprises:

[0019] Based on the initial registration result, the matching point weights of the first registration target are adjusted to obtain the second registration target.

[0020] In some embodiments, the method further comprises:

[0021] According to the first registration target and the second registration target, a target registration result is determined based on the initial registration result.

[0022] In some embodiments, the first registration target and the second registration target are modeled based on a preset function, and the preset function includes a state variable value and a state function value;

[0023] The determining, according to the first registration target and the second registration target, a target registration result based on the initial registration result, comprises:

[0024] Acquire a state variable value of the initial registration result, and determine a state function value of the second registration target according to the state variable value;

[0025] Based on the state function value, and according to the second registration target, registering the first image data and the second image data to obtain an optimized registration result;

[0026] Based on the optimized registration result, determining an updated initial registration result according to the first registration target;

[0027] Based on the updated initial registration result, determining an updated optimized registration result according to the second registration target;

[0028] Based on the updated optimized registration result, a target registration result is determined according to the first registration target.

[0029] In a second aspect, an image registration device is provided in this embodiment, the device comprising: an acquisition module and a registration module;

[0030] The acquisition module is used to acquire the first image data and the second image data;

[0031] The registration module is used to align the first image data and the second image data according to the first registration target to obtain an initial registration result; it is also used to adjust the initial registration result based on the second registration target to obtain a target registration result; the first registration target and the second registration target are different registration targets based on the same registration task.

[0032] In a third aspect, an electronic device is provided in this embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image configuration method described in the first aspect when executing the computer program.

[0033] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored. When the program is executed by a processor, the image configuration method described in the first aspect is implemented.

[0034] Compared with the related art, an image registration method, device, electronic device and storage medium provided in this embodiment align the first image data and the second image data of different modalities through an initial first registration target, and then obtain an initial registration result. The initial registration result at this time includes obtaining a locally optimal posture; at the same time, the initial registration result is adjusted according to the second registration target to obtain a target registration result that jumps out of the local optimal position, and image registration is performed according to the target registration result, thereby improving the accuracy of image registration.

[0035] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0037] Figure 1 This is a hardware structure block diagram of a terminal for the image registration method provided in an embodiment of the present application;

[0038] Figure 2 is a schematic diagram of a skeleton matching algorithm provided in an embodiment of the present application;

[0039] Figure 3 It is a schematic diagram of a large number of local optimal poses in the registration algorithm;

[0040] Figure 4 This is a schematic diagram of the projection effect of an existing 3D cube;

[0041] Figure 5 is a flow chart of the image registration method provided in an embodiment of the present application;

[0042] Figure 6 is a schematic diagram of an image registration method based on variable weight optimization provided in this specific embodiment;

[0043] Figure 7 is a schematic diagram of the weight change method provided in this specific embodiment;

[0044] Figure 8 is a flow chart of the iterative optimization registration method provided in this specific embodiment;

[0045] Figure 9 is a schematic diagram of an image registration result provided in this specific embodiment;

[0046] Figure 10 This is a structural block diagram of the image registration device provided in this embodiment. DETAILED DESCRIPTION

[0047] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0048] Unless otherwise defined, the technical terms or scientific terms involved in this application should have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "an", "a", "the", "these" and the like in this application do not indicate quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, and B exists alone. Generally, the character " / " indicates that the related objects are in an "or" relationship. The terms "first," "second," "third," etc. used in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0049] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 This is a hardware structure diagram of the terminal of the image registration method provided in the embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown) a processor 102 and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0050] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the image configuration method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0051] The transmission device 106 is used to receive or send data via a network. The network may include a wireless network provided by the terminal's telecommunications provider. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0052] Medical image registration is the process of transforming medical images from the same or different imaging modalities so that their spatial positions and spatial coordinates match. The result of registration is that all anatomical points on the two images, at least all points of diagnostic significance and points of surgical interest, are matched. With the development of medical imaging engineering and computer technology, medical imaging has become an indispensable part of modern medicine. Its application runs through the entire clinical work. It is not only widely used in the diagnosis of diseases, but also plays an important role in the planning, implementation, and efficacy evaluation of surgical and radiotherapy operations. However, due to the different imaging principles of different devices, multiple modal medical images have emerged in clinical diagnosis. The morphological and functional information obtained by different imaging technologies on the same anatomical structure of the human body is different and complementary.

[0053] In clinical diagnosis, it is often necessary to integrate information from multiple images to help doctors understand the comprehensive condition of diseased tissues or organs, making more accurate diagnoses or developing more appropriate treatment plans. The primary challenge in integrating multiple images is strict image alignment, or image registration. The effectiveness of image registration directly impacts the quality of image fusion, so only accurate registration can provide doctors with a reliable basis for diagnosis.

[0054] Medical image matching, including 3D images and 2D images, is mainly divided into data-driven and model-driven. Among them, data-driven requires a large amount of training data, but in many scenarios, it is not possible to obtain a large amount of data. Therefore, in many medical scenarios, data-driven methods are not used for image matching, but model-driven methods are used for image matching. Existing model-driven algorithms are mainly divided into: image similarity matching algorithm, skeleton feature matching algorithm and "3D skeleton + 2D gradient" matching algorithm. Model-driven algorithms are mainly used to process cross-modal image matching, but image similarity matching algorithms are difficult to obtain image matching results in real time. "3D skeleton + 2D gradient" projects the 3D skeleton onto the 2D image, and the algorithm determines whether it matches based on their direction vectors.

[0055] The skeleton feature matching algorithm unifies cross-modal 3D and 2D images on line topology features (i.e., 3D and 2D skeletons). Therefore, the consistency of line topology features can be used to estimate the pose of the DSA (Digital Subtraction Angiography) sensor. Figure 2 This is a schematic diagram of the skeleton matching algorithm provided in the embodiment of this application. Figure 2 , obtain the initial pose of the 2D image camera, and then obtain the 3D image and 2D image respectively. According to the 3D image, a 3D skeleton (i.e., the image centerline) is obtained, and according to the 2D image, a 2D skeleton (i.e., the image centerline) is obtained. Subsequently, the initial pose of the 2D image camera and the camera intrinsic parameters can be used to project the 3D skeleton onto the 2D camera plane to obtain the projection of the 3D skeleton at this initial pose; the projection of the 3D skeleton at this initial pose is then matched with the 2D skeleton, and the optimal pose of the 3D camera is obtained through iterative optimization, so that the projection of the 3D skeleton on the 2D plane best matches the observed 2D skeleton.

[0056] Furthermore, one way to define the best match between the 3D skeleton and the 2D skeleton is to define the best match as the Gaussian kernel distance between all 3D reprojection points and the nearest 2D skeleton point, which can be expressed as:

[0057]

[0058] Among them, p i represents the 3D reprojection point, Ω i For all 2D skeleton points that are close to pi, the K function is a Gaussian kernel. The optimal pose can be obtained by maximizing this Gaussian kernel function.

[0059] Other methods for defining the best match between a 3D skeleton and a 2D skeleton are to define it as the minimum distance between all 3D reprojection points and the nearest 2D skeleton point, or to directly search for descriptors in the 2D and 3D images and match them directly. Among them, the descriptor is a data or structure used to describe, represent or quantify an object, feature or attribute, and is used to describe the local features of key points in the image (such as corner points, edge points, etc.). It has a certain degree of robustness to rotation, scaling, illumination changes, etc. Furthermore, when solving the optimal pose, in order to ensure the real-time search (for example, a processing speed of more than 5Hz), a gradient-based method needs to be used, and a second-order search is preferably used.

[0060] In addition, regarding the modeling method of 3D skeletal motion, skeletal motion modeling is currently divided into "rigid body" and "rigid body + non-rigid body" forms. The rigid body is mainly expressed through Euler angles, but the linearization process of the rotation represented by Euler angles is computationally intensive and may also cause the gimbal lock problem, resulting in the loss of a degree of freedom.

[0061] Based on the above analysis, we adopted an alternative 3D skeletal motion modeling approach for image registration. Specifically, we used a Gaussian kernel function for non-parametric modeling and an Iterative Reweighted Least Squares (IRLS) solution. Furthermore, we expressed the rigid body transformations (i.e., rotations and translations) of the 3D data in a Special Euclidean Group (SE3). This approach can further improve the convergence speed of the algorithm iterations. However, all of these 3D-2D registration algorithms still suffer from a large number of local optima, which significantly impacts registration accuracy, particularly in the estimation of rigid body pose. Figure 3 This is a schematic diagram of a large number of local optimal poses in the registration algorithm. Figure 3 When registering 3D and 2D images, rotation and translation operations in a 3D object (for example, a cube) cannot replace each other, meaning that optimal registration cannot be achieved using only translation or only rotation. However, in the registration method between a 3D object (for example, a cube) and a 2D projection, since the projection of the 3D object is specifically registered with the 2D object, it is difficult to distinguish between rotation and translation in the projection of the 3D object. Figure 4 This is a schematic diagram of the projection effect of an existing 3D cube. Figure 4 As shown in the figure, the square on the left is the original 3D cube's logo on the 2D plane, and the two squares on the right are the logos on the 2D plane obtained by translating and rotating the original 3D cube. It can be understood that no matter whether it is translated or rotated, the projection of the original 3D cube on the 2D camera plane completely overlaps, so there are countless rotations and translations that can achieve the same 2D projection effect. Therefore, it will lead to Figure 3 There are a large number of valleys in , each of which corresponds to a local optimal pose. That is, there are a large number of local optimal poses. When searching for the optimal pose of a rigid body, it is easy to enter a "very bad" local optimum, resulting in poor image registration results.

[0062] In order to solve the problem of poor image registration results caused by multiple local optimal poses during image registration, an image registration method is provided in this embodiment. Figure 5 is a flow chart of the image registration method provided in the embodiment of the present application, such as Figure 5 As shown, the process includes the following steps:

[0063] Step S510 , acquiring first image data and second image data; registering the first image data and the second image data according to a first registration target to obtain an initial registration result.

[0064] In this step, when performing image registration, it is first necessary to obtain the first image data and the second image data to be registered, and the first image data and the second image data are image data obtained based on the same target object in different scenes. The first image data and the second image data can be data of the same modality or data of different modalities. For example, in clinical medicine, the first and / or second image data obtained by the above different imaging methods, such as X-ray imaging image data, computed tomography (CT) image data, magnetic resonance imaging (MRI) image data, ultrasound imaging data, and digital subtraction angiography (DSA) image data, can be of the same modality or different modalities, and the imaging methods of the first image data and the second image data are not specifically limited here.

[0065] Preferably, the image registration method provided by the present application is further elaborated by taking the first image data and the second image data as data of different modalities as an example. Specifically, after obtaining the first image data and the second image data of different modalities, the first registration target is determined, and the first registration target includes an algorithm for registering the first image data and the second image data; the first registration target can be determined according to conventional registration methods in the field, and can also be determined according to actual registration scenarios and registration requirements. For example, the first image data is 3D data, and the second image data is 2D data. The first registration target can be to project the 3D data onto a 2D imaging plane so that the projection of the 3D image data in the initial posture of the 2D image matches the 2D data. Furthermore, in order to improve the efficiency and accuracy of image matching, a skeleton feature matching algorithm is used to obtain the skeleton features of the 3D data and the 2D data respectively. The skeleton features generally used in clinical medicine can be bones or blood vessels, and then the consistency of the linear topological features is used to perform posture estimation.

[0066] The skeleton features corresponding to the first image data and the second image data are registered using a first registration target to obtain an initial registration result. Exemplarily, the initial registration result includes the relative pose of the 3D data in the 2D data coordinate system. Furthermore, based on the relative pose, the 3D data is projected into the 2D coordinate system, the 2D data point closest to the 3D projection point is determined, and data association is performed between the 3D projection point and the 2D data point closest to the 3D projection point to determine the corresponding relationship between the 3D data point and the 2D data point.

[0067] Step S520 , adjusting the initial registration result based on the second registration target to obtain a target registration result; the first registration target and the second registration target are different registration targets based on the same registration task.

[0068] In this embodiment, after the initial registration result is determined according to the first registration target, since the projection of the 3D data on the 2D plane may correspond to multiple combinations of motion modes after rotation and translation, multiple local optimal postures will be obtained when the first image data and the second image data are registered. Therefore, it is necessary to adjust the initial registration result through the second registration target to reduce the influence of the local optimal posture and obtain the best image registration result. The second registration target is also used to align the first image data and the second image data, but it is not the same as the first registration target. Exemplarily, the second registration target can be based on the first registration target to identify key feature points or markers with anatomical significance, such as vascular branch points, bony landmarks, etc. Thereafter, the initial registration result obtained by the first registration target is optimized and adjusted according to the second registration target to obtain the target registration result, so as to avoid the target registration result of the image registration being the initial registration result including the local optimal posture, thereby achieving the accuracy of the image registration.

[0069] Through the above steps, the first image data and the second image data of different modalities are registered according to the initial first registration target, and then an initial registration result is obtained. The initial registration result at this time includes multiple local optimal poses; then, in order to avoid the influence of the local optimal poses, the initial registration result is adjusted according to the second registration target, and then a target registration result that jumps out of the local optimal pose is obtained. Image registration is performed based on the target registration result, thereby improving the accuracy of image registration.

[0070] In some embodiments, in step S510, the initial registration result is adjusted based on the second registration target to obtain a target registration result, including: based on the second registration target, the first image data and the second image data are registered to obtain an optimized registration result; and the initial registration result is adjusted according to the optimized registration result to obtain a target registration result.

[0071] In this embodiment, after the first image data and the second image data are aligned according to the first alignment target, multiple initial alignment results including local optimal postures will be obtained; at this time, the second alignment target is required to re-align the first image data and the second image data to obtain an optimized alignment result, and then the initial alignment result obtained by the first alignment target is adjusted according to the optimized alignment result.

[0072] Furthermore, the method of adjusting the initial registration result according to the optimized registration result to obtain the target registration result includes initializing the first registration target according to the optimized registration result, and then adjusting the state variable value corresponding to the initial registration result according to the initialized first registration target to obtain the target registration result.

[0073] The two sets of image data are further registered using the second registration target, thereby overcoming limitations that may be encountered when using only the first registration target, such as the problem of multiple local optima, and obtaining a more accurate and globally optimal registration result, namely an optimized registration result. Subsequently, the first registration target is further processed using the optimized registration result, so that when the first registration target registers the first image data and the second image data, it can escape the local optimal situation. That is, the first registration target adjusts the state variable value corresponding to the local optimal initial registration result, and then obtains the state function value corresponding to the adjusted state variable value. The target registration result is determined based on the state function value, which is conducive to obtaining a more robust target registration result.

[0074] In addition, aligning the first image data and the second image data according to the second registration target will also obtain multiple optimized registration results, that is, there are multiple local optimal poses; therefore, it is necessary to alternately optimize the optimized registration results and the initial registration results obtained by the alignment according to the first registration target and the second registration target to improve the accuracy of the target registration results obtained by the final alignment.

[0075] The initial registration result obtained by the first registration target is adjusted through the second registration target to avoid the situation where multiple local relative poses are obtained by image registration based on the first registration target, and the best relative pose with the highest image registration accuracy, that is, the target registration result, cannot be obtained. This is conducive to solving the technical problem that multiple local relative poses of the first image data in the second image data coordinate system appear during image registration due to the inability to distinguish the projections of rotation and translation motion corresponding to the same image data.

[0076] Some of the embodiments provide another image registration method, including: adjusting the weights of matching points of a first registration target according to preset constraints to obtain a second registration target.

[0077] In this embodiment, a method for image registration of 2D image data and 3D image data is taken as an example. Non-parametric modeling is mainly performed using a Gaussian kernel function and solved using a least squares algorithm. Subsequently, the rigid body transformation (including rotation and translation) of the 3D data is expressed in a special Euclidean group. Specifically, the following formula for the relative pose parameters, i.e., the parameters of the initial registration result, is referred to:

[0078]

[0079] Where Ω3 is the set of skeleton feature points of 3D image data, and Ω2 is the set of skeleton feature points of 2D skeleton image data. π() represents the projection of the 3D skeleton onto the 2D coordinate plane. c∈R 3 is the center of the DSA device. L is the hyperparameter variance. w i Is a weight proportional to the 3D radius. dsa_cb is the transformation matrix from the known CB to the camera coordinate system of the DSA device. T∈SE(3) is the rotation and translation of the 3D skeleton point to be determined.

[0080] However, since the formula for the relative pose parameters is difficult to express in a standard least squares form, in this embodiment, an iterative reweighted least squares algorithm (IRLS) is used for optimization. Optimization is performed under the SE(3) special Euclidean group, which can determine the search step size at the second order and significantly improve the optimization speed. The optimization process is expressed as follows:

[0081]

[0082] Among them, the formula for expressing the weight w using the preset relative posture parameter ΔT is expressed as:

[0083]

[0084] Where π() represents the projection of the 3D skeleton onto the 2D coordinate plane. c∈R 3 is the center of the DSA device. L is the hyperparameter variance. w i is a weight proportional to the 3D radius, i is the set of skeleton feature points of 3D image data, and j is the set of skeleton feature points of 2D skeleton image data. dsa_cb is the transformation matrix from the known CB to the camera coordinate system of the DSA device. T∈SE(3) is the rotation and translation of the 3D skeleton point to be determined.

[0085] By continuously changing the weight w during the optimization process to achieve the purpose of second-order search, the relative posture parameter ΔT is obtained. Furthermore, the hyperparameter L can be changed from large to small during the optimization process to achieve the goal of matching at different scales.

[0086] Based on the above-mentioned image registration optimization method provided in the embodiments of this application, in this embodiment, the matching point weights of the first registration target can be further adjusted according to preset constraints to obtain a second registration target that implements a second-order search, i.e., optimization method. The preset constraints can be determined based on the first registration target; for example, regularization terms can be added to the registration algorithm corresponding to the first registration target, certain parameters in the weight w of the first registration target must meet specific conditions, or the weight of the distance error between corresponding points can be increased.

[0087] For example, taking a method for image registration of 2D image data and 3D image data corresponding to first image data and second image data as an example, a first registration target includes projecting the blood vessels where the skeleton points of the 3D image are located onto a 2D imaging plane to obtain the relative pose of the 3D image data in the 2D image data coordinate system. However, due to the small difference between the rotational and translational motions of the 3D image data within the 2D imaging plane, multiple relative poses will be obtained based on this first registration target; therefore, it is necessary to further constrain the first registration target to avoid obtaining multiple relative poses. For example, based on the first registration target, cross-matching points of the blood vessels where the skeleton points of the 3D image are located are obtained, and the weights of the cross-matching points and the weights of the non-cross-matching points are set to obtain a second registration target with changed matching point weights. The second registration target is used to optimize the multiple relative postures, i.e., multiple initial registration results, obtained according to the first registration target. That is, after a locally optimal relative posture is obtained through the first registration target, the image registration problem of the first image data and the second image data is transferred to the second registration target, and the image registration is performed again through the second registration target to obtain an optimized registration result. The initial registration result is replaced by the optimized registration result, thereby jumping out of the initial registration result obtained by the first registration target, which helps to improve the accuracy of image registration.

[0088] Furthermore, when image registration is performed through the second registration target, multiple optimized registration results can be obtained, that is, when performing image registration according to the second registration target, multiple local optimal postures may be obtained. After the second registration target obtains a local optimal relative posture, the image registration problem of the first image data and the second image data is transferred to the first registration target again, and image registration is performed again through the first registration target to obtain an updated initial registration result. Through iterative optimization registration between the first registration target and the second registration target, a target registration result that breaks away from the local optimal posture based on the initial registration result can eventually be obtained.

[0089] In some embodiments, another image registration method is provided, comprising: adjusting the matching point weights of a first registration target based on an initial registration result to obtain a second registration target.

[0090] In this embodiment, after the first image data and the second image data are image-aligned through the first registration target, a local optimal pose corresponding to the initial registration result will be obtained. According to the image data association relationship corresponding to the initial registration result, the matching point weight of the first registration target is adjusted to obtain the second registration target; thereafter, the image registration problem of the first image data and the second image data is transferred to the second registration target, and image alignment is performed again through the second registration target to obtain a local optimal pose corresponding to the optimized registration result. According to the image data association relationship corresponding to the optimized registration result, the matching point weight of the second registration target is adjusted to obtain the adjusted first registration target, and the second registration target and the first registration target are iteratively optimized in turn to finally obtain the final optimized first registration target, and image registration is performed according to the first registration target to obtain the target registration result.

[0091] This embodiment also provides an image registration method, which includes determining a target registration result based on an initial registration result according to a first registration target and a second registration target.

[0092] Furthermore, the first registration target and the second registration target are modeled based on a preset function, and the preset function includes a state variable value and a state function value; based on the first registration target and the second registration target, the target registration result is determined based on the initial registration result, including: obtaining the state variable value of the initial registration result, and determining the state function value corresponding to the state variable value in the second registration target; based on the state function value, according to the second registration target, aligning the first image data and the second image data to obtain an optimized registration result; based on the optimized registration result, determining an updated initial registration result according to the first registration target; based on the updated initial registration result, determining an updated optimized registration result according to the second registration target; based on the updated optimized registration result, determining the target registration result according to the first registration target.

[0093] In this embodiment, an initial registration result is obtained by performing image registration of the first image data and the second image data according to the first registration target. However, since there may be multiple local optimal postures corresponding to the initial registration result, it is necessary to determine the target registration result from the multiple initial registration results according to the first registration target and the second registration target to improve the accuracy of the image registration result. Specifically, according to the relative posture parameters corresponding to the initial registration result, the state variable values in the initial registration result are determined, that is, the variables to be optimized in the association relationship between the first image data and the second image data obtained based on the first registration target. For example, according to the relative posture parameters corresponding to the initial registration result, the relative posture parameters here are the representation of the association relationship between the first image data and the second image data; the relative posture parameters are used to describe the result of projecting the first image data onto the plane where the second image data is located. However, since the relative pose parameters are obtained by initial registration with the first registration target and are not the most accurate optimal pose parameters, it is necessary to optimize the variables in the relative pose parameters, that is, to optimize the variables to be optimized in the association relationship between the first image data and the second image data. The variables to be optimized can be the transformation relationship between the first image data and the second image data in the corresponding coordinate system, such as the rotation transformation amount, the translation transformation amount, and the scale transformation amount, etc., which are not specifically limited here. By optimizing the variables to be optimized through the second registration target, a target registration result with higher registration accuracy is ultimately obtained.

[0094] Specifically according to the second registration target, the variables to be optimized in the correlation relationship between the first image data and the second image data, that is, the state variable value in the second registration target and the corresponding state function value are calculated; then the state function value is optimized by the iterative reweighted least squares algorithm according to the second registration target, and then the optimized registration result corresponding to the second registration target is obtained, so as to help the first registration target jump out of the local optimal posture and obtain the target registration result.

[0095] For example, the first registration target and the second registration target are both based on the Gaussian kernel function distance Modeled, where Ω i For all 3D reprojection points p i The K function is a Gaussian kernel for 2D skeleton points with close distances. The state variable values corresponding to the first and second registration targets both include the reprojection point p of the skeleton point of the 3D image data corresponding to the Gaussian kernel distance. i and the skeleton points q of the 2D image data j ; The state function value includes the relative position of the 3D data corresponding to the state variable value in the 2D data coordinate system.

[0096] When the first image data and the second image data are registered using the first registration target, the above-mentioned iterative reweighted least squares algorithm (IRLS) is used for optimization to obtain an initial registration result that converges to a local optimum. After obtaining the initial registration result, it is indicated that further optimization cannot be performed, but the relative position corresponding to the initial registration result is not the optimal relative position. In this case, the state variable value corresponding to the initial registration result is obtained. Based on the state variable value, the first image data and the second image data are registered using the space where the second registration target is located to obtain the state function value corresponding to the state variable value in the second registration target. At the same time, the iterative reweighted least squares method is used for optimization and convergence to obtain the locally optimal optimized registration result obtained when the second registration target is registered.

[0097] Furthermore, based on the state variable values corresponding to the optimized registration result obtained by the second registration target, and based on the first registration target and the iterative reweighted least squares method for optimization convergence, the target registration result obtained by registering the first registration target is determined. This method allows the local optimal initial registration result to be jumped out with the help of the registration result of the second registration target when registering the first registration target to obtain the target registration result. Through the above steps, based on the relative posture parameters corresponding to the initial registration result, the state function value corresponding to the state variable value in the second registration target is determined; that is, the value of the corresponding Gaussian kernel function, i.e., the target registration result, is calculated based on the state function value including the variable to be optimized, thereby improving the accuracy of image registration.

[0098] The present embodiment is described and illustrated below through specific examples.

[0099] In a specific embodiment, taking the registration of 3D image data obtained before surgery and 2D image data obtained during surgery as an example, it is first necessary to perform blood vessel segmentation on the 3D image data and 2D image data respectively, and then perform registration based on the segmented blood vessel centerlines, wherein the registration is to obtain the relative position and posture of the 3D image data in the 2D image data coordinate system; after determining the relative position and posture, the 3D image data is projected to the 2D image data coordinate system, and the data association between the 3D projection point and the nearest 2D data point, that is, the 3D-2D correspondence, is determined.

[0100] Furthermore, the following formula is used to explain the reason why multiple local optimal poses are obtained when registering two images. Figure 4 The projection effect in the i is the i-th point of the 3D geometry, and {R, t} represents the rotation and translation operations respectively. are the motions caused by rotation and translation respectively. x ,R y ,Rz Represents the three degrees of freedom row vectors of R in the rotation direction, f x ,f y ,c x ,c y They all represent the intrinsic parameters of the camera. Among them, the rotation and translation of the 3D geometry are expressed by the formulas:

[0101]

[0102] In the camera coordinate system, the i-th point p of the 3D geometry i The points are mainly located in front of the camera and relatively far away from the camera. i The z coordinate value of the point is much larger than the x coordinate value and the y coordinate value, that is, the formula p i | z >>p i | x 、p i | z >>p i | y All of them are established. At this time, the first-order Taylor expansion of the rotation matrix is obtained as follows:

[0103]

[0104] At this time, the motion generated by rotation and translation Respectively expressed as:

[0105]

[0106] The difference in motion caused by rotation and translation is expressed as:

[0107]

[0108] (i)p i | z >>p i | x ;p i | z >>p i | y ;φ i <<1.

[0109] (ii) Approximately 1.

[0110] Therefore, we can know that The difference is very small. Therefore, when performing 3D-2D registration, since the rotation and translation of the 3D rigid body posture are numerically inseparable, multiple local optimal postures will be obtained when registering the two images.

[0111] In view of the problem that there are multiple local optimal poses when performing image registration based on the above method, this specific embodiment provides an image registration method with variable weight optimization. Figure 6 Schematic diagram of the image registration method based on variable weight optimization provided by this specific embodiment. Figure 6 The horizontal axis represents the state variable value, and the vertical axis represents the target state function value. The solid line represents the first registration objective in the aforementioned embodiment—the original problem—and the dashed line represents the second registration objective in the aforementioned embodiment—the auxiliary problem. The auxiliary problem is derived from the original problem, so when performing image registration using the auxiliary problem, a large number of local optima also exist. Although both problems have a large number of local optima, their local optima do not necessarily overlap, provided the problems are properly formulated. After the first step (Step 1) reaches a local optimum in the solid line, by transitioning to the auxiliary problem (Step 2), optimization can be performed in the auxiliary problem space. Step 3 then allows optimization to be performed outside the local optimum of the original problem and into the local optimum of the subproblem. This helps the original problem escape a local optimum, namely the local optimum of the solid line on the far left of the figure. Similarly, through Steps 4 and 5, it is possible to escape from the local optimum of the auxiliary problem to another local optimum of the original problem. Through repeated iterative optimization, the local optimum of the original problem can be effectively escaped, ultimately obtaining an object registration result that includes the optimal relative pose.

[0112] In a preferred embodiment, the original problem is defined as projecting 3D blood vessels onto a 2D imaging plane. The auxiliary problem is defined as projecting the intersection of 3D blood vessels onto the 2D imaging plane. In this case, the auxiliary problem is a sub-problem of the original problem. That is, based on the original problem, the auxiliary problem is defined as only the intersection point has a weight of 1 and the other points have a weight of 0, thus achieving variable weight optimization of the auxiliary problem and the original problem. Figure 7 , Figure 7 This is a schematic diagram of the weighting method provided in this specific embodiment. In the figure, the line segments represent 3D blood vessels, and the circles represent the intersection points of 3D blood vessels, that is, Figure 7 The weight corresponding to the middle circle is 1, and the weights corresponding to the rest are 0.

[0113] In one possible embodiment, the auxiliary problem is defined as projecting the intersection points with the most connected vessel segments among the 3D vessel intersection points into the 2D imaging plane.

[0114] Figure 8 Schematic diagram of the iterative optimization registration method provided in this specific embodiment. Figure 8First, the 3D image data and 2D image data are registered through the original problem to obtain an initial registration result, namely the relative pose including the 3D-2D data association; then, the auxiliary problem is initialized based on the relative pose including the 3D-2D data association, and then the auxiliary problem with the changed weight is estimated. Then, the 3D image data and 2D image data are registered through the auxiliary problem to obtain an optimized registration result, namely the relative pose including the 3D-2D data association; then, the original problem is initialized based on the optimized registration result including the relative pose including the 3D-2D data association, and then the original problem with the changed weight is estimated. By continuously iteratively optimizing the original problem and the auxiliary problem, the target registration result with the best registration result is finally determined through the original problem.

[0115] Preferably, the images of preoperative CT angiography (CTA, CT angiography) and intraoperative DSA (Digital Subtraction Angiography, vascular subtraction under X-ray) are registered, and the steps are as follows: First, the blood vessels are segmented using a 3D neural network and a 2D neural network respectively. Considering that CTA data and DSA data are different modalities, it is very difficult to directly register by image brightness. Since the blood vessels define invariance (the segmented blood vessels do not have multimodality), they can be used for registration. Thereafter, the variable weight alternating optimization method as in the above embodiment is used to estimate the rigid body posture and establish the corresponding relationship of the blood vessels. A larger scale parameter can be set. Finally, the optimal rigid body posture and blood vessel correspondence are obtained.

[0116] refer to Figure 9 , Figure 9 This is a schematic diagram of an image registration result provided in this specific embodiment. According to the experimental results based on the above-mentioned image registration method, during the DSA angiography process, due to the small amount of initial angiography, the initial posture of the blood vessels is very poor, and there will be initial mismatching of blood vessels. When the angiography is normal, the image registration method provided in this application can correct the above-mentioned errors and obtain the best image registration results.

[0117] Furthermore, the image registration method provided in the present application can be applied to vascular tissue, as well as pulmonary vascular tissue, such as skin, bones, etc.; both methods can be used to improve the image registration effect.

[0118] This embodiment also provides an image registration device for implementing the above-mentioned embodiments and preferred implementations. Details already described will not be repeated. The terms "module," "unit," "subunit," etc. used below may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0119] Figure 10 This is a structural block diagram of the image registration device provided in this embodiment. Figure 10 As shown, the device includes: an acquisition module 10 and a registration module 20.

[0120] The acquisition module 10 is configured to acquire first image data and second image data.

[0121] The registration module 20 is used to align the first image data and the second image data according to the first registration target to obtain an initial registration result; it is also used to adjust the initial registration result based on the second registration target to obtain a target registration result; the first registration target and the second registration target are different registration targets based on the same registration task.

[0122] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0123] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0124] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0125] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0126] S1, acquiring first image data and second image data.

[0127] S2, registering the first image data and the second image data according to the first registration target to obtain an initial registration result.

[0128] S3, adjusting the initial registration result based on the second registration target to obtain a target registration result; the first registration target and the second registration target are different registration targets based on the same registration task.

[0129] It should be noted that, for specific examples in this embodiment, reference may be made to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0130] In addition, in combination with the image registration method provided in the above embodiments, a storage medium may be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the image registration methods in the above embodiments is implemented.

[0131] It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0132] Obviously, the accompanying drawings are merely examples or embodiments of the present application. A person skilled in the art can also apply the present application to other similar situations based on these drawings without inventive effort. Furthermore, it is understandable that, although the work involved in this development process may be complex and lengthy, certain design, manufacturing, or production changes based on the technical content disclosed in this application are merely routine technical means for a person skilled in the art and should not be considered to constitute a deficiency in the disclosure of the present application.

[0133] The term "embodiment" as used in this application refers to specific features, structures, or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily mean that the embodiment is the same, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is understood, either explicitly or implicitly, by those skilled in the art that the embodiments described in this application can be combined with other embodiments when there is no conflict.

[0134] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An image registration method, characterized in that: The method comprises: acquiring first image data and second image data; Registering the first image data and the second image data according to the first registration target to obtain an initial registration result; The initial registration result is adjusted based on the second registration target to obtain a target registration result; the first registration target and the second registration target are different registration targets based on the same registration task.

2. The image registration method according to claim 1, wherein: The adjusting the initial registration result based on the second registration target to obtain the target registration result includes: Based on the second registration target, registering the first image data and the second image data to obtain an optimized registration result; The initial registration result is adjusted according to the optimized registration result to obtain a target registration result.

3. The image registration method according to claim 2, wherein: The adjusting the initial registration result according to the optimized registration result to obtain a target registration result includes: Initialize the first registration target according to the optimized registration result to obtain an initialized first registration target; According to the initialized first registration target, the state variable value corresponding to the initial registration result is adjusted to obtain a target registration result.

4. The image registration method according to any one of claims 1 to 3, characterized in that: The method further comprises: According to preset constraints, the matching point weights of the first registration target are adjusted to obtain the second registration target.

5. The image registration method according to claim 4, characterized in that: The method further comprises: Based on the initial registration result, the matching point weights of the first registration target are adjusted to obtain the second registration target.

6. The image registration method according to claim 1, wherein: The method further comprises: According to the first registration target and the second registration target, a target registration result is determined based on the initial registration result.

7. The image registration method according to claim 6, characterized in that: The first registration target and the second registration target are obtained based on a preset function model, and the preset function includes a state variable value and a state function value; The determining, according to the first registration target and the second registration target, a target registration result based on the initial registration result, comprises: Acquire a state variable value of the initial registration result, and determine a state function value of the second registration target according to the state variable value; Based on the state function value, and according to the second registration target, registering the first image data and the second image data to obtain an optimized registration result; Based on the optimized registration result, determining an updated initial registration result according to the first registration target; Based on the updated initial registration result, determining an updated optimized registration result according to the second registration target; Based on the updated optimized registration result, a target registration result is determined according to the first registration target.

8. An image registration device, characterized in that: The device comprises: an acquisition module and a registration module; The acquisition module is used to acquire the first image data and the second image data; The registration module is used to align the first image data and the second image data according to the first registration target to obtain an initial registration result; it is also used to adjust the initial registration result based on the second registration target to obtain a target registration result; the first registration target and the second registration target are different registration targets based on the same registration task.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the image registration method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image registration method according to any one of claims 1 to 7 are implemented.

Images