Multi-bone-block scene image registration method, device and equipment based on 2D-3D contour registration

Abstract

The invention relates to the technical field of image processing, and provides a multi-bone-block scene image registration method and device based on 2D-3D contour registration, electronic equipment and a computer program product. The method comprises the following steps: acquiring a two-dimensional image and a three-dimensional image to be registered in a multi-bone-block scene; selecting a plurality of fixed contour points with significant features from the three-dimensional image; projecting the plurality of fixed contour points to the two-dimensional image to obtain a plurality of initial contour points in the two-dimensional image, the plurality of initial contour points being in one-to-one correspondence with the plurality of fixed contour points; and completing registration of the two-dimensional image and the three-dimensional image based on the plurality of initial contour points and the plurality of fixed contour points. According to the method, a plurality of fixed contour points with remarkable characteristics are selected as registration points, so that the problem of unstable contour extraction caused by interactive point selection in a registration process can be solved, and the accuracy of 2D-3D image registration for a multi-bone-block scene is improved.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular, to a multi-bone block scene image registration method, apparatus, electronic device, and computer program product based on 2D-3D contour registration. Background Art

[0002] 2D-3D image registration refers to the process of spatially aligning two-dimensional images such as X-ray images and RGB images with three-dimensional images such as CT images and MRI images. This process establishes a mapping relationship between two-dimensional pixels and three-dimensional point clouds to achieve cross-modal or cross-dimensional data consistency, and has extensive applications in fields such as medical image analysis, radiotherapy, and surgical navigation. Existing technologies have proposed contour-based 2D-3D image registration methods. This method extracts the contour features of two-dimensional and three-dimensional images and establishes corresponding geometric correspondence relationships to achieve spatial alignment, and can obtain good image registration results. However, in a multi-bone block scene, since there may be overlap or occlusion between the bone blocks, and the relative positions and postures of the bone blocks may also change during the surgical process, it is difficult to stably extract the bone block contours, reducing the accuracy of image registration. Summary of the Invention

[0003] In view of this, the embodiments of the present application provide a multi-bone block scene image registration method, apparatus, electronic device, and computer program product based on 2D-3D contour registration, which can improve the accuracy of 2D-3D image registration for multi-bone block scenes.

[0004] The first aspect of the embodiments of the present application provides a multi-bone block scene image registration method based on 2D-3D contour registration, including:

[0005] Obtain a two-dimensional image and a three-dimensional image to be registered in a multi-bone block scene;

[0006] Select a plurality of fixed contour points with significant features from the three-dimensional image;

[0007] Project the plurality of fixed contour points onto the two-dimensional image to obtain a plurality of initial contour points in the two-dimensional image, and the plurality of initial contour points correspond to the plurality of fixed contour points one by one;

[0008] Complete the registration of the two-dimensional image and the three-dimensional image based on the plurality of initial contour points and the plurality of fixed contour points.

[0009] In the technical solution of the embodiment of the present application, first, a two-dimensional image and a three-dimensional image of a multi-bone block scene are acquired, and a plurality of fixed contour points with significant features are selected from the three-dimensional image; then, the plurality of fixed contour points are projected onto the two-dimensional image to obtain a plurality of initial contour points in the two-dimensional image, and the plurality of initial contour points correspond to the plurality of fixed contour points one by one; finally, based on the plurality of initial contour points and the plurality of fixed contour points, the registration of the two-dimensional image and the three-dimensional image is completed. Existing 2D-3D image registration methods need to continuously and iteratively reselect registration points, and in a multi-bone block scene, due to reasons such as occlusion or overlap of bone blocks, the registration points may be occluded, affecting the registration accuracy. Different from this, the embodiment of the present application selects a plurality of fixed contour points with significant features as registration points, and these fixed contour points exhibit obvious features in the bone structure, making their visual and geometric performances very significant. Even if the bone blocks are occluded or overlapped, it will not affect the accurate extraction of these fixed contour points. Therefore, the problem of unstable contour extraction caused by interactive point selection during the registration process is solved, and the accuracy of 2D-3D image registration can be effectively improved in a multi-bone block scene.

[0010] In one implementation manner of the embodiment of the present application, based on a plurality of initial contour points and a plurality of fixed contour points, the registration of the two-dimensional image and the three-dimensional image is completed, including:

[0011] For each initial contour point, based on the normalized cross-correlation algorithm, a plurality of candidate contour points with the highest gradient correlation corresponding to the initial contour point are searched from the gradient direction of the two-dimensional image, and by introducing the distance and angle constraints between point pairs, a candidate contour point is selected from the plurality of candidate contour points as the target contour point corresponding to the initial contour point;

[0012] Based on the target contour points corresponding to the plurality of initial contour points respectively and the plurality of fixed contour points, the registration of the two-dimensional image and the three-dimensional image is completed.

[0013] In one implementation manner of the embodiment of the present application, by introducing the distance and angle constraints between point pairs, a candidate contour point is selected from the plurality of candidate contour points as the target contour point corresponding to the initial contour point, including:

[0014] For each candidate contour point, based on the perspective-n-point algorithm, the matching contour point corresponding to the candidate contour point is calculated, and according to the gradient correlation between the candidate contour point and the corresponding initial contour point, and the distance similarity between the candidate contour point and the matching contour point, the matching score of the candidate contour point is calculated;

[0015] From the plurality of candidate contour points, a candidate contour point with the highest matching score is selected as the target contour point corresponding to the initial contour point.

[0016] In an implementation manner of the embodiment of the present application, according to the gradient correlation between the candidate contour point and the corresponding initial contour point, and the distance similarity between the candidate contour point and the matching contour point, the matching degree score of the candidate contour point is calculated, including:

[0017] The matching degree score of the candidate contour point is calculated by using the following formula:

[0018] S(p i ,p′ i )=w GC *GC(p i ,p′ i )-w dist *D(p′ i ,p″ i )

[0019] Wherein, p′ i represents the candidate contour point, p i represents the initial contour point corresponding to the candidate contour point, p″ i represents the matching contour point corresponding to the candidate contour point, S(p i ,p′ i ) represents the matching degree score of the candidate contour point, GC(p i ,p′ i ) represents the gradient correlation between p i and p′ i , D(p′ i ,p″ i ) represents the distance similarity between p′ i and p″ i , w GC and w dist are weight coefficients.

[0020] In an implementation manner of the embodiment of the present application, the distance similarity between p′ i and p″ i is calculated by the following formula:

[0021]

[0022] Wherein, and are respectively the abscissa and ordinate of p′ i in the two-dimensional image, and are respectively the abscissa and ordinate of p″ i in the two-dimensional image.

[0023] In an implementation manner of the embodiment of the present application, from the three-dimensional image, multiple fixed contour points with significant features are selected, including:

[0024] Extract the three-dimensional contour points of the bone mass from the three-dimensional image;

[0025] Select multiple ridge points on the bone ridge line from the three-dimensional contour points as multiple fixed contour points.

[0026] In one implementation manner of the embodiment of the present application, selecting multiple ridge points on the bone ridge line from the three-dimensional contour points includes:

[0027] Select multiple contour points with curvatures greater than a preset threshold from the three-dimensional contour points as multiple ridge points through a curvature analysis algorithm.

[0028] The second aspect of the embodiment of the present application provides a multi-bone mass scene image registration device based on 2D-3D contour registration, including:

[0029] An image acquisition module for acquiring a two-dimensional image and a three-dimensional image to be registered in a multi-bone mass scene;

[0030] A contour point selection module for selecting multiple fixed contour points with significant features from the three-dimensional image;

[0031] A contour point projection module for projecting multiple fixed contour points onto the two-dimensional image to obtain multiple initial contour points in the two-dimensional image, and the multiple initial contour points correspond one-to-one to the multiple fixed contour points;

[0032] An image registration module for completing the registration of the two-dimensional image and the three-dimensional image based on the multiple initial contour points and the multiple fixed contour points.

[0033] The third aspect of the embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multi-bone mass scene image registration method based on 2D-3D contour registration provided in the first aspect of the embodiment of the present application.

[0034] The fourth aspect of the embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, it causes the electronic device to execute the multi-bone mass scene image registration method based on 2D-3D contour registration provided in the first aspect of the embodiment of the present application.

[0035] The fifth aspect of the embodiment of the present application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the multi-bone mass scene image registration method based on 2D-3D contour registration provided in the first aspect of the embodiment of the present application.

[0036] It can be understood that the beneficial effects of the second to fifth aspects described above can be referred to the relevant descriptions in the first aspect, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flowchart of a multi-bone block scene image registration method based on 2D-3D contour registration provided by an embodiment of the present application;

[0038] Figure 2 is an operation schematic diagram of three-dimensional pelvic ridge line extraction provided by an embodiment of the present application;

[0039] Figure 3 is a schematic diagram of the instability of single-dependent image gradient-based NCC matching point search provided by an embodiment of the present application;

[0040] Figure 4 is an operation schematic diagram of selecting registration points by combining gradient correlation and distance similarity provided by an embodiment of the present application;

[0041] Figure 5 is a schematic diagram of a registration framework adopted by a multi-bone block scene image registration method based on 2D-3D contour registration provided by an embodiment of the present application;

[0042] Figure 6 is a schematic diagram of the structure of a multi-bone block scene image registration device based on 2D-3D contour registration provided by an embodiment of the present application;

[0043] Figure 7 is a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] In the following description, specific details such as specific system structures and technologies are proposed for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application. In addition, in the description of the specification and the appended claims of the present application, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0045] With the development of medical imaging technology, contour-based 2D-3D image registration methods have been widely studied and applied in the field of medical image registration. These methods mainly use the boundary or contour information of images to achieve accurate and stable image registration. However, in multi-bone scenarios, due to possible overlap or occlusion between bones, the features and boundaries of bones may be confused, and misleading similarity measures may occur during the registration point search process, resulting in difficulty for the registration algorithm to accurately find the corresponding registration points during the iteration process, affecting the accuracy of image registration. Moreover, by processing each bone separately, the registration algorithm can only obtain a local optimal solution and cannot obtain a global optimal solution that comprehensively considers the mutual relationships of all bones.

[0046] To address the above technical problems, embodiments of the present application propose a multi-bone scenario image registration method, apparatus, electronic device, and computer program product based on 2D-3D contour registration. By selecting multiple fixed contour points with significant features as registration points and introducing distance and angle constraints between point pairs as a strategy to improve the accuracy and stability of matching point search, the accuracy of 2D-3D image registration for multi-bone scenarios can be effectively improved. For more specific technical implementation details of the embodiments of the present application, please refer to the various method embodiments described below.

[0047] It should be understood that the execution subject of each method embodiment proposed in the present application can be various types of electronic devices. For example, it can be a mobile phone, a tablet computer, a desktop computer, a wearable device, a medical device, an endoscope host, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a large-screen TV, and so on. The embodiments of the present application do not impose any restrictions on the specific type of this electronic device.

[0048] Please refer to Figure 1 , which shows a multi-bone scenario image registration method based on 2D-3D contour registration provided by an embodiment of the present application, including:

[0049] 101. Obtain a two-dimensional image and a three-dimensional image to be registered in a multi-bone scenario;

[0050] First, obtain the two-dimensional image and three-dimensional image to be registered in the multi-bone block scenario. Among them, the two-dimensional image can be an image such as an X-ray image, RGB image, etc., and the three-dimensional image can be an image such as a CT image, MRI image, etc. The specific types of the two-dimensional image and three-dimensional image to be registered in the embodiments of the present application are not limited in any way. The two-dimensional image and three-dimensional image are images collected for the multi-bone block scenario, and the images contain multiple bone blocks, and there may be overlap or occlusion between the bone blocks. For example, the edge of one bone block may be confused with the features of another bone block, or may be completely invisible due to occlusion.

[0051] 102. Select multiple fixed contour points with significant features from the three-dimensional image;

[0052] In the process of image registration, selecting representative registration points is a key step. These registration points can provide important information about the global changes of the image or point cloud. Generally, key feature points in the image or point cloud, such as corner points, edge points, or other significant structural points, are selected as registration points. By selecting some registration points instead of all points for image registration, the calculation time and complexity can be greatly reduced, while still being able to obtain accurate registration results.

[0053] In the process of traditional registration algorithms, interactive point selection is generally based on visible contour points. In the multi-bone block scenario, due to reasons such as occlusion or overlap of bone blocks, the registration points selected iteratively may be occluded, affecting the registration accuracy. Different from this, after obtaining the two-dimensional image and three-dimensional image to be registered in the embodiments of the present application, multiple fixed contour points with significant features (such as feature points on the pelvic edge) are selected from the three-dimensional image. These fixed contour points usually have stable features in the image or point cloud and contain sufficient information for estimating global changes. In actual operation, a certain number of multiple fixed contour points with significant features can be selected based on the anatomical structure features of the bone block scenario. These fixed contour points show obvious features in the bone structure, making their visual and geometric performances very significant, and even if there is occlusion or overlap of bone blocks, it will not affect the accurate extraction of these fixed contour points. That is, using fixed contour points with significant features as registration points instead of the unstable iterative interactive point selection process can avoid problems such as occlusion of registration points.

[0054] In an implementation manner of the embodiments of the present application, selecting multiple fixed contour points with significant features from the three-dimensional image includes:

[0055] (1) Extract the three-dimensional contour points of the bone block from the three-dimensional image;

[0056] (2) Select multiple ridge points on the bone ridge line from the three-dimensional contour points as the multiple fixed contour points.

[0057] In the embodiments of the present application, through the analysis of the pelvic anatomical structure, it is found that the matching degree between the bone ridge line and the visible contour line of the pelvis is relatively high. The points on the bone ridge line, that is, the ridge line points, are generally not blocked in the multi-bone block scenario, and the ridge line points exhibit obvious characteristics in the bone structure, making their visual and geometric performances very significant. Therefore, multiple ridge line points on the bone ridge line can be selected as the above-mentioned multiple fixed contour points. Specifically, first, from the above three-dimensional image, the three-dimensional contour points of each bone block are extracted, and then the characteristics of the bone ridge line are identified for these three-dimensional contour points. After identifying the bone ridge line, a certain number of ridge line points can be selected from the bone ridge line in a random selection or fixed interval selection manner, etc., as the above-mentioned fixed contour points. By selecting fixed points on the bone ridge line instead of unstable iterative interactive point selection, the risk of misregistration caused by image overlap and occlusion can be significantly reduced, and the registration error caused by soft tissue deformation or other factors during the operation can be effectively reduced, thereby improving the accuracy and robustness of image registration. In addition, using these ridge line points with significant characteristics as registration points can improve the time efficiency of surgical navigation because these ridge line points are usually easier to automatically match between different medical images.

[0058] In one implementation manner of the embodiments of the present application, selecting multiple ridge line points on the bone ridge line from the three-dimensional contour points includes:

[0059] Through the curvature analysis algorithm, multiple contour points with curvatures greater than a preset threshold are selected from the three-dimensional contour points as multiple ridge line points.

[0060] When identifying the bone ridge line from the three-dimensional contour points of the bone block, a curvature analysis algorithm, such as the principal curvature analysis algorithm, etc., can be used to identify the linear region with a curvature greater than the preset threshold from the three-dimensional contour points as the identified bone ridge line, and then multiple contour points are selected from the bone ridge line in a certain manner (such as random selection or fixed interval selection, etc.) as the selected multiple ridge line points. Considering that in three-dimensional space, the ridge line is a linear feature region with a very large surface curvature of the bone block, these ridge lines are often relatively prominent visually and have certain significance in the anatomy of the bone. Therefore, the curvature analysis algorithm can more accurately identify and extract these ridge line points from the three-dimensional contour points of the bone block.

[0061] As an example, Figure 2 is an operation schematic diagram of three-dimensional pelvic ridge line extraction provided by the embodiments of the present application. Among them, Figure 2 (a) are all the peak lines of the three-dimensional pelvic ridge line, Figure 2 (b) are the peak lines of the three-dimensional pelvic ridge line filtered by an intensity threshold of 1.5, Figure 2 (c) are the peak lines of the three-dimensional pelvic ridge line filtered by a sharpness threshold of 80000. Through Figure 2It can be seen that since the bone ridge line has the characteristic of extremely large curvature, by selecting the linear region where the crest line located above among the three-dimensional pelvic contour points, the ridge line points can be accurately identified and extracted.

[0062] 103. Project multiple fixed contour points onto a two-dimensional image to obtain multiple initial contour points in the two-dimensional image;

[0063] After selecting multiple fixed contour points with significant features from the three-dimensional image, project the multiple fixed contour points onto the above-mentioned two-dimensional image to obtain multiple initial contour points in the two-dimensional image. The multiple initial contour points correspond one-to-one to the multiple fixed contour points. For example, assume that N fixed contour points are selected from the three-dimensional image. Then, projecting the N fixed contour points onto the two-dimensional image can obtain N projection points, and the N projection points are the N initial contour points in the two-dimensional image. The N fixed contour points and the N initial contour points are in a one-to-one correspondence relationship.

[0064] 104. Complete the registration of the two-dimensional image and the three-dimensional image based on the multiple initial contour points and the multiple fixed contour points.

[0065] Finally, complete the registration of the two-dimensional image and the three-dimensional image based on the multiple initial contour points and the multiple fixed contour points. Specifically, use the multiple initial contour points and the multiple fixed contour points as registration points, and complete the alignment operation of the two-dimensional image and the three-dimensional image in the same coordinate system according to the basic method of 2D-3D image registration in the prior art, and then the registration of the two-dimensional image and the three-dimensional image can be realized.

[0066] In the traditional normalized cross-correlation (NCC) matching method based on image gradient, it mainly relies on the local gradient features of the image to find registration points, and this method is very effective in many applications. However, when dealing with images with non-uniform feature distributions, this method may encounter problems. Especially in the fields of medical imaging and surgical navigation, for multi-bone block scenarios, the feature distribution within the image region may be significantly uneven due to tissue heterogeneity, different imaging conditions, or interference during operations, resulting in unstable search for registration points. As an example, Figure 3 is a schematic diagram showing the instability of the NCC matching point search that solely relies on image gradient provided by an embodiment of the present application. Through Figure 3 it can be seen that the high NCC value in the gradient direction may only reflect the high similarity of local features, and thus the globally optimal registration points cannot be obtained.

[0067] To solve the above problems existing in the traditional NCC matching method, in the embodiments of the present application, on the basis of the NCC matching method, the distance and angle constraints between point pairs are introduced, which can improve the accuracy and stability of the registration point search. During the registration point search process, not only the similarity of image features is relied on, but also the geometric position relationship of points in space is considered, enhancing the anatomical correctness and physical feasibility of the matching. By introducing the distance and angle constraints, the point matching process becomes more rigorous and accurate, and can significantly improve the accuracy, stability and reliability of image registration. For example, in the multi-bone block scenario, introducing the distance and angle constraints can provide additional context information by maintaining the geometric relationship between points, thereby improving the accuracy of the registration point search. The specific operation method is described below.

[0068] In one implementation manner of the embodiments of the present application, based on multiple initial contour points and multiple fixed contour points, the registration of the two-dimensional image and the three-dimensional image is completed, including:

[0069] (1) For each initial contour point, based on the normalized cross-correlation algorithm, search from the gradient direction of the two-dimensional image to obtain multiple candidate contour points with the highest gradient correlation corresponding to the initial contour point, and by introducing the distance and angle constraints between point pairs, select one candidate contour point from the multiple candidate contour points as the target contour point corresponding to the initial contour point;

[0070] (2) Based on the target contour points corresponding to the multiple initial contour points and the multiple fixed contour points, complete the registration of the two-dimensional image and the three-dimensional image.

[0071] Assume that the traditional normalized cross-correlation algorithm based on image gradient is used. Then for each initial contour point, one contour point with the highest gradient correlation corresponding to the initial contour point will be searched from the gradient direction of the two-dimensional image as the target contour point corresponding to the initial contour point. Based on the above Figure 3 analysis, it can be known that such an NCC matching point search that solely relies on image gradient has instability and may cause problems of non-homogeneous feature matching. In contrast, in the embodiments of the present application, for each initial contour point, first, based on the normalized cross-correlation algorithm, search from the gradient direction of the two-dimensional image to obtain multiple candidate contour points with the highest gradient correlation corresponding to the initial contour point, and then, by introducing the distance and angle constraints between point pairs, select one candidate contour point from the multiple candidate contour points as the target contour point corresponding to the initial contour point. For example, assume that an initial contour point is p i , then search along the gradient direction of the two-dimensional image for n candidate contour points p i with the highest gradient correlation with p i ′ , and then, by introducing the distance and angle constraints between point pairs, select from the n candidate contour points pi ′ Select a candidate contour point from the candidates as the target contour point corresponding to the initial contour point p i Finally, based on the target contour points corresponding to each of the multiple initial contour points and the above-mentioned multiple fixed contour points, the registration of the two-dimensional image and the three-dimensional image is completed. By introducing distance and angle constraints between point pairs, the accuracy and stability of the registration point search can be improved, thereby improving the accuracy of image registration.

[0072] In one implementation manner of the embodiment of the present application, by introducing distance and angle constraints between point pairs, selecting a candidate contour point from multiple candidate contour points as the target contour point corresponding to the initial contour point includes:

[0073] (1) For each candidate contour point, calculate the matching contour point corresponding to the candidate contour point based on the Perspective-n-Point algorithm, and calculate the matching score of the candidate contour point according to the gradient correlation between the candidate contour point and the corresponding initial contour point, and the distance similarity between the candidate contour point and the matching contour point;

[0074] (2) Select the candidate contour point with the highest matching score from the multiple candidate contour points as the target contour point corresponding to the initial contour point.

[0075] When selecting a target contour point from multiple candidate contour points by introducing distance and angle constraints between point pairs, for each candidate contour point, calculate the matching contour point corresponding to the candidate contour point based on the Perspective-n-Point algorithm, and calculate the matching score of the candidate contour point according to the gradient correlation between the candidate contour point and the corresponding initial contour point, and the distance similarity between the candidate contour point and the matching contour point. Then, select the candidate contour point with the highest matching score from the multiple candidate contour points as the target contour point corresponding to the initial contour point. This process utilizes the invariant rigid distance between 3D point pairs and introduces geometric constraints through the Perspective-n-Point algorithm (i.e., the Perspective-n-Point, PnP algorithm), which can ensure the registration accuracy under different perspectives. Specifically, the PnP algorithm calculates the corresponding camera pose by imposing spatial geometric constraints on the points in 3D space, thereby effectively solving the problem of matching point errors in complex anatomical structures. This geometric constraint helps to eliminate abnormal matching points during the matching process and improves the robustness of the registration algorithm, especially in multi-bone scenarios such as the pelvis with complex contours and multiple bone blocks overlapping.

[0076] In one implementation manner of the embodiment of the present application, calculating the matching score of the candidate contour point according to the gradient correlation between the candidate contour point and the corresponding initial contour point, and the distance similarity between the candidate contour point and the matching contour point includes:

[0077] The matching score of the candidate contour point is calculated using the following formula:

[0078] S(p i , p′ i ) = w GC *GC(p i , p′ i ) - w dist *D(p′ i , p″ i )

[0079] where p′ i represents the candidate contour point, p i represents the initial contour point corresponding to the candidate contour point, p″ i represents the matching contour point corresponding to the candidate contour point, S(p i , p′ i ) represents the matching score of the candidate contour point, GC(p i , p′ i ) represents the gradient correlation between p i and p′ i , D(p′ i , p″ i ) represents the distance similarity between p′ i and p″ i , and w GC and w dist are weight coefficients.

[0080] Assume that p i represents any initial contour point, p′ i represents any candidate contour point corresponding to the initial contour point, p″ i represents the matching contour point corresponding to p′ i (which can be obtained by calculating based on the PnP algorithm), then the matching score calculation function S(p i , p′ i ) as defined by the above formula can be defined. This function combines the gradient correlation and the distance similarity to evaluate the matching score of each candidate contour point. Specifically, GC(p i , p′ i ) represents the gradient correlation between p i and p′ i , which can be calculated by the NCC algorithm; D(p′ i , p″ i ) represents the distance similarity between p′ i and p″ i , which can be calculated according to the coordinates of p′ i and p″ i ; w GCand w dist are preset weight coefficients, which are used to determine the calculation weights of gradient correlation and distance similarity. By using the above matching score calculation function S(p i , p′ i ), it can simultaneously consider the gradient correlation and distance similarity between point pairs, thereby more robustly determining the matching relationship between images and avoiding the problem of non-uniform feature matching caused by local area similarity. Using the above matching score calculation function S(p i , p′ i ) can be calculated separately from the initial contour point p i The matching scores of all corresponding candidate contour points are then used as the candidate contour point with the highest matching score. i The corresponding target contour points.

[0081] In one implementation of the embodiment of the present application, p′ i and p″ i The distance similarity is calculated by the following formula:

[0082]

[0083] in, and are p′ i The horizontal and vertical coordinates in a two-dimensional image, and are p″ i The horizontal and vertical coordinates in a two-dimensional image.

[0084] In calculating p′ i and p″ i The distance similarity D(p′ i , p″ i ), due to p′ i The horizontal coordinate in a two-dimensional image and the vertical axis is known, so we first use the PnP algorithm to calculate p″ i The horizontal coordinate in a two-dimensional image and the vertical axis Then substitute the four coordinate values into the above D(p′ i , p″ i ) can be calculated by the formula.

[0085] As an example, Figure 4 This is a schematic diagram of an operation of selecting registration points by combining gradient correlation and distance similarity provided in an embodiment of the present application. Figure 4 In the example, an initial contour point is represented by p iIt is shown that within the search range along the gradient direction, the gradient correlation and distance similarity of each candidate contour point are calculated, and the matching score of each candidate contour point is calculated. Specifically, among the candidate contour points, the one with the highest gradient correlation (GC value) with p i is a candidate contour point p1′ i , but the candidate contour point with the highest matching score S is p′ i . Therefore, p′ i is selected instead of p1′ i as the target contour point corresponding to p i . Additionally, in Figure 4 , the vector dp i = p′ i - p i can be used to represent the deviation between the two-dimensional image and the three-dimensional image.

[0086] Generally speaking, the embodiment of the present application proposes a registration framework based on a point-to-plane correspondence model with fixed contour points, which can process all bone blocks in a multi-bone block scenario under a unified registration framework, and consider the relative positions and constraints between bone blocks, so it is more likely to obtain accurate and robust image registration results. As an example, Figure 5 is a schematic diagram of a registration framework adopted by the multi-bone block scenario image registration method based on 2D-3D contour registration provided by the embodiment of the present application. For the preoperative CT three-dimensional image, fixed contour points are selected for searching registration points. For example, the ridge points with significant features in the pelvic contour are selected as registration points. The selected ridge points are projected onto the X-ray two-dimensional image, and the invariant rigid distance between three-dimensional points is used to introduce geometric constraints through the PnP algorithm to ensure the registration accuracy under different perspectives. For the multi-bone block scenario, the above registration framework can achieve efficient and accurate image registration in complex regions such as the pelvis by introducing geometric constraints and the matching strategy of fixed contour points, providing strong technical support for further surgical navigation.

[0087] In the technical solution of the embodiment of the present application, first, a two-dimensional image and a three-dimensional image of a multi-bone block scene are obtained, and a plurality of fixed contour points with significant features are selected from the three-dimensional image; then, the plurality of fixed contour points are projected onto the two-dimensional image to obtain a plurality of initial contour points in the two-dimensional image, and the plurality of initial contour points correspond to the plurality of fixed contour points one by one; finally, based on the plurality of initial contour points and the plurality of fixed contour points, the registration of the two-dimensional image and the three-dimensional image is completed. Existing 2D-3D image registration methods need to continuously and iteratively reselect registration points, and in a multi-bone block scene, the registration points may be occluded due to occlusion or overlap of bone blocks, etc., affecting the registration accuracy. Different from this, the embodiment of the present application selects a plurality of fixed contour points with significant features as registration points, and these fixed contour points exhibit obvious features in the bone structure, making their visual and geometric performances very significant. Even if the bone blocks are occluded or overlapped, it will not affect the accurate extraction of these fixed contour points. Therefore, the problem of unstable contour extraction caused by interactive point selection during the registration process is solved, and the accuracy of 2D-3D image registration can be effectively improved in a multi-bone block scene.

[0088] In summary, the image registration method proposed in the embodiment of the present application uses fixed contour points to replace iterative interactive point selection, solving problems such as instability and complexity of interactive point selection during the registration process. Moreover, by introducing distance and angle constraints between point pairs, the stability and accuracy of matching point search can be improved, thereby further improving the accuracy of image registration.

[0089] It should be understood that the magnitudes of the sequence numbers of the steps in the above various embodiments do not mean the order of execution. The order of execution of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiment of the present application.

[0090] The above mainly describes a multi-bone block scene image registration method based on 2D-3D contour registration. Next, a multi-bone block scene image registration device based on 2D-3D contour registration will be described.

[0091] Please refer to Figure 6 , which shows a multi-bone block scene image registration device provided by the embodiment of the present application, including:

[0092] An image acquisition module 601, configured to acquire a two-dimensional image and a three-dimensional image to be registered in a multi-bone block scene;

[0093] A contour point selection module 602, configured to select a plurality of fixed contour points with significant features from the three-dimensional image;

[0094] The contour point projection module 603 is configured to project a plurality of fixed contour points onto a two-dimensional image to obtain a plurality of initial contour points in the two-dimensional image, and the plurality of initial contour points correspond to the plurality of fixed contour points one by one;

[0095] The image registration module 604 is configured to complete the registration of the two-dimensional image and the three-dimensional image based on the plurality of initial contour points and the plurality of fixed contour points.

[0096] In an implementation manner of the embodiment of the present application, the image registration module includes:

[0097] The contour point selection unit is configured to, for each initial contour point, search in the gradient direction of the two-dimensional image based on the normalized cross-correlation algorithm to obtain a plurality of candidate contour points with the highest gradient correlation corresponding to the initial contour point, and select one candidate contour point from the plurality of candidate contour points as the target contour point corresponding to the initial contour point by introducing distance and angle constraints between point pairs;

[0098] The image registration unit is configured to complete the registration of the two-dimensional image and the three-dimensional image based on the target contour points corresponding to the plurality of initial contour points and the plurality of fixed contour points.

[0099] In an implementation manner of the embodiment of the present application, the contour point selection unit includes:

[0100] The matching degree score calculation sub-unit is configured to, for each candidate contour point, calculate the matching contour point corresponding to the candidate contour point based on the perspective-n-point algorithm, and calculate the matching degree score of the candidate contour point according to the gradient correlation between the candidate contour point and the corresponding initial contour point and the distance similarity between the candidate contour point and the matching contour point;

[0101] The contour point selection sub-unit is configured to select one candidate contour point with the highest matching degree score from the plurality of candidate contour points as the target contour point corresponding to the initial contour point.

[0102] In an implementation manner of the embodiment of the present application, the matching degree score calculation sub-unit includes:

[0103] The first formula calculation sub-unit is configured to calculate the matching degree score of the candidate contour point by using the following formula:

[0104] S(p i , p′ i ) = w GC *GC(p i , p′ i ) - w dist *D(p′ i , p″ i )

[0105] where p′i Denote the candidate contour point as p i Denote the initial contour point corresponding to the candidate contour point as p″ i Denote the matching contour point corresponding to the candidate contour point as S(p i , p′ i ), and denote the matching score of the candidate contour point as GC(p i , p′ i ), which represents the gradient correlation between p i and p′ i . Denote the distance similarity between p′ i and p″ i as D(p′ i , p″ i ), and w GC and w dist are weight coefficients.

[0106] In an implementation manner of the embodiment of the present application, the matching score calculation sub-unit further includes:

[0107] A second formula calculation sub-unit, configured to calculate the distance similarity between p′ i and p″ i by using the following formula:

[0108]

[0109] wherein, and are respectively the abscissa and ordinate of p′ i in the two-dimensional image, and are respectively the abscissa and ordinate of p″ i in the two-dimensional image.

[0110] In an implementation manner of the embodiment of the present application, the contour point selection module includes:

[0111] A contour point extraction unit, configured to extract three-dimensional contour points of a bone mass from a three-dimensional image;

[0112] A ridge line point selection unit, configured to select multiple ridge line points on a bone ridge from the three-dimensional contour points as multiple fixed contour points.

[0113] In an implementation manner of the embodiment of the present application, the ridge line point selection unit includes:

[0114] A ridge line point selection sub-unit, configured to select multiple contour points with curvatures greater than a preset threshold from the three-dimensional contour points as multiple ridge line points by using a curvature analysis algorithm.

[0115] An embodiment of the present application further provides a computer-readable storage medium storing a computer program, which when executed by a processor, implements the multi-bone block scene image registration method based on 2D-3D contour registration described in any of the above embodiments.

[0116] An embodiment of the present application further provides a computer program product, which when running on an electronic device, causes the electronic device to execute the multi-bone block scene image registration method based on 2D-3D contour registration described in any of the above embodiments.

[0117] Figure 7 FIG. is a schematic diagram of an electronic device provided by an embodiment of the present application. As Figure 7 shown, the electronic device 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70. When the processor 70 executes the computer program 72, it implements the steps in the above embodiments of the various multi-bone block scene image registration methods based on 2D-3D contour registration, such as Figure 1 the steps 101-104 shown. Alternatively, when the processor 70 executes the computer program 72, it implements the functions of each module / unit in the above device embodiments, such as implementing Figure 6 the functions of module 601-module 604 of the device shown.

[0118] The computer program 72 can be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program 72 in the electronic device 7.

[0119] The so-called processor 70 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0120] The memory 71 may be an internal storage unit of the electronic device 7, such as a hard disk or memory of the electronic device 7. The memory 71 may also be an external storage device of the electronic device 7, such as a plug-in hard disk equipped on the electronic device 7, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 71 may also include both an internal storage unit and an external storage device of the electronic device 7. The memory 71 is used to store the computer program and other programs and data required by the electronic device. The memory 71 may also be used to temporarily store data that has been output or will be output.

[0121] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the above division of each functional unit and module is used as an example. In practical applications, the above functions can be assigned to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiment and will not be elaborated herein.

[0122] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiment and will not be elaborated herein.

[0123] In the above embodiments, the descriptions of the various embodiments have their own emphases. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0124] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0125] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0126] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiments of the present application.

[0127] In addition, each functional unit in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0128] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above method embodiments of the present application, it can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form. The computer-readable medium can include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0129] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A multi-bone block scene image registration method based on 2D-3D contour registration, characterized in that, Including: Obtain a two-dimensional image and a three-dimensional image to be registered in a multi-bone block scenario; Select a plurality of fixed contour points with significant features from the three-dimensional image; Project the plurality of fixed contour points onto the two-dimensional image to obtain a plurality of initial contour points in the two-dimensional image, and the plurality of initial contour points correspond to the plurality of fixed contour points one by one; Based on the plurality of initial contour points and the plurality of fixed contour points, complete the registration of the two-dimensional image and the three-dimensional image.

2. The method according to claim 1, characterized in that The completing the registration of the two-dimensional image and the three-dimensional image based on the plurality of initial contour points and the plurality of fixed contour points includes: For each of the initial contour points, based on the normalized cross-correlation algorithm, search for a plurality of candidate contour points with the highest gradient correlation corresponding to the initial contour point from the gradient direction of the two-dimensional image, and select one candidate contour point from the plurality of candidate contour points as the target contour point corresponding to the initial contour point by introducing distance and angle constraints between point pairs; Based on the target contour points corresponding to the plurality of initial contour points respectively and the plurality of fixed contour points, complete the registration of the two-dimensional image and the three-dimensional image.

3. The method according to claim 2, characterized in that, The selecting one candidate contour point from the plurality of candidate contour points as the target contour point corresponding to the initial contour point by introducing distance and angle constraints between point pairs includes: For each of the candidate contour points, calculate the matching contour point corresponding to the candidate contour point based on the perspective-n-point algorithm, and calculate the matching score of the candidate contour point according to the gradient correlation between the candidate contour point and the corresponding initial contour point and the distance similarity between the candidate contour point and the matching contour point; Select the candidate contour point with the highest matching score from the plurality of candidate contour points as the target contour point corresponding to the initial contour point.

4. The method according to claim 3, wherein The calculating the matching score of the candidate contour point according to the gradient correlation between the candidate contour point and the corresponding initial contour point and the distance similarity between the candidate contour point and the matching contour point includes: Calculate the matching score of the candidate contour point by using the following formula: S(p i , p′ i ) = w GC * GC(p i , p′ i ) - w dist * D(p′ i , p″ i ) Among them, p′ i represents the candidate contour point, p i represents the initial contour point corresponding to the candidate contour point, p″ i represents the matching contour point corresponding to the candidate contour point, S(p i , p′ i ) represents the matching score of the candidate contour point, GC(p i , p′ i ) represents the gradient correlation between p i and p′ i , D(p′ i , p″ i ) represents the distance similarity between p′ i and p″ i , w GC and w dist are weight coefficients.

5. The method according to claim 4, characterized in that p′ i The distance similarity with p″ i is calculated by the following formula: Among them, and are the abscissa and ordinate of p′ i in the two-dimensional image, respectively, and are the abscissa and ordinate of p″ i in the two-dimensional image, respectively.

6. The method according to any one of claims 1 to 5, characterized in that, The selecting a plurality of fixed contour points with significant features from the three-dimensional image includes: Extract the three-dimensional contour points of the bone block from the three-dimensional image; Select a plurality of ridge points on the bone ridge line from the three-dimensional contour points as the plurality of fixed contour points.

7. The method according to claim 6, wherein The selecting a plurality of ridge points on the bone ridge line from the three-dimensional contour points includes: Select a plurality of contour points with curvature greater than a preset threshold from the three-dimensional contour points as the plurality of ridge points by using a curvature analysis algorithm.

8. A multi-bone block scene image registration device based on 2D-3D contour registration, characterized in that Including: An image acquisition module for obtaining a two-dimensional image and a three-dimensional image to be registered in a multi-bone block scenario; A contour point selection module for selecting a plurality of fixed contour points with significant features from the three-dimensional image; A contour point projection module for projecting the plurality of fixed contour points onto the two-dimensional image to obtain a plurality of initial contour points in the two-dimensional image, and the plurality of initial contour points correspond to the plurality of fixed contour points one by one; An image registration module, configured to complete the registration of the two-dimensional image and the three-dimensional image based on the multiple initial contour points and the multiple fixed contour points.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multi-bone block scene image registration method based on 2D-3D contour registration according to any one of claims 1 to 7.

10. A computer program product, characterized in that, When the computer program product runs on an electronic device, it causes the electronic device to execute the multi-bone block scene image registration method based on 2D-3D contour registration according to any one of claims 1 to 7.

Images