Registration method of three-dimensional model and entity object, computer equipment and storage medium

Abstract

The invention is suitable for the technical field of medical images, and provides a registration method of a three-dimensional model and an entity object, computer equipment and a storage medium, and the method comprises the steps: calculating an initial registration matrix between a target entity object and the three-dimensional model of the target entity object; and projecting the three-dimensional model, the preset landmark on the three-dimensional model, the intraoperative landmark on the target entity object and the intraoperative point cloud to a preset plane based on the initial registration matrix, so that a user can drag a first projection point corresponding to the preset landmark up, down, left and right in a two-dimensional projection image; updating of the display position of the first projection point in the two-dimensional projection image is achieved, and then accurate correction of the initial registration matrix is achieved; the three-dimensional model and the intraoperative point cloud are registered based on the target registration matrix obtained through correction, and the display positions of the projection points in the two-dimensional projection image are updated, so that accurate registration of the target entity object and the three-dimensional model can be realized. According to the method, the registration precision is improved while the difficulty of landmark registration correction is reduced.

Description

Technical Field

[0001] This application belongs to the field of medical imaging technology, and in particular relates to a method for registering a three-dimensional model with a physical object, a computer device, and a storage medium. Background Technology

[0002] In computer-assisted surgery (CAS) or robot-assisted surgery (RAS), to accurately reproduce the surgical plan planned preoperatively based on medical imaging data within the patient's body, it is necessary to first register a 3D model of the actual anatomical structure with the actual anatomical structure. The registration process between the 3D model and the actual anatomical structure typically includes landmark registration and surface registration. Landmark registration usually involves solving for the initial values ​​of the rigid transformation between the 3D model and the actual anatomical structure based on pre-defined landmarks on the 3D model and their corresponding intraoperative landmarks. Surface registration, based on this rigid transformation, achieves precise registration between the intraoperative point cloud of the 3D model and the surface of the actual anatomical structure.

[0003] In related techniques, intraoperative landmarks on actual anatomical structures are typically marked manually by the surgeon. However, in minimally invasive surgeries with small incisions, the surgeon's field of vision is limited, making it difficult to accurately locate and mark the intraoperative landmarks corresponding to the pre-set landmarks. This can easily introduce rotational deviations during landmark registration, reducing the accuracy of subsequent surface registration and even leading to registration failure. Although related techniques can provide an adjustment interface for the initial value of rigid transformations, their operation is difficult, preventing surgeons from promptly correcting registration errors caused by landmark registration. Summary of the Invention

[0004] In view of this, embodiments of this application provide a registration method, computer device and storage medium for a three-dimensional model and an entity object, in order to solve the technical problem that traditional registration methods for entity objects and their three-dimensional models cannot promptly correct registration errors generated during the landmark registration process, which easily leads to registration failure.

[0005] In a first aspect, embodiments of this application provide a registration method for a three-dimensional model and a solid object, including: Obtain a 3D model of the target entity object, the 3D coordinates of each preset landmark on the 3D model, the 3D coordinates of each intraoperative landmark on the target entity object corresponding to each preset landmark, and the 3D coordinates of the intraoperative point cloud on the target entity object; Based on the three-dimensional coordinates of all the preset landmarks and the three-dimensional coordinates of all the intraoperative landmarks, calculate the initial registration matrix between the three-dimensional model and the target entity object; Based on the initial registration matrix and preset projection parameters, the 3D model, the preset landmarks, the intraoperative landmarks, and the intraoperative point cloud are projected onto a preset plane to generate and display a corresponding 2D projection image. The 2D projection image includes the projection outline of the 3D model, each first projection point corresponding to each preset landmark, each second projection point corresponding to each intraoperative landmark, and multiple third projection points corresponding to the intraoperative point cloud. In response to a drag operation on the first projection point, the display position of the first projection point in the two-dimensional projection image is updated in real time. Based on the preset projection parameters and the updated display position of the first projection point, the three-dimensional coordinates of the preset landmark corresponding to the first projection point are updated. Based on the updated three-dimensional coordinates of all preset landmarks and the three-dimensional coordinates of all intraoperative landmarks, the target registration matrix between the three-dimensional model and the target entity object is calculated. The 3D model is registered with the intraoperative point cloud based on the target registration matrix, and the display positions of all projection points in the 2D projection image are updated.

[0006] In one optional implementation of the first aspect, calculating the initial registration matrix between the 3D model and the target entity object based on the 3D coordinates of all the preset landmarks and the 3D coordinates of all the intraoperative landmarks includes: Based on the three-dimensional coordinates of all the preset landmarks, determine the first centroid of all the preset landmarks, and based on the first centroid, perform decentralization processing on each of the preset landmarks respectively; Based on the three-dimensional coordinates of all the intraoperative landmarks, the second centroid of all the intraoperative landmarks is determined, and based on the second centroid, each of the intraoperative landmarks is decentralized. Based on the three-dimensional coordinates of all the preset landmarks after decentralization and the three-dimensional coordinates of all the intraoperative landmarks, calculate the first optimal rotation matrix between all the preset landmarks and all the intraoperative landmarks; Based on the three-dimensional coordinates of the first centroid, the three-dimensional coordinates of the second centroid, and the first optimal rotation matrix, calculate the first translation vector between all the preset landmarks and all the intraoperative landmarks; The first optimal rotation matrix and the first translation vector are used to construct the initial registration matrix.

[0007] In one optional implementation of the first aspect, projecting the 3D model, the preset landmarks, the intraoperative landmarks, and the intraoperative point cloud onto a preset plane based on the initial registration matrix and preset projection parameters includes: The three-dimensional coordinates of the intraoperative landmark and the three-dimensional coordinates of the intraoperative point cloud are transformed to the coordinate system of the three-dimensional model by using the inverse matrix of the initial registration matrix, so as to obtain the three-dimensional coordinates of the intraoperative landmark and the three-dimensional coordinates of the intraoperative point cloud after coordinate transformation. Based on preset projection parameters, the 3D model, the preset landmarks, the intraoperative landmarks after coordinate transformation, and the intraoperative point cloud after coordinate transformation are projected onto a preset plane.

[0008] In one alternative implementation of the first aspect, the first projection point, the second projection point, and the third projection point in the two-dimensional projection image are each a semi-transparent dot of a different color.

[0009] In one optional implementation of the first aspect, calculating the target registration matrix between the three-dimensional model and the target entity object based on the updated three-dimensional coordinates of all the preset landmarks and the three-dimensional coordinates of all the intraoperative landmarks includes: Based on the updated 3D coordinates of all the preset landmarks, determine the third centroid of all the preset landmarks, and based on the third centroid, decentralize the updated preset landmarks respectively. Based on the three-dimensional coordinates of all the intraoperative landmarks, the second centroid of all the intraoperative landmarks is determined, and based on the second centroid, each of the intraoperative landmarks is decentralized. Based on the three-dimensional coordinates of all the updated preset landmarks after decentralization and the three-dimensional coordinates of all intraoperative landmarks, calculate the second optimal rotation matrix between all the updated preset landmarks and all the intraoperative landmarks; Based on the three-dimensional coordinates of the third centroid, the three-dimensional coordinates of the second centroid, and the optimal rotation matrix, calculate the second translation vector between all the preset landmarks and all the intraoperative landmarks; The second optimal rotation matrix and the second translation vector are used to construct the target registration matrix.

[0010] In one optional implementation of the first aspect, the target registration matrix is ​​used as the initial value of the rigid transformation of the iterative nearest point algorithm, and the three-dimensional model and the intraoperative point cloud are registered by the iterative nearest point algorithm to obtain the final registration matrix and the corresponding registration quality parameters. If the registration quality parameters meet the preset requirements, then based on the final registration matrix, the display positions of the first projection point, the second projection point, and the third projection point in the two-dimensional projection image are updated again.

[0011] Secondly, embodiments of this application provide a computer device, including: The first acquisition unit is used to acquire the three-dimensional model of the target entity object, the three-dimensional coordinates of each preset landmark on the three-dimensional model, the three-dimensional coordinates of each intraoperative landmark on the target entity object corresponding to each preset landmark, and the three-dimensional coordinates of the intraoperative point cloud on the target entity object. The first calculation unit is used to calculate the initial registration matrix between the three-dimensional model and the target entity object based on the three-dimensional coordinates of all the preset landmarks and the three-dimensional coordinates of all the intraoperative landmarks. A virtual projection unit is used to project the three-dimensional model, the preset landmarks, the intraoperative landmarks, and the intraoperative point cloud onto a preset plane based on the initial registration matrix and preset projection parameters, so as to generate and display the corresponding two-dimensional projection image; the two-dimensional projection image includes the projection outline of the three-dimensional model, each first projection point corresponding to each preset landmark, each second projection point corresponding to each intraoperative landmark, and multiple third projection points corresponding to the intraoperative point cloud; The first processing unit is configured to respond to a drag operation on the first projection point by updating the display position of the first projection point in the two-dimensional projection image in real time, and update the three-dimensional coordinates of the preset landmark corresponding to the first projection point according to the preset projection parameters and the updated display position of the first projection point, and calculate the target registration matrix between the three-dimensional model and the target entity object according to the updated three-dimensional coordinates of all the preset landmarks and the three-dimensional coordinates of all the intraoperative landmarks. The second processing unit is used to register the three-dimensional model with the intraoperative point cloud based on the target registration matrix, and update the display position of all projection points in the two-dimensional projection image.

[0012] Thirdly, embodiments of this application provide another computer device, including a memory and a computer program stored in the memory and executable on a processor, wherein the processor executes the computer program to implement the method as described in any of the optional implementations of the first aspect above.

[0013] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the optional implementations of the first aspect above.

[0014] Fifthly, embodiments of this application provide a computer program product that, when run on a computer device, enables the computer device to implement the method described in any optional implementation of the first aspect.

[0015] Implementing the registration method for three-dimensional models and solid objects, the computer equipment, the computer-readable storage medium, and the computer program product provided in the embodiments of this application has the following beneficial effects: The registration method for a 3D model and an entity object provided in this application calculates an initial registration matrix between the target entity object and its 3D model. Based on this initial registration matrix, the 3D model, preset landmarks on the 3D model, intraoperative landmarks on the target entity object, and intraoperative point cloud are projected onto a preset plane. This allows users to update the display position of the first projection point in the 2D projection image by dragging the first projection point corresponding to the preset landmark in the up, down, left, and right directions, thereby achieving accurate correction of the initial registration matrix. By registering the 3D model and the intraoperative point cloud based on the corrected target registration matrix and updating the display position of the projection point in the 2D projection image, precise registration of the target entity object and the 3D model can be achieved. In other words, this method can improve the registration accuracy and efficiency of the target entity object and its 3D model while reducing the difficulty of manually correcting the initial registration matrix. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic flowchart illustrating a registration method between a three-dimensional model and a solid object provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the implementation process of step S102 in a registration method between a 3D model and an entity object provided in an embodiment of this application. Figure 3 A schematic diagram of a two-dimensional projection image provided in an embodiment of this application; Figure 4 A schematic diagram illustrating the dragging process of a first projection point in a two-dimensional projection image provided in this application embodiment; Figure 5 A schematic diagram illustrating the updating principle of intraoperative landmarks provided in this application embodiment; Figure 6 This is a schematic diagram illustrating the implementation process of S104 in a registration method between a 3D model and an entity object provided in an embodiment of this application. Figure 7 A schematic diagram of an updated two-dimensional projection image provided in an embodiment of this application; Figure 8A schematic diagram comparing the registration process provided by known technologies and embodiments of this application; Figure 9 A schematic diagram of the structure of a computer device provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of a computer device provided in another embodiment of this application. Detailed Implementation

[0018] The following embodiments are only used to illustrate the technical solutions of this application more clearly, and are therefore only examples and should not be used to limit the scope of protection of this application.

[0019] In the description of the embodiments of this application, the technical terms "comprising," "including," "having," and any variations thereof all mean "including but not limited to," unless otherwise specifically emphasized. In the description of the embodiments of this application, unless otherwise stated, the technical term "multiple" refers to two or more, and the technical terms "at least one" or "one or more" refer to one, two, or more than two. The technical terms "first," "second," etc., are only used to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary / secondary relationship of the indicated technical features. The technical term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0020] This application first provides a method for registering a 3D model with a solid object. This method can be applied to computer devices. Computer devices may include, for example, mobile phones, tablets, laptops, or desktop computers; this application does not limit the type of computer device.

[0021] Figure 1 This is a schematic flowchart illustrating a registration method between a 3D model and a solid object, provided as an embodiment of this application. Figure 1 As shown, the method may include S101 to S105, as detailed below: S101, obtain the three-dimensional model of the target entity object, the three-dimensional coordinates of each preset landmark on the three-dimensional model, the three-dimensional coordinates of each intraoperative landmark on the target entity object corresponding to each preset landmark, and the three-dimensional coordinates of the intraoperative point cloud on the target entity object.

[0022] The target entity can refer to the actual anatomical part of the human body or other animal body that requires surgical intervention, such as the acetabulum, vertebral body or femur of the human body.

[0023] The 3D model of the target entity can be a virtual model obtained by reconstructing the target entity's medical imaging data in 3D. Medical imaging data can include magnetic resonance imaging (MRI) data or computed tomography (CT) data, etc.

[0024] Pre-defined landmarks can be a series of feature points marked on a 3D model before surgery. These feature points can be selected at locations with clear anatomical significance, such as bone protrusions, depressions, or intersections of edges, to ensure the uniqueness and identifiability of each pre-defined landmark in space.

[0025] Intraoperative landmarks can be a series of feature points collected on the surface of the target object by the probe of the surgical navigation system during the operation. These feature points correspond one-to-one with the various preset landmarks on the three-dimensional model.

[0026] Intraoperative point clouds are collections of numerous surface points acquired during surgery by scanning equipment within a surgical navigation system on the surface of the target object. The number of surface points in an intraoperative point cloud is far greater than that of intraoperative landmarks.

[0027] Optionally, the computer device can obtain a 3D model of the target entity and the 3D coordinates of each preset landmark on the 3D model from local memory. For example, the 3D model can be represented by isosurfaces. An isosurface is a 3D mesh model reconstructed from CT data based on a preset density threshold, representing the surface of the target entity. The preset density threshold can be set according to actual conditions. The 3D coordinates of the preset landmarks can be 3D coordinates in the CT coordinate system corresponding to the CT device.

[0028] Optionally, the computer device can obtain the three-dimensional coordinates of each intraoperative landmark and the three-dimensional coordinates of the intraoperative point cloud from the surgical navigation system. For example, both the three-dimensional coordinates of the intraoperative landmarks and the three-dimensional coordinates of the intraoperative point cloud can be three-dimensional coordinates in the navigation coordinate system corresponding to the surgical navigation system.

[0029] S102, calculate the initial registration matrix between the 3D model and the target entity based on the 3D coordinates of all preset landmarks and the 3D coordinates of all intraoperative landmarks.

[0030] The initial registration matrix can be an initial rigid transformation matrix used to transform the three-dimensional coordinates of the intraoperative landmarks and the three-dimensional coordinates of the intraoperative point cloud to the coordinate system of the three-dimensional model and the preset landmarks.

[0031] It should be understood that since the intraoperative landmarks collected during the operation may not accurately correspond to the preset landmarks, the initial rigid transformation matrix may be inaccurate. A more accurate target rigid transformation matrix needs to be calculated through manual correction.

[0032] Optionally, the computer device can be used through, for example Figure 2 The steps S1021~S1025 shown are used to calculate the initial registration matrix between the 3D model and the target entity object, detailed below: S1021. Based on the three-dimensional coordinates of all preset landmarks, determine the first centroid of all preset landmarks, and perform decentralization processing on each preset landmark based on the first centroid.

[0033] The first centroid is the geometric center of all preset landmarks, and its three-dimensional coordinates in the CT coordinate system are... x The axis components are all preset coordinates. x The average value of the axial components, y The axis components are all preset coordinates. y The average value of the axial components, z The axis components are all preset coordinates. z The average value of the axial components.

[0034] Optionally, the computer device can achieve decentralized processing of each preset landmark by subtracting the three-dimensional coordinates of each preset landmark from the three-dimensional coordinates of the first centroid.

[0035] S1022, Based on the three-dimensional coordinates of all intraoperative landmarks, determine the second centroid of all intraoperative landmarks, and based on the second centroid, perform decentralization processing on each intraoperative landmark.

[0036] The second centroid is the geometric center of all intraoperative landmarks, and its three-dimensional coordinates in the navigation coordinate system are... x The axial component is the basis for all intraoperative landmarks. x The average value of the axial components, y The axial component is the basis for all intraoperative landmarks. y The average value of the axial components, z The axial component is the basis for all intraoperative landmarks. z The average value of the axial components.

[0037] Optionally, the computer device can achieve decentralized processing of each intraoperative landmark by subtracting the three-dimensional coordinates of each intraoperative landmark from the three-dimensional coordinates of the second centroid.

[0038] S1023, Calculate the first optimal rotation matrix between all preset landmarks and all intraoperative landmarks based on the three-dimensional coordinates of all preset landmarks and the three-dimensional coordinates of all intraoperative landmarks after the decentralization process.

[0039] The first optimal rotation matrix is ​​the rotation matrix that minimizes the overall deviation between all the pre-defined decentralized landmarks and all the intraoperative decentralized landmarks after rotation. The dimension of the first optimal rotation matrix can be 3×3. It should be noted that since the calculation process of the optimal rotation matrix for two three-dimensional coordinates in different coordinate systems is a known technique, the calculation process of the first optimal rotation matrix can be referred to the description in related technologies, and will not be described in detail in the embodiments of this application.

[0040] S1024, based on the three-dimensional coordinates of the first centroid, the three-dimensional coordinates of the second centroid, and the first optimal rotation matrix, calculate the first translation vector between all preset landmarks and all intraoperative landmarks.

[0041] The first translation vector is defined as the amount of displacement required to maximize the spatial overlap between all pre-defined landmarks and all intraoperative landmarks after rotation. The dimension of the first translation vector can be 3×1.

[0042] Specifically, the computer device can calculate the product of the first optimal rotation matrix and the three-dimensional coordinates of the first centroid, and subtract the product from the three-dimensional coordinates of the second centroid to obtain the first translation vector.

[0043] S1025, construct the initial registration matrix by combining the first optimal rotation matrix and the first translation vector.

[0044] The initial registration matrix is ​​a 4×4 dimension standard homogeneous coordinate transformation matrix.

[0045] Specifically, the computer device can fill the top left 3×3 region of an empty 4×4 matrix with the first optimal rotation matrix, fill the top right 3×1 region of the 4×4 matrix with the first translation vector, and fill the last row of the 4×4 matrix with [0,0,0,1]. The completed 4×4 matrix is ​​the initial registration matrix.

[0046] S103, based on the initial registration matrix and preset projection parameters, the three-dimensional model, preset landmarks, intraoperative landmarks and intraoperative point cloud are projected onto a preset plane to generate and display the corresponding two-dimensional projection image; the two-dimensional projection image includes the projection outline of the three-dimensional model, each first projection point corresponding to each preset landmark, each second projection point corresponding to each intraoperative landmark and multiple third projection points corresponding to the intraoperative point cloud.

[0047] The preset projection parameters can be used to define the projection parameters required to project the 3D model, preset landmarks, intraoperative landmarks, and intraoperative point cloud onto a preset plane (e.g., the acetabular cup plane of the hip joint). For example, the preset projection parameters may include the 3D coordinates of the virtual projection source in the CT coordinate system and the orientation of the virtual projection source. It should be noted that the specific representation of the orientation of the virtual projection source can be found in descriptions in related technologies, and will not be detailed in this embodiment.

[0048] Optionally, the computer device can project the 3D model, preset landmarks, intraoperative landmarks, and intraoperative point cloud onto a preset plane through the following steps 1.1 to 1.2: Step 1.1: Using the inverse of the initial registration matrix, transform the 3D coordinates of the intraoperative landmarks and the 3D coordinates of the intraoperative point cloud to the coordinate system of the 3D model, and obtain the 3D coordinates of the intraoperative landmarks and the 3D coordinates of the intraoperative point cloud after coordinate transformation.

[0049] Specifically, the computer equipment can perform matrix multiplication operations on the three-dimensional coordinates of each intraoperative landmark with the inverse of the initial registration matrix to obtain the three-dimensional coordinates of each intraoperative landmark after coordinate transformation. The three-dimensional coordinates of the intraoperative landmark after coordinate transformation refer to the three-dimensional coordinates of the intraoperative landmark in the CT coordinate system.

[0050] Similarly, computer equipment can perform matrix multiplication between the 3D coordinates of the intraoperative point cloud and the inverse of the initial registration matrix to obtain the 3D coordinates of the intraoperative point cloud after coordinate transformation. The 3D coordinates of the intraoperative point cloud after coordinate transformation refer to the 3D coordinates of the intraoperative point cloud in the CT coordinate system.

[0051] Step 1.2: Based on preset projection parameters, project the 3D model, preset landmarks, intraoperative landmarks after coordinate transformation, and intraoperative point cloud after coordinate transformation onto the preset plane.

[0052] Optionally, the computer device can use a preset projection transformation method (e.g., orthogonal projection) to calculate the two-dimensional coordinates of the first projection point, second projection point, and third projection point corresponding to each preset landmark, each intraoperative landmark after coordinate transformation, and the intraoperative point cloud after coordinate transformation on a preset plane, based on preset projection parameters, the isosurface of the 3D model, the 3D coordinates of each preset landmark, each intraoperative landmark after coordinate transformation, and the intraoperative point cloud after coordinate transformation. It can also calculate the two-dimensional coordinates of each contour point included in the projection contour of the 3D model on the preset plane, thereby projecting the 3D model, preset landmarks, intraoperative landmarks after coordinate transformation, and intraoperative point cloud after coordinate transformation onto the preset plane.

[0053] Based on this, the computer device can draw the projection outline, first projection point, second projection point and third projection point on a preset plane according to the two-dimensional coordinates of each outline point, the two-dimensional coordinates of each first projection point, the two-dimensional coordinates of each second projection point and the two-dimensional coordinates of the midpoint cloud, thereby obtaining a two-dimensional projection image and displaying the two-dimensional projection image.

[0054] Optionally, to facilitate the differentiation of different types of projection points in a two-dimensional projection image, the first, second, and third projection points in the two-dimensional projection image can be different colors. For example, such as... Figure 3 As shown, Figure 3 In the diagram, 31 can be the projection outline, the red dot can be the first projection point 32, the green dot can be the second projection point 33, and the blue dot can be the third projection point 34.

[0055] Furthermore, to facilitate users in determining the overlap rate between the first projection point and the second projection point after the first projection point is dragged to the second projection point, the first projection point, the second projection point, and the third projection point in the two-dimensional projection image can all be semi-transparent dots of different colors.

[0056] S104, in response to the drag operation on the first projection point, update the display position of the first projection point in the two-dimensional projection image in real time, and update the three-dimensional coordinates of the preset landmark corresponding to the first projection point according to the preset projection parameters and the updated display position of the first projection point. Based on the updated three-dimensional coordinates of all preset landmarks and the three-dimensional coordinates of all intraoperative landmarks, calculate the target registration matrix between the three-dimensional model and the target entity object.

[0057] It should be understood that, in order to correct the positional deviation between the first and second projection points caused by inaccurate initial registration matrix, the user can drag the first projection points in the two-dimensional projection image to maximize the overlap between each first projection point and its corresponding second projection point. For example, as... Figure 4 As shown in (a), the yellow arrow indicates a drag operation on the first projection point 32. Since this embodiment restricts the drag operation to a two-dimensional plane, allowing users to drag the first projection point up and down or left and right only within the two-dimensional plane, it reduces the difficulty of manually correcting the initial registration matrix and improves the registration efficiency between the target entity and its three-dimensional model compared to related technologies that correct the initial registration matrix in six dimensions (i.e., three rotational degrees of freedom and three translational degrees of freedom).

[0058] Based on this, when a computer device detects a drag operation targeting any first projection point in a two-dimensional projected image, it can respond to the drag operation by updating the display positions of all first projection points in the two-dimensional projected image in real time. It should be understood that when any first projection point is dragged, the display positions of all first projection points will change to maintain the inherent geometry of the three-dimensional model. For example, as... Figure 4 As shown in (b), the updated display position of the first projection point roughly coincides with the display position of the second projection point.

[0059] Furthermore, the computer device can update the three-dimensional coordinates of the preset landmark corresponding to the first projection point based on preset projection parameters and the updated display position of the first projection point. For example, for any first projection point, the computer device can map the updated display position of the first projection point to a virtual ray emitted from the virtual projection point to the surface of the three-dimensional model based on preset projection parameters, and determine the three-dimensional coordinates of the intersection of the virtual ray and the three-dimensional model in the CT coordinate system as the updated three-dimensional coordinates of the preset landmark corresponding to the first projection point, thereby realizing the update of the preset landmark.

[0060] For example, such as Figure 5 As shown, if the two-dimensional projection image is 51, the original display position of a first projection point in the two-dimensional projection image 51 is 52, the intraoperative landmark corresponding to the first projection point on the three-dimensional model 53 is 54, and the updated display position of the first projection point after the drag operation represented by the yellow arrow in the figure is 55, and the virtual projection source is 56, then the computer device can map the updated display position 55 of the first projection point to a virtual ray 57 pointing from the virtual projection source 56 to the three-dimensional model 53 based on preset projection parameters. The three-dimensional coordinates of the intersection point 58 of the virtual ray 57 and the three-dimensional model 53 are the updated three-dimensional coordinates of the preset landmark corresponding to the first projection point. Optionally, computer equipment can be used through, for example Figure 6 S1041~S1045, shown below, calculates the target registration matrix between the 3D model and the target entity object, detailed as follows: S1041, based on the updated three-dimensional coordinates of all preset landmarks, determine the third centroid of all preset landmarks, and based on the third centroid, perform decentralization processing on the updated preset landmarks respectively.

[0061] The third centroid is the geometric center of all updated preset landmarks, and its three-dimensional coordinates in the CT coordinate system are... x The axis components are updated after all preset landmarks are updated. x The average value of the axial components, y The axis components are updated after all preset landmarks are updated. y The average value of the axial components, zThe axis components are updated after all preset landmarks are updated. z The average value of the axial components.

[0062] Optionally, the computer device can achieve decentralized processing of each updated preset landmark by subtracting the updated three-dimensional coordinates of each preset landmark from the three-dimensional coordinates of the third centroid.

[0063] S1042, based on the three-dimensional coordinates of all intraoperative landmarks, determine the second centroid of all intraoperative landmarks, and based on the second centroid, perform decentralization processing on each intraoperative landmark.

[0064] It should be noted that S1042 is exactly the same as S1022. Therefore, the specific implementation process of S1042 can be referred to the relevant description in S1022, and will not be repeated here.

[0065] S1043, based on the three-dimensional coordinates of all updated preset landmarks after decentralization and the three-dimensional coordinates of all intraoperative landmarks, calculate the second optimal rotation matrix between all updated preset landmarks and all intraoperative landmarks.

[0066] The second optimal rotation matrix is ​​the rotation matrix that minimizes the overall deviation between all updated preset landmarks after decentralization and all intraoperative landmarks after decentralization. The dimension of the second optimal rotation matrix can be 3×3. It should be noted that since the calculation process of the optimal rotation matrix for two three-dimensional coordinates in different coordinate systems is a known technique, the calculation process of the second optimal rotation matrix can be referred to the description in related technologies, and will not be described in detail in the embodiments of this application.

[0067] S1044, based on the three-dimensional coordinates of the third centroid, the three-dimensional coordinates of the second centroid, and the optimal rotation matrix, calculate the second translation vector between all preset landmarks and all intraoperative landmarks.

[0068] The second translation vector is defined as the amount of displacement required to maximize the spatial overlap between all the updated preset landmarks after rotation and all intraoperative landmarks. The dimension of the second translation vector can be 3×1.

[0069] Specifically, the computer device can calculate the product of the second optimal rotation matrix and the three-dimensional coordinates of the third centroid, and subtract the product from the three-dimensional coordinates of the second centroid to obtain the second translation vector.

[0070] S1045, construct the target registration matrix by combining the second optimal rotation matrix and the second translation vector.

[0071] The target registration matrix is ​​a 4×4 dimension standard homogeneous coordinate transformation matrix.

[0072] Specifically, the computer device can fill the upper left 3×3 region of an empty 4×4 matrix with the second optimal rotation matrix, fill the upper right 3×1 region of the 4×4 matrix with the second translation vector, and fill the last row of the 4×4 matrix with [0,0,0,1]. The completed 4×4 matrix is ​​the target registration matrix.

[0073] S105, based on the target registration matrix, register the 3D model with the intraoperative point cloud, and update the display position of all projection points in the 2D projection image.

[0074] Optionally, the computer device can use the target registration matrix as the initial value of the rigid transformation of the iterative closest point (ICP) algorithm, and register the 3D model with the intraoperative point cloud using the ICP algorithm to obtain the final registration matrix and the corresponding registration quality parameters. When the registration quality parameters meet the preset requirements, the display positions of the first projection point, the second projection point, and the third projection point in the 2D projection image are updated based on the final registration matrix. It should be noted that since the ICP algorithm is a known technology, the specific process of registering the 3D model with the intraoperative point cloud using the ICP algorithm can be found in the descriptions in related technologies, and will not be detailed in the embodiments of this application.

[0075] Registration quality parameters can be used to measure the registration quality between the 3D model and the intraoperative point cloud. The specific types of registration quality parameters can be set according to actual needs. For example, registration quality parameters can include any one or more of the following: root mean square error, inlier ratio, and overlap rate. Based on this, preset requirements can also be set according to actual needs. For example, preset requirements can include any one or more of the following: root mean square error cannot be less than a preset error threshold, inlier ratio relative to a preset ratio threshold, and overlap rate greater than a preset overlap rate.

[0076] For example, the final two-dimensional projected image obtained after updating the display positions of the first projection point, the second projection point, and the third projection point in the two-dimensional projected image can be as follows: Figure 7 As shown. Relative to Figure 3 The original two-dimensional projection image shown, Figure 7 All projection points in the projected image shown are within the projection outline.

[0077] Furthermore, related techniques typically perform registration by directly searching the 3D model for pre-defined landmarks that perfectly correspond to the intraoperative landmarks. However, this registration method has multiple solutions, especially when the number of landmarks is small. It can find an infinite number of sets of results that satisfy the constraints in the 3D model, such as... Figure 8 As shown by the dashed line in (a) above, this results in the above search having exponential complexity. And as... Figure 8As shown in (b) of the paper, this application can avoid global search, reduce registration efficiency, and improve registration efficiency by combining two-dimensional drag and local constraints.

[0078] As can be seen, the registration method between the 3D model and the entity object provided in this embodiment calculates an initial registration matrix between the target entity object and its 3D model. Based on this initial registration matrix, the 3D model, preset landmarks on the 3D model, intraoperative landmarks on the target entity object, and intraoperative point cloud are projected onto a preset plane. This allows users to update the display position of the first projection point in the 2D projection image by dragging the first projection point corresponding to the preset landmark up, down, left, and right, thereby achieving accurate correction of the initial registration matrix. By registering the 3D model and the intraoperative point cloud based on the corrected target registration matrix and updating the display position of the projection point in the 2D projection image, accurate registration between the target entity object and the 3D model can be achieved. In other words, this method can improve the registration accuracy and efficiency between the target entity object and its 3D model while reducing the difficulty of manually correcting the initial registration matrix.

[0079] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0080] Based on the registration method for three-dimensional models and entity objects provided in the above embodiments, this application further provides an embodiment of a computer device that implements the above method embodiments. Figure 9 This is a schematic diagram of a computer device provided in an embodiment of this application. For ease of explanation, only the parts relevant to this embodiment are shown. Figure 9 As shown, the computer device 90 may include: a first acquisition unit 901, a first calculation unit 902, a virtual projection unit 903, a first processing unit 904, and a second processing unit 905. Wherein: The first acquisition unit 901 is used to acquire the three-dimensional model of the target entity object, the three-dimensional coordinates of each preset landmark on the three-dimensional model, the three-dimensional coordinates of each intraoperative landmark on the target entity object corresponding to each preset landmark, and the three-dimensional coordinates of the intraoperative point cloud on the target entity object.

[0081] The first calculation unit 902 is used to calculate the initial registration matrix between the three-dimensional model and the target entity object based on the three-dimensional coordinates of all the preset landmarks and the three-dimensional coordinates of all the intraoperative landmarks.

[0082] The virtual projection unit 903 is used to project the three-dimensional model, the preset landmarks, the intraoperative landmarks, and the intraoperative point cloud onto a preset plane based on the initial registration matrix and preset projection parameters, so as to generate and display the corresponding two-dimensional projection image; the two-dimensional projection image includes the projection outline of the three-dimensional model, each first projection point corresponding to each preset landmark, each second projection point corresponding to each intraoperative landmark, and multiple third projection points corresponding to the intraoperative point cloud.

[0083] The first processing unit 904 is configured to respond to a drag operation on the first projection point, update the display position of the first projection point in the two-dimensional projection image in real time, update the three-dimensional coordinates of the preset landmark corresponding to the first projection point according to the preset projection parameters and the updated display position of the first projection point, and calculate the target registration matrix between the three-dimensional model and the target entity object according to the updated three-dimensional coordinates of all preset landmarks and the three-dimensional coordinates of all intraoperative landmarks.

[0084] The second processing unit 905 is used to register the three-dimensional model with the intraoperative point cloud based on the target registration matrix, and update the display position of all projection points in the two-dimensional projection image.

[0085] Optionally, the first computing unit 902 is specifically used for: Based on the three-dimensional coordinates of all the preset landmarks, determine the first centroid of all the preset landmarks, and based on the first centroid, perform decentralization processing on each of the preset landmarks respectively; Based on the three-dimensional coordinates of all the intraoperative landmarks, the second centroid of all the intraoperative landmarks is determined, and based on the second centroid, each of the intraoperative landmarks is decentralized. Based on the three-dimensional coordinates of all the preset landmarks after decentralization and the three-dimensional coordinates of all the intraoperative landmarks, calculate the first optimal rotation matrix between all the preset landmarks and all the intraoperative landmarks; Based on the three-dimensional coordinates of the first centroid, the three-dimensional coordinates of the second centroid, and the first optimal rotation matrix, calculate the first translation vector between all the preset landmarks and all the intraoperative landmarks; The first optimal rotation matrix and the first translation vector are used to construct the initial registration matrix.

[0086] Optionally, the virtual projection unit 903 is specifically used for: The three-dimensional coordinates of the intraoperative landmark and the three-dimensional coordinates of the intraoperative point cloud are transformed to the coordinate system of the three-dimensional model by using the inverse matrix of the initial registration matrix, so as to obtain the three-dimensional coordinates of the intraoperative landmark and the three-dimensional coordinates of the intraoperative point cloud after coordinate transformation. Based on preset projection parameters, the 3D model, the preset landmarks, the intraoperative landmarks after coordinate transformation, and the intraoperative point cloud after coordinate transformation are projected onto a preset plane.

[0087] Optionally, the first projection point, the second projection point, and the third projection point in the two-dimensional projection image are each a semi-transparent dot of a different color.

[0088] Optionally, the first processing unit 904 is specifically used for: Based on the updated 3D coordinates of all the preset landmarks, determine the third centroid of all the preset landmarks, and based on the third centroid, decentralize the updated preset landmarks respectively. Based on the three-dimensional coordinates of all the intraoperative landmarks, the second centroid of all the intraoperative landmarks is determined, and based on the second centroid, each of the intraoperative landmarks is decentralized. Based on the three-dimensional coordinates of all the updated preset landmarks after decentralization and the three-dimensional coordinates of all intraoperative landmarks, calculate the second optimal rotation matrix between all the updated preset landmarks and all the intraoperative landmarks; Based on the three-dimensional coordinates of the third centroid, the three-dimensional coordinates of the second centroid, and the optimal rotation matrix, calculate the second translation vector between all the preset landmarks and all the intraoperative landmarks; The second optimal rotation matrix and the second translation vector are used to construct the target registration matrix.

[0089] Optionally, the second processing unit 905 is specifically used for: The target registration matrix is ​​used as the initial value of the rigid transformation of the iterative nearest point algorithm, and the three-dimensional model and the intraoperative point cloud are registered by the iterative nearest point algorithm to obtain the final registration matrix and the corresponding registration quality parameters. If the registration quality parameters meet the preset requirements, then based on the final registration matrix, the display positions of the first projection point, the second projection point, and the third projection point in the two-dimensional projection image are updated again.

[0090] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units is used as an example. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the computer device can be divided into different functional units to complete all or part of the functions described above. The functional units in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of each unit in the above-described computer device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0091] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a computer device provided for another embodiment of this application. For example... Figure 10 As shown, the computer device 11 provided in this embodiment may include: a processor 110, a memory 111, and a computer program 112 stored in the memory 111 and executable on the processor 110, such as a program corresponding to a registration method between a 3D model and a solid object. When the processor 110 executes the computer program 112, it implements the steps in the above-described embodiment of the registration method between a 3D model and a solid object, for example... Figure 1 S101~S105 are shown. Alternatively, when the processor 110 executes the computer program 112, it implements the functions of each unit in the above-described computer device embodiment.

[0092] For example, computer program 112 can be divided into one or more modules / units, one or more of which are stored in memory 111 and executed by processor 110 to complete this application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of computer program 112 in computer device 11. For example, computer program 112 can be divided into a first acquisition unit, a first calculation unit, a virtual rendering unit, a first processing unit, and a second processing unit. For the specific functions of each unit, please refer to [link to relevant documentation]. Figure 9 The relevant descriptions in the corresponding embodiments are not repeated here.

[0093] Those skilled in the art will understand that Figure 10 This is merely an example of computer device 11 and does not constitute a limitation on computer device 11. It may include more or fewer components than shown, or combine certain components, or use different components.

[0094] The processor 110 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0095] The memory 111 can be an internal storage unit of the computer device 11, such as a hard disk or RAM of the computer device 11. The memory 111 can also be an external storage device of the computer device 11, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, or flash card equipped on the computer device 11. Furthermore, the memory 111 can include both internal and external storage units of the computer device 11. The memory 111 is used to store computer programs and other programs and data required by the computer device. The memory 111 can also be used to temporarily store data that has been output or will be output.

[0096] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the various steps in the registration method between a 3D model and an entity object described in the above method embodiments.

[0097] This application provides a computer program product that, when run on a computer device, enables the computer device to perform the steps described in the various method embodiments above.

[0098] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, refer to the relevant descriptions of other embodiments.

[0099] It should be noted that, unless otherwise specified, all technical terms used in the embodiments of this application have the same meaning as commonly understood by those skilled in the art to which this application belongs. The technical terms used in the embodiments of this application are only used to explain specific embodiments of this application and are not intended to limit this application.

[0100] The term "embodiment" as used in the description of embodiments in this application means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various display locations throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0101] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0102] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for registering a 3D model with a solid object, characterized in that, include: Obtain a 3D model of the target entity object, the 3D coordinates of each preset landmark on the 3D model, the 3D coordinates of each intraoperative landmark on the target entity object corresponding to each preset landmark, and the 3D coordinates of the intraoperative point cloud on the target entity object; Based on the three-dimensional coordinates of all the preset landmarks and the three-dimensional coordinates of all the intraoperative landmarks, calculate the initial registration matrix between the three-dimensional model and the target entity object; Based on the initial registration matrix and preset projection parameters, the 3D model, the preset landmarks, the intraoperative landmarks, and the intraoperative point cloud are projected onto a preset plane to generate and display a corresponding 2D projection image. The 2D projection image includes the projection outline of the 3D model, each first projection point corresponding to each preset landmark, each second projection point corresponding to each intraoperative landmark, and multiple third projection points corresponding to the intraoperative point cloud. In response to a drag operation on the first projection point, the display position of the first projection point in the two-dimensional projection image is updated in real time. Based on the preset projection parameters and the updated display position of the first projection point, the three-dimensional coordinates of the preset landmark corresponding to the first projection point are updated. Based on the updated three-dimensional coordinates of all preset landmarks and the three-dimensional coordinates of all intraoperative landmarks, the target registration matrix between the three-dimensional model and the target entity object is calculated. The 3D model is registered with the intraoperative point cloud based on the target registration matrix, and the display positions of all projection points in the 2D projection image are updated.

2. The method according to claim 1, characterized in that, Based on the three-dimensional coordinates of all the preset landmarks and the three-dimensional coordinates of all the intraoperative landmarks, calculate the initial registration matrix between the three-dimensional model and the target entity object, including: Based on the three-dimensional coordinates of all the preset landmarks, determine the first centroid of all the preset landmarks, and based on the first centroid, perform decentralization processing on each of the preset landmarks respectively; Based on the three-dimensional coordinates of all the intraoperative landmarks, the second centroid of all the intraoperative landmarks is determined, and based on the second centroid, each of the intraoperative landmarks is decentralized. Based on the three-dimensional coordinates of all the preset landmarks after decentralization and the three-dimensional coordinates of all the intraoperative landmarks, calculate the first optimal rotation matrix between all the preset landmarks and all the intraoperative landmarks; Based on the three-dimensional coordinates of the first centroid, the three-dimensional coordinates of the second centroid, and the first optimal rotation matrix, calculate the first translation vector between all the preset landmarks and all the intraoperative landmarks; The first optimal rotation matrix and the first translation vector are used to construct the initial registration matrix.

3. The method according to claim 1, characterized in that, Based on the initial registration matrix and preset projection parameters, the 3D model, the preset landmarks, the intraoperative landmarks, and the intraoperative point cloud are projected onto a preset plane, including: The three-dimensional coordinates of the intraoperative landmark and the three-dimensional coordinates of the intraoperative point cloud are transformed to the coordinate system of the three-dimensional model by using the inverse matrix of the initial registration matrix, so as to obtain the three-dimensional coordinates of the intraoperative landmark and the three-dimensional coordinates of the intraoperative point cloud after coordinate transformation. Based on preset projection parameters, the 3D model, the preset landmarks, the intraoperative landmarks after coordinate transformation, and the intraoperative point cloud after coordinate transformation are projected onto a preset plane.

4. The method according to claim 1, characterized in that, The first projection point, the second projection point, and the third projection point in the two-dimensional projection image are all semi-transparent dots of different colors.

5. The method according to claim 1, characterized in that, Based on the updated 3D coordinates of all the preset landmarks and the 3D coordinates of all the intraoperative landmarks, calculate the target registration matrix between the 3D model and the target entity object, including: Based on the updated three-dimensional coordinates of all the preset landmarks, determine the third centroid of all the preset landmarks, and based on the third centroid, perform decentralization processing on the updated preset landmarks respectively. Based on the three-dimensional coordinates of all the intraoperative landmarks, the second centroid of all the intraoperative landmarks is determined, and based on the second centroid, each of the intraoperative landmarks is decentralized. Based on the three-dimensional coordinates of all the updated preset landmarks after decentralization and the three-dimensional coordinates of all intraoperative landmarks, calculate the second optimal rotation matrix between all the updated preset landmarks and all the intraoperative landmarks; Based on the three-dimensional coordinates of the third centroid, the three-dimensional coordinates of the second centroid, and the optimal rotation matrix, calculate the second translation vector between all the preset landmarks and all the intraoperative landmarks; The second optimal rotation matrix and the second translation vector are used to construct the target registration matrix.

6. The method according to any one of claims 1-5, characterized in that, Based on the target registration matrix, the 3D model is registered with the intraoperative point cloud, and the display positions of all projection points in the 2D projection image are updated, including: The target registration matrix is ​​used as the initial value of the rigid transformation of the iterative nearest point algorithm, and the three-dimensional model and the intraoperative point cloud are registered by the iterative nearest point algorithm to obtain the final registration matrix and the corresponding registration quality parameters. If the registration quality parameters meet the preset requirements, then based on the final registration matrix, the display positions of the first projection point, the second projection point, and the third projection point in the two-dimensional projection image are updated again.

7. A computer device, characterized in that, include: The first acquisition unit is used to acquire the three-dimensional model of the target entity object, the three-dimensional coordinates of each preset landmark on the three-dimensional model, the three-dimensional coordinates of each intraoperative landmark on the target entity object corresponding to each preset landmark, and the three-dimensional coordinates of the intraoperative point cloud on the target entity object. The first calculation unit is used to calculate the initial registration matrix between the three-dimensional model and the target entity object based on the three-dimensional coordinates of all the preset landmarks and the three-dimensional coordinates of all the intraoperative landmarks. A virtual projection unit is used to project the three-dimensional model, the preset landmarks, the intraoperative landmarks, and the intraoperative point cloud onto a preset plane based on the initial registration matrix and preset projection parameters, so as to generate and display the corresponding two-dimensional projection image; the two-dimensional projection image includes the projection outline of the three-dimensional model, each first projection point corresponding to each preset landmark, each second projection point corresponding to each intraoperative landmark, and multiple third projection points corresponding to the intraoperative point cloud; The first processing unit is configured to respond to a drag operation on the first projection point by updating the display position of the first projection point in the two-dimensional projection image in real time, and update the three-dimensional coordinates of the preset landmark corresponding to the first projection point according to the preset projection parameters and the updated display position of the first projection point, and calculate the target registration matrix between the three-dimensional model and the target entity object according to the updated three-dimensional coordinates of all the preset landmarks and the three-dimensional coordinates of all the intraoperative landmarks. The second processing unit is used to register the three-dimensional model with the intraoperative point cloud based on the target registration matrix, and update the display position of all projection points in the two-dimensional projection image.

8. A computer device, characterized in that, It includes a memory and a computer program stored in the memory and executable on a processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, When the computer program product is run on a computer device, the computer device causes the computer device to perform the method as described in any one of claims 1-6.

Images