Structured light image registration method, device, equipment, medium and program product

Abstract

The invention provides a structured light image registration method, device and equipment, a medium and a program product, which can be applied to the technical field of medical instruments. The structured light image registration method comprises the steps that a reference three-dimensional image and an initial three-dimensional image are obtained, and the initial three-dimensional image is a three-dimensional image obtained by irradiating intraoperative structured light on an intraoperative object; cutting the initial three-dimensional image to obtain a target three-dimensional image; performing coarse registration on the target three-dimensional image and the reference three-dimensional image to obtain a first coordinate system conversion relation between the target three-dimensional image and the reference three-dimensional image; based on the first coordinate system conversion relation, obtaining a coarse registration three-dimensional image; performing fine registration on the coarse registration three-dimensional image and the reference three-dimensional image to obtain a second coordinate system conversion relation between the coarse registration three-dimensional image and the reference three-dimensional image; and based on the second coordinate system conversion relation, obtaining a precisely registered three-dimensional image.

Description

Technical Field

[0001] The present disclosure relates to the field of medical device technology, and more specifically to a structured light image registration method, device, equipment, medium, and program product. Background Art

[0002] In computer-assisted high tibial osteotomy surgery, a paired point registration-based scheme is adopted. The patient's CT image is obtained before the operation, and anatomical landmarks are predefined. During the operation, medical staff use a handheld probe to select the corresponding position on the real bone surface according to the predefined landmarks, and finally obtain the mapping relationship between the spatial coordinates of the CT model and the real bone surface.

[0003] However, existing registration schemes still have several technical limitations in practical applications, such as low registration accuracy and low registration efficiency. Summary of the Invention

[0004] In view of the above problems, the present disclosure provides a structured light image registration method, apparatus, device, medium and program product for improving registration accuracy and efficiency.

[0005] According to a first aspect of the present disclosure, a structured light image registration method is provided, comprising: acquiring a reference 3D image and an initial 3D image, wherein the initial 3D image is a 3D image obtained by irradiating an intraoperative object with structured light during surgery; cropping the initial 3D image to obtain a target 3D image; coarsely registering the target 3D image with the reference 3D image to obtain a first coordinate system transformation relationship between the target 3D image and the reference 3D image; based on the first coordinate system transformation relationship, transforming a source point cloud in the reference 3D image into a coordinate system of the target 3D image to obtain a coarsely registered 3D image; finely registering the coarsely registered 3D image with the reference 3D image to obtain a second coordinate system transformation relationship between the coarsely registered 3D image and the reference 3D image; and based on the second coordinate system transformation relationship, transforming a source point cloud in the reference 3D image into a coordinate system of the coarsely registered 3D image to obtain a finely registered 3D image.

[0006] According to an embodiment of the present disclosure, the above-mentioned target three-dimensional image and the above-mentioned reference three-dimensional image are roughly aligned to obtain a first coordinate system transformation relationship between the above-mentioned target three-dimensional image and the above-mentioned reference three-dimensional image, including: obtaining at least three pairs of feature points, each pair of feature points including a reference feature point and a target feature point; wherein the above-mentioned reference feature points are obtained from the above-mentioned reference three-dimensional image, the above-mentioned target feature points are obtained from the above-mentioned target three-dimensional image, and the positions of the reference feature points and the target feature points of each pair of feature points in the above-mentioned reference three-dimensional image and the above-mentioned target three-dimensional image are corresponding; based on the three-dimensional coordinate vectors of the above-mentioned at least three reference feature points in the above-mentioned reference three-dimensional image and the three-dimensional coordinate vectors of the above-mentioned at least three target feature points in the above-mentioned target three-dimensional image, calculating the first center of mass coordinate vectors of the at least three reference feature points and the second center of mass coordinate vectors of the at least three target feature points; based on the above-mentioned first center of mass coordinate vector and the above-mentioned second center of mass coordinate vector, determining a first rotation matrix and a first translation matrix; based on the above-mentioned first rotation matrix and the above-mentioned first translation matrix, determining the above-mentioned first coordinate system transformation relationship.

[0007] According to an embodiment of the present disclosure, the above-mentioned first rotation matrix and the first translation matrix are determined based on the above-mentioned first center-of-mass coordinate vector and the above-mentioned second center-of-mass coordinate vector, including: calculating the covariance matrix based on the above-mentioned first center-of-mass coordinate vector and the above-mentioned second center-of-mass coordinate vector; performing singular value decomposition on the above-mentioned covariance matrix to obtain the above-mentioned first rotation matrix; and obtaining the above-mentioned first translation matrix based on the above-mentioned first rotation matrix and the three-dimensional coordinate vectors of a pair of feature points.

[0008] According to an embodiment of the present disclosure, the above-mentioned coarsely registered three-dimensional image and the above-mentioned reference three-dimensional image are finely registered to obtain a second coordinate system transformation relationship between the above-mentioned coarsely registered three-dimensional image and the above-mentioned reference three-dimensional image, including: downsampling the target point cloud of the above-mentioned coarsely registered three-dimensional image and the source point cloud of the above-mentioned reference three-dimensional image; using a registration algorithm, performing three-dimensional point cloud registration on the downsampled target point cloud and the downsampled source point cloud to obtain a second rotation matrix and a second translation matrix; based on the above-mentioned second rotation matrix and the above-mentioned second translation matrix, the above-mentioned second coordinate system transformation relationship is obtained.

[0009] According to an embodiment of the present disclosure, the above-mentioned acquisition of the reference three-dimensional image and the initial three-dimensional image includes: acquiring a first distorted image and a second distorted image respectively obtained by two structured light cameras, wherein the two structured light cameras are respectively arranged on both sides of a projector that projects the intraoperative structured light; and determining the above-mentioned initial three-dimensional image based on the intraoperative structured light, the first distorted image and the second distorted image.

[0010] According to an embodiment of the present disclosure, the above-mentioned initial three-dimensional image is determined based on the above-mentioned intraoperative structured light, the above-mentioned first distorted image and the above-mentioned second distorted image, including: calculating the first distortion information between the above-mentioned first distorted image and the above-mentioned intraoperative structured light; calculating the second distortion information between the above-mentioned second distorted image and the above-mentioned intraoperative structured light; and determining the above-mentioned initial three-dimensional image based on the above-mentioned first distortion information and the above-mentioned second distortion information according to binocular stereo vision technology.

[0011] According to a second aspect of the present disclosure, a structured light image registration device is provided, comprising: an acquisition module for acquiring a reference three-dimensional image and an initial three-dimensional image; wherein the initial three-dimensional image is a three-dimensional image obtained by irradiating an intraoperative object with structured light during surgery; a cropping module for cropping the initial three-dimensional image to obtain a target three-dimensional image; a first registration module for coarsely registering the target three-dimensional image with the reference three-dimensional image to obtain a first coordinate system transformation relationship between the target three-dimensional image and the reference three-dimensional image; a first conversion module for converting a source point cloud in the reference three-dimensional image into a coordinate system of the target three-dimensional image based on the first coordinate system transformation relationship to obtain a coarsely registered three-dimensional image; a second registration module for finely registering the coarsely registered three-dimensional image with the reference three-dimensional image to obtain a second coordinate system transformation relationship between the coarsely registered three-dimensional image and the reference three-dimensional image; and a second conversion module for converting a source point cloud in the reference three-dimensional image into a coordinate system of the coarsely registered three-dimensional image based on the second coordinate system transformation relationship to obtain a finely registered three-dimensional image.

[0012] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0013] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0014] The fifth aspect of the present disclosure further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0015] According to the embodiments of the present disclosure, structured light is used to acquire three-dimensional information of the object during surgery, resulting in an initial three-dimensional image composed of a three-dimensional point cloud. This allows for the rapid acquisition of a large number of precise points that can be used for registration. By cropping the target three-dimensional image of the area to be operated on, most interference information is removed, registration noise is reduced, registration efficiency is improved, and surgical time is shortened. The coarse and fine registration operations improve registration accuracy, providing more reliable three-dimensional spatial information, allowing the surgeon to more accurately identify and locate the surgical site, more accurately plan the angle and position of the osteotomy, ensure that the surgical operation complies with the preoperative plan, further enhance the surgical effect, and improve the success rate and safety of the surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0017] Figure 1 A diagram schematically illustrates an application scenario of the structured light image registration method, apparatus, device, medium, and program product according to an embodiment of the present disclosure;

[0018] Figure 2 The following schematically shows a flow chart of a structured light image registration method according to an embodiment of the present disclosure;

[0019] FIG3(A) schematically shows a flow chart of acquiring a reference 3D image and an initial 3D image according to an embodiment of the present disclosure;

[0020] FIG3(B) schematically shows a flow chart of coarse registration and fine registration according to an embodiment of the present disclosure;

[0021] Figure 4 Schematically shows a structural block diagram of a structured light image registration device according to an embodiment of the present disclosure; and

[0022] Figure 5 A block diagram of an electronic device suitable for implementing a structured light image registration method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0024] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0027] High tibial osteotomy (HTO) precisely resects the proximal tibial fragment and adjusts coronal plane alignment, shifting knee joint load from the medial lesion to the relatively normal lateral compartment. Based on the principle of biomechanical compensation, high tibial osteotomy utilizes wedge-shaped osteotomy combined with locking plate fixation to effectively delay the progression of end-stage osteoarthritis, preserve proprioception, and maintain secondary arthroplasty. Its advantages lie in its adaptability to young, active patients. Compared with total knee arthroplasty, it maintains motion function and reduces the risk of prosthesis revision. Postoperative progressive weight-bearing training can achieve early functional recovery. However, traditional high tibial osteotomy relies on two-dimensional radiographic projections to assess lower limb alignment. Mechanical axis positioning is susceptible to three-dimensional projection deviations caused by knee flexion contracture, resulting in sagittal and coronal errors in osteotomy angle correction. Furthermore, frequent fluoroscopic procedures expose the surgeon to a cumulative radiation dose exceeding 1.7 times that of standard orthopedic surgery, further limiting the accuracy and safety of biomechanical reconstruction.

[0028] In computer-assisted high tibial osteotomy surgery, registration technology is used to achieve precise mapping of the three-dimensional model to the real bone space coordinates. Preoperative planning parameters (including osteotomy angle, depth, and correction amount) are accurately converted to intraoperative navigation benchmarks through point cloud matching or image feature alignment. Computer-assisted high tibial osteotomy surgery overcomes the geometric deviation problems caused by traditional high tibial osteotomy surgery due to reliance on visual inspection or template positioning. It ensures millimeter-level accuracy in intraoperative osteotomy plane and force line adjustment, while simultaneously compensating for dynamic deformation of the surgical field in real time. This improves the reliability of open wedge osteotomy opening angle control and locking plate posture matching, providing closed-loop verification for personalized biomechanical reconstruction.

[0029] Computer-assisted high tibial osteotomy surgery typically uses a paired point registration-based approach. The standard process involves obtaining a CT scan of the patient and predefining anatomical landmarks before surgery. During surgery, a probe is used to select corresponding landmarks on the real bone surface, thereby establishing a spatial coordinate mapping relationship between the CT model and the real bone. However, this paired point registration-based approach has the following technical limitations:

[0030] Low registration accuracy: The surgeon needs to manually select more than ten sets of paired points, relying on limited characteristic anatomical points, resulting in insufficient spatial information; the positioning error of the marker points will be directly transmitted to the registration result.

[0031] Low registration efficiency: Increasing the number of paired points collected to increase the amount of information will lead to a nonlinear increase in time cost; the surgeon needs to repeatedly confirm the matching degree of the point cloud, which prolongs the critical path time of the surgery; the positioning accuracy of the marker points has a high operation learning curve, which significantly prolongs the surgery time.

[0032] Stability defects: The scheme based on paired point registration is easily disturbed by the intraoperative environment. The displacement of the surgical drape or the obstruction of metal instruments can cause the optical tracking to be interrupted, and the registration process needs to be repeated.

[0033] The embodiments of the present disclosure provide a structured light image registration method, apparatus, device, medium, and program product, in order to solve at least one of the above technical problems.

[0034] Figure 1 The application scenario diagram of the structured light image registration method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown.

[0035] like Figure 1As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or optical fiber cables.

[0036] A user may use a first terminal device 101, a second terminal device 102, or a third terminal device 103 to interact with a server 105 via a network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, or the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (for example only).

[0037] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0038] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal devices.

[0039] It should be noted that the structured light image registration method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the structured light image registration device provided in the embodiment of the present disclosure can generally be set in the server 105. The structured light image registration method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the structured light image registration device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0040] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0041] Figure 2 The flowchart of the structured light image registration method according to an embodiment of the present disclosure is schematically shown.

[0042] like Figure 2 As shown, the structured light image registration method of this embodiment includes operations S210 to S260.

[0043] In operation S210 , a reference three-dimensional image and an initial three-dimensional image are acquired, wherein the initial three-dimensional image is a three-dimensional image obtained by irradiating an intraoperative object with intraoperative structured light.

[0044] According to an embodiment of the present disclosure, the reference three-dimensional image can be a simulated image obtained by scanning the patient before surgery. In one embodiment, the reference three-dimensional image can be a CT three-dimensional image. Before surgery, a CT scan is used to scan the patient's tibia in the position that requires correction. The obtained CT three-dimensional image can more clearly display the three-dimensional information of the tibia. Based on the CT three-dimensional image, the site of the patient's tibia to be operated on can be preliminarily calculated, and the position of the osteotomy surface and the distraction angle can be designed. This result can be used as one of the target values ​​for alignment during surgery. The process is essentially the process of preoperative planning for the doctor.

[0045] According to an embodiment of the present disclosure, by irradiating intraoperative structured light on an intraoperative object, an initial three-dimensional image composed of a three-dimensional point cloud can be obtained, and three-dimensional information of the surface of the intraoperative object can be acquired.

[0046] In operation S220 , the initial 3D image is cropped to obtain a target 3D image.

[0047] According to an embodiment of the present disclosure, an interactive point cloud cropping method can be used to crop the initial three-dimensional image, retaining the three-dimensional point cloud of the part of the intraoperative object where surgery is to be performed, and obtaining a target three-dimensional image.

[0048] In operation S230 , the target three-dimensional image and the reference three-dimensional image are coarsely registered to obtain a first coordinate system transformation relationship between the target three-dimensional image and the reference three-dimensional image.

[0049] In operation S240 , based on the first coordinate system conversion relationship, the source point cloud in the reference 3D image is converted into the coordinate system of the target 3D image to obtain a coarsely registered 3D image.

[0050] In operation S250 , the coarsely registered 3D image and the reference 3D image are finely registered to obtain a second coordinate system transformation relationship between the coarsely registered 3D image and the reference 3D image.

[0051] In operation S260 , based on the second coordinate system conversion relationship, the source point cloud in the reference 3D image is converted into the coordinate system of the coarsely registered 3D image to obtain a finely registered 3D image.

[0052] According to the embodiments of the present disclosure, structured light is used to acquire three-dimensional information of the object during surgery, resulting in an initial three-dimensional image composed of a three-dimensional point cloud. This allows for the rapid acquisition of a large number of precise points that can be used for registration. By cropping the target three-dimensional image of the area to be operated on, most interference information is removed, registration noise is reduced, registration efficiency is improved, and surgical time is shortened. The coarse and fine registration operations improve registration accuracy, providing more reliable three-dimensional spatial information, allowing the surgeon to more accurately identify and locate the surgical site, more accurately plan the angle and position of the osteotomy, ensure that the surgical operation complies with the preoperative plan, further enhance the surgical effect, and improve the success rate and safety of the surgery.

[0053] According to an embodiment of the present disclosure, a first distorted image and a second distorted image are obtained respectively by two structured light cameras, wherein the two structured light cameras are respectively arranged on either side of a projector that projects intraoperative structured light; an initial three-dimensional image is determined based on the intraoperative structured light, the first distorted image, and the second distorted image.

[0054] According to an embodiment of the present disclosure, the intraoperative structured light can be structured light with black and white stripes, and the structured light with black and white stripes is transmitted on the intraoperative object. For example, the surface of the tibia exposed during the operation presents distorted black and white stripes on the surface of the tibia, and two structured light cameras respectively obtain a first distorted image and a second distorted image of the surface of the tibia.

[0055] According to the embodiments of the present disclosure, the use of structured light instantaneous imaging can achieve rapid acquisition of three-dimensional information of the object during surgery, greatly shortening the registration time. If the registration process is interrupted, only 2-3 seconds of supplementary scanning is required to complete the data update, which can effectively solve the occlusion problem and will not cause a significant impact on the surgical process.

[0056] According to an embodiment of the present disclosure, first distortion information between a first distorted image and intraoperative structured light is calculated; second distortion information between a second distorted image and intraoperative structured light is calculated; and an initial three-dimensional image is determined based on the first distortion information and the second distortion information according to binocular stereo vision technology.

[0057] According to the embodiments of the present disclosure, binocular stereo vision technology, based on the principle of parallax, can determine an initial 3D image based on the positional relationship between two structured light cameras and first and second distortion information, thereby obtaining a 3D point cloud of the intraoperative object with 3D geometric information. A 3D reconstruction algorithm based on phase shift analysis can also be used to determine the initial 3D image.

[0058] According to the embodiments of the present disclosure, binocular stereo vision technology is used to perform three-dimensional reconstruction of the surface of an intraoperative object, generating a digital model with a point cloud density of millions. Because a three-dimensional point cloud is a collection of massive points that are consistent with the surface characteristics of the intraoperative object, and is derived using the principles of structured light photogrammetry, typically including three-dimensional coordinate information, the initial three-dimensional image can have a greater number of feature points and feature point information, enabling the automatic identification of anatomical landmarks and surface features, replacing traditional manual marking methods, eliminating subjective sources of operator error, lowering the operational threshold, and improving registration accuracy.

[0059] In one embodiment, an interactive point cloud cropping method can be used to crop the initial 3D image. The operator can manually select the edge points of a pre-reserved point cloud region on the display using a mouse, drawing a polygonal region. The point cloud cropping method then retains only the 3D point cloud within the polygonal region, resulting in the cropped target 3D image.

[0060] According to the embodiment of the present disclosure, the target 3D image is registered with the reference 3D image, and the conversion relationship between the coordinate systems of the target 3D image and the reference 3D image is calculated so that the target 3D image and the reference 3D image are aligned to a unified coordinate system. The source point cloud of the reference 3D image can be used To represent, the 3D point cloud of the target 3D image can be represented by To express.

[0061] According to an embodiment of the present disclosure, at least three pairs of feature points are obtained, each pair of feature points including a reference feature point and a target feature point; wherein the reference feature points are obtained from a reference three-dimensional image, and the target feature points are obtained from a target three-dimensional image, and the positions of the reference feature points and the target feature points of each pair of feature points in the reference three-dimensional image and the target three-dimensional image correspond to each other. Based on the three-dimensional coordinate vectors of the at least three reference feature points in the reference three-dimensional image and the three-dimensional coordinate vectors of the at least three target feature points in the target three-dimensional image, the first center of mass coordinate vectors of the at least three reference feature points and the second center of mass coordinate vectors of the at least three target feature points are calculated. Based on the first center of mass coordinate vector and the second center of mass coordinate vector, a first rotation matrix and a first translation matrix are determined. Based on the first rotation matrix and the first translation matrix, a first coordinate system transformation relationship is determined.

[0062] According to an embodiment of the present disclosure, a covariance matrix is ​​calculated based on the first centroid coordinate vector and the second centroid coordinate vector. A singular value decomposition is performed on the covariance matrix to obtain a first rotation matrix. A first translation matrix is ​​obtained based on the first rotation matrix and the three-dimensional coordinate vectors of a pair of feature points.

[0063] In one embodiment, an interactive coarse registration method can be used, in which the operator selects the source point cloud of the reference 3D image. 3D point cloud with target 3D image In the example, three pairs of feature points are selected. The three pairs of feature points are the corresponding three reference feature points. and three target feature points ,in, . Calculate and obtain three benchmark feature points The first centroid coordinate vector , three target feature points The second centroid coordinate vector .

[0064] Based on the first centroid coordinate vector and the second centroid coordinate vector , the calculated covariance matrix It can be expressed as:

[0065] (1).

[0066] The covariance matrix can be decomposed using the singular value decomposition algorithm Decompose the covariance matrix The singular value decomposition of can be expressed as:

[0067] (2).

[0068] Matrix-based and matrix , the first rotation matrix of the coordinate system from the source point cloud of the reference 3D image to the target 3D image can be calculated :

[0069] (3).

[0070] Based on the first rotation matrix And the three-dimensional coordinate vector of a pair of feature points, calculate the first translation matrix :

[0071] (4).

[0072] First coordinate system transformation relationship It can be expressed as:

[0073] (5).

[0074] According to an embodiment of the present disclosure, the target point cloud of the coarsely registered 3D image and the source point cloud of the reference 3D image are downsampled. A registration algorithm is used to perform 3D point cloud registration on the downsampled target point cloud and the downsampled source point cloud, thereby obtaining a second rotation matrix and a second translation matrix. Based on the second rotation matrix and the second translation matrix, a second coordinate system transformation relationship is obtained.

[0075] In one embodiment, an iterative closest point (ICP) registration algorithm can be used for precise registration. The iterative closest point registration algorithm can be used to solve the registration problem based on free-form surfaces. The optimal coordinate transformation, i.e., the second rotation matrix, is calculated iteratively using the least squares method. and the second translation matrix , so that the error function is minimized, the second coordinate system transformation relationship that accurately transforms the source point cloud of the reference three-dimensional image to the coordinate system of the target three-dimensional image can be obtained .

[0076] Second coordinate system transformation relationship It can be expressed as:

[0077] (6).

[0078] Figure 3 (A) schematically illustrates a flow chart of acquiring a reference 3D image and an initial 3D image according to an embodiment of the present disclosure. Figure 3 (B) schematically illustrates a flow chart of coarse registration and fine registration according to an embodiment of the present disclosure.

[0079] In one embodiment, before the high tibial osteotomy surgery, the patient's tibia can be scanned using CT to obtain a CT three-dimensional digital image that is used as a reference three-dimensional image in the embodiment of the present disclosure.

[0080] As shown in FIG3(A), in operation S311, a CT scan is performed on the patient's tibia to obtain a CT three-dimensional digital image. In operation S312, the position of the osteotomy surface and the distraction angle are planned.

[0081] As shown in Figure 3(A), in operation S321, structured light is projected onto the exposed bone surface of the tibia to create a distortion. In operation S322, an image of the distorted image is captured. In operation S323, the captured image is transmitted to a computer. In operation S324, a structured light 3D image of the exposed bone surface is reconstructed using binocular vision technology. In operation S330, the structured light 3D image is registered with the 3D CT digital image.

[0082] As shown in Figure 3(B), in operation S331, the structured light 3D image is cropped to retain the target area. In operation S332, the doctor selects three sets of paired points for coarse registration. In operation S333, the computer performs fine registration, calculating the transformation matrix and registration error between the structured light 3D image and the CT 3D digital image. In operation S334, the registration results are output.

[0083] Based on the above structured light image registration method, the present disclosure also provides a structured light image registration device.

[0084] Figure 4 The structural block diagram of the structured light image registration device according to an embodiment of the present disclosure is schematically shown.

[0085] like Figure 4 As shown, the structured light image registration device 400 of this embodiment includes an acquisition module 410 , a cropping module 420 , a first registration module 430 , a first conversion module 440 , a second registration module 450 and a second conversion module 460 .

[0086] The acquisition module 410 is configured to acquire a reference 3D image and an initial 3D image, wherein the initial 3D image is a 3D image obtained by irradiating an intraoperative object with intraoperative structured light.

[0087] The cropping module 420 is configured to crop the initial 3D image to obtain a target 3D image.

[0088] The first registration module 430 is configured to perform a coarse registration between the target 3D image and the reference 3D image to obtain a first coordinate system transformation relationship between the target 3D image and the reference 3D image.

[0089] The first conversion module 440 is configured to convert the source point cloud in the reference 3D image into the coordinate system of the target 3D image based on the first coordinate system conversion relationship to obtain a coarsely registered 3D image.

[0090] The second registration module 450 is configured to perform fine registration on the coarsely registered 3D image and the reference 3D image to obtain a second coordinate system transformation relationship between the coarsely registered 3D image and the reference 3D image.

[0091] The second conversion module 460 is configured to convert the source point cloud in the reference 3D image into the coordinate system of the coarsely registered 3D image based on the second coordinate system conversion relationship to obtain a finely registered 3D image.

[0092] According to an embodiment of the present disclosure, the first registration module 430 includes a first acquisition submodule, a first calculation submodule, a first determination submodule, and a second determination submodule.

[0093] The first acquisition submodule is used to acquire at least three pairs of feature points, each pair of feature points including a reference feature point and a target feature point; wherein the reference feature points are acquired from the reference three-dimensional image, and the target feature points are acquired from the target three-dimensional image, and the positions of the reference feature point and the target feature point of each pair of feature points in the reference three-dimensional image and the target three-dimensional image correspond to each other.

[0094] The first calculation submodule is used to calculate the first centroid coordinate vectors of at least three reference feature points and the second centroid coordinate vectors of at least three target feature points based on the three-dimensional coordinate vectors of the at least three reference feature points in the reference three-dimensional image and the three-dimensional coordinate vectors of the at least three target feature points in the target three-dimensional image.

[0095] The first determining submodule is configured to determine a first rotation matrix and a first translation matrix based on the first center-of-mass coordinate vector and the second center-of-mass coordinate vector.

[0096] The second determining submodule is configured to determine a first coordinate system transformation relationship based on the first rotation matrix and the first translation matrix.

[0097] According to an embodiment of the present disclosure, the first determining submodule includes a first calculating unit, a first obtaining unit, and a second obtaining unit.

[0098] The first calculation unit is used to calculate a covariance matrix based on the first centroid coordinate vector and the second centroid coordinate vector.

[0099] The first obtaining unit is used to perform singular value decomposition on the covariance matrix to obtain a first rotation matrix.

[0100] The second obtaining unit is used to obtain a first translation matrix based on the first rotation matrix and the three-dimensional coordinate vectors of a pair of feature points.

[0101] According to an embodiment of the present disclosure, the second registration module 450 includes a downsampling submodule, a registration submodule, and a obtaining submodule.

[0102] The downsampling submodule is used to downsample the target point cloud of the coarsely registered 3D image and the source point cloud of the reference 3D image.

[0103] The registration submodule is used to perform three-dimensional point cloud registration on the downsampled target point cloud and the downsampled source point cloud using a registration algorithm to obtain a second rotation matrix and a second translation matrix.

[0104] The submodule is used to obtain a second coordinate system transformation relationship based on the second rotation matrix and the second translation matrix.

[0105] According to an embodiment of the present disclosure, the acquisition module 410 includes a second acquisition submodule and a third determination submodule.

[0106] The second acquisition submodule is used to acquire a first distorted image and a second distorted image respectively obtained by two structured light cameras, wherein the two structured light cameras are respectively configured on both sides of a projector of structured light in projection surgery.

[0107] The third determination submodule is configured to determine an initial three-dimensional image according to the intraoperative structured light, the first distorted image, and the second distorted image.

[0108] According to an embodiment of the present disclosure, the third determining submodule includes a second calculating unit, a third calculating unit, and a determining unit.

[0109] The second calculation unit is used to calculate first distortion information between the first distorted image and the intraoperative structured light.

[0110] The third calculation unit is used to calculate second distortion information between the second distorted image and the intraoperative structured light.

[0111] The determining unit is configured to determine an initial three-dimensional image based on the first distortion information and the second distortion information according to the binocular stereo vision technology.

[0112] According to embodiments of the present disclosure, any multiple modules among the acquisition module 410, cropping module 420, first registration module 430, first conversion module 440, second registration module 450, and second conversion module 460 may be combined into a single module, or any one of these modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in a single module. According to embodiments of the present disclosure, at least one of the acquisition module 410, cropping module 420, first registration module 430, first conversion module 440, second registration module 450, and second conversion module 460 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or may be implemented in hardware or firmware through any other reasonable means of circuit integration or packaging, or implemented in any one of software, hardware, and firmware, or any suitable combination of any of these. Alternatively, at least one of the acquisition module 410, the cropping module 420, the first registration module 430, the first conversion module 440, the second registration module 450 and the second conversion module 460 can be at least partially implemented as a computer program module, which can perform corresponding functions when executed.

[0113] Figure 5 A block diagram of an electronic device suitable for implementing a structured light image registration method according to an embodiment of the present disclosure is schematically shown.

[0114] like Figure 5As shown, the electronic device 500 according to an embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage unit 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present disclosure.

[0115] Various programs and data required for the operation of the electronic device 500 are stored in the RAM 503. The processor 501, ROM 502, and RAM 503 are connected to each other via a bus 504. The processor 501 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 502 and / or RAM 503. It should be noted that the programs may also be stored in one or more memories other than the ROM 502 and RAM 503. The processor 501 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0116] According to an embodiment of the present disclosure, electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to bus 504. Electronic device 500 may also include one or more of the following components connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 508 including a hard disk; and a communication section 509 including a network interface card such as a LAN card or modem. Communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 510 as needed, so that computer programs read from the removable media can be installed into storage section 508 as needed.

[0117] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0118] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 502 and / or RAM 503 described above, and / or one or more memories other than ROM 502 and RAM 503.

[0119] The present disclosure also includes a computer program product comprising a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to enable the computer system to implement the structured light image registration method provided in the present disclosure.

[0120] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the computer program is executed by the processor 501. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0121] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 509, and / or installed from a removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0122] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 509, and / or installed from a removable medium 511. When the computer program is executed by the processor 501, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0123] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0125] Those skilled in the art will appreciate that the features described in the various embodiments of the present disclosure may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present disclosure. In particular, the features described in the various embodiments of the present disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of the present disclosure. All such combinations and / or couplings fall within the scope of the present disclosure.

[0126] The above describes the embodiments of the present disclosure. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A structured light image registration method, characterized in that: The method comprises: Acquiring a reference three-dimensional image and an initial three-dimensional image, wherein the initial three-dimensional image is a three-dimensional image obtained by irradiating an intraoperative object with intraoperative structured light; cropping the initial three-dimensional image to obtain a target three-dimensional image; performing coarse registration of the target three-dimensional image and the reference three-dimensional image to obtain a first coordinate system transformation relationship between the target three-dimensional image and the reference three-dimensional image; Based on the first coordinate system transformation relationship, transforming the source point cloud in the reference three-dimensional image into the coordinate system of the target three-dimensional image to obtain a coarsely registered three-dimensional image; Performing fine registration on the coarsely registered three-dimensional image and the reference three-dimensional image to obtain a second coordinate system transformation relationship between the coarsely registered three-dimensional image and the reference three-dimensional image; Based on the second coordinate system conversion relationship, the source point cloud in the reference three-dimensional image is converted into the coordinate system of the coarsely registered three-dimensional image to obtain a finely registered three-dimensional image.

2. The method according to claim 1, characterized in that The coarsely registering the target three-dimensional image with the reference three-dimensional image to obtain a first coordinate system transformation relationship between the target three-dimensional image and the reference three-dimensional image includes: Acquire at least three pairs of feature points, each pair of feature points comprising a reference feature point and a target feature point; wherein the reference feature points are acquired from the reference three-dimensional image, and the target feature points are acquired from the target three-dimensional image, and the reference feature point and the target feature point of each pair of feature points correspond to each other in the reference three-dimensional image and the target three-dimensional image; Calculating first centroid coordinate vectors of at least three reference feature points and second centroid coordinate vectors of at least three target feature points based on the three-dimensional coordinate vectors of the at least three reference feature points in the reference three-dimensional image and the three-dimensional coordinate vectors of the at least three target feature points in the target three-dimensional image; Determine a first rotation matrix and a first translation matrix based on the first center-of-mass coordinate vector and the second center-of-mass coordinate vector; The first coordinate system transformation relationship is determined based on the first rotation matrix and the first translation matrix.

3. The method according to claim 2, characterized in that The determining of a first rotation matrix and a first translation matrix based on the first mass center coordinate vector and the second mass center coordinate vector includes: Calculating a covariance matrix based on the first centroid coordinate vector and the second centroid coordinate vector; Performing singular value decomposition on the covariance matrix to obtain the first rotation matrix; The first translation matrix is ​​obtained based on the first rotation matrix and the three-dimensional coordinate vectors of a pair of feature points.

4. The method according to claim 1, wherein The finely registering the coarsely registered three-dimensional image and the reference three-dimensional image to obtain a second coordinate system transformation relationship between the coarsely registered three-dimensional image and the reference three-dimensional image includes: downsampling the target point cloud of the coarsely registered three-dimensional image and the source point cloud of the reference three-dimensional image; Using a registration algorithm, the downsampled target point cloud and the downsampled source point cloud are registered to obtain a second rotation matrix and a second translation matrix; Based on the second rotation matrix and the second translation matrix, the second coordinate system transformation relationship is obtained.

5. The method according to claim 1, characterized in that The obtaining of the reference three-dimensional image and the initial three-dimensional image comprises: Acquire a first distorted image and a second distorted image respectively obtained by two structured light cameras, wherein the two structured light cameras are respectively configured on either side of a projector that projects the intraoperative structured light; The initial three-dimensional image is determined according to the intraoperative structured light, the first distorted image, and the second distorted image.

6. The method according to claim 5, characterized in that The determining the initial three-dimensional image according to the intraoperative structured light, the first distorted image, and the second distorted image includes: calculating first distortion information between the first distorted image and the intraoperative structured light; calculating second distortion information between the second distorted image and the intraoperative structured light; According to binocular stereo vision technology, the initial three-dimensional image is determined based on the first distortion information and the second distortion information.

7. A structured light image registration device, characterized in that: The device comprises: An acquisition module, configured to acquire a reference three-dimensional image and an initial three-dimensional image; wherein the initial three-dimensional image is a three-dimensional image obtained by irradiating an intraoperative object with intraoperative structured light; A cropping module, configured to crop the initial three-dimensional image to obtain a target three-dimensional image; a first registration module, configured to perform coarse registration on the target three-dimensional image and the reference three-dimensional image to obtain a first coordinate system transformation relationship between the target three-dimensional image and the reference three-dimensional image; a first conversion module, configured to convert the source point cloud in the reference 3D image into the coordinate system of the target 3D image based on the first coordinate system conversion relationship, to obtain a coarsely registered 3D image; a second registration module, configured to perform fine registration on the coarsely registered three-dimensional image and the reference three-dimensional image to obtain a second coordinate system transformation relationship between the coarsely registered three-dimensional image and the reference three-dimensional image; The second conversion module is used to convert the source point cloud in the reference 3D image into the coordinate system of the coarsely registered 3D image based on the second coordinate system conversion relationship to obtain a finely registered 3D image.

8. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images