Skeleton registration apparatus and electronic device

Abstract

The embodiments of the present invention belong to the field of data processing. Provided are a skeleton registration apparatus and an electronic device. The apparatus is used for: performing coarse registration on a skeletal surface model constructed on the basis of three-dimensional image data and a physical skeleton in a real-scene video image, so as to obtain a superimposed and aligned image; performing point cloud registration on the skeletal surface model and the physical skeleton on the basis of a registration point selected by a user on the physical skeleton, and scoring the point cloud registration result; and when a score value does not reach a preset pass threshold value, calculating a registration score deduction area, removing a registration point located in the registration score deduction area, and returning to the step of point cloud registration until the score value reaches the preset pass threshold value. Thus, when a point cloud registration result is unsatisfactory, a registration point in a registration score deduction area is reselected to continuously perform point cloud registration, such that the point cloud registration result is subjected to targeted correction until the registration result reaches a pass threshold value, thereby greatly improving the accuracy of skeleton registration.

Description

Skeleton registration device and electronic equipment

[0001] Related applications

[0002] This application claims priority to Chinese patent application No. 2024101379907, filed on February 1, 2024, entitled “Skeleton Alignment Device and Electronic Device,” the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the field of data processing, and in particular to a skeleton registration device and electronic equipment. Background Art

[0004] An orthopedic surgical robot is a medical device system designed to improve surgical precision. It consists of a robotic arm, a preoperative planning module, an intraoperative navigation module, and a visualization module. During use, the orthopedic surgical robot uses registration technology to align a virtual 3D skeleton model in a computer with the actual skeleton.

[0005] Currently, skeletal registration is typically performed using a first-stage coarse or initial registration followed by a second-stage fine registration. In the first stage, users collect registration points on the physical object based on experience and then associate these points with points on the virtual model of the object in sequence and position. During fine registration, key center points are typically found by rotating relevant parts of the object, and these center points are used to correct errors in the registration points. However, this registration method is prone to errors and mismatches, resulting in low registration accuracy.

[0006] Summary of the Invention

[0007] In view of this, the purpose of this application is to provide a skeleton alignment device and electronic equipment that can improve the accuracy of alignment while eliminating the need for repeated initial alignment, greatly shortening the alignment time and improving the alignment efficiency.

[0008] In order to achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows:

[0009] In a first aspect, an embodiment of the present application provides a skeleton registration device, comprising a coarse registration module, a fine registration module, an evaluation module, and a correction module;

[0010] The coarse registration module is configured to construct a bone surface model based on the three-dimensional image data of the physical bone, obtain a real-scene video image of the area where the physical bone is located in real time, and coarsely register the bone surface model with the physical bone in the real-scene video image to obtain a superimposed aligned image;

[0011] The fine registration module is configured to perform point cloud registration between the bone surface model and the physical bone according to the registration points selected by the user on the physical bone, and score the point cloud registration result in combination with the registration points and the bone surface model;

[0012] The evaluation module is configured to determine whether the score value reaches a preset passing threshold, and if so, complete the registration and obtain a final registration matrix based on the registration result;

[0013] The correction module is configured to calculate the registration deduction area if not, eliminate the registration points located in the registration deduction area, and display the registration deduction area on the superimposed alignment image to instruct the user to reselect the registration points on the registration deduction area of ​​the solid skeleton, and return to the fine registration module to execute the step of performing point cloud registration between the bone surface model and the solid skeleton according to the registration points selected by the user on the solid skeleton, until the score value reaches a preset passing threshold.

[0014] In a possible implementation, the number of the registration points is set to: p<40.

[0015] In a possible implementation, the number of the registration points is set to: 15≤p≤30.

[0016] In a possible implementation, the point cloud registration result includes a skeleton registration matrix;

[0017] The fine registration module is configured to score the point cloud registration result by combining the registration points and the bone surface model, including:

[0018] For each of the registration points, mapping the registration point back to the image coordinate system of the superimposed alignment image according to the skeletal registration matrix to obtain a mapping coordinate value of the registration point;

[0019] Calculating the distance between each registration point and the surface of the bone surface model according to the mapping coordinate value;

[0020] Based on the distances corresponding to all the registration points, the point cloud registration result is scored to obtain a score value.

[0021] In a possible implementation, a first tracking device is installed on the physical skeleton;

[0022] The fine registration module is configured to map the registration points back to the image coordinate system of the superimposed aligned image according to the skeletal registration matrix to obtain the mapping coordinate values ​​of the registration points, including:

[0023] Obtaining a first pose matrix of the first tracking device and a second pose matrix of the registration point, combining the skeletal registration matrix, the first pose matrix, and the second pose matrix, and using a registration point mapping formula to calculate a mapping coordinate value of the registration point;

[0024] The registration point mapping formula includes:

[0025] Among them, T spt Characterize the mapping coordinate value, The inverse matrix of the skeleton registration matrix representing the nth point cloud registration, Characterize the inverse matrix of the first pose matrix, T tpt Represents the second pose matrix.

[0026] In a possible implementation, the fine registration module is configured to score the point cloud registration result based on the distances corresponding to all the registration points, and the step of obtaining a score value includes:

[0027] The distance corresponding to each registration point is compared with a preset distance threshold, the total number of registration points whose distance is greater than the distance threshold is counted, and points are deducted based on the total number to obtain a score value.

[0028] In a possible implementation, the coarse registration module is configured to coarsely register the bone surface model with the physical bone in the real-scene video image to obtain a superimposed aligned image, including:

[0029] The bone surface model is superimposed on the real-scene video image by using augmented reality technology, and the bone surface model is aligned and overlapped with the physical bone in the real-scene video image to obtain a superimposed aligned image.

[0030] In a possible implementation, after the step of constructing a bone surface model based on the three-dimensional image data of the physical bone and before the step of coarsely registering the bone surface model with the physical bone in the real-scene video image to obtain a superimposed aligned image, the coarse registration module is further configured to:

[0031] marking a plurality of feature points on the bone surface model;

[0032] After the step of coarsely registering the bone surface model with the physical bone in the real-scene video image to obtain a superimposed aligned image, and before the step of performing point cloud registration between the bone surface model and the physical bone according to the registration points selected by the user on the physical bone, the coarse registration module is further configured to:

[0033] The feature points of the aligned and superimposed bone surface models are used as recommended positions and displayed in the superimposed alignment image to instruct the user to select registration points on the physical bone according to the recommended positions.

[0034] In a possible implementation, a first tracking device is installed on the physical skeleton;

[0035] The registration module is configured to perform point cloud registration between the bone surface model and the physical bone according to the registration points selected by the user on the physical bone, comprising:

[0036] After obtaining the superimposed alignment image, obtaining a relative pose matrix between the first tracking device and the bone surface model;

[0037] Obtaining registration points selected by a user on the physical skeleton, and using an ICP algorithm to align the bone surface model with the physical skeleton according to the registration points to obtain a precise registration matrix;

[0038] A skeleton registration matrix is ​​calculated according to the relative pose matrix and the fine registration matrix.

[0039] In a possible implementation, the evaluation module is configured to obtain a final registration matrix according to the registration result, including:

[0040] The product of all the skeleton registration matrices is calculated to obtain the final registration matrix.

[0041] In a possible implementation, the coarse registration module is configured to align the bone surface model with the physical bone in the real-life video image, comprising:

[0042] Binding the model pose matrix of the bone surface model to the probe pose matrix of a preset probe; wherein a second tracking device is installed on the probe;

[0043] Based on the feature points, reference positions are drawn on the bone surface model to instruct the user to align the bone surface model with the physical bone in the real-life video image by moving the probe based on the reference positions; wherein the reference positions include feature points and extension lines of feature axes.

[0044] In a second aspect, an embodiment of the present application provides an electronic device comprising a skeletal registration device as described in any possible implementation of the first aspect.

[0045] In a third aspect, an embodiment of the present application provides a skeleton registration method, the method comprising:

[0046] Constructing a bone surface model based on the three-dimensional image data of the physical bone, acquiring a real-life video image of the area where the physical bone is located in real time, and coarsely registering the bone surface model with the physical bone in the real-life video image to obtain a superimposed aligned image;

[0047] Performing point cloud registration of the bone surface model with the physical bone according to the registration points selected by the user on the physical bone, and scoring the point cloud registration result based on the registration points and the bone surface model;

[0048] Determine whether the score reaches the preset pass threshold. If so, complete the registration and obtain the final registration matrix based on the registration result.

[0049] If not, the registration deduction area is calculated, the registration points located in the registration deduction area are eliminated, and the registration deduction area is displayed as a prompt on the superimposed alignment image to instruct the user to reselect registration points on the registration deduction area of ​​the solid bone, and return to the step of performing point cloud registration between the bone surface model and the solid bone according to the registration points selected by the user on the solid bone, until the score value reaches the preset passing threshold.

[0050] The embodiments of the present application provide a bone registration device and an electronic device, which are used to: acquire a real-life video image of the area where the physical bone is located in real time, perform a rough registration of a bone surface model constructed based on three-dimensional image data of the physical bone with the physical bone in the real-life video image to obtain a superimposed aligned image; perform point cloud registration of the bone surface model with the physical bone according to the registration points selected by the user on the physical bone, and score the point cloud registration result; when the score value does not reach a preset pass threshold, calculate the registration deduction area, eliminate the registration points located in the registration deduction area, and display a prompt for the registration deduction area on the superimposed aligned image to instruct the user to reselect the registration points on the registration deduction area of ​​the physical bone, and return to the step of performing point cloud registration of the bone surface model with the physical bone according to the registration points selected by the user on the physical bone, until the score value reaches a preset pass threshold; if the score value reaches the preset pass threshold, the registration is completed, and a final registration matrix is ​​obtained according to the registration result. In this way, when the point cloud registration result is not ideal, the registration points in the registration deduction area are reselected to continue the point cloud registration, and the point cloud registration result is corrected in a targeted manner until the registration result reaches the passing threshold, which greatly improves the accuracy of bone registration.

[0051] In addition, only one coarse registration is required. If the score value does not reach the passing threshold, there is no need to re-perform coarse registration, which reduces the number of coarse registrations, simplifies the registration process and registration complexity, and thus can shorten the registration time and improve the registration efficiency.

[0052] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0054] FIG1 shows a schematic structural diagram of a skeleton registration system provided in an embodiment of the present application.

[0055] FIG2 shows a schematic structural diagram of a skeleton registration device provided in an embodiment of the present application.

[0056] FIG3 shows one of the method flow diagrams of the skeleton registration device provided in an embodiment of the present application.

[0057] FIG. 4 is a schematic flow chart showing some sub-steps of step S11 in FIG. 3 .

[0058] FIG5 shows a second schematic diagram of the method flow of the skeleton registration device provided in an embodiment of the present application.

[0059] FIG. 6 shows one of the flowcharts of some sub-steps of step S13 in FIG. 3 and FIG. 5 .

[0060] FIG. 7 shows a second flowchart of some sub-steps of step S13 in FIG. 3 and FIG. 5 .

[0061] FIG8 shows a schematic structural diagram of an electronic device provided in an embodiment of the present application.

[0062] Explanation of the accompanying drawings: 1000 - skeletal registration system; 10 - orthopedic surgical robot; 101 - control processor; 20 - image acquisition device; 30 - imaging shooting device; 40 - first tracking device; 50 - second tracking device; 60 - probe; 70 - skeletal registration device; 701 - coarse registration module; 702 - fine registration module; 703 - evaluation module; 704 - correction module; 80 - electronic device. DETAILED DESCRIPTION

[0063] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0064] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present application.

[0065] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0066] The skeletal alignment device 70 provided in an embodiment of the present application can be applied to the skeletal alignment system 1000 as shown in Figure 1. The skeletal alignment system 1000 may include an orthopedic surgical robot 10, an image acquisition device 20, an imaging shooting device 30, a first tracking device 40 and a second tracking device 50. The orthopedic surgical robot 10 may include a control processor 101 and a display device.

[0067] The orthopedic surgical robot 10 can communicate with the image acquisition device 20, the imaging device 30, the first tracking device 40, and the second tracking device 50 via wired or wireless communication. The first tracking device 40 is mounted on a physical bone, the second tracking device 50 is mounted on a probe 60, and the image acquisition device 20 is oriented toward the physical bone to be registered. The orthopedic surgical robot 10 can also be equipped with an optical positioning system.

[0068] The image acquisition device 20 is used to capture real-time video images of the area where the physical bones are located and transmit them to the orthopedic surgery robot 10. The physical bones can be any human bones such as the pelvis, femur, tibia, spine and scapula, or any animal bones.

[0069] The imaging device 30 is used to photograph the physical skeleton, obtain three-dimensional image data of the physical skeleton, and transmit the three-dimensional image data to the orthopedic surgery robot 10 .

[0070] The first tracking device 40 is used to track and obtain first position information of the physical skeleton, wherein the first position information includes a first pose matrix.

[0071] The second tracking device 50 is used to track and obtain second position information of the probe 60 , wherein the second position information may include a pose matrix of the probe 60 .

[0072] The control processor 101 is used to implement the skeleton registration device 70 and method provided in the embodiment of the present application based on the real-scene video image, the three-dimensional image data, the first position information and the second position information.

[0073] Image acquisition device 20 may be, but is not limited to, a camera, mobile phone, tablet, mobile terminal, or webcam. Imaging device 30 may be, but is not limited to, a CT device or an MRI device. First tracking device 40 and second tracking device 50 may be, but are not limited to, RF trackers and optical trackers. For example, they may be reference arrays in optical positioning systems. Optical positioning systems may be, but are not limited to, infrared binocular positioning systems, radar positioning systems, and multi-link arm positioning systems.

[0074] In a possible implementation, a skeleton registration device 70 is further provided. Referring to FIG. 2 , the device may include a coarse registration module 701 , a fine registration module 702 , an evaluation module 703 , and a correction module 704 .

[0075] The coarse registration module 701 is configured to construct a bone surface model based on the three-dimensional image data of the physical bone, and obtain a real-life video image of the area where the physical bone is located in real time, and coarsely align the bone surface model with the physical bone in the real-life video image to obtain a superimposed aligned image.

[0076] The fine registration module 702 is configured to perform point cloud registration between the bone surface model and the physical bone according to the registration points selected by the user on the physical bone, and score the point cloud registration result based on the registration points and the bone surface model.

[0077] The evaluation module 703 is configured to determine whether the score value reaches a preset passing threshold. If so, the registration is completed and a final registration matrix is ​​obtained based on the registration result.

[0078] The correction module 704 is configured to calculate the registration deduction area if the score value does not reach the preset passing threshold, eliminate the registration points located in the registration deduction area, and display the registration deduction area on the superimposed alignment image to instruct the user to reselect the registration points on the registration deduction area of ​​the physical skeleton, and return to the fine registration module 702 to execute the step of performing point cloud registration between the bone surface model and the physical skeleton according to the registration points selected by the user on the physical skeleton, until the score value reaches the preset passing threshold.

[0079] In the above-mentioned skeletal registration device 70, through the coordinated action of the coarse registration module 701, the fine registration module 702, the evaluation module 703, and the correction module 704, if the point cloud registration result is not ideal, the registration points in the registration deduction area are reselected to continue point cloud registration, and the point cloud registration result is corrected in a targeted manner until the registration result reaches the passing threshold, thereby greatly improving the accuracy of skeletal registration. In addition, only one coarse registration is required, and if the score does not reach the passing threshold, there is no need to re-perform coarse registration, which reduces the number of coarse registrations, simplifies the registration process and registration complexity, thereby shortening the registration time and improving the registration efficiency.

[0080] The skeleton alignment device 70 provided in the embodiment of the present application can be the control processor 101 in Figure 1, or it can be a functional module of the control processor 101. The skeleton alignment device 70 or the control processor 101 where the skeleton alignment device 70 is located executes the various steps in the following embodiments to achieve corresponding technical effects.

[0081] In a possible implementation, referring to FIG. 3 , the skeleton registration device may perform the following steps.

[0082] S11, constructing a bone surface model based on the three-dimensional image data of the physical bone, and obtaining a real-scene video image of the area where the physical bone is located in real time, and roughly aligning the bone surface model with the physical bone in the real-scene video image to obtain a superimposed aligned image.

[0083] S13, based on the registration points selected by the user on the physical skeleton, the bone surface model is point cloud registered with the physical skeleton, and the point cloud registration result is scored based on the registration points and the bone surface model.

[0084] S15: Determine whether the score reaches a preset passing threshold. If not, proceed to step S17; if so, proceed to step S19.

[0085] S17: Calculate the registration deduction area, remove the registration points located in the registration deduction area, and display a prompt for the registration deduction area on the superimposed aligned image to instruct the user to reselect registration points in the registration deduction area of ​​the physical skeleton. After step S17, the process returns to step S13.

[0086] S19, completing the registration, and obtaining the final registration matrix based on the registration results.

[0087] It can be understood that the user can use the imaging device 30 to capture (perhaps in the form of a three-dimensional scan) the physical part of the subject (i.e., the physical part including the physical skeleton) to obtain three-dimensional image data that can distinguish the bony structure required for registration. The imaging device 30 transmits the three-dimensional image data to the control processor 101 of the bone surgery robot. At the same time, the technician can use the image acquisition device 20 to capture real-time images of the area including the physical skeleton of the subject to obtain a real-scene video image (i.e., a video stream). The image acquisition device 20 transmits the real-scene video image to the control processor 101 of the bone surgery robot.

[0088] After receiving the 3D image data, the control processor 101 of the orthopedic surgical robot 10 reconstructs a bone surface model based on the 3D image data. Furthermore, the control processor 101 performs a coarse registration between the bone surface model and the real bone in the received endoscopic video image, generating an overlaid aligned image. At this point, the real bone in the overlaid aligned image and the bone surface model are aligned and overlap.

[0089] The user can use the physical probe 60 to select a registration point on the physical skeleton. The second tracking device 50 identifies and obtains the position information of the user-selected registration point and transmits the configuration point information (including the position information and configuration point of the registration point) to the control processor 101 of the orthopedic surgical robot 10. The control processor 101 performs point cloud registration between the bone surface model and the physical skeleton based on the user-selected registration point. The control processor 101 then scores the point cloud registration result based on the registration point and the bone surface model to obtain a score.

[0090] When the score value reaches the preset pass threshold, the control processor 101 completes the registration and obtains the final registration matrix based on the registration result. On the contrary, if the score value does not reach the preset pass threshold, the registration deduction area is calculated, the registration points located in the registration deduction area are eliminated, and the registration deduction area is displayed on the superimposed alignment image to instruct the user to reselect the registration point on the registration deduction area of ​​the physical skeleton. After the user selects a new registration point in the registration deduction area, the control processor 101 continues point cloud registration based on the new registration point and the registration points outside the registration deduction area, repeating the process of fine registration and scoring until the score value reaches the pass threshold.

[0091] It should be noted that the passing threshold may be positively correlated with the number of registration points, that is, the more the number of registration points, the higher the passing threshold.

[0092] Traditional bone registration methods include: (1) during initial registration (i.e., coarse registration), the user uses anatomical knowledge to infer and collect initial registration points on the physical skeleton, and the orthopedic surgical robot performs initial registration, matching the points on the virtual model with the order and position of the initial registration points one by one; (2) during fine registration, the key center point is generally found by rotating the relevant parts of the entity, and the registration point error is corrected with the key center point. For example, when performing thigh bone registration, the center of the femoral head or the center of the acetabulum can be found by rotating the thigh. However, when selecting the initial registration point, it is easy to produce incorrect registration or exceed the allowable error, so the initial registration needs to be repeated, which is time-consuming. In addition, when rotating the bone to find the key center point, the traction of the surrounding muscle tissue affects the center point, resulting in the actual key center point (such as the center of the femoral head or the center of the acetabulum) not matching the actual one, making the registration accuracy low and the registration time-consuming.

[0093] In the skeleton registration device provided in the embodiment of the present application, through the above steps S11 to S19, when the point cloud registration result is not ideal, the registration points in the registration deduction area are reselected to continue the point cloud registration, and the point cloud registration result is corrected in a targeted manner until the registration result reaches the passing threshold, thereby greatly improving the accuracy of the skeleton registration.

[0094] In addition, only one coarse registration is required. If the score value does not reach the passing threshold, there is no need to re-perform coarse registration, which reduces the number of coarse registrations, simplifies the registration process and registration complexity, and thus can shorten the registration time and improve the registration efficiency.

[0095] Optionally, in order to further reduce the registration time and improve the registration efficiency, augmented reality technology can be introduced into the coarse registration of step S11. Specifically, augmented reality technology can be used to superimpose the bone surface model on the real-scene video image, and the bone surface model is aligned with the physical bones in the real-scene video image to obtain a superimposed aligned image.

[0096] In this way, coarse registration can be completed by moving the bone surface model so that the bone surface model is aligned and overlapped with the physical bone. There is no need to collect registration points to correspond one-to-one with the points on the bone surface model for coarse registration. This greatly simplifies the operational complexity and calculation amount of coarse registration, thereby further shortening the registration time and improving the registration efficiency.

[0097] The method of moving the bone surface model so that the bone surface model is aligned and overlapped with the physical bone in the real-life video image to complete the coarse alignment can be flexibly set. For example, the bone surface model can be moved by moving the mouse or touch screen, or by inputting the displacement direction and displacement distance. In this embodiment, no specific limitation is made.

[0098] In order to improve the accuracy of coarse registration and simplify the operation of coarse registration, a probe 60 and a reference position for moving the bone surface model can be introduced. The probe 60 can be equipped with a second tracking device (which can be a reference array of an optical positioning system) that is in communication with the orthopedic surgical robot 10. Optionally, referring to FIG. 4 , the process of coarse registration module 701 executing step S11 to perform coarse registration can include the following implementations.

[0099] S111, marking multiple feature points on the bone surface model.

[0100] S112, binding the model pose matrix of the bone surface model to the probe pose matrix of the preset probe.

[0101] S113, drawing a reference position on the bone surface model according to the feature point, to instruct the user to align the bone surface model with the physical bone in the real-scene video image by moving the probe according to the reference position.

[0102] The optical positioning system deployed on the orthopedic surgical robot 10 captures the probe 60 pose matrix T of the probe 60 in the spatial coordinate system. Tracker_Probe After that, the control processor 101 converts the pose matrix of the probe 60 and the pose matrix T of the bone surface model into Surface Bind and get the binding pose matrix T Probe_Surface After the two are bound, the bone surface model will move along with the movement of the probe 60 in the coordinate system of the optical positioning system.

[0103] In this embodiment, the feature points may include key areas and key points of the skeleton, and the reference positions may include extension lines of the feature axes of the feature points and key areas. The selection of feature points may be flexibly set and is not specifically limited in this embodiment.

[0104] It can be understood that after constructing the bone surface model, the user can divide the bone surface model into multiple areas according to arbitrary rules, and mark a certain number of key points in each area. At the same time, at least one characteristic axis can be marked (which can be the center line of the bone surface model, the center line of the area, etc.).

[0105] After completing pose binding and feature point annotation, the control processor 101 can use augmented reality technology to overlay the skeletal surface model annotated with the feature points onto the real-life video image. The user can then view the real-life video image's physical parts (including the physical skeleton) and the overlaid skeletal surface model on a three-dimensional display device such as a display screen, a projection screen, or the screen of an augmented display helmet. Simultaneously, the control processor 101 draws reference points on the skeletal surface model based on the feature points.

[0106] On this basis, the user uses the reference position as an aid to intuitively move the probe 60 to control the posture of the bone surface model, so as to move the bone surface model to align with the physical part. After confirming the alignment, the user can send a confirmation signal to the control processor 101 by clicking a confirmation button or issuing a confirmation voice. After receiving the signal, the control processor 101 calculates the coarse registration posture matrix of the current bone surface model in the coordinate system of the optical positioning system: T Tracker_SfI =T Tracker_Probe ×T Probe_SfI Among them, T Probe_SfI The bind pose matrix representing the alignment.

[0107] Through the above S111 to S113, the coarse registration module 701 is assisted based on the reference position, combined with the enhanced display technology and the probe binding movement method, and can quickly coarsely align the bone surface model with the physical bone in the video image, and can ensure the accuracy of the coarse registration, further shortening the time consumption of the coarse registration.

[0108] Optionally, in order to shorten the time consumption of fine registration (i.e., point cloud registration), a concept of registration point position recommendation is introduced in a possible embodiment. Optionally, referring to FIG5 , after step S11 and before step S13 , the coarse registration module 701 of the skeleton registration device 70 may further perform step S12 .

[0109] S12: The feature points of the aligned and coincident bone surface models are used as recommended positions and displayed in the superimposed alignment image to instruct the user to select registration points on the physical bone according to the recommended positions.

[0110] Feature points can be displayed as recommended positions in special colors (such as red, blue), special recommended points, etc. At this time, since the bone surface model and the physical bones in the real-life video image are aligned and overlapped, after the recommended position is displayed, the user can use the probe 60 unbound from the bone surface model to select the alignment point from the physical part position corresponding to the recommended position.

[0111] In this way, there is no need to rely on user experience when selecting registration points, which can shorten the time spent on selecting registration points, improve the accuracy of registration point selection, reduce errors, and help improve the accuracy of fine registration.

[0112] It should be noted that feature points are the result of dividing the bone surface model into multiple regions, with each region labeled. Correspondingly, each region of the bone surface model has a recommended location. When selecting registration points, at least one registration point is selected for each region of the physical bone corresponding to each region of the bone surface model.

[0113] In this way, each area of ​​the physical skeleton has a registration point, which can improve the coverage of the registration range and help improve the registration accuracy.

[0114] In this embodiment, the number of registration points may be set to: p<40.

[0115] In traditional fine registration, the number of selected registration points is typically 40 ≤ p ≤ 60. In this implementation, the number of registration points is controlled below 40. After point cloud registration, scoring is performed, and point cloud registration is repeated based on the score until the score meets the passing threshold. This approach ensures registration accuracy while reducing the amount of registration computation. This minimizes the time required for fine registration and reduces the number of registration points.

[0116] Optionally, the number of registration points may be set to 15 ≤ p ≤ 30. This can significantly reduce the amount of registration calculations, shorten the time required for precise registration, and also ensure registration accuracy.

[0117] Optionally, for step S13 performed by the fine registration module 702 , referring to FIG. 6 , the process of performing point cloud registration may include the following implementations.

[0118] S131: After obtaining the superimposed alignment image, obtain the relative pose matrix between the first tracking device and the bone surface model.

[0119] S132, obtaining the registration points selected by the user on the physical skeleton, using the ICP algorithm to align the bone surface model with the physical skeleton according to the registration points, and obtaining a precise registration matrix.

[0120] S133, calculating the skeleton registration matrix according to the relative pose matrix and the fine registration matrix.

[0121] In step S131, the first tracking device 40 can be represented by RF, and the relative pose matrix between the first tracking device 40 and the bone surface model can be expressed as: T RF_SfI =T RF_Tracker × Among them, T RF_SfI Represents the relative pose matrix, The inverse matrix representing the pose matrix of the first tracking device 40 in the optical positioning system, T Tracker_SfI Represents the coarse registration pose matrix of the bone surface model in the coordinate system of the optical positioning system.

[0122] Regarding step S132, the ICP algorithm is only an example of point cloud registration and can be replaced by a point cloud registration algorithm such as the NICP algorithm, the IMLS-ICP algorithm, etc. In addition, the ICP algorithm is a relatively mature algorithm in this field and will not be described in detail here.

[0123] In addition, the fine registration matrix can be expressed as T SfI_Surface_n , represents the precise matching matrix obtained after the n-th point cloud registration.

[0124] In step S133, the relative pose matrix is ​​multiplied by the fine registration matrix to obtain the skeleton registration matrix. The skeleton registration matrix can be expressed as: RFS_n =T RF_SfI ×T SfI_Surface_n , where T RFS_n Skeletal registration matrix representing the nth point cloud registration.

[0125] Through the above steps S131 to S133, the fine registration module 702 can quickly and accurately perform point cloud registration and obtain a skeleton registration matrix after point cloud registration.

[0126] In step S13, the process of scoring the point cloud registration results can be flexibly set. For example, the scoring can be performed according to preset rules, or the registration results can be scored using a machine learning model. In this embodiment, there is no limitation.

[0127] It should be understood that after each point cloud registration is completed, the position of the bone surface model in the superimposed alignment image will be updated and displayed based on the point cloud registration results, so that the user can intuitively see the fit between the bone surface model and the physical bone.

[0128] In a possible implementation, referring to FIG. 7 , the procedure for the fine registration module 702 to score the point cloud registration result may be implemented as follows.

[0129] S134 , for each registration point, mapping the registration point back to the image coordinate system of the superimposed aligned image according to the skeleton registration matrix to obtain a mapping coordinate value of the registration point.

[0130] S135, calculating the distance between each registration point and the surface of the bone surface model according to the mapping coordinate value.

[0131] S136 , scoring the point cloud registration result based on the distances corresponding to all registration points to obtain a scoring value.

[0132] For step S134, the first pose matrix of the first tracking device 40 and the second pose matrix of the registration point can be obtained, and the mapping coordinate value of the registration point can be calculated by combining the skeleton registration matrix, the first pose matrix and the second pose matrix using the registration point mapping formula.

[0133] The registration point mapping formula includes:

[0134] Among them, T spt Characterize the mapping coordinate value, The inverse matrix of the skeleton registration matrix representing the nth point cloud registration, Characterize the inverse matrix of the first pose matrix, T tpt Represents the second pose matrix.

[0135] In step S135, at least one coordinate point can be selected on the surface of each region of the bone surface model as a surface coordinate. For each registration point, the distance between the mapped coordinate value of the registration point and the surface coordinate of the region to which the registration point belongs can be calculated. Distance calculation methods such as Euclidean distance and Mahalanobis distance can be used, which are not limited in this embodiment.

[0136] For step S136, the method of scoring the point cloud registration results according to the distance can be flexibly set. For example, each registration point can be scored according to the distance value, and the scores of all registration points can be used as the total score. Alternatively, the difference between the distance and the preset distance can be calculated and the score can be assigned based on the difference. In this embodiment, there is no limitation.

[0137] In a possible implementation, the distance corresponding to each registration point may be compared with a preset distance threshold, the total number of registration points whose distance is greater than the distance threshold may be counted, and points may be deducted based on the total number to obtain a score value.

[0138] Through the above steps S134 to S136, the fine registration module 702 can quickly obtain the scoring value, and the scoring value is related to the distance between the registration point mapped back to the image coordinate system based on the registration result and the bone surface model, so that the scoring value can reflect (i.e., represent) the degree of fit between the bone surface model and the physical device, making the obtained scoring value more referenceable.

[0139] When the score does not reach the passing threshold, all the registration points that lead to the deduction (which can become deduction points) can be counted and the area where the deduction points are located can be registered as the deduction area. After the deduction area is determined, the registration points in the deduction area are removed and new registration points are selected in the deduction area.

[0140] Then, the control processor 101 continues to perform the next point cloud registration based on the new registration points in the registration deduction area and the registration points without deduction, and continuously performs fine registration until the score reaches the passing threshold.

[0141] It should be noted that the number of newly selected registration points in the registration penalty area is greater than the number of registration points selected in the previous round, that is, the number of registration points in the registration penalty area increases successively. For example, if two registration points were selected in the registration penalty area in the previous round, three registration points can be selected in the same registration penalty area in the current round, and the positions of these three registration points can be different from the positions of the removed registration points.

[0142] In this way, the density of registration points in the registration deduction area is increased to improve the registration precision of the registration deduction area, thereby helping to improve the registration accuracy of the area.

[0143] In step S19 executed by the evaluation module 703, when point cloud registration is performed only once, the skeleton registration matrix of that time is used as the final registration matrix. When point cloud registration is performed twice or more, the product of all skeleton registration matrices is calculated to obtain the final registration matrix.

[0144] The final registration matrix can be expressed as: T RFS_final =T RF_SfI ×T SfI_Surface_1 ×…×T SfI_Surface_n .

[0145] The final registration matrix is ​​the relative transformation relationship between the first tracking device installed on the physical skeleton and the bone surface model. As long as the optical positioning system can track the first tracking device installed on the skeleton, the final registration matrix is ​​used for posture transformation to track the real-time spatial posture T of the physical skeleton. Tracker_Surface , T Tracker_Surface =T Tracker_RF ×T RFS_final .

[0146] Optionally, the coarse alignment module 701, the fine alignment module 702, the evaluation module 703 and the correction module 704 can be set separately or integrated into one unit, namely the processing unit. The embodiment of the present application does not make any specific restrictions on the specific implementation method of the coarse alignment module 701, the fine alignment module 702, the evaluation module 703 and the correction module 704.

[0147] Optionally, the skeleton registration device 70 may further include a storage unit storing a program or instruction. When the coarse registration module 701, the fine registration module 702, the evaluation module 703, and the correction module 704 execute the program or instruction, the skeleton registration device 70 may execute any possible implementation of the skeleton registration device 70 of the embodiment of the present application.

[0148] The skeletal alignment device 70 can be the central control system of the orthopedic surgical robot 10, the orthopedic surgical robot 10, the central control system of the skeletal alignment device, the skeletal alignment device and the server, or it can be a computer device connected to the orthopedic surgical robot 10 for communication, such as a mobile phone, tablet computer, laptop computer and server, etc. The embodiments of the present application do not limit this.

[0149] Based on the same inventive concept as the above-mentioned skeleton registration device 70 , this embodiment further provides a skeleton registration method, which may include the following steps with reference to FIG3 : The skeleton registration method may be executed by the control processor 101 in FIG1 .

[0150] S11, constructing a bone surface model based on the three-dimensional image data of the physical bone, and obtaining a real-scene video image of the area where the physical bone is located in real time, and roughly aligning the bone surface model with the physical bone in the real-scene video image to obtain a superimposed aligned image.

[0151] S13, based on the registration points selected by the user on the physical skeleton, the bone surface model is point cloud registered with the physical skeleton, and the point cloud registration result is scored based on the registration points and the bone surface model.

[0152] S15: Determine whether the score reaches a preset passing threshold. If not, proceed to step S17; if so, proceed to step S19.

[0153] S17: Calculate the registration deduction area, remove the registration points located in the registration deduction area, and display a prompt for the registration deduction area on the superimposed aligned image to instruct the user to reselect registration points in the registration deduction area of ​​the physical skeleton. After step S17, the process returns to step S13.

[0154] S19, completing the registration, and obtaining the final registration matrix based on the registration results.

[0155] The skeletal registration method provided in the embodiments of the present application, when the point cloud registration results are unsatisfactory, reselects the registration points in the registration deduction area and continues to perform point cloud registration, making targeted corrections to the point cloud registration results until the registration results reach the passing threshold, greatly improving the accuracy of skeletal registration. In addition, only one coarse registration is required; if the score does not reach the passing threshold, there is no need to re-perform coarse registration, reducing the number of coarse registrations, simplifying the registration process and registration complexity, thereby shortening the registration time and improving registration efficiency.

[0156] It should be noted that the basic principles, technical effects and further implementation methods of the skeletal registration method provided in the embodiment of the present application are the same as those of the above-mentioned embodiment of the skeletal registration device 70. For the sake of brief description, the parts not mentioned in this embodiment will not be repeated here, and reference may be made to the corresponding contents in the above-mentioned embodiment of the skeletal registration device 70.

[0157] An electronic device 80 is provided below, which can be the central control system of the orthopedic surgical robot 10, the orthopedic surgical robot 10, the central control system of the skeletal registration device, the skeletal registration device and the server, or it can be a computer device connected to the orthopedic surgical robot 10, such as a mobile phone, a tablet computer, a laptop computer and a server, etc. The electronic device 80 is shown in Figure 8 and can implement the method of the above-mentioned skeletal registration device 70, that is, the skeletal registration method; specifically, the electronic device 80 includes a processor, a memory and a communication module connected via a system bus. The processor can be a CPU. The memory is used to store one or more programs, and when one or more programs are executed by the processor, the method of the skeletal registration device 70 provided in the above embodiment, that is, the skeletal registration method, is executed. The memory, processor and communication module are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines.

[0158] The memory is used to store programs or data. The memory may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM).

[0159] The processor is used to read / write data or programs stored in the memory, such as the programs and data implemented by the functions of the above-mentioned LVDS, TDC and processing unit, and execute the method of the skeleton alignment device 70 provided in any embodiment of the present application.

[0160] The communication module is used to establish a communication connection between the electronic device 80 and other communication terminals through a network, and to send and receive data through the network.

[0161] It should be understood that the structure shown in FIG8 is merely a schematic structural diagram of the electronic device 80 , and the electronic device 80 may further include more or fewer components than those shown in FIG8 , or have a configuration different from that shown in FIG8 .

[0162] An embodiment of the present application also provides an electronic device 80, including a skeletal registration device 70, which includes a processor and a memory, and the memory is used to store one or more programs; when the one or more programs are executed by the processor, the steps of the skeletal registration device described in any possible implementation method in the embodiment of the device of the present application are implemented.

[0163] An embodiment of the present application also provides an electronic device 80, including a processor and a memory, wherein the memory is used to store one or more programs; when the one or more programs are executed by the processor, the skeletal registration method described in any possible implementation method in the method embodiment of the present application is implemented.

[0164] The present application provides a computer-readable storage medium comprising a computer program (also referred to as code or instructions) that, when executed, causes a computer to perform the skeletal registration method described in any possible implementation of the method embodiments of the present application. The storage medium may comprise memory, flash memory, registers, or a combination thereof.

[0165] In summary, the skeleton registration device and electronic device provided in the embodiments of the present application have the following beneficial effects:

[0166] (1) Fewer registration points are required. The coarse registration method based on the overlay alignment of augmented reality display technology replaces the traditional coarse registration method using coarse registration points. This method reduces the number of coarse registration points and is more accurate than coarse registration. It also reduces the number of registration points required for fine registration, thus shortening the registration time.

[0167] (2) The registration result is more accurate. The traditional method of coarse registration plus fine registration is highly dependent on user experience. If the coarse registration points collected by the user deviate from the marker points on the three-dimensional skeleton model, a higher registration error will be caused. In the skeleton registration device 70 provided in the embodiment of the present application, the user can directly observe the deviation between the position of the skeleton surface model and the target position through enhanced display technology, and the initial position of the placed skeleton surface model is more accurate than the coarse registration.

[0168] (3) Reduce the probability of registration failure. The traditional coarse registration plus fine registration technology is very dependent on the accuracy of coarse registration, and the selection of actual coarse registration points needs to correspond one-to-one with the coarse registration points marked on the three-dimensional model. If the coarse registration deviation is large, it is easy to cause registration failure. In the embodiment of the present application, there is no need to perform point-based coarse registration. The fine registration allows the collected registration points to only cover a certain feature area, without the need for one-to-one correspondence with the markings on the bone surface model. In addition, when the results are not ideal, continuous fine registration is supported, and registration points are added in the area with large errors for correction, thereby preventing the occurrence of registration failure.

[0169] (4) Reduce repeated registration and registration time. Using enhanced display technology, users can intuitively observe the degree of fit between the bone surface model and the physical bone at each stage, and judge whether the registration is correct in advance. If it is not correct, the inaccurate registration points are removed and new registration points are collected in the area, and then the registration is re-performed without having to repeat the entire registration process.

[0170] (5) Solve the problem of being limited by a small exposure area. This system uses enhanced display technology to overlay a 3D bone model on the real physical part, allowing the user to intuitively perceive the orientation of the bone. Combined with the reference position overlaid on the bone surface model, the bone surface model is aligned with the orientation of the physical part. Based on the correct alignment, surface precision registration can be performed in a smaller exposure area.

[0171] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0172] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0173] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0174] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A skeleton registration device, comprising a coarse registration module, a fine registration module, an evaluation module, and a correction module; The coarse registration module is configured to construct a bone surface model based on the three-dimensional image data of the physical bone, obtain a real-scene video image of the area where the physical bone is located in real time, and coarsely register the bone surface model with the physical bone in the real-scene video image to obtain a superimposed aligned image; The fine registration module is configured to perform point cloud registration between the bone surface model and the physical bone according to the registration points selected by the user on the physical bone, and score the point cloud registration result in combination with the registration points and the bone surface model; The evaluation module is configured to determine whether the score value reaches a preset passing threshold, and if so, complete the registration and obtain a final registration matrix based on the registration result; The correction module is configured to calculate the registration deduction area if not, eliminate the registration points located in the registration deduction area, and display the registration deduction area on the superimposed alignment image to instruct the user to reselect the registration points on the registration deduction area of the solid skeleton, and return to the fine registration module to execute the step of performing point cloud registration between the bone surface model and the solid skeleton according to the registration points selected by the user on the solid skeleton, until the score value reaches a preset passing threshold.

2. The skeleton registration device according to claim 1, wherein: The number of the registration points is set to: p<40.

3. The skeleton registration device according to claim 2, wherein: The number of the registration points is set to: 15≤p≤30.

4. The skeleton registration device according to claim 1, wherein: The point cloud registration result includes a skeleton registration matrix; The fine registration module is configured to score the point cloud registration result by combining the registration points and the bone surface model, including: For each of the registration points, mapping the registration point back to the image coordinate system of the superimposed alignment image according to the skeletal registration matrix to obtain a mapping coordinate value of the registration point; Calculating the distance between each registration point and the surface of the bone surface model according to the mapping coordinate value; Based on the distances corresponding to all the registration points, the point cloud registration result is scored to obtain a score value.

5. The skeleton registration device according to claim 4, wherein: A first tracking device is installed on the physical skeleton; The fine registration module is configured to map the registration points back to the image coordinate system of the superimposed aligned image according to the skeletal registration matrix to obtain the mapping coordinate values of the registration points, including: Obtaining a first pose matrix of the first tracking device and a second pose matrix of the registration point, combining the skeletal registration matrix, the first pose matrix, and the second pose matrix, and using a registration point mapping formula to calculate a mapping coordinate value of the registration point; The registration point mapping formula includes: Among them, T spt Characterize the mapping coordinate value, The inverse matrix of the skeleton registration matrix representing the nth point cloud registration, Characterize the inverse matrix of the first pose matrix, T tpt Represents the second pose matrix.

6. The skeleton registration device according to claim 4, wherein: The fine registration module is configured to score the point cloud registration result based on the distances corresponding to all the registration points to obtain a score value, including: The distance corresponding to each registration point is compared with a preset distance threshold, the total number of registration points whose distance is greater than the distance threshold is counted, and points are deducted based on the total number to obtain a score value.

7. The skeleton registration device according to any one of claims 1 to 6, wherein: The coarse registration module is configured to coarsely register the bone surface model with the physical bone in the real-scene video image to obtain a superimposed aligned image, including: The bone surface model is superimposed on the real-scene video image by using augmented reality technology, and the bone surface model is aligned and overlapped with the physical bone in the real-scene video image to obtain a superimposed aligned image.

8. The skeleton registration device according to claim 7, wherein: After the step of constructing a bone surface model based on the three-dimensional image data of the physical bone and before the step of coarsely registering the bone surface model with the physical bone in the real-scene video image to obtain a superimposed aligned image, the coarse registration module is further configured to: marking a plurality of feature points on the bone surface model; After the step of coarsely registering the bone surface model with the physical bone in the real-scene video image to obtain a superimposed aligned image, and before the step of performing point cloud registration between the bone surface model and the physical bone according to the registration points selected by the user on the physical bone, the coarse registration module is further configured to: The feature points of the aligned and superimposed bone surface models are used as recommended positions and displayed in the superimposed alignment image to instruct the user to select registration points on the physical bone according to the recommended positions.

9. The skeleton registration device according to claim 8, wherein: A first tracking device is installed on the physical skeleton; The fine registration module is configured to perform point cloud registration between the bone surface model and the physical bone according to the registration points selected by the user on the physical bone, including: After obtaining the superimposed alignment image, obtaining a relative pose matrix between the first tracking device and the bone surface model; Obtaining registration points selected by a user on the physical skeleton, and using an ICP algorithm to align the bone surface model with the physical skeleton according to the registration points to obtain a precise registration matrix; A skeleton registration matrix is calculated according to the relative pose matrix and the fine registration matrix.

10. The skeleton registration device according to claim 9, wherein: The evaluation module is configured to obtain a final registration matrix according to the registration result, including: The product of all the skeleton registration matrices is calculated to obtain the final registration matrix.

11. The skeleton registration device according to claim 9, wherein: The coarse registration module is configured to align the bone surface model with the physical bone in the real-scene video image, comprising: Binding the model pose matrix of the bone surface model to the probe pose matrix of a preset probe; wherein a second tracking device is installed on the probe; Based on the feature points, reference positions are drawn on the bone surface model to instruct the user to align the bone surface model with the physical bone in the real-life video image by moving the probe based on the reference positions; wherein the reference positions include feature points and extension lines of feature axes.

12. An electronic device comprising the skeleton registration device according to any one of claims 1 to 11.

Images