Image display method and system for bone grinding state, electronic device and storage medium

Abstract

This invention provides a method, system, electronic device, and storage medium for displaying images of bone grinding states, relating to the field of image processing technology. It includes: establishing a three-dimensional model of a target bone based on a user's bone scan image, and obtaining the three-dimensional model data of the target bone; determining the position information of a first target registration point in the camera coordinate system of a surgical robot; determining an initial registration matrix based on the position information and the three-dimensional model data; determining a secondary registration matrix based on the initial registration matrix and a second target registration point, to obtain a target pose matrix based on the secondary registration matrix; transforming the bone prosthesis model into the coordinate system of the three-dimensional model of the target bone based on the pose matrix of the bone prosthesis model; determining the grinding state of the target grinding point in the coordinate system of the three-dimensional model based on the distance from the target grinding point on the three-dimensional model to the bone prosthesis model; and displaying the target grinding point on the three-dimensional model with a preset color corresponding to the grinding state.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method and system for displaying images of a bone grinding state, an electronic device, and a storage medium. Background Technology

[0002] In joint replacement surgery, because the contact surface between the joint prosthesis and the patient's joint area is generally non-planar, bone grinding is necessary. The degree of matching during bone grinding directly determines the precision of the joint replacement surgery. Therefore, real-time monitoring of the bone grinding status is crucial.

[0003] In current methods for displaying the state of bone grinding, the geometric model of the removed bone is obtained by performing three-dimensional reconstruction of the removed bone, and then rendering these models to obtain the rendered geometric model of the removed bone.

[0004] However, the inventors discovered that the geometric model with the bones removed contains a large amount of facet data, and the transmission and rendering of such a large amount of facet data requires a lot of resources and is quite time-consuming, so the timeliness and accuracy of facet data rendering cannot be guaranteed.

[0005] The content in the background section is merely technology known to the public and does not necessarily represent existing technology in this field. Summary of the Invention

[0006] According to one aspect of the present invention, an image display method for bone grinding state is provided, comprising: establishing a three-dimensional model of a target bone based on a user's bone scan image; obtaining three-dimensional model data of the target bone; determining the position information of a first target registration point in the camera coordinate system of a surgical robot, wherein the first target registration point is determined based on a planned registration point in the bone scan image; determining an initial registration matrix based on the position information and the three-dimensional model data; determining a secondary registration matrix based on the initial registration matrix and a second target registration point, to obtain a target pose matrix based on the secondary registration matrix, wherein the target pose matrix includes at least a bone prosthesis model pose matrix; converting the bone prosthesis model to the coordinate system of the three-dimensional model of the target bone based on the bone prosthesis model pose matrix; determining the grinding state of the target grinding point in the coordinate system of the three-dimensional model based on the distance from the target grinding point on the three-dimensional model to the bone prosthesis model, wherein the grinding state includes at least a state to be ground, a ground state, and an over-ground state; and displaying the target grinding point on the three-dimensional model with a preset color corresponding to the grinding state.

[0007] According to some embodiments of the present invention, determining the initial registration matrix based on location information and three-dimensional model data includes: calculating the location information and three-dimensional model data based on a preset registration algorithm to obtain a registration matrix between the first target registration point and the planned registration point; and determining the registration matrix with the smallest error between the first target registration point and the planned registration point as the initial registration matrix.

[0008] According to some embodiments of the present invention, establishing a three-dimensional model of a target bone based on a user's bone scan image further includes: determining target trimming rules based on the bone region to be ground; and trimming the three-dimensional model of the target bone according to the target trimming rules.

[0009] According to some embodiments of the present invention, in the coordinate system of the three-dimensional model, determining the grinding state of the target grinding point based on the distance from the target grinding point of the three-dimensional model to the bone prosthesis model includes: when the target grinding point is located inside the bone prosthesis model, determining the target grinding point as a state to be ground; when the target grinding point is located outside the bone prosthesis model and the distance between the target grinding point and the bone prosthesis model is not greater than a preset threshold, determining the target grinding point as a state already ground; and when the target grinding point is located outside the bone prosthesis model and the distance between the target grinding point and the bone prosthesis model is greater than a preset threshold, determining the target grinding point as a state over-ground.

[0010] According to another aspect of the present invention, an image display system for bone grinding states is provided, comprising a model processing module, a registration processing module, a state determination module, and a state display module. The model processing module establishes a three-dimensional model of the target bone based on a user's bone scan image and obtains the three-dimensional model data of the target bone. The registration processing module determines the position information of a first target registration point in the camera coordinate system of the surgical robot, wherein the first target registration point is determined based on a planned registration point in the bone scan image; an initial registration matrix is ​​determined based on the position information and the three-dimensional model data; a secondary registration matrix is ​​determined based on the initial registration matrix and the second target registration point, to obtain a target pose matrix, the target pose matrix including at least the pose matrix of the bone prosthesis model. The state determination module transforms the bone prosthesis model into the coordinate system of the three-dimensional model of the target bone according to the pose matrix of the bone prosthesis model; in the coordinate system of the three-dimensional model, the grinding state of the target grinding point is determined based on the distance from the target grinding point on the three-dimensional model to the bone prosthesis model, the grinding state including at least a state to be ground, a ground state, and an over-ground state. The status display module presents the target grinding point on the 3D model with a preset color corresponding to the grinding status.

[0011] According to some embodiments of the present invention, the registration processing module calculates the position information and three-dimensional model data based on a preset registration algorithm to obtain the registration matrix between the first target registration point and the planned registration point; the registration processing module determines the registration matrix with the smallest error between the first target registration point and the planned registration point in the registration matrix as the initial registration matrix.

[0012] According to some embodiments of the present invention, the model processing module determines the target trimming rules based on the bone region to be ground; the model processing module trims the three-dimensional model of the target bone according to the target trimming rules.

[0013] According to some embodiments of the present invention, when the target grinding point is located inside the bone prosthesis model, the state determination module determines the target grinding point to be in a grinding state; when the target grinding point is located outside the bone prosthesis model and the distance between the target grinding point and the bone prosthesis model is not greater than a preset threshold, the state determination module determines the target grinding point to be in a ground state; when the target grinding point is located outside the bone prosthesis model and the distance between the target grinding point and the bone prosthesis model is greater than a preset threshold, the state determination module determines the target grinding point to be in an over-grinding state.

[0014] According to another aspect of the present invention, an electronic device is also provided. The electronic device includes: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement the image display method for bone grinding states as described above.

[0015] According to another aspect of the present invention, a non-volatile computer-readable storage medium is also provided. The storage medium stores a computer program that, when executed by a processor, enables the image display method for the bone grinding state as described above.

[0016] Beneficial effects

[0017] This invention enables initial and secondary registration using a 3D model of the target bone and its data. This initial and secondary registration achieves precise alignment between the user's physical bone surface region and the space of the bone scan image. Therefore, this invention can transform the bone prosthesis model into the same reference coordinate system as the target bone's 3D model based on the pose matrix obtained from the registration. This invention also converts the determination of the grinding state at the target grinding point into a distance determination from the target grinding point to the bone prosthesis model to obtain the grinding state. Furthermore, this invention can visually present the grinding state in the image using different preset colors.

[0018] Compared with existing technologies, this invention eliminates the need for 3D reconstruction and rendering of the removed bone, thus avoiding the processing of large amounts of polygon data and saving significant computational resources and time. This invention features low computational load, fast computation, high accuracy, and intuitive display. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic diagram of the structure of a three-dimensional model of an existing unicompartmental femoral prosthesis is shown;

[0021] Figure 2 A schematic diagram showing the matching of an existing unicompartmental femoral prosthesis with a target femur is shown;

[0022] Figure 3 A flowchart illustrating an embodiment of the image display method of the present invention is shown;

[0023] Figure 4 This diagram illustrates yet another flow chart of the image display method according to an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of a skeletal prosthesis model according to an embodiment of the present invention is shown;

[0025] Figure 6 A schematic diagram of the target grinding point according to an embodiment of the present invention is shown;

[0026] Figure 7 A schematic diagram illustrating different grinding states using preset colors, representing an embodiment of the present invention;

[0027] Figure 8 This diagram illustrates yet another flow chart of the image display method according to an embodiment of the present invention;

[0028] Figure 9 A schematic diagram of the structure of an image display system according to an embodiment of the present invention is shown.

[0029] Explanation of reference numerals in the attached figures:

[0030] Image display system 1; model processing module 10; registration processing module 20; status determination module 30; status display module 40. Detailed Implementation

[0031] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that the invention will be thorough and complete, and the concept of the exemplary embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0032] The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of these specific details, or other methods, components, materials, devices, etc. In these cases, well-known structures, methods, devices, implementations, materials, or operations will not be shown or described in detail.

[0033] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0034] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, rather than to describe a specific order.

[0035] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] According to one aspect of the present invention, an image display method for displaying the grinding state of a bone is provided, for displaying the grinding state of a target bone. The grinding state of the target bone can be a state to be ground, a state that has been ground, or a state that has been over-ground.

[0037] As an example, this image display method can be applied to joint replacement scenarios based on surgical robots. Joint replacement includes, but is not limited to, total hip replacement, hemi-hip replacement, total knee replacement, patellofemoral replacement, and unicompartmental joint replacement, etc., and this invention does not limit these applications.

[0038] For example, taking unicompartmental arthroplasty as an application scenario, Figure 1 A schematic diagram of the structure of a three-dimensional model of an existing unicompartmental femoral prosthesis is shown; Figure 2 A schematic diagram showing the matching of an existing unicompartmental femoral prosthesis with a target femur is shown.

[0039] like Figure 1 As shown, this unicompartmental femoral prosthesis has an osteotomy plane and a spherical curved surface. Figure 2 As shown, to ensure the fit of unicompartmental femoral replacement, the target femur also requires a spherical surface to match the unicompartmental femoral prosthesis. Therefore, the target femur needs to be ground to obtain this spherical surface. The image display method for the bone grinding state provided by this invention, based on image processing, can display the grinding state of the target femur in real time in image form.

[0040] It is understood that the above description is merely an exemplary application scenario of the present invention, and not an application of the present invention in surgical operations based on surgical robots. The present invention only provides an image presentation method for the grinding state of the target bone, and has no interaction with the surgical operation, nor does it provide any guidance or assistance for the surgical operation.

[0041] Figure 3 A flowchart illustrating an embodiment of the image display method of the present invention is shown. Figure 3 As shown, the image display method may include steps S100-S700. For example, the image display method may be performed by an image display system for bone grinding states that has computational capabilities.

[0042] According to the example embodiment, in step S100, the image display system builds a three-dimensional model of the target bone based on the user's bone scan image and obtains the three-dimensional model data of the target bone.

[0043] For example, a user's bone scan image can be a CT scan image (Computed Tomography). The image display system acquires this CT scan image and performs segmentation and reconstruction processing on the target bone to obtain a three-dimensional model of the target bone. Based on this three-dimensional model of the target bone, the image display system can then obtain the three-dimensional model data of the target bone.

[0044] For example, the target bone can be any type of bone that needs to be ground down, such as the femur or tibia, and the present invention does not limit this.

[0045] For ease of description, the femur will be used as an example in the following text.

[0046] In step S200, the image display system determines the position information of the first target registration point in the camera coordinate system of the surgical robot. The first target registration point is determined based on the planned registration point in the bone scan image.

[0047] For example, the planned registration points are a certain number (e.g., 3) of bony landmarks pre-selected by the user in the CT scan image. Based on the planned registration points in the CT scan image, the image display system can determine the corresponding actual physical location of the point in the user's physical bone surface region, i.e., the first target registration point.

[0048] The image display system uses the surgical robot's probes to acquire the position information of the first target registration point within the surgical robot's camera coordinate system over the user's physical bone surface area. The surgical robot's camera coordinate system is a crucial component, used to describe and locate the physical positions of surgical instruments and the user's body.

[0049] In step S300, the image display system determines the initial registration matrix based on the position information and the three-dimensional model data.

[0050] For example, since the image display system knows the position information of the first target registration point in the camera coordinate system of the surgical robot, it can perform image registration between the first target registration point and the planned registration point using a preset registration algorithm. This spatially aligns the 3D model of the target skeleton with the user's physical bone surface area, facilitating subsequent image analysis and processing.

[0051] Figure 4 This diagram illustrates yet another flow chart of the image display method according to an embodiment of the present invention.

[0052] Optionally, such as Figure 4 As shown, step S300 may also include steps S310-S320.

[0053] In step S310, the image display system calculates the position information and three-dimensional model data based on a preset registration algorithm to obtain the registration matrix between the first target registration point and the planned registration point.

[0054] In step S320, the image display system determines the initial registration matrix as the registration matrix that minimizes the error between the first target registration point and the planned registration point in the registration matrix.

[0055] For example, the image display system employs a rigid registration algorithm, which calculates the registration matrix using the 3D model data of the target skeleton based on the position information of the first target registration point in the camera coordinate system of the surgical robot. This calculation yields the registration matrix that minimizes the error between the first target registration point and the planned registration point, and this registration matrix is ​​then determined as the initial registration matrix.

[0056] With this configuration, the present invention establishes an alignment relationship between the actual physical points (i.e., the first target registration point) of the user's physical bone surface region and the planned points (i.e., the planned registration points) on the 3D model by determining the position information of the first target registration point in the camera coordinate system of the surgical robot. This enables initial registration of the user's physical bone surface region with the spatial alignment of the bone scan image.

[0057] In step S400, the image display system determines a secondary registration matrix based on the initial registration matrix and the second target registration point, so as to obtain the target pose matrix according to the secondary registration matrix.

[0058] For example, the second target registration points are a certain number of actual physical points on the user's physical bone surface area. The image display system can acquire a certain number (e.g., 20-50) of second target registration points on the user's physical bone surface area using the probe of the surgical robot. This certain number of second target registration points is dispersed and uniformly distributed on the user's physical bone surface area.

[0059] The image display system registers a certain number of second target registrations based on the initial registration matrix to initially achieve spatial alignment between the user's physical bone surface region and the three-dimensional model of the target bone.

[0060] The image display system then uses the 3D model data of the target skeleton for calculations, and the secondary registration matrix can be calculated through an iterative nearest-point algorithm. For example, the iterative formula for the secondary registration matrix can be:

[0061] p'i = Rpi + t;

[0062] Where R is the rotation matrix of the secondary registration matrix; pi is the second target registration point acquired; and t is the translation matrix of the secondary registration matrix.

[0063] For example, an image display system can transform the source point cloud pi based on the iterative nearest point algorithm to obtain the target point p'i in the new point cloud. Through successive iterations, a secondary registration matrix with the required accuracy can be obtained.

[0064] According to an example embodiment, the image display system can convert the secondary registration matrix into a 4*4 homogeneous transformation matrix, and the target pose matrix can be obtained from the transformation matrix.

[0065] For example, the target pose matrix includes at least the pose matrix of the skeletal prosthesis model, the femur pose matrix, and the tibia pose matrix.

[0066] For example, the skeletal prosthesis pose matrix can be The femoral pose matrix can be Tibial pose matrix can be

[0067] Where T is the transformation matrix; P is the skeletal prosthesis model; I is the skeletal prosthesis model image, i.e., the pose of the skeletal prosthesis model in the image; F1 is the femoral tracker; I1 is the femoral scan image, i.e., the pose of the femoral scan image coordinates in the user coordinate base coordinate system; F2 is the tibia tracker; I2 is the tibia scan image, i.e., the pose of the tibia scan image coordinates in the user coordinate base coordinate system.

[0068] In step S500, the image display system converts the skeletal prosthesis model into the coordinate system of the three-dimensional model of the target bone based on the pose matrix of the skeletal prosthesis model.

[0069] For example, the bone prosthesis model is configured to correspond to the target bone prosthesis. The target bone prosthesis is determined based on the user's surgical plan. This surgical plan may include configuration information such as the required implantation location, size, and angle of the bone prosthesis.

[0070] For example, Figure 5 A schematic diagram of a skeletal prosthesis model according to an embodiment of the present invention is shown. Figure 5 As shown, this skeletal prosthesis model can be used for Figure 1 The cutting model shown corresponds to the unicompartmental femoral prosthesis. This cutting model is consistent with the spherical surface of the unicompartmental femoral prosthesis.

[0071] After determining the skeletal prosthesis model, the image display system can achieve spatial alignment between the skeletal prosthesis model and the target bone's 3D model based on the solved pose matrix of the skeletal prosthesis model. In other words, the image display system transforms the skeletal prosthesis model into the coordinate system of the target bone's 3D model, thus placing the skeletal prosthesis model and the target bone's 3D model in the same reference coordinate system.

[0072] In step S600, the image display system determines the grinding state of the target grinding point in the coordinate system of the three-dimensional model based on the distance from the target grinding point on the three-dimensional model to the bone prosthesis model. The grinding state includes at least the state to be ground, the state that has been ground, and the state that has been over-ground.

[0073] Figure 6 A schematic diagram of the target grinding point according to an embodiment of the present invention is shown.

[0074] For example, such as Figure 6 As shown, the target grinding point is a point on the 3D model where the grinding operation is to be performed. The target grinding point can be customized according to user needs, and this invention does not impose any restrictions on it.

[0075] Since the 3D model of the bone prosthesis model and the 3D model of the target bone are in the same reference coordinate system, it can be known that the point of intersection between the 3D model of the bone prosthesis model and the 3D model of the target bone in this reference coordinate system is the target grinding point.

[0076] Therefore, by determining the distance between the target grinding point and the bone prosthesis model, this invention can determine the positional relationship between the target grinding point and the intersecting area. Based on this positional relationship, the image display system can determine whether the target grinding point is in a state to be ground, a state that has been ground, or a state that has been over-ground.

[0077] In step S700, the image display system presents the target grinding point on the three-dimensional model in a preset color corresponding to the grinding state.

[0078] For example, different grinding states correspond to different preset colors.

[0079] Figure 7 This diagram illustrates different grinding states presented in preset colors according to an embodiment of the present invention.

[0080] For example, such as Figure 7 As shown, the preset color corresponding to the state to be ground can be the first preset color (such as green); the preset color corresponding to the state that has been ground can be the second preset color (such as white); and the preset color corresponding to the state that has been over-ground can be the third preset color (red).

[0081] After determining the grinding state of the target grinding point, the image display system displays the corresponding color on the image of the 3D model, thus providing a more intuitive view of the grinding state of the target bone.

[0082] Through the above embodiments, this invention establishes a three-dimensional model of the target bone based on the user's bone scan image, and obtains the three-dimensional model data of the target bone. Furthermore, this invention determines the position information of the first target registration point in the camera coordinate system of the surgical robot, and determines the initial registration matrix based on the position information and the three-dimensional model data. Based on the initial registration matrix and the second target registration point, this invention determines a secondary registration matrix, and obtains the target pose matrix based on the secondary registration matrix. This allows the bone prosthesis model to be transformed into the coordinate system of the three-dimensional model of the target bone based on the pose matrix of the bone prosthesis model. Finally, in the coordinate system of the three-dimensional model, this invention determines the grinding state of the target grinding point based on the distance from the target grinding point on the three-dimensional model to the bone prosthesis model, and then presents the target grinding point on the three-dimensional model with a preset color corresponding to the grinding state.

[0083] This invention enables initial and secondary registration using a 3D model of the target bone and its data. This initial and secondary registration achieves precise alignment between the user's physical bone surface region and the space of the bone scan image. Therefore, this invention can transform the bone prosthesis model into the same reference coordinate system as the target bone's 3D model based on the pose matrix obtained from the registration. This invention also converts the determination of the grinding state at the target grinding point into a distance determination from the target grinding point to the bone prosthesis model to obtain the grinding state. Furthermore, this invention can visually present the grinding state in the image using different preset colors.

[0084] Compared with existing technologies, this invention eliminates the need for 3D reconstruction and rendering of the removed bone, thus avoiding the processing of large amounts of polygon data and saving significant computational resources and time. This invention features low computational load, fast computation, high accuracy, and intuitive display.

[0085] Figure 8 This diagram illustrates yet another flow chart of the image display method according to an embodiment of the present invention.

[0086] Optionally, such as Figure 8 As shown, step S100 may also include steps S110-S120.

[0087] In step S110, the image display system determines the target trimming rules based on the bone area to be ground.

[0088] In step S120, the image display system clips the three-dimensional model of the target skeleton according to the target clipping rules.

[0089] For example, the area of ​​the bone to be ground can be a custom-defined grinding area based on user requirements. The image display system can determine the corresponding target clipping rules based on the grinding area. Then, the image display system can clip some unnecessary areas of the target bone's 3D model according to these target clipping rules. With this setup, the present invention can optimize the number of calculation points by clipping the target bone's 3D model, thereby further improving the computational efficiency of the present invention.

[0090] For example, taking unicompartmental arthroplasty as an application scenario, in the case of medial condyle arthroplasty, the target clipping rule is to retain the 3D model data of the medial region of the femoral condyle center and clip the 3D model data of the lateral region of the femoral condyle center. In the case of lateral condyle arthroplasty, the target clipping rule is to retain the 3D model data of the lateral region of the femoral condyle center and clip the 3D model data of the medial region of the femoral condyle center.

[0091] Optionally, in step S600, if the target grinding point is located inside the bone prosthesis model, the image display system determines that the target grinding point is in a state to be ground.

[0092] When the target grinding point is located outside the bone prosthesis model and the distance between the target grinding point and the bone prosthesis model is not greater than a preset threshold, the image display system determines that the target grinding point has been ground.

[0093] If the target grinding point is located outside the bone prosthesis model and the distance between the target grinding point and the bone prosthesis model is greater than a preset threshold, the image display system determines that the target grinding point is in an over-grinding state.

[0094] For example, if the target grinding point is located inside the bone prosthesis model, and the target grinding point is located in the intersection area of ​​the bone prosthesis model and the three-dimensional model of the target bone, then the image display system determines that the target grinding point is in a state to be ground.

[0095] The preset threshold can be customized according to user needs. For example, the preset threshold can be 1.5mm.

[0096] If the target grinding point is located outside the bone prosthesis model, and the distance between the target grinding point and the outside of the bone prosthesis model is less than or equal to 1.5 mm, indicating that the three-dimensional model of the bone prosthesis model and the target bone has no intersecting area, and the grinding error of the target grinding point is within the user's acceptable range, then the image display system determines that the target grinding point is in a ground state.

[0097] If the target grinding point is located outside the bone prosthesis model and the distance between the target grinding point and the outside of the bone prosthesis model is greater than 1.5mm, indicating that the three-dimensional model representing the bone prosthesis model and the target bone has no intersecting area, and the grinding error of the target grinding point is within the user's unacceptable range, then the image display system determines that the target grinding point is in an over-grinding state.

[0098] With this configuration, the present invention can determine the internal or external distance relationship between the target grinding point and the bone prosthesis model under the same reference coordinate system by judging the distance between the target grinding point and the bone prosthesis model. Furthermore, the present invention can determine whether the grinding error is within the user's acceptable range based on the specific distance between the target grinding point and the external part of the bone prosthesis model, thereby accurately determining whether over-grinding has occurred.

[0099] According to another aspect of the present invention, an image display system for displaying the grinding state of a target bone is provided. The grinding state of the target bone can be a state to be ground, a ground state, or an over-ground state.

[0100] Figure 9 A schematic diagram of the structure of an image display system according to an embodiment of the present invention is shown. Figure 9 As shown, the image display system 1 includes a model processing module 10, a registration processing module 20, a state determination module 30, and a state display module 40.

[0101] According to the example embodiment, the model processing module 10 establishes a three-dimensional model of the target bone based on the user's bone scan image, and obtains the three-dimensional model data of the target bone.

[0102] For example, a user's bone scan image can be a CT scan image (Computed Tomography). The model processing module 10 acquires the CT scan image and performs segmentation and reconstruction processing on the target bone to obtain a three-dimensional model of the target bone. Based on the three-dimensional model of the target bone, the model processing module 10 can then obtain three-dimensional model data of the target bone.

[0103] For example, the target bone can be any type of bone that needs to be ground down, such as the femur or tibia, and the present invention does not limit this.

[0104] For ease of description, the femur will be used as an example in the following text.

[0105] The registration processing module 20 determines the position information of the first target registration point in the camera coordinate system of the surgical robot. The first target registration point is determined based on the planned registration points in the bone scan image.

[0106] For example, the planned registration points are a certain number (e.g., 3) of bony landmarks pre-selected by the user in the CT scan image. The registration processing module 20 can determine the corresponding real physical location of the planned registration points in the user's physical bone surface region based on the planned registration points in the CT scan image, i.e., the first target registration point.

[0107] The registration processing module 20 can obtain the position information of the first target registration point in the camera coordinate system of the surgical robot in the user's physical bone surface area through the probe of the surgical robot. The camera coordinate system of the surgical robot is an important component of the surgical robot, used to describe and locate the physical positions of surgical instruments and the user's body.

[0108] The registration processing module 20 determines the initial registration matrix based on the position information and the 3D model data.

[0109] For example, since the registration processing module 20 knows the position information of the first target registration point in the camera coordinate system of the surgical robot, it can perform image registration between the first target registration point and the planned registration point using a preset registration algorithm. This allows for spatial registration and alignment of the three-dimensional model of the target skeleton with the user's physical bone surface region, facilitating subsequent image analysis and processing.

[0110] Optionally, the registration processing module 20 calculates the location information and three-dimensional model data based on a preset registration algorithm to obtain the registration matrix between the first target registration point and the planned registration point.

[0111] The registration processing module 20 determines the registration matrix with the smallest error between the first target registration point and the planned registration point in the registration matrix as the initial registration matrix.

[0112] For example, the registration processing module 20 employs a rigid registration algorithm, using the 3D model data of the target skeleton to calculate the registration matrix that minimizes the error between the first target registration point and the planned registration point, based on the position information of the first target registration point in the camera coordinate system of the surgical robot. This registration matrix is ​​then determined as the initial registration matrix.

[0113] With this configuration, the present invention establishes an alignment relationship between the actual physical points (i.e., the first target registration point) of the user's physical bone surface region and the planned points (i.e., the planned registration points) on the 3D model by determining the position information of the first target registration point in the camera coordinate system of the surgical robot. This enables initial registration of the user's physical bone surface region with the spatial alignment of the bone scan image.

[0114] The registration processing module 20 determines the secondary registration matrix based on the initial registration matrix and the second target registration point, so as to obtain the target pose matrix according to the secondary registration matrix.

[0115] For example, the second target registration points are a certain number of actual physical points in the user's physical bone surface area. The registration processing module 20 can collect a certain number (e.g., 20-50) of the second target registration points in the user's physical bone surface area using the probe of the surgical robot. This certain number of second target registration points is dispersed and uniformly distributed in the user's physical bone surface area.

[0116] The registration processing module 20 registers a certain number of second target registrations based on the initial registration matrix to initially achieve spatial alignment between the user's physical bone surface region and the three-dimensional model of the target bone.

[0117] The registration processing module 20 then uses the 3D model data of the target skeleton to perform calculations. The secondary registration matrix can be calculated using an iterative nearest-point algorithm. For example, the iterative formula for the secondary registration matrix can be:

[0118] p'i = Rpi + t;

[0119] Where R is the rotation matrix of the secondary registration matrix; pi is the second target registration point acquired; and t is the translation matrix of the secondary registration matrix.

[0120] For example, the registration processing module 20 performs a transformation on the source point cloud pi based on the iterative nearest point algorithm to obtain the new point cloud p'i target point. After successive iterations, a secondary registration matrix with the required accuracy can be obtained.

[0121] According to the example embodiment, the registration processing module 20 can convert the secondary registration matrix into a 4*4 homogeneous transformation matrix, and the target pose matrix can be obtained from the transformation matrix.

[0122] For example, the target pose matrix includes at least the pose matrix of the skeletal prosthesis model, the femur pose matrix, and the tibia pose matrix.

[0123] For example, the skeletal prosthesis pose matrix can be The femoral pose matrix can be Tibial pose matrix can be

[0124] Where T is the transformation matrix; P is the skeletal prosthesis model; I is the skeletal prosthesis model image, i.e., the pose of the skeletal prosthesis model in the image; F1 is the femoral tracker; I1 is the femoral scan image, i.e., the pose of the femoral scan image coordinates in the user coordinate base coordinate system; F2 is the tibia tracker; I2 is the tibia scan image, i.e., the pose of the tibia scan image coordinates in the user coordinate base coordinate system.

[0125] The state determination module 30 converts the skeletal prosthesis model into the coordinate system of the target bone's three-dimensional model based on the pose matrix of the skeletal prosthesis model.

[0126] For example, the bone prosthesis model is configured to correspond to the target bone prosthesis. The target bone prosthesis is determined based on the user's surgical plan. This surgical plan may include configuration information such as the required implantation location, size, and angle of the bone prosthesis.

[0127] For example, Figure 5 A schematic diagram of a skeletal prosthesis model according to an embodiment of the present invention is shown. Figure 5 As shown, this skeletal prosthesis model can be used for Figure 1 The cutting model shown corresponds to the unicompartmental femoral prosthesis. This cutting model is consistent with the spherical surface of the unicompartmental femoral prosthesis.

[0128] After determining the skeletal prosthesis model, the state determination module 30 can achieve spatial alignment between the skeletal prosthesis model and the 3D model of the target bone based on the solved pose matrix of the skeletal prosthesis model. That is, the state determination module 30 transforms the skeletal prosthesis model into the coordinate system of the 3D model of the target bone, so that the skeletal prosthesis model and the 3D model of the target bone are in the same reference coordinate system.

[0129] The state determination module 30 determines the grinding state of the target grinding point in the coordinate system of the 3D model based on the distance from the target grinding point on the 3D model to the bone prosthesis model. The grinding state includes at least the state to be ground, the state that has been ground, and the state that has been over-ground.

[0130] Since the 3D model of the bone prosthesis model and the 3D model of the target bone are in the same reference coordinate system, it can be known that the point of intersection between the 3D model of the bone prosthesis model and the 3D model of the target bone in this reference coordinate system is the target grinding point.

[0131] Therefore, by determining the distance between the target grinding point and the bone prosthesis model, the present invention can determine the positional relationship between the target grinding point and the intersecting area. Based on this positional relationship, the state determination module 30 can determine whether the target grinding point is in a state to be ground, a state that has been ground, or a state that has been over-ground.

[0132] The status display module 40 displays the target grinding point on the 3D model in a preset color corresponding to the grinding status.

[0133] For example, different grinding states correspond to different preset colors.

[0134] Figure 7 This diagram illustrates different grinding states presented in preset colors according to an embodiment of the present invention.

[0135] For example, such as Figure 7 As shown, the preset color corresponding to the state to be ground can be the first preset color (such as green); the preset color corresponding to the state that has been ground can be the second preset color (such as white); and the preset color corresponding to the state that has been over-ground can be the third preset color (red).

[0136] After determining the grinding state of the target grinding point, the status display module 40 displays the state on the image of the three-dimensional model based on the corresponding color, so as to display the grinding state of the target bone in a more intuitive way.

[0137] Through the above embodiments, this invention establishes a three-dimensional model of the target bone based on the user's bone scan image, and obtains the three-dimensional model data of the target bone. Furthermore, this invention determines the position information of the first target registration point in the camera coordinate system of the surgical robot, and determines the initial registration matrix based on the position information and the three-dimensional model data. Based on the initial registration matrix and the second target registration point, this invention determines a secondary registration matrix, and obtains the target pose matrix based on the secondary registration matrix. This allows the bone prosthesis model to be transformed into the coordinate system of the three-dimensional model of the target bone based on the pose matrix of the bone prosthesis model. Finally, in the coordinate system of the three-dimensional model, this invention determines the grinding state of the target grinding point based on the distance from the target grinding point on the three-dimensional model to the bone prosthesis model, and then presents the target grinding point on the three-dimensional model with a preset color corresponding to the grinding state.

[0138] This invention enables initial and secondary registration using a 3D model of the target bone and its data. This initial and secondary registration achieves precise alignment between the user's physical bone surface region and the space of the bone scan image. Therefore, this invention can transform the bone prosthesis model into the same reference coordinate system as the target bone's 3D model based on the pose matrix obtained from the registration. This invention also converts the determination of the grinding state at the target grinding point into a distance determination from the target grinding point to the bone prosthesis model to obtain the grinding state. Furthermore, this invention can visually present the grinding state in the image using different preset colors.

[0139] Compared with existing technologies, this invention eliminates the need for 3D reconstruction and rendering of the removed bone, thus avoiding the processing of large amounts of polygon data and saving significant computational resources and time. This invention features low computational load, fast computation, high accuracy, and intuitive display.

[0140] Optionally, the model processing module 10 determines the target trimming rules based on the bone region to be ground.

[0141] The model processing module 10 trims the 3D model of the target skeleton according to the target trimming rules.

[0142] For example, the area of ​​the bone to be ground can be a grinding area customized according to user needs. The model processing module 10 can determine the corresponding target trimming rules based on the corresponding grinding area. Then, the model processing module 10 can trim some unnecessary areas of the three-dimensional model of the target bone according to the target trimming rules. With this setting, the present invention can optimize the number of calculation points by trimming the three-dimensional model of the target bone, thereby further improving the computational efficiency of the present invention.

[0143] For example, taking unicompartmental arthroplasty as an application scenario, in the case of medial condyle arthroplasty, the target clipping rule is to retain the 3D model data of the medial region of the femoral condyle center and clip the 3D model data of the lateral region of the femoral condyle center. In the case of lateral condyle arthroplasty, the target clipping rule is to retain the 3D model data of the lateral region of the femoral condyle center and clip the 3D model data of the medial region of the femoral condyle center.

[0144] Optionally, if the target grinding point is located inside the bone prosthesis model, the state determination module 30 determines the target grinding point as a state to be ground.

[0145] When the target grinding point is located outside the bone prosthesis model and the distance between the target grinding point and the bone prosthesis model is not greater than a preset threshold, the state determination module 30 determines that the target grinding point is in the ground state.

[0146] If the target grinding point is located outside the bone prosthesis model and the distance between the target grinding point and the bone prosthesis model is greater than a preset threshold, the state determination module 30 determines that the target grinding point is in an over-grinding state.

[0147] For example, if the target grinding point is located inside the bone prosthesis model, indicating that the target grinding point is located in the intersection area of ​​the bone prosthesis model and the three-dimensional model of the target bone, then the state determination module 30 determines the target grinding point as a state to be ground.

[0148] The preset threshold can be customized according to user needs. For example, the preset threshold can be 1.5mm.

[0149] If the target grinding point is located outside the bone prosthesis model, and the distance between the target grinding point and the outside of the bone prosthesis model is less than or equal to 1.5 mm, it indicates that there is no intersection between the three-dimensional model of the bone prosthesis model and the target bone, and the grinding error of the target grinding point is within the acceptable range of the user. Then, the state determination module 30 determines the target grinding point as being in the ground state.

[0150] If the target grinding point is located outside the bone prosthesis model and the distance between the target grinding point and the outside of the bone prosthesis model is greater than 1.5mm, indicating that the three-dimensional model of the bone prosthesis model and the target bone has no intersecting area, and the grinding error of the target grinding point is within the user's unacceptable range, then the state determination module 30 determines that the target grinding point is in an over-grinding state.

[0151] With this configuration, the present invention can determine the internal or external distance relationship between the target grinding point and the bone prosthesis model under the same reference coordinate system by judging the distance between the target grinding point and the bone prosthesis model. Furthermore, the present invention can determine whether the grinding error is within the user's acceptable range based on the specific distance between the target grinding point and the external part of the bone prosthesis model, thereby accurately determining whether over-grinding has occurred.

[0152] According to another aspect of the present invention, an electronic device is also provided. The electronic device includes: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement the image display method for bone grinding states as described above.

[0153] For example, the electronic device may be a surgical robot operating device, but the invention is not limited thereto.

[0154] According to another aspect of the present invention, a non-volatile computer-readable storage medium is also provided. The storage medium stores a computer program that, when executed by a processor, enables the image display method for the bone grinding state as described above.

[0155] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions of the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for displaying images of bone grinding processes, characterized in that, include: A three-dimensional model of the target bone is built based on the user's bone scan image, and the three-dimensional model data of the target bone is obtained. The position information of the first target registration point in the camera coordinate system of the surgical robot is determined, wherein the first target registration point is determined based on the planned registration point in the bone scan image; Determine the initial registration matrix based on the location information and the 3D model data; A secondary registration matrix is ​​determined based on the initial registration matrix and the second target registration point, so as to obtain the target pose matrix according to the secondary registration matrix. The target pose matrix includes at least the pose matrix of the skeletal prosthesis model. Based on the pose matrix of the skeletal prosthesis model, the skeletal prosthesis model is transformed into the coordinate system of the three-dimensional model of the target bone; In the coordinate system of the three-dimensional model, the grinding state of the target grinding point is determined according to the distance from the target grinding point on the three-dimensional model to the bone prosthesis model. The grinding state includes at least the state to be ground, the state that has been ground, and the state that has been over-ground. The target grinding point is displayed on the 3D model with a preset color corresponding to the grinding state.

2. The image display method according to claim 1, characterized in that, Determining the initial registration matrix based on the location information and the 3D model data includes: The location information and the three-dimensional model data are calculated based on a preset registration algorithm to obtain the registration matrix between the first target registration point and the planned registration point; The initial registration matrix is ​​determined by the registration matrix that minimizes the error between the first target registration point and the planned registration point.

3. The image display method according to claim 1, characterized in that, The process of building a three-dimensional model of the target skeleton based on the user's skeletal scan images also includes: Determine the target trimming rules based on the area of ​​bone to be ground; The three-dimensional model of the target bone is trimmed according to the target trimming rules.

4. The image display method according to claim 1, characterized in that, Determining the grinding state of the target grinding point based on the distance from the target grinding point of the three-dimensional model to the bone prosthesis model in the coordinate system of the three-dimensional model includes: When the target grinding point is located inside the bone prosthesis model, the target grinding point is determined to be the state to be ground; If the target grinding point is located outside the bone prosthesis model and the distance between the target grinding point and the bone prosthesis model is not greater than a preset threshold, the target grinding point is determined to be in the ground state. If the target grinding point is located outside the bone prosthesis model and the distance between the target grinding point and the bone prosthesis model is greater than a preset threshold, the target grinding point is determined to be in the over-grinding state.

5. An image display system for bone grinding, characterized in that, include: The model processing module builds a three-dimensional model of the target skeleton based on the user's bone scan image and obtains the three-dimensional model data of the target skeleton. The registration processing module determines the position information of the first target registration point in the camera coordinate system of the surgical robot, wherein the first target registration point is determined based on the planned registration point in the bone scan image; and determines the initial registration matrix based on the position information and the three-dimensional model data. A secondary registration matrix is ​​determined based on the initial registration matrix and the second target registration point, so as to obtain the target pose matrix according to the secondary registration matrix. The target pose matrix includes at least the pose matrix of the skeletal prosthesis model. The state determination module transforms the skeletal prosthesis model into the coordinate system of the three-dimensional model of the target bone based on the pose matrix of the skeletal prosthesis model. In the coordinate system of the three-dimensional model, the grinding state of the target grinding point is determined according to the distance from the target grinding point on the three-dimensional model to the bone prosthesis model. The grinding state includes at least the state to be ground, the state that has been ground, and the state that has been over-ground. The status display module displays the target grinding point on the three-dimensional model in a preset color corresponding to the grinding status.

6. The image display system according to claim 5, characterized in that, The registration processing module calculates the location information and the three-dimensional model data based on a preset registration algorithm to obtain the registration matrix between the first target registration point and the planned registration point; The registration processing module determines the initial registration matrix as the registration matrix that minimizes the error between the first target registration point and the planned registration point in the registration matrix.

7. The image display system according to claim 5, characterized in that, The model processing module determines the target trimming rules based on the bone region to be ground. The model processing module trims the three-dimensional model of the target skeleton according to the target trimming rules.

8. The image display system according to claim 5, characterized in that, When the target grinding point is located inside the bone prosthesis model, the state determination module determines that the target grinding point is the state to be ground. The state determination module determines the target grinding point as the ground state when the target grinding point is located outside the bone prosthesis model and the distance between the target grinding point and the bone prosthesis model is not greater than a preset threshold. The state determination module determines that the target grinding point is in the over-grinding state when the target grinding point is located outside the bone prosthesis model and the distance between the target grinding point and the bone prosthesis model is greater than a preset threshold.

9. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image display method as described in any one of claims 1-4.

10. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image display method as described in any one of claims 1-4.

Images