Sensor assembly, positioning device and virtual image display method

Abstract

The invention provides a virtual image display method. The method comprises the following steps: determining a first position conversion relation between an electromagnetic sensor on a positioning device and the positioning device; the coordinate system transformation relation between an image coordinate system and a magnetic field coordinate system is determined, the image coordinate system is a coordinate system of an image space where the three-dimensional model of the bone connected with the fixing nail is located, the fixing nail in the fixing nail positioning device is located, and the magnetic field coordinate system is a coordinate system of a magnetic field space where an electromagnetic sensor in the positioning device is located; based on the pose information of the electromagnetic sensor in the magnetic field coordinate system, the first position conversion relation between the electromagnetic sensor and the positioning device and the coordinate system conversion relation between the image coordinate system and the magnetic field coordinate system, the pose information of the skeleton in the image coordinate system is determined; and displaying the skeleton in a virtual image corresponding to the image coordinate system based on the pose information of the skeleton in the image coordinate system.

Description

[0001] This application is a divisional application of the Chinese application entitled “Sensor component, positioning device and virtual image display method”, filed on March 12, 2024, and with application number 202410281236.0. Technical Field

[0002] The present disclosure relates to the technical field of medical devices, and in particular to a sensor component, a positioning device and a virtual image display method. Background Art

[0003] With the development of medical imaging technology and its application in surgery, the difficulty of performing surgery has been greatly improved. For example, in orthopedic surgery, by displaying virtual images of the patient's bones on the monitor in real time, the surgery can be easily guided.

[0004] In order to accurately detect and display the virtual image of the bone, it is usually necessary to use a positioning frame to fix the sensor on the bone. How to reduce the volume of the positioning frame has gradually become a research hotspot. Summary of the invention

[0005] In a first aspect, the present disclosure provides a virtual image display method, including: determining a first position conversion relationship between an electromagnetic sensor on a positioning device and the positioning device; determining a coordinate system transformation relationship between an image coordinate system and a magnetic field coordinate system, wherein the image coordinate system is a coordinate system of an image space where a three-dimensional model of a bone connected to a fixing pin is located, the fixing pin is a fixing pin in a positioning device, and the magnetic field coordinate system is a coordinate system of a magnetic field space where an electromagnetic sensor in the positioning device is located; based on the posture information of the electromagnetic sensor in the magnetic field coordinate system, the first position conversion relationship between the electromagnetic sensor and the positioning device, and the coordinate system transformation relationship between the image coordinate system and the magnetic field coordinate system, determining the posture information of the bone in the image coordinate system; based on the posture information of the bone in the image coordinate system, displaying the bone in a virtual image corresponding to the image coordinate system.

[0006] In a second aspect, the present disclosure provides a virtual image display device, comprising: a first determination unit, used to determine a first position conversion relationship between an electromagnetic sensor on a positioning device and the positioning device; a second determination unit, used to determine a coordinate system transformation relationship between an image coordinate system and a magnetic field coordinate system, wherein the image coordinate system is a coordinate system of an image space where a three-dimensional model of a bone connected to a fixing pin is located, the fixing pin is a fixing pin in the positioning device, and the magnetic field coordinate system is a coordinate system of a magnetic field space where an electromagnetic sensor in the positioning device is located; a third determination unit, used to determine the posture information of the bone in the image coordinate system based on the posture information of the electromagnetic sensor in the magnetic field coordinate system, the first position conversion relationship between the electromagnetic sensor and the positioning device, and the coordinate system transformation relationship between the image coordinate system and the magnetic field coordinate system; a display unit, used to display the bone in a virtual image corresponding to the image coordinate system based on the posture information of the bone in the image coordinate system.

[0007] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any method in the embodiments of the present disclosure.

[0008] In a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute any method according to the embodiments of the present disclosure.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments provided according to the present disclosure and should not be regarded as limiting the scope of the present disclosure.

[0011] Figure 1 is a schematic diagram of the structure of a sensor assembly provided by an embodiment of the present disclosure;

[0012] Figure 2A is a structural schematic diagram of a positioning device provided by an embodiment of the present disclosure;

[0013] Figure 2B for Figure 2A A cross-sectional view of

[0014] Figure 3for Figure 2A Structural diagram of the mounting parts;

[0015] Figure 4 for Figure 2A Bottom view of the middle mounting part;

[0016] Figure 5 for Figure 2A A schematic diagram of the structure of the middle protective shell;

[0017] Figure 6 for Figure 2A Another structural schematic diagram of the middle protective shell;

[0018] Figure 7 for Figure 2A Schematic diagram of the structure of the connecting parts;

[0019] Figure 8 for Figure 2A Schematic diagram of the structure of the middle fixing nail;

[0020] Fig. 9 for Figure 2A Installation diagram of the middle fixing nail and the non-return nail;

[0021] Fig.10 is a structural diagram of a system for applying a virtual image display method provided by an embodiment of the present disclosure;

[0022] Fig.11 is a flowchart of a virtual image display method provided by an embodiment of the present disclosure;

[0023] Fig.12 yes Fig.11 Schematic diagram of the process of step S1102;

[0024] Fig.13 is a schematic block diagram of a virtual image display device provided by an embodiment of the present disclosure;

[0025] Fig.14 It is a block diagram of an electronic device used to implement the virtual image display method of the embodiment of the present disclosure.

[0026] Description of reference numerals:

[0027] 100: electromagnetic sensor; 200: mounting mechanism;

[0028] 210: mounting member; 211: first boss;

[0029] 212: second boss; 213: side surface;

[0030] 214: mounting plane; 215: spline;

[0031] 215a, 215b: protrusions; 216: first through hole;

[0032] 217: internal thread; 220: protective shell;

[0033] 221: first limiting surface; 222: limiting groove;

[0034] 223: side wall; 224: calibration hole;

[0035] 225: first identification portion; 230: receiving cavity;

[0036] 240: positioning pin; 300: connecting piece;

[0037] 310: operation section; 320: middle section;

[0038] 321: external thread; 330: thread section;

[0039] 400: fixing pin; 410: spline groove;

[0040] 420: threaded hole; 430: feature portion;

[0041] 431: characteristic hole; 432: second identification portion;

[0042] 440: limit table; 441: second through hole;

[0043] 450: nail body; 500: check nail. DETAILED DESCRIPTION

[0044] The present disclosure will be further described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0045] The positioning frame used in surgical navigation in the related art is large in size and inconvenient to use. At the same time, in order to fix the larger positioning frame, a larger fixing screw is required, which causes greater surgical trauma.

[0046] In order to solve at least one of the above problems, the embodiments of the present disclosure provide a sensor assembly, a positioning device and a virtual image display method, which realizes the connection between the electromagnetic sensor and the mounting mechanism by using the accommodating cavity between the protective shell and the mounting member, and realizes the connection between the mounting mechanism and the fixing nail by using the connecting member. Since the fixing nail can be fixed to the bone, the electromagnetic sensor and the bone can be fixedly connected, and then the position and posture of the bone can be obtained by using the electromagnetic sensor, so as to facilitate navigation of orthopedic surgery. In addition, the mounting mechanism is small in size and compact in structure, which can reduce the volume of the sensor assembly and the positioning device.

[0047] Figure 1 is a structural schematic diagram of a sensor assembly provided by an embodiment of the present disclosure; Figure 2A is a schematic diagram of the structure of a positioning device provided by an embodiment of the present disclosure; Figure 2B for Figure 2A A cross-sectional view of Figure 3 for Figure 2A Structural diagram of the mounting parts; Figure 4 for Figure 2A Bottom view of the middle mounting part; Figure 5 for Figure 2A A schematic diagram of the structure of the middle protective shell; Figure 6 for Figure 2A Another structural schematic diagram of the middle protective shell; Figure 7 for Figure 2A The structural diagram of the connecting parts; Figure 8 for Figure 2A Schematic diagram of the structure of the middle fixing nail; Fig. 9 for Figure 2A Installation diagram of the fixing pin and the non-return pin.

[0048] Please refer to Figures 1 to 9 The embodiment of the present disclosure provides a sensor assembly, including: an electromagnetic sensor 100, a mounting mechanism 200 and a connecting member 300, the mounting mechanism 200 includes a protective shell 220 and a mounting member 210 extending along a first direction, the protective shell 220 is sleeved and connected to the outside of the mounting member 210, and a receiving cavity 230 is provided between the mounting member 210 and the protective shell 220, and the electromagnetic sensor 100 is installed in the receiving cavity 230; the connecting member 300 passes through the mounting mechanism 200 along the first direction, and the connecting member 300 is used to connect the mounting mechanism 200 to the fixing nail 400.

[0049] The sensor assembly can be used in orthopedic surgery, and orthopedic surgery includes but is not limited to fracture reduction surgery. It can be understood that in orthopedic surgery, the electromagnetic sensor 100 can be fixed to the bone of the surgical site using a positioning device. Specifically, the positioning device includes a sensor assembly and a fixing nail 400. The bone can be fixed with a fixing nail 400. The fixing nail 400 can be a titanium nail commonly used in orthopedic surgery, which can be nailed into the bone to achieve fixation. The sensor assembly can be fixed on the fixing nail 400, thereby achieving fixation of the electromagnetic sensor and the bone.

[0050] The sensor assembly includes an electromagnetic sensor 100, a mounting mechanism 200, and a connector 300. The electromagnetic sensor 100 may be a common sensor that can convert a measured physical quantity into an induced electromotive force, for example, it may be a six-degree-of-freedom electromagnetic sensor. It is understood that when the electromagnetic sensor is in a magnetic field space, the spatial position and posture of the electromagnetic sensor can be accurately sensed and calculated.

[0051] The mounting mechanism 200 is a component for mounting the electromagnetic sensor 100, which may include a protective shell 220 and a mounting member 210. The mounting member 210 may be a rod-shaped structure extending along a first direction. The first direction may be Figure 2B The material of the mounting member 210 can be titanium alloy or 304 stainless steel.

[0052] The protective shell 220 may be a shell-like structure, which is sleeved outside the mounting member 210, and the protective shell 220 and the mounting member 210 are connected to each other. The protective shell 220 may have various shapes. For example, the protective shell 220 may also extend along the first direction, and may have a receiving hole extending along the first direction inside, and the mounting member 210 may be installed in the receiving hole. For another example, the protective shell 220 may be a hollow truncated cone structure.

[0053] The material of the protective shell 220 can also be set according to needs. For example, the material of the protective shell can be selected from engineering plastics or ceramics. Since the operation requires the use of CT (Computed Tomography) images to reconstruct the three-dimensional model of the bone and the fixation pin 400, by setting the protective shell 220 to engineering plastics or ceramics, it is possible to avoid artifacts or scattering during CT scanning, thereby improving the clarity and accuracy of the CT image.

[0054] There is a housing cavity 230 between the protective shell 220 and the mounting member 210, and the electromagnetic sensor 100 is mounted in the housing cavity 230. It can be understood that there are many ways to mount the electromagnetic sensor 100, for example, the electromagnetic sensor 100 can be fixed to at least one of the protective shell 220 or the mounting member 210 by fasteners such as screws or rivets. The protective shell 220 can protect the electromagnetic sensor 100 inside it and reduce the impact of high temperature or water vapor on the electromagnetic sensor 100 during disinfection and sterilization.

[0055] In addition, the material of the connecting member 300 may be titanium alloy or 304 stainless steel, etc. The connecting member 300 can penetrate the mounting mechanism 200 along the first direction, for example, it can penetrate the protective shell 220 or the mounting member 210, and the connecting member 300 can play a connecting role to fix the mounting mechanism 200 to the fixing nail 400. In some embodiments, the connecting member 300 may be a fastener such as a screw or a rivet to achieve the fixing of the mounting mechanism 200.

[0056] It can be understood that the sensor assembly can be used in bone surgery navigation to update the position and posture of the bones in real time in the image space. For the accuracy of surgical navigation, preoperative preparation is required before the formal operation. The preoperative preparation includes the calibration, installation and registration of fixation pins and sensor assemblies. In some embodiments, the registration process can also include two processes: alignment and determining the coordinate transformation relationship between the magnetic field coordinate system and the image coordinate system to ensure the accuracy of the surgical implementation.

[0057] The following will briefly describe the preoperative preparation process in conjunction with the positioning device, and the specific methods of calibration and registration will be explained in the following embodiments of the virtual image display method.

[0058] Specifically, when the positioning device leaves the factory, the positional relationship between the electromagnetic sensor and the positioning device is calibrated. The calibration process can use a calibration tool. The calibration tool is a component produced by precision machining, and its geometric dimensions are very precise. Another six-degree-of-freedom electromagnetic sensor is fixed on the calibration tool. In the magnetic field space, the position and posture of the tip of the calibration tool can be tracked by the magnetic field generator. Therefore, after the sensor assembly is assembled, it can be calibrated. During the calibration process, the tip of the calibration tool can touch the calibration point on the sensor assembly to determine the position of the calibration point in the magnetic field space, and then the position of the sensor assembly in the magnetic field is obtained. Then, the position relationship between the electromagnetic sensor and the sensor assembly is calculated through the position of the electromagnetic sensor in the magnetic field, and then the position relationship between the electromagnetic sensor and the positioning device can be obtained based on the position relationship between the sensor assembly and the positioning device in the standard three-dimensional model. Calibration at the time of leaving the factory can save waiting time for the patient to be operated on. Of course, calibration can also be implemented in the preoperative preparation stage.

[0059] In the preoperative preparation stage, the fixing pin 400 is fixed to the bone of the surgical site, and then a CT scan is performed to determine the CT image of the bone with the fixing pin 400, and then a segmented image of the bone with the fixing pin 400 is obtained by segmentation, and then the segmented image is reconstructed in three dimensions by three-dimensional reconstruction to obtain a reconstructed three-dimensional model of the bone and the fixing pin (i.e., a segmented model), which can be displayed in the image space (display space of the display). Of course, the positioning device can also be installed as a whole to shoot CT.

[0060] Then, alignment can be performed by importing the standard three-dimensional model of the positioning device (i.e., the design model designed in the three-dimensional design software) into the image space, and then aligning the standard three-dimensional model with the segmentation model so that the two are approximately overlapped, that is, the alignment is completed when the overlap meets the requirements. At this time, the standard three-dimensional model can replace the model of the fixing pin obtained by three-dimensional reconstruction, and its accuracy is higher, so that the position (coordinates) of the fixing pin or the feature points on it in the image space can be accurately obtained.

[0061] Then determine the coordinate system transformation relationship. For example, use the registration result (the position of the fixing pin in the image space) and the calibration result (the position relationship between the electromagnetic sensor and the positioning device) to obtain the conversion relationship between the image space and the magnetic field space, that is, the coordinate system transformation relationship between the image coordinate system and the magnetic field coordinate system.

[0062] Of course, in other embodiments, the method for determining the coordinate system transformation relationship can also be carried out by finding the first position coordinates of the feature points on the fixing pin 400 in the image coordinate system in the registered standard three-dimensional model, selecting these feature points on the fixing pin 400 by using a calibration tool, obtaining the second position coordinates of the feature points in the magnetic field space, and then calculating the coordinate system transformation relationship based on the first position coordinates and the second position coordinates.

[0063] During the subsequent surgical process, since the electromagnetic sensor 100 remains relatively fixed to the bone with the fixing pin, the position and posture changes of the electromagnetic sensor 100 are converted into the position and posture changes of the bone based on the calibrated positional relationship between the electromagnetic sensor and the positioning device. The position and posture of the electromagnetic sensor 100 in the magnetic field space can be obtained through the electromagnetic tracking device, and then according to the coordinate system transformation relationship, the position and posture of the electromagnetic sensor 100 in the magnetic field space can be finally converted into the position and posture of the bone connected with the fixing pin or positioning device in the image space, and then the virtual image of the bone in the image space (i.e., displayed on the display) is updated, so as to perform surgical navigation.

[0064] The sensor assembly provided in this embodiment can realize surgical navigation, and has a small size and compact structure, which can reduce the volume of the sensor assembly and the positioning device. At the same time, due to the small size of the installation mechanism, the fixing nail can also be selected to be smaller in size, that is, the fixing nail driven into the bone is also smaller, which can reduce surgical trauma.

[0065] In some embodiments, the outer surface of the mounting member 210 is protruding with a first boss 211 extending along the circumference of the mounting member 210, and the inner surface of the protective shell 220 is provided with a first limiting surface 221 perpendicular to the first direction, and the first boss is used to abut against the first limiting surface.

[0066] It can be understood that the inner surface of the protective shell 220 is the hole wall of the receiving hole in the protective shell 220, and the outer surface of the mounting member 210 can be the annular side surface of the mounting member 210 surrounding the first direction. Figure 2B , Figure 3 and Figure 6 A first boss 211 is provided on the outer surface of the mounting member 210 , and the first boss 211 can extend along the circumference of the mounting member 210 , for example, it can extend into a ring shape or an arc shape (non-circular), thereby forming a shoulder on the outer surface of the mounting member 210 .

[0067] The inner surface of the protective shell 220 may be provided with a first limiting surface 221, and the first limiting surface 221 may be perpendicular to the first direction. In addition, the first limiting surface 221 is oriented away from the fixing pin 400, that is, Figure 2B In the embodiment, the fixing pin 400 is located at the bottom of the sensor assembly, and the first limiting surface 221 can be set toward the top.

[0068] A side surface of the first boss 211 at one end along the first direction can abut against the first limiting surface 221 to achieve cooperation between the two, thereby completing the axial limiting of the mounting member 210 and the protective shell 220, which helps to improve the installation accuracy of the protective shell 220 and the mounting member 210.

[0069] In some embodiments, the inner surface of the protective shell 220 is provided with a limiting groove 222 extending along the first direction, which is used to limit the protective shell in the circumferential direction. The limiting groove 222 has two side walls 223 arranged opposite to each other along the second direction, wherein there is a preset angle between the second direction and the first direction; the outer surface of the mounting member 210 is provided with a second boss 212, and the second boss 212 has two side surfaces 213 arranged opposite to each other along the second direction; the second boss 212 is engaged in the limiting groove 222, and the two side surfaces 213 of the second boss 212 are respectively against the two side walls 223 of the limiting groove 222.

[0070] It can be understood that the limiting groove 222 can be a groove on the inner surface of the protective shell 220, and the second boss 212 can be a protrusion protruding from the outer surface of the mounting member 210, and the two can be matched. Specifically, the limiting groove 222 is a through groove extending along the first direction, so that the second boss 212 can enter the limiting groove 222 and match with the limiting groove 222.

[0071] The second direction may have a preset angle with the first direction, and the preset angle may be 30 degrees, 60 degrees, or 90 degrees. For example, taking the second direction being perpendicular to the first direction as an example, the second direction may be Figure 2B In the direction perpendicular to the paper, the two side walls 223 of the limiting groove 222 can be spaced apart along the second direction, and similarly, the two side surfaces 213 of the second boss 212 can also be spaced apart along the second direction. It can be understood that when the limiting groove 222 is matched with the second boss 212, the two side walls 223 can respectively abut against the two side surfaces 213, thereby realizing the circumferential limitation between the protective shell 220 and the mounting member 210, that is, limiting the relative rotation of the two along the circumference of the mounting member 210, which helps to improve the installation accuracy of the protective shell 220 and the mounting member 210.

[0072] It can be understood that the above embodiment can ensure the installation accuracy of the protective shell 220 and the mounting member 210 by limiting the circumferential and axial directions of the protective shell 220 and the mounting member 210, which helps to improve the installation accuracy of the electromagnetic sensor and the calibration accuracy of the positioning device.

[0073] Of course, in other embodiments, the circumferential and axial limiting between the protective shell 220 and the mounting member 210 can also be achieved by other means, for example, by achieving axial limiting through the cooperation of the conical outer surface of the mounting member 210 and the conical inner surface of the protective shell 220, etc.

[0074] It can be understood that when calibrating and / or aligning the standard three-dimensional model with the segmentation model, it can be achieved through a characteristic structure on the positioning device, and the characteristic structure can be set in the protective shell 220 in the positioning device. There can be many kinds of characteristic structures. For example, the outer surface of the protective shell 220 is provided with N calibration holes 224 with pointed tips, where N is a positive integer greater than or equal to 3. Alternatively, the outer surface of the protective shell 220 is provided with a first identification portion 225, and the first identification portion 225 is protruding or recessed in the protective shell. Alternatively, the outer surface of the protective shell 220 is provided with N calibration holes 224 with pointed tips, and the outer surface of the protective shell 220 is provided with a first identification portion 225, and the first identification portion 225 is protruding or recessed in the protective shell 220.

[0075] The calibration hole 224 may be disposed on the outer surface of the protective shell 220 , for example, on the top end surface of the protective shell 220 facing away from the fixing pin 400 .

[0076] The calibration hole 224 can be a conical hole or a conical groove or other structure, so that it can have a pointed end, which is convenient for calibration by using the tip of the calibration tool to select the tip of the calibration hole 224, so as to accurately obtain the position of the tip of the calibration hole 224 in the magnetic field space. By setting three or more calibration holes 224, the position of the entire sensor assembly in the magnetic field space can be accurately calibrated.

[0077] The first identification portion 225 may be disposed on the outer surface of the protective shell 220. For example, it may be recessed to form a groove or a notch, or it may be protruded to form a protrusion. The shape of the first identification portion 225 may be rectangular, circular, etc. Figure 5 As shown, the first identification portion 225 is located at the top of the protective shell 220 and can be a rectangular notch.

[0078] It can be understood that since the outer surface of the protective shell is a rotating body structure, the first identification portion 225 can serve as a feature surface and play an identification role, so that it obtains a unique spatial position, thereby facilitating the improvement of the alignment accuracy when aligning the segmentation model and the standard three-dimensional model, and can also indicate a unique installation direction during installation.

[0079] In some embodiments, Figure 2B and Figure 3 As shown, the outer surface of the mounting member 210 has a mounting plane 214 parallel to the first direction, the electromagnetic sensor 100 is mounted on the mounting plane 214 through the positioning pins 240, and the accommodating cavity 230 is also filled with an adhesive that wraps the electromagnetic sensor 100.

[0080] The mounting plane 214 may extend in the first direction to be a rectangle, and may be provided with a plurality of pin holes ( Figure 3 Two pin holes are provided in the cavity). The electromagnetic sensor 100 can be attached to the mounting plane 214, and the electromagnetic sensor 100 can be fixed to the mounting plane 214 by the cooperation of the positioning pin 240 and the pin hole. Further, by filling the accommodating cavity with an adhesive, the electromagnetic sensor 100 can be further fixed and protected. The adhesive can be a high temperature resistant sealant, such as a high temperature resistant epoxy resin potting glue, to prevent damage during the high temperature of the operation and the sterilization process.

[0081] In this embodiment, the electromagnetic sensor 100 is fixed by installing the plane 214, the positioning pin 240 and the adhesive, so that the positional relationship between the assembled electromagnetic sensor 100 and the sensor component can be kept consistent with the positional relationship in the standard (design) model, so that the aligned standard three-dimensional model can accurately represent the position of the sensor component actually installed on the bone, which helps to increase the accuracy of surgical navigation.

[0082] In some embodiments, the first end of the mounting member 210 along the first direction protrudes out of the protective shell 220 , and the first end of the mounting member 210 is provided with a spline 215 , which is used to be engaged with the spline groove 410 of the fixing nail 400 .

[0083] like Figure 2B As shown, the first end of the mounting member 210 may be its lower end, and the end may be provided with a spline 215 , which may be a spline structure in the related art.

[0084] The fixing nail 400 may be provided with a spline groove 410, and the spline 215 may be engaged with the spline groove by shape matching, thereby limiting the relative rotation between the fixing nail 400 and the sensor assembly, which helps to ensure the installation accuracy between the fixing nail 400 and the sensor assembly.

[0085] In some embodiments, Figure 4 As shown, the spline 215 includes a plurality of protrusions sequentially connected along the circumferential direction of the mounting member, and a shape of one of the plurality of protrusions 215b is different from shapes of the remaining protrusions 215a of the plurality of protrusions.

[0086] Please refer to Figure 4, the spline 215 includes four protrusions 215a and one protrusion 215b, the shape of 215b is different from the shape of the protrusion 215a, and similarly, Figure 8 The cross-sectional shape of the spline groove 410 is also similar to Figure 4 The top view shape of the spline 215 is the same, so that the matching relationship between the spline 215 and the spline groove 410 can be guaranteed to be unique, and then the assembly relationship between the fixing pin 400 and the sensor assembly is unique, which is helpful for the subsequent alignment and registration process.

[0087] In some embodiments, please refer to Figure 3 , Figure 4 and Figure 7 A first through hole 216 extending along a first direction is provided in the mounting member 210, and the connecting member 300 includes an operating section 310, an intermediate section 320 and a threaded section 330 which are sequentially connected along the first direction; the intermediate section 320 is located in the first through hole 216, the operating section 310 and the threaded section 330 are located outside the first through hole 216, and the operating section 310 is used to press against the mounting member 210, and the threaded section 330 is used to be screwed with the threaded hole 420 in the fixing nail 400.

[0088] The mounting member 210 is provided with a first through hole 216 , which may be a hollow circular hole. The connecting member 300 may pass through the first through hole 216 and be connected to the fixing nail 400 .

[0089] like Figure 7 As shown, the connector 300 includes an operating section 310, an intermediate section 320, and a threaded section 330 from top to bottom. The operating section 310 may protrude from the intermediate section along the circumference of the connector 300, and its outer surface may be provided with vertical stripes to facilitate manual rotation. The threaded section 330 may protrude from the outside of the spline 215, such as Figure 2B and Figure 8 As shown, a threaded hole 420 is provided on the fixing nail 400 , and the threaded section 330 can be screwed into the threaded hole 420 .

[0090] When in use, the connecting member 300 can be inserted into the first through hole 216, and the threaded section 330 can be screwed into the threaded hole 420 by rotating the operating section 310. The lower end of the operating section 310 will press on the top end surface of the mounting member 210 to press the bottom end surface of the spline 215 against the bottom end surface of the spline groove 410, thereby fixing the mounting member 210 and the fixing nail 400 with high reliability.

[0091] In some embodiments, Figure 2B and Figure 7 The middle section 320 is provided with an external thread 321 at a position away from the threaded section 330 , and the first through hole 216 is provided with an internal thread 217 for threading with the external thread 321 at a position away from the threaded section 330 .

[0092] In this embodiment, Figure 2B The upper end of the first through hole 216 may be provided with an internal thread 217 (in the dotted box), and the upper end of the connector may also be provided with a corresponding external thread 321, so that the electromagnetic sensor 100, the mounting mechanism 200 and the connector 300 can be assembled together as a part of the positioning device to prevent the connector 300 from being dislodged and lost during transportation of the sensor assembly.

[0093] The present disclosure also provides a positioning device, such as Figure 2B The positioning device includes: a sensor assembly and a fixing nail 400 for connecting to the bone; the fixing nail 400 is provided with a feature portion 430 for providing feature points and / or feature surfaces; the connecting piece 300 of the sensor assembly is connected to the fixing nail 400.

[0094] Among them, the structure and function of the sensor component are the same as those in the above embodiment, and the details can be referred to the above embodiment and will not be repeated here.

[0095] The fixing nail 400 may be an orthopedic titanium nail, which may be connected to the connector 300 . For example, the fixing nail 400 may be provided with a threaded hole 420 , and the threaded section 330 of the connector 300 may be screwed into the threaded hole 420 .

[0096] The fixation pin 400 may be driven into the bone before a pre-operative CT scan, and the sensor assembly may be mounted to the fixation pin 400 at the start of the surgery.

[0097] The feature portion 430 can be set on the fixing pin 400, which can provide feature points and / or feature surfaces that can serve as identification. For example, it can be a structure such as a groove or protrusion with a pointed tip, which is used in the process of aligning and / or determining the coordinate system transformation relationship.

[0098] The electromagnetic sensor 100 can be fixed to the bone through the positioning device, so that the position and posture changes of the electromagnetic sensor 100 can represent the position and posture changes of the bone. Therefore, during the operation, the position and posture of the electromagnetic sensor 100 in the magnetic field space can be obtained through the electromagnetic tracking device, and converted into its position and posture in the image space, and then the virtual image of the bone in the image space (i.e., displayed on the display) is updated, so as to perform surgical navigation.

[0099] In addition, since the sensor assembly in the positioning device is small in size and compact in structure, the size of the positioning device can be reduced. At the same time, since the size of the mounting mechanism is small, the fixing nail can also be smaller in size, that is, the fixing nail driven into the bone is also smaller, which can reduce surgical trauma.

[0100] In some embodiments, please refer to Figure 8The fixing nail 400 includes a limiting platform 440 and a nail body 450 connected to the limiting platform 440 . The limiting platform 440 protrudes from the outer peripheral surface of the nail body 450 , and the characteristic portion 430 is disposed on the limiting platform 440 .

[0101] The characteristic portion 430 includes M characteristic holes 431 with pointed tips, and the points corresponding to the pointed tips constitute characteristic points, wherein M is a positive integer greater than or equal to 3; and / or, the characteristic portion 430 includes a second identification portion 432, which is protruding or recessed on the limit platform, and the surface of the second identification portion constitutes a characteristic surface.

[0102] Specifically, the limit platform 440 is connected to one end of the nail body 450, and has a cross-sectional area that is greater than that of the nail body 450, taking a plane perpendicular to the first direction as the cross-sectional area, so that the limit platform 440 can protrude from the outer peripheral surface of the nail body 450, thereby preventing the fixing nail 400 from drilling too deeply into the bone (cortical bone) and causing trauma to the patient.

[0103] It can be understood that in this embodiment, the limiting platform 440 can be provided with M characteristic holes 431 , or the limiting platform 440 can be provided with the second identification portion 432 , or the limiting platform 440 can be provided with the second identification portion and M characteristic holes 431 .

[0104] In some embodiments, the feature portion 430 may include a feature hole 431 with a pointed end disposed on the surface of the stop platform 440. The feature hole 431 may be a tapered groove or a tapered hole, and the number is 3 or more. In addition, the feature hole may be disposed on the upper surface or on the side surface of the stop platform 440, and may be used for alignment in registration and / or the process of determining the coordinate system transformation relationship in registration to improve accuracy.

[0105] In some embodiments, the feature portion 430 may include a second identification portion 432, which may be disposed on the limiting platform 440. For example, it may be recessed to form a groove or a notch, or it may be protruded to form a protrusion. The shape of the second identification portion 432 may be rectangular, circular, etc. Figure 8 As shown, the second identification portion 432 can be a cut surface to provide a distinct geometric feature.

[0106] It can be understood that since the fixing pin 400 is a rotating body structure and cannot be axially positioned, the second identification portion 432 can be used as a feature surface to play a role of identification, so that it obtains a unique spatial position, which is convenient for determining its position and posture in the CT image. In addition, it can also be convenient to improve the registration accuracy when the segmentation model and the standard three-dimensional model are registered. When the fixing pin 400 is installed with the sensor assembly, the second identification portion 432 and the first identification portion 225 can also be used as installation instructions, that is, Figure 2A Align the two during installation to ensure a single installation direction.

[0107] In some embodiments, please refer to Figure 6 The limiting groove 222 and the first identification portion 225 can be located on the same side of the protective shell 220, and the center planes of the two coincide. Figure 8 The part of the spline groove 410 corresponding to the protrusion 215b can also be located on the same side as the second identification portion 432, and the center planes of the two coincide. Similarly, please refer to Figure 3 , the protrusion 215b, the second boss 212 and the mounting plane 214 can also be located on the same side, and the center planes of the three coincide. The above features allow the installation to be performed by aligning these features during assembly, for example, aligning the second boss 212 with the first identification portion 225 to install the protective shell 220 and the mounting member 210, or aligning the first identification portion 225 with the second identification portion 432 to install the sensor assembly and the fixing pin 400, thereby playing the role of installation indication and ensuring a unique installation direction.

[0108] In some embodiments, the positioning device may also include: at least two check pins 500; at least two second through holes 441 are arranged on the limit platform 440 and are spaced apart along the circumference of the fixing pin 400, and the axis of each of the at least two second through holes 441 is arranged in a non-planar manner with the axis of the fixing pin 400; each of the at least two check pins 500 is installed in one of the at least two second through holes 441.

[0109] like Figure 2A and Fig. 9 As shown, the anti-return nail 500 can be tilted on the fixing nail 400 to strengthen the connection between the fixing nail 400 and the bone and prevent the fixing nail 400 from rotating after being driven into the bone to affect the registration or registration. The anti-return nail 500 can be a common standard titanium nail with a smaller diameter than the fixing nail 400.

[0110] At least two check screws 500 are provided on the limiting platform 440 to support and reinforce the fixing screw 400 at a limited depth, so as to prevent the fixing screw 400 from loosening due to osteoporosis after the fixing screw 400 is screwed into the bone, thereby affecting the accuracy of registration and the like and thus affecting the positioning accuracy of surgical navigation. In addition, the depth of the check screw 500 entering the bone is similar to that of the fixing screw 400.

[0111] Assuming that the axial direction (first direction) of the fixing pin 400 is the z direction in the coordinate system, the non-return pin 500 can be tilted outward at a certain angle along the x and / or y direction, so that its axis is not in the same plane as the axis of the fixing pin 400, that is, the two are not located in the same plane.

[0112] In a specific embodiment, as shown in FIG. 2 , the positioning device provided in this embodiment may include a sensor assembly and a fixing pin 400 .

[0113] A spline groove 410 may be provided at the top of the fixing pin 400, and the spline groove 410 is used to cooperate with the spline 215 at the first end of the mounting member 210 to limit and ensure the uniqueness of the installation position. It can be understood that before surgery, the fixing pin 400 can be screwed into the body without installing the sensor assembly. After taking the CT image, the doctor determines the position of the fixing pin in the image space through preoperative registration, and then installs the sensor assembly on the basis of the fixing pin 400 during surgery. The fixing pin 400 includes a limit table 440, and three characteristic holes 431 are designed on the outer surface of the limit table 440 for the registration in the above-mentioned registration and / or the stage of determining the coordinate system transformation relationship in the registration. At least two second through holes 441 are also designed on the limit table 440 to install the non-return pin 500, which is used to support and reinforce the fixing pin 400 based on a limited drilling depth and prevent the fixing pin 400 from rotating. In addition, the non-return pin 500 is inclined at a certain angle relative to the axis of the fixing pin 400.

[0114] In this embodiment, the sensor assembly includes an electromagnetic sensor 100 , a mounting member 210 , a protective shell 220 and a connecting member 300 .

[0115] The protective shell 220 is sleeved on the mounting member 210 , and a receiving cavity 230 is defined between the two. The electromagnetic sensor 100 is installed in the receiving cavity 230 , and the receiving cavity 230 is filled with adhesive.

[0116] The mounting member 210 and the fixing nail 400 are connected by the spline 215 and the spline groove 410 to limit the position and ensure the uniqueness of the position.

[0117] The upper surface of the protective shell 220 is provided with three calibration holes 224, which are conical grooves with pointed tips, for calibrating the spatial position relationship between the electromagnetic sensor 100 and the protective shell 220. It can be understood that due to the existence of manufacturing errors and the errors in the assembly of the electromagnetic sensor 100 on the mounting member 210 and the assembly errors caused by factors such as the glue filling in the protective shell, the accuracy of the position relationship between the electromagnetic sensor, the mounting member and the protective shell will be affected. Therefore, after the electromagnetic sensor 100 is installed on the mounting member 210, and the protective shell 220 is installed and glue is poured, the calibration tool can be used to calibrate the exact position of the electromagnetic sensor 100. In addition, a first identification portion 225 such as a rectangular notch can be provided on the protective shell 220 to serve as a feature surface to improve the registration accuracy and indicate a unique installation relationship.

[0118] When assembling the sensor assembly, the following steps may be included:

[0119] (1) The electromagnetic sensor 100 is mounted on a corresponding position of the mounting plane 214 of the mounting member 210 using the positioning pins 240 .

[0120] (2) Install the protective shell 220 on the mounting member 210, and use the second boss 212 on the side surface of the mounting member 210 to match the limiting groove 222 in the protective shell 220 to limit the protective shell 220 circumferentially to prevent the two from rotating relative to each other. At the same time, axial limiting is achieved through the cooperation of the first boss 211 and the first limiting surface 221, thereby ensuring that the protective shell 220 is installed in a unique position on the mounting member 210.

[0121] (3) High temperature resistant sealant (such as high temperature resistant epoxy resin potting glue) is injected between the mounting member 210 and the protective shell 220 to achieve waterproof and high temperature protection and sensor fixation, and also to limit the upward and downward movement of the protective shell 220 along the first direction.

[0122] (4) The connecting member 300 may be screwed into the second through hole 441 of the mounting member 210 .

[0123] After assembly, the spatial position relationship between the electromagnetic sensor 100 and the sensor assembly including the mounting member 210 and the protective shell 220 can also be calibrated.

[0124] The assembly steps of the sensor assembly can be completed before leaving the factory. When surgery is required, the fixing nail 400 can be fixed to the bone and tightened before the surgery, and the sensor assembly including the electromagnetic sensor 100 can be installed on the fixing nail 400 (the fixing nail 400 is equivalent to the base for fixing the electromagnetic sensor), and then the connector 300 can be tightened manually.

[0125] Of course, in some embodiments, the assembly step of the sensor assembly may also be performed during surgery, and the sensor assembly may also be fixed to the bone together with the fixing pin 400 before surgery, and no specific limitation is made here.

[0126] The disclosed embodiments further provide a surgical navigation system, including: a display, a processing device, an electromagnetic tracking device, a calibration tool, and a positioning device according to any one of the above embodiments.

[0127] The display can be used to show virtual images of bones and surgical instruments, etc., for surgical guidance.

[0128] The electromagnetic tracking device may include a magnetic field generator, which may generate an electromagnetic field to form a magnetic field space. The electromagnetic tracking device may also sense and calculate the position and posture of the electromagnetic sensor in the magnetic field space.

[0129] The calibration tool is a component produced by precision machining with very precise geometric dimensions. Another six-degree-of-freedom electromagnetic sensor is fixed to the calibration tool. In the magnetic field space generated by the electromagnetic tracking device, the position and posture of the tip of the calibration tool can be tracked by it.

[0130] The processing device can be a data processing device such as a computer, which can use the electromagnetic tracking device to obtain the spatial position and posture of the electromagnetic sensor in real time, and display virtual images such as the patient's bones and surgical instruments reconstructed in three dimensions from the CT image on the display in real time to guide the operation.

[0131] In addition to the sensor components and positioning devices in the above embodiments, the embodiments of the present disclosure also provide a virtual image display method, which can be applied to the above surgical navigation system. Specifically, the virtual image display method of the embodiment of the present disclosure can be executed by a processing device in the surgical navigation system, wherein the processing device can be a terminal or a server. The terminal can be a tablet computer, a laptop computer, an intelligent voice interaction device, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0132] The following describes a method for displaying virtual images provided by an embodiment of the present disclosure in conjunction with the accompanying drawings.

[0133] Fig.10 is a structural diagram of a system for applying a virtual image display method provided by an embodiment of the present disclosure; please refer to Fig.10 The system includes a terminal 1010 and a server 1020, etc. The terminal 1010 and the server 1020 are connected via a network, for example, via a wired or wireless network connection, etc.

[0134] Among them, the server 1020 can be used to determine the coordinate system transformation relationship between the image coordinate system and the magnetic field coordinate system, wherein the image coordinate system is the coordinate system of the image space where the three-dimensional model of the bone connected with the fixing pin is located, the fixing pin is the fixing pin in the positioning device, and the magnetic field coordinate system is the coordinate system of the magnetic field space where the electromagnetic sensor in the positioning device is located; based on the posture information of the electromagnetic sensor in the magnetic field coordinate system and the coordinate system transformation relationship, the posture information of the bone in the image coordinate system is determined; based on the posture information of the bone in the image coordinate system, the bone is displayed in the virtual image corresponding to the image coordinate system. Terminal 1010 can be used to display a graphical user interface. In this embodiment, terminal 1010 can be used to display a virtual image.

[0135] It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0136] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0137] Fig.11 is a flowchart of a virtual image display method provided by an embodiment of the present disclosure; please refer to Fig.11 , an embodiment of the present disclosure provides a virtual image display method 1100, including steps S1101 to S1104.

[0138] Step S1101, determining a first position conversion relationship between the electromagnetic sensor on the positioning device and the positioning device.

[0139] Step S1102, determine the coordinate transformation relationship between the image coordinate system and the magnetic field coordinate system, wherein the image coordinate system is the coordinate system of the image space where the three-dimensional model of the bone connected with the fixing pin is located, the fixing pin is the fixing pin in the positioning device, and the magnetic field coordinate system is the coordinate system of the magnetic field space where the electromagnetic sensor in the positioning device is located.

[0140] Step S1103, determining the posture information of the skeleton in the image coordinate system based on the posture information of the electromagnetic sensor in the magnetic field coordinate system, the first position conversion relationship between the electromagnetic sensor and the positioning device, and the coordinate system transformation relationship between the image coordinate system and the magnetic field coordinate system.

[0141] Step S1104, based on the posture information of the skeleton in the image coordinate system, display the skeleton in the virtual image corresponding to the image coordinate system.

[0142] In this embodiment, method 1100 can be used in orthopedic surgery, including but not limited to fracture reduction surgery. The positioning device is a positioning device of any of the above embodiments, and the above embodiments can be referred to for details. The positioning device includes a fixing nail 400 and a sensor assembly, and the bone is a bone of the surgical site to be fixed with a fixing nail, that is, a bone to which the sensor assembly needs to be installed.

[0143] It can be understood that the magnetic field coordinate system is a coordinate system in the magnetic field space, that is, a coordinate system in the electromagnetic field generated by the electromagnetic tracking device. The electromagnetic tracking device can obtain the position information (such as coordinates) of the electromagnetic sensor 100 and the tip of the calibration tool, and the position information obtained by the electromagnetic tracking device is the position information in the magnetic field coordinate system.

[0144] The image coordinate system is a coordinate system in the image space. It can be understood that the image space is the space where the three-dimensional model is located, that is, the space where the three-dimensionally reconstructed skeleton is located. Since the virtual image is displayed based on the three-dimensional model, the image space is also the space of the virtual image displayed on the display.

[0145] During surgery, since the position information of the electromagnetic sensor 100 obtained directly by the electromagnetic tracking device is in the magnetic field coordinate system, in order to display the virtual image of the bone, it is necessary to convert the position information of the electromagnetic sensor 100 (magnetic field coordinate system) into the position information of the bone in the image coordinate system. The virtual image display method can realize the conversion process.

[0146] Step S1101 can be implemented by calibration. For example, when the positioning device leaves the factory, a calibration tool is used to calibrate the first position conversion relationship between the electromagnetic sensor and the positioning device.

[0147] Step S1102 can determine the coordinate transformation relationship between the image coordinate system and the magnetic field coordinate system, that is, the coordinate transformation relationship between the image space and the magnetic field space. There are many specific methods, such as selecting a reference object, measuring its respective position information in the image space and the magnetic field space, and then calculating the coordinate transformation relationship between the two coordinate systems.

[0148] In step S1103, the electromagnetic tracking device can be used to obtain the position and posture information of the electromagnetic sensor 100 in the positioning device in the magnetic field coordinate system, that is, the position and posture data, and the position information can be coordinates. Then, according to the coordinate system transformation relationship in step S1102, the position and posture information of the electromagnetic sensor 100 in the magnetic field coordinate system is transformed into the position and posture information of the electromagnetic sensor 100 in the image coordinate system.

[0149] Since the fixing pin of the positioning device is fixed on the bone, based on the first position conversion relationship obtained in step S1101, the posture information change of the positioning device can be obtained by using the posture information change of the electromagnetic sensor 100, thereby obtaining the posture information change of the bone, and thus obtaining the posture information of the bone in the image coordinate system.

[0150] In one embodiment, before surgery, a CT image of a bone with a fixation pin can be scanned, and a reconstructed 3D model of the fixation pin and the bone can be obtained through 3D reconstruction. After registration, the initial virtual image of the bone can be displayed on a display. During surgery, since the electromagnetic sensor and the bone remain relatively fixed, based on the positional relationship between the calibrated electromagnetic sensor and the positioning device, the position change of the electromagnetic sensor can be converted into the position change of the fixation pin and the bone, and then the real-time position information of the bone in the image coordinate system can be obtained through the coordinate system transformation relationship between the image coordinate system and the magnetic field coordinate system.

[0151] Next, according to step S1104, the position and posture of the bones can be updated in real time in the virtual image displayed on the display using the posture information in the image coordinate system of the bones.

[0152] The virtual image display method provided in this embodiment can update the position and posture of the bones during surgery in real time in the virtual image of the display, so that the surgeon can accurately observe the current position of the bones, and can perform surgical navigation and guidance simply and accurately, which is conducive to improving the success rate and convenience of the surgery.

[0153] In some embodiments, determining the coordinate system transformation relationship between the image coordinate system and the magnetic field coordinate system may include the following steps S1201 to S1203.

[0154] Step S1201, determining first position information of the fixing pin in the image coordinate system based on a standard three-dimensional model of a positioning device including at least the fixing pin and a registration result with a reconstructed three-dimensional model including the fixing pin and the bone.

[0155] Step S1202: based on the second position information of the electromagnetic sensor in the magnetic field coordinate system and the first position conversion relationship, obtain the third position information of the positioning device including the fixing pin in the magnetic field coordinate system.

[0156] Step S1203: determining a coordinate system transformation relationship between the image coordinate system and the magnetic field coordinate system based on the first position information and the third position information.

[0157] In this embodiment, step S1201 can determine the precise position information (ie, first position information) of the fixing pin installed on the skeleton in the image coordinate system. It can be understood that the position information can be information such as coordinates that can represent the exact position.

[0158] In some embodiments, step S1201 may include: performing three-dimensional reconstruction of an initial image of a bone connected to a fixing pin to obtain a reconstructed three-dimensional model including the fixing pin and the bone; and aligning the reconstructed three-dimensional model with a standard three-dimensional model of a positioning device including at least the fixing pin to obtain first position information of the fixing pin in the image coordinate system.

[0159] Among them, the initial image of the bone can be obtained through CT scanning. Specifically, the patient can carry the fixation nail 400 for CT scanning, and then the obtained CT image is segmented to obtain a segmented image of the bone including the retention nail area (i.e., the initial image of the bone), and then the segmented image is three-dimensionally reconstructed through a three-dimensional reconstruction method to obtain a reconstructed three-dimensional model of the bone and the fixation nail 400 (i.e., the segmentation model).

[0160] In addition, the standard three-dimensional model of the positioning device including at least the fixing pin can be a pre-designed three-dimensional model of at least part of the positioning device, for example, it can be a standard three-dimensional model of the fixing pin, or a standard three-dimensional model of the entire positioning device.

[0161] Entering the registration stage, we can first import the standard 3D model into the image space. When importing, the standard 3D model can be Figure 8 As shown, only the three-dimensional model of the fixing pin 400 can be imported. Fig. 9 The three-dimensional model of the fixing nail 400 and the non-return nail 500 is shown in FIG. Figure 1 As shown, the three-dimensional model of the entire positioning device is imported, so that the three-dimensional model of the positioning device and the skeleton can be directly obtained after alignment.

[0162] At this time, two fixing pins are displayed in the image space at the same time, one is the fixing pin in the standard 3D model, and the other is the fixing pin in the reconstructed 3D model after 3D reconstruction. Then the standard 3D model and the reconstructed 3D model can be registered, that is, the standard 3D model is brought closer to the reconstructed 3D model, and the positions of the two are fine-tuned so that the fixing pins of the two are approximately overlapped. When the overlap meets the requirements, it can be determined that the registration is completed, and the registered standard 3D model, that is, the registration result, is obtained.

[0163] After the registration is completed, the position information of the fixing pin in the standard three-dimensional model at this time can be obtained and used as the first position information of the fixing pin 400 in the image coordinate system.

[0164] It can be understood that since the accuracy of the reconstructed three-dimensional model after three-dimensional reconstruction is lower than that of the standard three-dimensional model, through alignment, the position information of the standard three-dimensional model with higher accuracy can be used to replace the position information of the fixed pins in the reconstructed three-dimensional model, which is beneficial to the accuracy of subsequent calibration and surgical navigation.

[0165] In some embodiments, the registration may further include a rough registration (primary registration) and a fine registration (secondary registration) stage to improve the registration accuracy, which will be described in detail below.

[0166] Specifically, the above steps of registering the reconstructed 3D model with the standard 3D model to obtain the first position information of the fixing pin in the image coordinate system may include: using the first feature point in the standard 3D model and the second feature point in the reconstructed 3D model corresponding to the first feature point, performing an initial registration on the reconstructed 3D model and the standard 3D model to obtain the standard 3D model after the initial registration, wherein the first feature point is a feature point in the feature part of the fixing pin; performing a secondary registration on the reconstructed 3D model and the standard 3D model after the initial registration to obtain the standard 3D model after the secondary registration; and determining the first position information of the fixing pin or the feature hole thereon in the image coordinate system based on the standard 3D model after the secondary registration.

[0167] The first feature point is a point in the feature part of the fixing pin in the standard three-dimensional model. For example, the first feature point can be the tip of the feature hole 431, or it can be a corner point on the second identification part 432. The second feature point is a point in the reconstructed three-dimensional model corresponding to the first feature point, that is, the two points are at the same position, but one is located in the standard three-dimensional model and the other is located in the reconstructed three-dimensional model. For example, the first feature point is the feature hole 431 ( Figure 8 When the second feature point is the tip point of the corresponding feature hole (i.e., the feature hole closest to the second identification portion 432) in the reconstructed three-dimensional model, the second feature point is also the tip point of the corresponding feature hole (i.e., the feature hole closest to the second identification portion) in the reconstructed three-dimensional model.

[0168] By selecting the first feature points at three different positions in the standard 3D model and the three second feature points in the reconstructed 3D model that correspond to the three first feature points, the initial registration can be performed. For example, the position of the standard 3D model is adjusted so that the first feature point and the corresponding second feature point approximately coincide with each other, and the initial registration (i.e., rough registration) is performed to obtain the standard 3D model after the initial registration. It can be understood that after each registration, the standard 3D model is closer to the reconstructed 3D model, and the position of the reconstructed 3D model does not change.

[0169] Then, a data registration algorithm can be used to perform a secondary registration based on the primary registration. For example, ICP (Iterative Closest Point, a point cloud registration algorithm used to align the coordinate systems of two or more point clouds) can be used to achieve a secondary registration (i.e., precise registration) between the reconstructed 3D model and the standard 3D model after the primary registration, and obtain a registration result, i.e., a standard 3D model after the secondary registration.

[0170] It can be understood that after two registrations, the fixed pins in the standard three-dimensional model and the reconstructed three-dimensional model can be considered to have almost identical positions, and the position information of the fixed pin in the standard three-dimensional model can be used as the first position information of the fixed pin 400 in the image coordinate system, so that the position of the fixed pin in the image space can be represented by a more accurate standard three-dimensional model, and the position information of the feature points on the fixed pin can also be found more accurately.

[0171] The registration method provided in this embodiment is simple and accurate, does not require fixing multiple marking points on the bones, and reduces damage to the human body.

[0172] It can be understood that in some embodiments, the sensor assembly can be installed on the fixing pin before the CT scan, and then the CT scan can be performed, that is, the initial image of the bone and the reconstructed three-dimensional model include the entire positioning device, and the feature points on the sensor assembly (such as the protective shell) can be used for alignment in the subsequent alignment stage.

[0173] In addition, in some embodiments, after the registration is completed, the registration result can also be verified. For example, after determining the first position information of the fixing pin in the image coordinate system, the method 1100 can also include: obtaining the first reference position information of the first verification point in the fixing pin in the reconstructed three-dimensional model after the secondary registration, and the second reference position information of the first verification point in the standard three-dimensional model after the secondary registration; and verifying the result of the secondary registration based on the first reference position information and the second reference position information.

[0174] Among them, the first verification point can also be a feature point in the feature part 430 that does not participate in the alignment, that is, it is different from the first feature point (second feature point) used in the above-mentioned alignment process. The first reference position information of the first verification point in the standard three-dimensional model after the secondary alignment and its second reference position information in the standard three-dimensional model after the secondary alignment can be determined through the image coordinate system.

[0175] Then, the first difference between the first reference position information and the second reference position information, such as the distance between the two, can be calculated, and the first difference can be compared with the first difference threshold. If the first difference threshold is met, it means that the alignment is qualified and the subsequent steps can be performed. Otherwise, it means that the alignment is unqualified and re-registration is required.

[0176] Of course, in other embodiments, verification of registration may also be achieved by comparing the difference map between the standard 3D model and the segmentation model.

[0177] In this embodiment, by verifying the registration result, an accurate data basis can be provided for subsequent registration, thereby avoiding the need to re-perform CT scanning, registration and other steps after registration problems occur, which affects the progress of the operation.

[0178] In addition, it can be understood that the registration stage uses a three-dimensional model, so it is performed in the image space and the data involved are also data in the image coordinate system.

[0179] In some embodiments, determining the first position conversion relationship between the electromagnetic sensor and the positioning device in step S1101 may include: using a calibration tool to determine the fourth position information of the third feature point in the positioning device in the magnetic field coordinate system; based on the fourth position information and the second position information of the electromagnetic sensor in the magnetic field coordinate system, determining the first position conversion relationship between the electromagnetic sensor and the positioning device.

[0180] Specifically, the calibration stage can be completed when the positioning device leaves the factory, or at the beginning of the operation (i.e., before the operation is officially performed), and then the sensor assembly is installed on the fixing pin 400 to complete the assembly of the entire positioning device. The calibration process can also be performed before the operation, or during the operation if the patient's waiting time is not considered, so the order of the calibration steps is not specifically limited.

[0181] Then, in the magnetic field space, the third feature point on the positioning device can be selected using a calibration tool to obtain fourth position information of the third feature point. It can be understood that the fourth position information is position information in the magnetic field coordinate system. The third feature point can be a feature point in the sensor component, such as a corner point or a sharp point.

[0182] In addition, in the magnetic field space, the electromagnetic tracking device can be directly used to obtain the second position information of the electromagnetic sensor 100 in the magnetic field coordinate system.

[0183] The first position conversion relationship between the electromagnetic sensor and the positioning device can be obtained according to the fourth position information and the second position information, so as to facilitate the subsequent acquisition of the coordinate system transformation relationship between the two coordinate systems, which is helpful for the implementation of surgical navigation.

[0184] In some embodiments, the third feature point is a feature point in the sensor assembly of the positioning device. The above steps of determining the first position conversion relationship between the electromagnetic sensor and the positioning device based on the fourth position information and the second position information of the electromagnetic sensor in the magnetic field coordinate system may include: determining the second position conversion relationship between the electromagnetic sensor and the sensor assembly based on the fourth position information and the second position information of the electromagnetic sensor in the magnetic field coordinate system; and determining the first position conversion relationship between the electromagnetic sensor and the positioning device based on the second position conversion relationship and the standard three-dimensional model of the positioning device, wherein the standard three-dimensional model of the positioning device is a pre-designed three-dimensional model of the positioning device.

[0185] The third characteristic point in this embodiment is a characteristic point in the sensor assembly. For example, it can be the tip of the calibration hole 224 in the protective shell 220, or it can be a corner point or a sharp point on other characteristic surfaces of the sensor assembly, for example, a characteristic point on the bottom surface of the spline 215 of the mounting member 210.

[0186] During the calibration process, multiple, for example, 3 or more third feature points may be selected to obtain the coordinates of these feature points in the magnetic field coordinate system (fourth position information).

[0187] The second position conversion relationship between the electromagnetic sensor and the sensor component, such as a second position conversion matrix, can be obtained through the fourth position information (magnetic field coordinate system) of the third characteristic point on the sensor component and the second position information (magnetic field coordinate system) of the electromagnetic sensor 100.

[0188] The second position transformation relationship can be used to characterize the transformation relationship between the posture (position information and attitude information) of the electromagnetic sensor and the posture of the design origin of the sensor component. The posture of any point in the sensor component in the magnetic field coordinate system can be calculated through the standard three-dimensional model of the sensor component.

[0189] It can be understood that the step of obtaining the second position conversion relationship can be performed during the operation, or it can be calibrated before the positioning device or sensor assembly leaves the factory. During the operation, the sensor assembly can be directly installed on the fixing pin, and there is no need to perform this calibration step during the operation. It is only necessary to calculate the position relationship between the fixing pin and the sensor assembly, which can shorten the operation time.

[0190] After determining the second position conversion relationship, the first position conversion relationship between the electromagnetic sensor and the positioning device, such as the first position relationship conversion matrix, can be obtained by combining the standard three-dimensional model of the positioning device (i.e., the design model of the positioning device) with the second position conversion relationship. It can be understood that since the position relationship between the sensor assembly and the fixing pin in the positioning device is unique and fixed, the position conversion relationship between the fixing pin and the sensor assembly can be obtained through the position relationship of each component in the standard three-dimensional model of the positioning device, and then the first position conversion relationship between the electromagnetic sensor and the positioning device can be obtained. The first position conversion relationship can be used to characterize the conversion relationship between the posture of the electromagnetic sensor and the posture of the design origin of the positioning device. The posture of any point in the positioning device can be calculated through the standard three-dimensional model of the positioning device.

[0191] It can be understood that in this embodiment, a point in the sensor assembly can be used as the third feature point, so that the step of obtaining the second position conversion relationship can be calibrated before the positioning device or the sensor assembly leaves the factory. During surgery, the sensor assembly can be directly installed on the fixing pin. There is no need to perform this calibration step during surgery. Only the position relationship between the fixing pin and the sensor assembly needs to be calculated, which can shorten the operation time.

[0192] It can be understood that in the above embodiment, the sensor assembly is first calibrated, and then the first position conversion relationship between the electromagnetic sensor and the positioning device is calculated through the standard three-dimensional model of the positioning device.

[0193] It is understandable that multiple third feature points can be selected during the calibration process, for example, third feature points whose distances between each other are greater than a distance threshold are selected, that is, three third feature points with a larger distribution distance are selected, thereby improving the accuracy of calibration.

[0194] In some embodiments, after the calibration is completed, a calibration verification step may be included to verify the calibration result. The surgery can be continued only if the verification result meets the accuracy requirement. Otherwise, re-calibration and other steps are required.

[0195] Calibration verification may include using a calibration tool to select a fourth feature point (point, corner point, etc.) on the sensor component to determine the third reference position information of the fourth feature point. Then, the fourth reference position information of the fourth feature point may be calculated through the second position information of the electromagnetic sensor and the first position conversion relationship. Then, the third reference position information and the fourth reference position information are compared, for example, the second difference between the two is calculated, and then whether the calibration is qualified is determined based on the relationship between the second difference and the second difference threshold.

[0196] In some embodiments, the fourth feature point is a feature point that is not used in the calibration process, that is, a feature point in the sensor component that is different from the third feature point.

[0197] In step S1202, third position information of the positioning device in the magnetic field coordinate system can be obtained by calculation based on the second position information of the electromagnetic sensor in the magnetic field coordinate system and the first position conversion relationship between the electromagnetic sensor and the positioning device obtained by the calibration method.

[0198] In step S1203, the fifth position information of the positioning device in the image coordinate system can be obtained based on the first position information of the fixing pin in the image coordinate system obtained in step S1201 and the standard three-dimensional model of the positioning device, and then the coordinate system transformation relationship between the image coordinate system and the magnetic field coordinate system can be obtained based on the fifth position information of the positioning device in the image coordinate system and the third position information of the positioning device in the magnetic field coordinate system.

[0199] In some embodiments, step S1102, determining the coordinate system transformation relationship between the image coordinate system and the magnetic field coordinate system, may include:

[0200] Determine the first position coordinates of the feature point on the fixing pin in the image coordinate system based on the standard three-dimensional model of the positioning device including at least the fixing pin and the registration result with the reconstructed three-dimensional model including the fixing pin and the bone;

[0201] Determine a second position coordinate of a characteristic point on the fixing pin in the magnetic field coordinate system;

[0202] Based on the first position coordinates and the second position coordinates, a coordinate system transformation relationship between the image coordinate system and the magnetic field coordinate system is determined.

[0203] In this embodiment, the method for determining the coordinate system transformation relationship can also be performed by finding the first position coordinates of the feature points on the fixing pin 400 in the image coordinate system in the registered standard three-dimensional model (i.e., the registration result), selecting these feature points on the fixing pin 400 by using a calibration tool, obtaining the second position coordinates of the feature points in the magnetic field space, and calculating the coordinate system transformation relationship according to the first position coordinates and the second position coordinates. The feature points on the fixing pin can be points on the feature part of the fixing pin.

[0204] It can be understood that after obtaining the coordinate system transformation matrix, the position information of the tip of the calibration tool in the magnetic field coordinate system can be converted into its position information in the image coordinate system, so that the calibration tool can be displayed in the virtual image of the display. Similarly, the surgical instrument (an electromagnetic sensor is also provided in the surgical tool) can also be displayed in the virtual image.

[0205] In certain embodiments, after determining the coordinate system transformation relationship, the coordinate system transformation relationship can also be verified before surgery.It is understandable that after the fixing nail is driven into the bone, when the steps such as CT scanning and installation of the sensor assembly are performed, once there is a situation of rotation or displacement, or the sensor appears loose and displaced after calibration, the result of the patient's spatial registration will be affected. Therefore, the calibration tool can be used to touch the bone surface for verification before surgery. If the calibration tool touches the bone surface, the calibration tool in the virtual image on the display is also close to the bone surface (i.e., the error is within a certain range, such as 2mm), it means that the accuracy is qualified, and formal surgical operation can be performed.In addition, in order to further improve the accuracy of verification, contact verification operation can be performed in all directions.

[0206] It is understandable that method 1100 may also include calibrating surgical instruments, so that images of bones, surgical instruments, calibration tools, etc. can be displayed in real time on the virtual image of the display, so as to utilize virtual image navigation to assist orthopedic surgery.

[0207] The method 1100 provided in this embodiment is simple and easy to implement, and only one CT scan is required before the operation, which reduces the radiation damage to the patient and the doctor, and has the advantages of small surgical trauma, no radiation during the operation, short operation time, accurate fracture reduction, low labor intensity for the operator, less pain for the patient, and short recovery time after the operation. The registration method provided by the method 1100 provided in this embodiment is simple and accurate, and does not require fixing multiple marking points on the bones, thereby reducing damage to the human body.

[0208] In a specific embodiment, the virtual image display method 1100 and the positioning device can be used in orthopedic surgery. The preoperative preparation for orthopedic surgery (before the formal surgical operation begins) can include fixed calibration, calibration verification, fixation pins 400, CT scanning, registration (alignment, alignment verification, and determination of coordinate system transformation between image space and magnetic field space) and preoperative verification stage.

[0209] (1) Calibration (calibration of electromagnetic sensor and positioning device): Install the sensor assembly, and use the calibration tool to select the third characteristic points of the positioning device (for example, the calibration holes on the protective shell and / or other characteristic surfaces of the sensor assembly), so as to obtain the fourth position information of these third characteristic points in the magnetic field coordinate system, and obtain the second position conversion relationship between the electromagnetic sensor and the sensor assembly. Then, through the standard three-dimensional model of the positioning device, the first position conversion relationship between the electromagnetic sensor and the positioning device is calculated to complete the calibration. This step can be calibrated before the positioning device leaves the factory, so that after it is installed during surgery, no calibration is required, thus shortening the surgery time for the patient. Of course, the positioning device can also be calibrated as a whole during surgery to obtain the first position conversion relationship.

[0210] (2) Calibration verification: Use the calibration tool to select the characteristic holes to verify the calibration results. If the calibration results meet the accuracy requirements, the surgery can be continued. Otherwise, the calibration steps need to be repeated.

[0211] (3) Fixing the fixing nail 400: Fix the fixing nail 400 to the bone of the surgical site. At this time, at least two anti-return nails 500 may be added to the bone, such as Figure 8 The three check nails 500 are further reinforced and locked to prevent the position change between the fixing nail 400 and the bone.

[0212] (4) CT scan: The patient can carry the fixation pin 400 for CT scanning, and then segment the obtained CT image to obtain a segmented image of the bone including the fixation pin area, and then reconstruct the segmented image in three dimensions through a three-dimensional reconstruction method to obtain a reconstructed three-dimensional model of the bone and the fixation pin 400 (i.e., the segmented model).

[0213] (5) The first stage of registration (alignment): select three or more feature points at the corresponding positions of the standard 3D model and the reconstructed 3D model of the fixation pin for rough alignment, and then use the ICP algorithm for fine alignment to determine the first position information of the fixation pin in the image coordinate system. After the alignment is completed, the alignment can be verified by finding the feature points in the fixation pin that are not involved in the alignment for comparison, or by using the difference map between the standard 3D model and the reconstructed 3D model for comparison. If the error is within a certain range, it means that the alignment is qualified and the next step can be continued.

[0214] (6) Registration second stage (determining the coordinate system transformation between the image space and the magnetic field space): using the first position transformation relationship and the first position information of the fixed pin in the image coordinate system, calculate the coordinate system transformation relationship between the image coordinate system and the magnetic field coordinate system.

[0215] It can be understood that in other embodiments, taking into account the errors in the processing of the fixing pins, more than three feature points on the fixing pins can also be used for this stage, that is, the feature points on the fixing pins (such as the tips of the feature holes) are selected using a calibration tool to find their exact coordinates in the magnetic field space, and the coordinates of these feature points in the image space are found in the aligned standard three-dimensional model, and the coordinate system transformation relationship between the image coordinate system and the magnetic field coordinate system is obtained by calculation.

[0216] (8) Preoperative verification: Before surgery, the calibration tool is used to touch the bone surface for verification. If the calibration tool in the virtual image on the display is also close to the bone surface when the calibration tool touches the bone surface (that is, the error is within a certain range, such as 2 mm), it means that the accuracy is qualified and the surgical operation can be performed.

[0217] (9) Surgical navigation: In the subsequent surgical process, since the electromagnetic sensor and the bone remain relatively fixed, the position and posture (posture) change of the electromagnetic sensor 100 can be converted into the posture change of the positioning device through the first position conversion relationship between the electromagnetic sensor and the positioning device, and the position relationship between the positioning device and the bone has been obtained through the image, so the position of the electromagnetic sensor can be converted into the position and posture change of the bone. Since the position and posture of the electromagnetic sensor in the magnetic field space can be obtained through the electromagnetic tracking device, then according to the coordinate system transformation relationship, the position and posture of the electromagnetic sensor in the magnetic field space can be finally converted into the position and posture of the bone in the image space, and then the virtual image of the bone in the image space (i.e., displayed on the display) is updated, so as to perform surgical navigation.

[0218] In addition, surgical instruments can be calibrated so that images of bones, surgical instruments, calibration tools, etc. can be displayed on the monitor in real time, that is, navigation can be used to assist orthopedic surgery.

[0219] The positioning device in this embodiment can be the positioning device described in any of the above embodiments. The positioning device can firmly connect the bones and protect the electromagnetic sensor, and can also be used to achieve registration and calibration. The virtual image display method disclosed in the present invention can achieve registration through the standard three-dimensional model and the reconstructed three-dimensional model, and use the registered standard three-dimensional model to achieve the conversion of the coordinate system of the image space and the magnetic field space, which is conducive to more accurate surgical navigation.

[0220] Fig.13 is a schematic block diagram of a virtual image display device provided by an embodiment of the present disclosure; please refer to Fig.13 The embodiment of the present disclosure provides a virtual image display device 1300, comprising the following units:

[0221] A first determining unit 1301 is used to determine a first position conversion relationship between the electromagnetic sensor on the positioning device and the positioning device as described in any one of the above items;

[0222] The second determining unit 1302 is used to determine a coordinate system transformation relationship between an image coordinate system and a magnetic field coordinate system, wherein the image coordinate system is a coordinate system of an image space where a three-dimensional model of a bone connected with a fixing pin is located, the fixing pin is a fixing pin in a positioning device, and the magnetic field coordinate system is a coordinate system of a magnetic field space where an electromagnetic sensor in the positioning device is located;

[0223] The third determining unit 1303 is used to determine the posture information of the skeleton in the image coordinate system based on the posture information of the electromagnetic sensor in the magnetic field coordinate system, the first position conversion relationship between the electromagnetic sensor and the positioning device, and the coordinate system transformation relationship between the image coordinate system and the magnetic field coordinate system;

[0224] The display unit 1304 is used to display the skeleton in the virtual image corresponding to the image coordinate system based on the posture information of the skeleton in the image coordinate system.

[0225] In some embodiments, the second determination unit 1302 is further used to: determine the first position information of the fixing pin in the image coordinate system based on the standard three-dimensional model of the positioning device at least including the fixing pin and the registration result with the reconstructed three-dimensional model including the fixing pin and the bone; obtain the third position information of the positioning device including the fixing pin in the magnetic field coordinate system based on the second position information of the electromagnetic sensor in the magnetic field coordinate system and the first position conversion relationship; determine the coordinate system conversion relationship between the image coordinate system and the magnetic field coordinate system based on the first position information and the third position information;

[0226] Alternatively, the second determination unit 1302 is also used to: determine the coordinate system transformation relationship between the image coordinate system and the magnetic field coordinate system, including: determining the first position coordinates of the feature point on the fixing pin in the image coordinate system based on a standard three-dimensional model of a positioning device that at least includes a fixing pin, and a registration result with a reconstructed three-dimensional model including the fixing pin and the bone; determining the second position coordinates of the feature point on the fixing pin in the magnetic field coordinate system; and determining the coordinate system transformation relationship between the image coordinate system and the magnetic field coordinate system based on the first position coordinates and the second position coordinates.

[0227] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0228] An embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of any one of the above embodiments.

[0229] An embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute a method according to any one of the above embodiments.

[0230] Fig.14 is a block diagram of an electronic device for implementing the virtual image display method of the embodiment of the present disclosure. Fig.14 As shown, the electronic device includes: a memory 1410 and a processor 1420, and the memory 1410 stores a computer program that can be run on the processor 1420. The number of memories 1410 and processors 1420 can be one or more. The memory 1410 can store one or more computer programs. When the one or more computer programs are executed by the electronic device, the electronic device executes the method provided by the above method embodiment. The electronic device may also include: a communication interface 1430, which is used to communicate with external devices and perform data exchange transmission.

[0231] If the memory 1410, the processor 1420 and the communication interface 1430 are implemented independently, the memory 1410, the processor 1420 and the communication interface 1430 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.14 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0232] Optionally, in a specific implementation, if the memory 1410, the processor 1420 and the communication interface 1430 are integrated on a chip, the memory 1410, the processor 1420 and the communication interface 1430 can communicate with each other through an internal interface.

[0233] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0234] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0235] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (for example: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (for example: infrared, Bluetooth, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc. It is worth noting that the computer-readable storage medium mentioned in the present disclosure may be a non-volatile storage medium, in other words, a non-transient storage medium.

[0236] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0237] In the description of the embodiments of the present disclosure, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0238] In the description of the embodiments of the present disclosure, unless otherwise specified, " / " means or, for example, A / B can mean A or B. "And / or" in this article is only a way to describe the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0239] In the description of the embodiments of the present disclosure, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, "plurality" means two or more.

[0240] In the description of the present disclosure, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present disclosure and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present disclosure.

[0241] In the present disclosure, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integrated connection. It can be a mechanical connection, an electrical connection, or a communication. It can be directly connected or indirectly connected through an intermediate medium. It can be the internal connection of two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present disclosure can be understood according to the specific circumstances.

[0242] In the present disclosure, unless otherwise expressly specified and limited, a first feature being “above” or “below” a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being “above”, “above”, and “above” a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being “below”, “below”, and “below” a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0243] The disclosure above provides many different embodiments or examples to implement different structures of the present disclosure. In order to simplify the present disclosure, the components and settings of specific examples are described above. Of course, they are only examples, and the purpose is not to limit the present disclosure. In addition, the present disclosure can repeat reference numbers and / or reference letters in different examples, and this repetition is for the purpose of simplification and clarity, which itself does not indicate the relationship between the various embodiments and / or settings discussed.

[0244] The above description is only an exemplary embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A virtual image display method, comprising: Determining a first position conversion relationship between the electromagnetic sensor on the positioning device and the positioning device; Determine a coordinate system transformation relationship between an image coordinate system and a magnetic field coordinate system, wherein the image coordinate system is a coordinate system of an image space where a three-dimensional model of a bone connected with a fixing pin is located, the fixing pin is a fixing pin in the positioning device, and the magnetic field coordinate system is a coordinate system of a magnetic field space where an electromagnetic sensor in the positioning device is located; Determine the posture information of the bone in the image coordinate system based on the posture information of the electromagnetic sensor in the magnetic field coordinate system, the first position conversion relationship between the electromagnetic sensor and the positioning device, and the coordinate system transformation relationship between the image coordinate system and the magnetic field coordinate system; Based on the posture information of the skeleton in the image coordinate system, the skeleton is displayed in a virtual image corresponding to the image coordinate system.

2. The method according to claim 1, wherein: Determine the coordinate transformation relationship between the image coordinate system and the magnetic field coordinate system, including: Determine first position information of the fixing pin in the image coordinate system based on a standard three-dimensional model of the positioning device including at least the fixing pin and a registration result with a reconstructed three-dimensional model including the fixing pin and the bone; Based on the second position information of the electromagnetic sensor in the magnetic field coordinate system and the first position conversion relationship, obtaining the third position information of the positioning device including the fixing pin in the magnetic field coordinate system; Determine a coordinate system transformation relationship between an image coordinate system and a magnetic field coordinate system based on the first position information and the third position information; or, Determine the coordinate transformation relationship between the image coordinate system and the magnetic field coordinate system, including: Determine the first position coordinates of the feature point on the fixing pin in the image coordinate system based on a standard three-dimensional model of the positioning device including at least the fixing pin and a registration result with a reconstructed three-dimensional model including the fixing pin and the bone; Determine a second position coordinate of a feature point on the fixing pin in the magnetic field coordinate system; Based on the first position coordinates and the second position coordinates, a coordinate system transformation relationship between an image coordinate system and a magnetic field coordinate system is determined.

3. A virtual image display device, comprising: A first determining unit, used to determine a first position conversion relationship between the electromagnetic sensor on the positioning device and the positioning device; A second determining unit is used to determine a coordinate system transformation relationship between an image coordinate system and a magnetic field coordinate system, wherein the image coordinate system is a coordinate system of an image space where a three-dimensional model of a bone connected with a fixing pin is located, the fixing pin is a fixing pin in a positioning device, and the magnetic field coordinate system is a coordinate system of a magnetic field space where an electromagnetic sensor in the positioning device is located; A third determining unit is used to determine the posture information of the bone in the image coordinate system based on the posture information of the electromagnetic sensor in the magnetic field coordinate system, the first position conversion relationship between the electromagnetic sensor and the positioning device, and the coordinate system transformation relationship between the image coordinate system and the magnetic field coordinate system; A display unit is used to display the skeleton in a virtual image corresponding to the image coordinate system based on the posture information of the skeleton in the image coordinate system.

4. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of claim 1 or 2.

5. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to claim 1 or 2.

Images