Image processing method and system in orthopedic surgery, electronic device and storage medium

Abstract

An image processing method and system in orthopedic surgery, electronic equipment and storage medium are disclosed. A plurality of setting components provided with positioning target balls are fixed on the target bone at the lesion of a patient. The image processing method comprises: acquiring a virtual bone image corresponding to the target bone at the lesion of the patient; acquiring target coordinate information of each setting component based on the positioning target ball; acquiring a first virtual image corresponding to the setting component based on the target coordinate information; acquiring a real image corresponding to the target bone and the setting component at the lesion of the patient; acquiring actual relative position information between each setting component and the target bone based on the real image; and adjusting the relative position between the first virtual image and the virtual bone image based on the actual relative position information. The present application improves the accuracy of orthopedic surgery while reducing the radiation dose during the operation process, effectively improving the safety of surgical operation.

Description

Technical Field

[0001] This application relates to the field of medical devices, and more particularly to an image processing method, system, electronic device, and storage medium for orthopedic surgery. Background Technology

[0002] With the development of technology, mixed reality devices are often used in orthopedic surgery to assist in the procedure, so that doctors can better locate the lesion and the surgical site.

[0003] The errors generated during the initial registration of existing mixed reality devices are added to the total error of the system, which can easily lead to a large deviation between the virtual image of the mixed reality device and the actual object. This reduces the accuracy of mixed reality glasses in orthopedic surgery and increases safety risks. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the defects in the prior art where the virtual images of mixed reality devices deviate significantly from the actual objects, resulting in poor accuracy and low safety in orthopedic surgery. The present invention provides an image processing method, system, electronic device and storage medium for orthopedic surgery.

[0005] The present invention solves the above-mentioned technical problems through the following technical solution:

[0006] An image processing method for orthopedic surgery involves fixing several pre-set components equipped with positioning target balls onto the target bone at the lesion site of the patient. The image processing method includes:

[0007] Obtain a virtual image of the bone corresponding to the target bone at the lesion site of the patient;

[0008] Based on the positioning target ball, obtain the target coordinate information of each of the set components;

[0009] Based on the target coordinate information, obtain the first virtual image corresponding to the designated component;

[0010] Acquire a real-world image of the target bone at the patient's lesion site and the corresponding set component;

[0011] Based on the real-world image, obtain the actual relative position information between each of the defined components and the target skeleton;

[0012] The relative position between the first virtual image and the skeletal virtual image is adjusted based on the actual relative position information.

[0013] Preferably, the image processing method further includes:

[0014] Pre-set the numbering information corresponding to different of the specified components;

[0015] Adjusting the relative position between the first virtual image and the skeletal virtual image based on the actual relative position information includes:

[0016] Based on the actual relative position information, the position of each designated component in the first virtual image is adjusted sequentially according to the numbering information until the relative position between the first virtual image and the skeletal virtual image reaches the actual relative position corresponding to the actual relative position information; or...

[0017] Based on the actual relative position information, the position of any set component in the first virtual image is randomly adjusted, and the adjustment of the position of all set components is completed until the relative position between the first virtual image and the skeletal virtual image reaches the actual relative position corresponding to the actual relative position information.

[0018] Preferably, the step of adjusting the position of each designated component in the first virtual image sequentially according to the numbering information based on the actual relative position information includes:

[0019] Choose any of the aforementioned settings as a reference adjustment point;

[0020] Obtain the first relative position information between the reference adjustment point and the target skeleton in the real image;

[0021] Based on the first relative position information, the relative position of the reference adjustment point between the first virtual image and the skeletal virtual image is adjusted until the relative position of the reference adjustment point reaches the first relative position corresponding to the first relative position information.

[0022] The adjustment range is determined based on the reference adjustment point;

[0023] Within the adjustment range, adjust the relative positions of all remaining set components in the first virtual image with respect to the skeletal virtual image until the relative positions of all set components reach the actual relative positions corresponding to the actual relative position information.

[0024] Preferably, determining the adjustment range based on the reference adjustment point includes:

[0025] Using the reference adjustment point as the center of the sphere, the distance between the reference adjustment point and any one of the remaining setting components is selected as the radius of the sphere to form a sphere;

[0026] Obtain the secant range formed by the sphere and the bone surface of the virtual bone image, and obtain the adjustment range based on the secant range.

[0027] Preferably, obtaining the target coordinate information of each of the designated components based on the positioning target ball includes:

[0028] Obtain the optical coordinate information of the positioning target ball;

[0029] The target coordinate information of the setting component corresponding to each positioning target ball is calculated based on the optical coordinate information.

[0030] Preferably, the step of adjusting the relative position between the first virtual image and the skeletal virtual image based on the actual relative position information further includes:

[0031] The first virtual image and the skeletal virtual image are combined to obtain the combined target virtual image.

[0032] Preferably, the step of combining the first virtual image and the skeletal virtual image to obtain the combined target virtual image further includes:

[0033] The target virtual image is sent to a mixed reality display device for display.

[0034] Preferably, acquiring the virtual bone image corresponding to the target bone at the patient's lesion site includes:

[0035] Obtain the skeletal model image corresponding to the target skeleton;

[0036] The skeletal model image is converted into a virtual skeletal image using a mixed reality display device;

[0037] And / or,

[0038] Obtaining the first virtual image corresponding to the designated component based on the target coordinate information includes:

[0039] A mixed reality display device is used to obtain the first virtual image corresponding to the set component based on the target coordinate information.

[0040] Preferably, acquiring the real-world image of the target bone at the patient's lesion site corresponding to the set component includes:

[0041] Several initial real-world images in multiple directions and dimensions are acquired using real-world imaging equipment;

[0042] The target real-world image corresponding to the target skeleton is obtained based on several initial real-world images.

[0043] An image processing system for orthopedic surgery includes several pre-set components equipped with positioning target balls fixed to the target bone at the lesion site of the patient. The image processing system includes: an optical positioning device, a mixed reality display device, a real image device, and a data processing server.

[0044] The mixed reality display device is used to acquire a virtual image of the skeleton corresponding to the patient's target skeleton;

[0045] The optical positioning device is used to obtain target coordinate information of each of the set components based on the positioning target ball;

[0046] The mixed reality display device is communicatively connected to the optical positioning device and is used to obtain a first virtual image corresponding to the set component based on the target coordinate information;

[0047] The real-image device is used to acquire real-images of the patient's target bone and the set component;

[0048] The data processing server is communicatively connected to the real-image device and is used to obtain the actual relative position information between each of the set components and the target skeleton based on the real-image.

[0049] The data processing server is also communicatively connected to the mixed reality display device, and is used to adjust the relative position between the first virtual image and the skeletal virtual image based on the actual relative position information.

[0050] Preferably, the data processing server includes:

[0051] A component numbering module is used to pre-set the numbering information corresponding to different specified components;

[0052] A sequential image adjustment module is used to adjust the position of each designated component in the first virtual image sequentially according to the numbering information, based on the actual relative position information, until the relative position between the first virtual image and the skeletal virtual image reaches the actual relative position corresponding to the actual relative position information; or...

[0053] The random image adjustment module is used to randomly adjust the position of any set component in the first virtual image based on the actual relative position information, and to iterate through all set components until the relative position between the first virtual image and the skeletal virtual image reaches the actual relative position corresponding to the actual relative position information.

[0054] Preferably, the sequential image adjustment module includes:

[0055] A reference adjustment point determination unit is used to arbitrarily select one of the set components as a reference adjustment point;

[0056] The first relative position acquisition unit is used to acquire the first relative position information between the reference adjustment point and the target bone in the real image.

[0057] The first relative position adjustment unit is used to adjust the relative position of the reference adjustment point in the first virtual image and the skeleton virtual image based on the first relative position information, until the relative position of the reference adjustment point reaches the first relative position corresponding to the first relative position information.

[0058] An adjustment range determination unit is used to determine the adjustment range based on the reference adjustment point;

[0059] The actual relative position adjustment unit is used to adjust the relative positions of all remaining set components in the first virtual image and the skeletal virtual image within the adjustment range, until the relative positions of all set components reach the actual relative positions corresponding to the actual relative position information.

[0060] Preferably, the adjustment range determining unit is further configured to:

[0061] Using the reference adjustment point as the center of the sphere, the distance between the reference adjustment point and any one of the remaining setting components is selected as the radius of the sphere to form a sphere;

[0062] Obtain the secant range formed by the sphere and the surface of the target bone, and obtain the adjustment range based on the secant range.

[0063] Preferably, the optical positioning device includes an optical locator and an optical positioning data processor connected in communication;

[0064] The optical locator is used to acquire the optical coordinate information of the positioning target ball;

[0065] The optical positioning data processor is used to calculate the target coordinate information of each of the set components based on the optical coordinate information.

[0066] Preferably, the data processing server is further configured to:

[0067] The first virtual image and the skeletal virtual image are combined to obtain the combined target virtual image.

[0068] Preferably, the mixed reality display device is also used to receive the target virtual image for display.

[0069] Preferably, the mixed reality display device further includes:

[0070] The image acquisition module is used to acquire the skeletal model image corresponding to the target skeleton;

[0071] An image conversion module is used to convert the skeleton model image into the skeleton virtual image;

[0072] And / or,

[0073] The mixed reality display device is also used to convert the target coordinate information into a first virtual image corresponding to the set component.

[0074] Preferably, the real-time imaging device includes:

[0075] The reality image acquisition module is used to acquire several initial reality images in multiple directions and dimensions;

[0076] The reality image conversion module is used to acquire the target reality image corresponding to the target skeleton and the set component based on several initial reality images.

[0077] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the image processing method in orthopedic surgery described above.

[0078] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method in orthopedic surgery described above.

[0079] The positive and progressive effects of this invention are as follows: the image processing method based on optical positioning equipment can meet the requirements of accuracy and real-time performance for guiding set components in orthopedic surgery, and improve the accuracy of virtual images in mixed reality display devices. This invention further enhances the accuracy of the invention by combining a set component equipped with a positioning target ball with a real-world imaging device to achieve image registration. Simultaneously, it eliminates the need for large, costly intraoperative imaging equipment, reducing hospital surgical costs, decreasing radiation doses to patients and doctors during surgery, and effectively improving the safety of surgical procedures. Attached Figure Description

[0080] Figure 1 A schematic flowchart of an image processing method in bone surgery provided as an exemplary embodiment of the present invention;

[0081] Figure 2 A schematic diagram of the structure of an image processing system in bone surgery provided as an exemplary embodiment of the present invention;

[0082] Figure 3A schematic diagram of a module of an image processing system in bone surgery is provided as an exemplary embodiment of the present invention;

[0083] Figure 4 An electronic device provided as an exemplary embodiment of the present invention. Detailed Implementation

[0084] The present invention will be further illustrated below by way of an exemplary embodiment, but the invention is not limited to the scope of the embodiments described herein.

[0085] An exemplary embodiment of the present invention provides an image processing method for orthopedic surgery, wherein the image processing method requires pre-fixing a plurality of setting components equipped with positioning target balls onto the target bone at the lesion site of the patient. The setting components can be any rigid surgical component that can be used for fixation in orthopedic surgery.

[0086] This application uses Kirschner wires as one embodiment of the setting component, but it is not limited to this and can be configured according to the type of orthopedic surgery. Kirschner wires can be inserted percutaneously into the lesion site of the patient, with one end of the wire abutting against the target bone at the lesion site. Since the contact point between the Kirschner wire and the target bone does not require special or excessively high requirements, using Kirschner wires as a setting component simplifies the surgeon's operation, and causes minimal trauma to the patient.

[0087] See Figure 1 The image processing method includes the following steps:

[0088] Step S101: Obtain the virtual image of the bone corresponding to the target bone at the lesion site of the patient.

[0089] In one embodiment, the virtual bone image can be acquired by preoperative imaging equipment, and the acquired virtual bone image is sent to a mixed reality display device for display. The mixed reality display device can display the virtual bone image corresponding to the target bone in the doctor's field of vision, but at this point it does not yet overlap with the real patient's bones.

[0090] In one embodiment, the mixed reality display device includes: mixed reality display glasses.

[0091] In one embodiment, step S101 specifically includes:

[0092] Step S1011: Obtain the bone model image corresponding to the target bone.

[0093] The bone model images are acquired by preoperative imaging equipment and sent to a mixed reality display device. The preoperative imaging equipment includes, but is not limited to, CT (computed tomography) equipment.

[0094] Step S1012: Convert the skeletal model image into the skeletal virtual image based on a mixed reality display device.

[0095] The mixed reality display device transforms the acquired skeletal model images. Generally, the skeletal model images acquired by preoperative imaging equipment are mostly two-dimensional or single-sided model images. Therefore, the mixed reality display device needs to synthesize the skeletal model images to form a three-dimensional or stereoscopic virtual skeletal image.

[0096] Step S102: Obtain the target coordinate information of each of the set components based on the positioning target ball.

[0097] In one embodiment, step S102 specifically includes:

[0098] S1021. Obtain the optical coordinate information of the positioning target ball.

[0099] Among them, the optical coordinate information of the positioning target ball can be collected by the optical positioning device.

[0100] S1022. Calculate the target coordinate information of the setting component corresponding to each positioning target ball based on the optical coordinate information.

[0101] Since the positioning target ball and the setting component are rigidly connected, the optical positioning device can perform relevant calculations based on the acquired optical coordinate information of the positioning target ball to obtain the target coordinate information of the setting component. The target coordinate information reflects the positional relationship between the setting component and the target skeleton.

[0102] Step S103: Obtain the first virtual image corresponding to the set component based on the target coordinate information.

[0103] In one embodiment, step S103 may employ a mixed reality display device to obtain the first virtual image corresponding to the set component based on the target coordinate information;

[0104] The mixed reality display device contains a mixed reality guidance program. After obtaining the target coordinate information of the set component, the mixed reality guidance program can form the first virtual image of the set component at the corresponding position of the virtual skeleton image according to the target coordinate information.

[0105] The positional relationship between the first virtual image and the skeletal virtual image depends on the positional relationship between the set component and the target skeleton. Therefore, the target coordinate information calculated by positioning the target ball can obtain a first virtual image with a relatively accurate position, reducing the workload of subsequent adjustments and improving the efficiency of image processing.

[0106] Step S104: Obtain the real-world image of the target bone at the patient's lesion site corresponding to the set component;

[0107] After generating the first virtual image of the set component, it is still necessary to further register the positional relationship between the first virtual image and the skeletal virtual image to make the virtual image more accurate. To achieve registration, a real image reflecting the actual positional relationship between the target skeleton and the set component is required as a reference.

[0108] In one embodiment, step S104 specifically includes:

[0109] Step S1041: Use a real-world imaging device to acquire several initial real-world images in multiple directions and dimensions.

[0110] The aforementioned real-image equipment refers to surgical imaging equipment that can be used during the surgical procedure. Specifically, the real-image equipment may include, but is not limited to, C-arm X-ray machines, O-arm X-ray machines, etc. For example, real-images in orthogonal directions can be obtained using a C-arm X-ray machine.

[0111] Step S1042: Obtain the target real-world image corresponding to the target skeleton and the set component based on several initial real-world images.

[0112] Specifically, a target real-world image is synthesized from several real-world images in multiple directions to accurately reflect the positional relationship between the target skeleton and the set component. The target real-world image can be a three-dimensional image displayed on a monitor or an AR image projected onto an AR display device. In reality, it is not limited to these two types.

[0113] Step S105: Obtain the actual relative position information between each of the set components and the target skeleton based on the real image.

[0114] In one embodiment, the actual relative position information between each of the set components and the target bone can be obtained by parsing the real image through a data processing server, or by a doctor interpreting the real image.

[0115] Step S106: Adjust the relative position between the first virtual image and the skeletal virtual image based on the actual relative position information.

[0116] In one embodiment, step S106 includes the following two different implementations, and either implementation can be selected in practice:

[0117] The first implementation involves pre-setting different numbering information corresponding to the set components, and adjusting the position of each set component in the first virtual image sequentially according to the actual relative position information based on the numbering information, until the relative position between the first virtual image and the skeletal virtual image reaches the actual relative position corresponding to the actual relative position information.

[0118] In one embodiment, the first implementation specifically includes:

[0119] Step S1061: Select any of the set components as a reference adjustment point.

[0120] Step S1062: Obtain the first relative position information between the reference adjustment point and the target skeleton in the real image.

[0121] The first relative position information between the reference adjustment point and the target bone can be obtained by the data processing server parsing the real image, or by the doctor interpreting the real image.

[0122] Step S1063: Based on the first relative position information, adjust the relative position of the reference adjustment point between the first virtual image and the skeletal virtual image until the relative position of the reference adjustment point reaches the first relative position corresponding to the first relative position information.

[0123] When adjusting the relative position of the reference adjustment point between the first virtual image and the virtual bone image, since one end of the setting component is always attached to the surface of the target bone at the patient's lesion in reality, the reference adjustment point in the first virtual image should also be attached to the surface of the virtual bone image during the adjustment process, so that the adjustment is more in line with the actual situation, reduces errors, and facilitates the subsequent adjustment process.

[0124] Step S1064: Determine the adjustment range based on the reference adjustment point;

[0125] In one embodiment, step S1064 specifically includes:

[0126] Using the reference adjustment point as the center of the sphere, the distance between the reference adjustment point and any one of the remaining setting components is selected as the radius of the sphere to form a sphere;

[0127] Obtain the secant range formed by the sphere and the bone surface of the virtual bone image, and obtain the adjustment range based on the secant range.

[0128] Since the relative positions of all the setting components are already determined, once the position of the reference adjustment point is determined, the adjustment range of each other setting component can be determined based on the positional information between the reference adjustment point and the setting component. In this embodiment, a sphere is drawn with the reference adjustment point as the center and the distance between the reference adjustment point and the setting component as the radius. The adjustment range of the setting component is determined by the secant line formed by the sphere and the bone surface of the skeletal virtual image. This greatly reduces the workload of adjustment and achieves more efficient and faster adjustment of the positional information between the setting components in the first virtual image and the skeletal virtual image. In reality, this implementation method is not limited to this one; adjustments can be made based on different relative positional information between the setting components, such as the relative distance and relative angle between the setting components.

[0129] Step S1065: Adjust the relative positions of all remaining set components in the first virtual image and the skeletal virtual image within the adjustment range until the relative positions of all set components reach the actual relative positions corresponding to the actual relative position information.

[0130] In one embodiment, this embodiment is further described with three setting components, but the number of setting components is not limited to three. The setting component numbered 1 is set as a reference adjustment point, and the relative position between the reference adjustment point and the target bone in the first virtual image is adjusted based on the first relative position information between the reference adjustment point and the target bone in the real image.

[0131] Once the position of the reference adjustment point, i.e. the setting component numbered 1, is determined, the adjustment range of the setting component numbered 2 can be determined based on the distance between the setting component numbered 2 and the reference adjustment point: a sphere is drawn with the reference adjustment point as the center and the distance between the setting component numbered 2 and the reference adjustment point as the radius, and the range of the secant line between the sphere and the bone surface of the virtual bone image is the adjustment range of the setting component numbered 2.

[0132] When adjusting the position of setting component number 3, the adjustment range is determined by drawing a sphere with the reference adjustment point as the center and the distance between setting component number 3 and the reference adjustment point as the radius. However, since the relative positions of setting component number 3 and setting component number 2 are fixed, the adjustment range of setting component number 3 must also satisfy the fixed relative position between it and setting component number 2. This further narrows the adjustment range between setting components number 2 and number 3, improving the accuracy and efficiency of image registration.

[0133] Finally, based on the adjustment ranges of the set component numbered 2 and the set component numbered 3, the relative positions of the set component numbered 2 and the set component numbered 3 in the first virtual image and the virtual skeleton image are adjusted respectively, until the relative positions of all set components reach the actual relative positions corresponding to the actual relative position information.

[0134] In one embodiment, in order to address situations where the real imaging equipment is not precise enough or the reference adjustment point deviates during the adjustment process, resulting in a deviation in the relative position between the set component in the first virtual image and the virtual skeletal image, the doctor can fine-tune the positional relationship between the first virtual image and the virtual skeletal image according to the actual situation.

[0135] The second implementation method is to randomly adjust the position of any set component in the first virtual image based on the actual relative position information, and then iterate through all the set components to complete the adjustment of their positions until the relative position between the first virtual image and the skeletal virtual image reaches the actual relative position corresponding to the actual relative position information.

[0136] In one embodiment, the image processing method further includes the following steps:

[0137] The first virtual image and the skeletal virtual image are combined to obtain the combined target virtual image.

[0138] In this process, after adjusting the relative positions of the components and the skeletal virtual image in the first virtual image, the data processing server combines the first virtual image and the skeletal virtual image to form a target virtual image with a fixed positional relationship. This reduces and avoids positional deviations in the image caused by movement in the mixed reality display device, thereby reducing errors and avoiding interference with the surgical process.

[0139] The target virtual image is sent to a mixed reality display device for display.

[0140] The data processing server sends the target virtual image to a mixed reality display device for display. The mixed reality display device presents the combined target virtual image to the doctor's field of vision. Because the positions of the reference components and the target skeleton are fixed, the doctor can determine the position of the target skeleton in the real world by aligning the reference components in the target virtual image with the reference components in the real world. The doctor can determine the surgical location through the mixed reality display device, making the surgical process more visual and improving surgical efficiency. At the same time, it reduces the harm to the patient caused by the continuous registration process of reference components, improving surgical safety.

[0141] An exemplary embodiment of the present invention also provides an image processing system for orthopedic surgery, wherein the image processing system requires pre-fixing a plurality of setting components equipped with positioning balls onto the target bone at the lesion site of the patient. The setting components can be any rigid surgical component that can be used for fixation in orthopedic surgery.

[0142] This application uses Kirschner wires as one embodiment of the setting component, but it is not limited to this and can be configured according to the type of orthopedic surgery. Kirschner wires can be inserted percutaneously into the lesion site of the patient, with one end of the wire abutting against the target bone at the lesion site. Since the contact point between the Kirschner wire and the target bone does not require special or excessively high requirements, using Kirschner wires as a setting component simplifies the surgeon's operation, and causes minimal trauma to the patient.

[0143] The image processing system includes: an optical positioning device 21, a mixed reality display device 23, a real-world image device 24, and a data processing server 22. Among them, Figure 2 The structure of the image processing system and the data transmission relationships between the devices in the system are shown.

[0144] See Figure 3 The image processing system includes:

[0145] Mixed reality display device 23 is used to acquire a virtual image of the skeleton corresponding to the target skeleton of the patient.

[0146] The virtual skeletal image can be acquired by preoperative imaging equipment and then sent to a mixed reality display device 23 for display. The mixed reality display device 23 can display the virtual skeletal image corresponding to the target bone in the doctor's field of vision, but at this point it does not yet overlap with the real patient's bones.

[0147] In one embodiment, the mixed reality display device 23 includes, but is not limited to, mixed reality display glasses.

[0148] In one embodiment, the mixed reality display device 23 specifically includes the following modules:

[0149] The image acquisition module is used to acquire the skeletal model image corresponding to the target skeleton.

[0150] The skeletal model images are acquired by preoperative imaging equipment and sent to mixed reality display device 23. The preoperative imaging equipment includes, but is not limited to, CT equipment.

[0151] An image conversion module is used to convert the skeletal model image into the skeletal virtual image.

[0152] The mixed reality display device 23 converts the acquired skeletal model images. Generally, the skeletal model images acquired by preoperative imaging equipment are mostly two-dimensional or single-sided model images. Therefore, the mixed reality display device 23 needs to synthesize the skeletal model images to form a three-dimensional or stereoscopic virtual skeletal image.

[0153] Optical positioning device 21 is used to obtain target coordinate information of each of the set components based on the positioning target ball.

[0154] In one embodiment, the optical positioning device 21 specifically includes an optical locator and an optical positioning data processor connected in communication.

[0155] An optical locator is used to acquire the optical coordinate information of the positioning target ball.

[0156] Among them, the optical coordinate information of the positioning target ball can be collected by the optical positioning device 21.

[0157] An optical positioning data processor is used to calculate the target coordinate information of each of the set components based on the optical coordinate information.

[0158] Since the positioning target ball and the setting component are rigidly connected, the optical positioning device 21 can perform relevant calculations based on the acquired optical coordinate information of the positioning target ball to obtain the target coordinate information of the setting component. The target coordinate information reflects the positional relationship between the setting component and the target skeleton.

[0159] The mixed reality display device 23 is communicatively connected to the optical positioning device 21 and is used to obtain the first virtual image corresponding to the set component based on the target coordinate information.

[0160] In one embodiment, the mixed reality display device 23 is used to convert the target coordinate information into a first virtual image corresponding to the set component.

[0161] The mixed reality display device 23 has a mixed reality guidance program. After obtaining the target coordinate information of the set component, the mixed reality guidance program can form a first virtual image of the set component at the corresponding position of the virtual skeleton image according to the target coordinate information.

[0162] The positional relationship between the first virtual image and the skeletal virtual image depends on the positional relationship between the set component and the target skeleton. Therefore, the target coordinate information calculated by positioning the target ball can obtain a first virtual image with a relatively accurate position, reducing the workload of subsequent adjustments and improving the efficiency of image processing.

[0163] The real-image device 24 is used to acquire a real-image of the patient's target skeleton corresponding to the set component.

[0164] After generating the first virtual image of the set component, it is still necessary to further register the positional relationship between the first virtual image and the skeletal virtual image to make the virtual image more accurate. To achieve registration, a real image reflecting the actual positional relationship between the target skeleton and the set component is required as a reference.

[0165] In one embodiment, the real-world imaging device 24 specifically includes the following modules:

[0166] The reality image acquisition module is used to acquire several initial reality images in multiple directions and dimensions.

[0167] The real-image device 24 is a surgical imaging device that can be used during the operation. Specifically, the real-image device 24 may include, but is not limited to, a C-arm X-ray machine, an O-arm X-ray machine, etc. For example, a real-image in an orthogonal direction can be obtained using a C-arm X-ray machine.

[0168] The reality image conversion module is used to acquire the target reality image corresponding to the target skeleton and the set component based on several initial reality images.

[0169] Specifically, a target real-world image is synthesized from several real-world images in multiple directions to accurately reflect the positional relationship between the target skeleton and the set component. The target real-world image can be a three-dimensional image displayed on a monitor or an AR image projected onto an AR display device. In reality, it is not limited to these two types.

[0170] The data processing server 22 is communicatively connected to the real-world imaging device 24 and is used to obtain the actual relative position information between each of the set components and the target skeleton based on the real-world imaging.

[0171] In one embodiment, the actual relative position information between each of the set components and the target bone can be obtained by the data processing server 22 parsing the real image, or by the doctor interpreting the real image.

[0172] The data processing server 22 is also communicatively connected to the mixed reality display device 23, and is used to adjust the relative position between the first virtual image and the skeletal virtual image based on the actual relative position information.

[0173] In one embodiment, the data processing server 22 includes the following two different types, and either type of data processing server 22 can be selected in practice:

[0174] The first type of data processing server 22 specifically includes the following modules:

[0175] The component numbering module is used to pre-set the numbering information corresponding to different components.

[0176] The sequential image adjustment module is used to adjust the position of each set component in the first virtual image sequentially according to the numbering information based on the actual relative position information, until the relative position between the first virtual image and the skeletal virtual image reaches the actual relative position corresponding to the actual relative position information.

[0177] In one embodiment, the sequential image adjustment module specifically includes the following units:

[0178] The reference adjustment point determination unit is used to arbitrarily select one of the set components as the reference adjustment point.

[0179] The first relative position acquisition unit is used to acquire the first relative position information between the reference adjustment point and the target skeleton in the real image.

[0180] The first relative position information between the reference adjustment point and the target bone can be obtained by the data processing server 22 through parsing the real image, or by the doctor's interpretation of the real image.

[0181] The first relative position adjustment unit is used to adjust the relative position of the reference adjustment point in the first virtual image and the skeletal virtual image based on the first relative position information, until the relative position of the reference adjustment point reaches the first relative position corresponding to the first relative position information.

[0182] When adjusting the relative position of the reference adjustment point between the first virtual image and the virtual bone image, since one end of the setting component is always attached to the surface of the target bone at the patient's lesion in reality, the reference adjustment point in the first virtual image should also be attached to the surface of the virtual bone image during the adjustment process, so that the adjustment is more in line with the actual situation, reduces errors, and facilitates the subsequent adjustment process.

[0183] The adjustment range determination unit is used to determine the adjustment range based on the reference adjustment point.

[0184] In one embodiment, the range determination unit is further configured to:

[0185] Using the reference adjustment point as the center of the sphere, the distance between the reference adjustment point and any one of the remaining setting components is selected as the radius of the sphere to form a sphere;

[0186] Obtain the secant range formed by the sphere and the bone surface of the virtual bone image, and obtain the adjustment range based on the secant range.

[0187] Since the relative positions of all the setting components are already determined, once the position of the reference adjustment point is determined, the adjustment range of each other setting component can be determined based on the positional information between the reference adjustment point and the setting component. In this embodiment, a sphere is drawn with the reference adjustment point as the center and the distance between the reference adjustment point and the setting component as the radius. The adjustment range of the setting component is determined by the secant line formed by the sphere and the bone surface of the skeletal virtual image. This greatly reduces the workload of adjustment and achieves more efficient and faster adjustment of the positional information between the setting components in the first virtual image and the skeletal virtual image. In reality, this implementation method is not limited to this one; adjustments can be made based on different relative positional information between the setting components, such as the relative distance and relative angle between the setting components.

[0188] The actual relative position adjustment unit is used to adjust the relative positions of all remaining set components in the first virtual image and the skeletal virtual image within the adjustment range, until the relative positions of all set components reach the actual relative positions corresponding to the actual relative position information.

[0189] In one embodiment, this embodiment is further described with three setting components, but the number of setting components is not limited to three. The setting component numbered 1 is set as a reference adjustment point, and the relative position between the reference adjustment point and the target bone in the first virtual image is adjusted based on the first relative position information between the reference adjustment point and the target bone in the real image.

[0190] Once the position of the reference adjustment point, i.e. the setting component numbered 1, is determined, the adjustment range of the setting component numbered 2 can be determined based on the distance between the setting component numbered 2 and the reference adjustment point: a sphere is drawn with the reference adjustment point as the center and the distance between the setting component numbered 2 and the reference adjustment point as the radius, and the range of the secant line between the sphere and the bone surface of the virtual bone image is the adjustment range of the setting component numbered 2.

[0191] When adjusting the position of setting component number 3, the adjustment range is determined by drawing a sphere with the reference adjustment point as the center and the distance between setting component number 3 and the reference adjustment point as the radius. However, since the relative positions of setting component number 3 and setting component number 2 are fixed, the adjustment range of setting component number 3 must also satisfy the fixed relative position between it and setting component number 2. This further narrows the adjustment range between setting components number 2 and number 3, improving the accuracy and efficiency of image registration.

[0192] Finally, based on the adjustment ranges of the set component numbered 2 and the set component numbered 3, the relative positions of the set component numbered 2 and the set component numbered 3 in the first virtual image and the virtual skeleton image are adjusted respectively, until the relative positions of all set components reach the actual relative positions corresponding to the actual relative position information.

[0193] In one embodiment, in order to address situations where the real imaging device 24 is not precise enough or the reference adjustment point deviates during the adjustment process, resulting in a deviation in the relative position between the set component and the virtual skeleton image in the first virtual image, the doctor can fine-tune the positional relationship between the first virtual image and the virtual skeleton image according to the actual situation.

[0194] The second type of data processing server 22 specifically includes the following modules:

[0195] The random image adjustment module is used to randomly adjust the position of any set component in the first virtual image based on the actual relative position information, and to iterate through all set components until the relative position between the first virtual image and the skeletal virtual image reaches the actual relative position corresponding to the actual relative position information.

[0196] In one embodiment, the data processing server 22 is further configured to: combine the first virtual image with the skeletal virtual image to obtain a combined target virtual image.

[0197] After adjusting the relative positions of the components and the skeletal virtual image in the first virtual image, the data processing server 22 combines the first virtual image and the skeletal virtual image to form a target virtual image with a fixed positional relationship. This reduces and avoids positional deviations in the image caused by movement in the mixed reality display device 23, thereby reducing errors and avoiding interference with the surgical process.

[0198] In one embodiment, the mixed reality display device 23 is also used to receive the target virtual image for display.

[0199] The data processing server 22 sends the target virtual image to the mixed reality display device 23 for display. The mixed reality display device 23 presents the combined target virtual image to the doctor's field of vision. Since the positions of the set components and the target skeleton are fixed, the doctor can determine the position of the target skeleton in the real world by aligning the set components in the target virtual image with the set components in the real world within their field of vision. The doctor can determine the surgical location through the mixed reality display device, making the surgical process more visual and improving surgical efficiency. Simultaneously, it reduces the harm to the patient caused by the continuous registration process of the set components, improving surgical safety.

[0200] The image processing solution of the present invention will be further illustrated below through a specific embodiment:

[0201] This invention provides an image processing method for orthopedic surgery using a mixed reality display device based on an optical positioning device. The orthopedic surgery is a minimally invasive pedicle screw placement surgery, which aims to improve the registration accuracy and ease of operation of the mixed reality display device-assisted minimally invasive pedicle screw placement surgery.

[0202] The mixed reality display device in the image processing system of this embodiment is a HoloLens head-mounted display (a type of mixed reality display glasses), the optical positioning device is an NDI (an optical positioning device), the data processing server is a mixed reality guidance program, and it also includes Kirschner wires numbered 1, 2, and 3 and equipped with NDI optical positioning target balls.

[0203] The NDI optical positioning device includes an NDI optical locator and an optical positioning data processor, which communicate with each other via a network cable. This enables the real-time transmission of the coordinate data of the NDI optical positioning target ball captured by the NDI optical locator to the optical positioning data processor. Since the instruments, including the Kirschner wire, are rigidly connected to the optical positioning target ball, the optical positioning data processor can perform relevant calculations based on the coordinates of the target ball to obtain the real-time coordinate position of the instruments, including the Kirschner wire.

[0204] This function loads preoperative spinal images of the patient's spine acquired by imaging equipment such as CT scans and displays them in the doctor's mixed reality glasses field of vision. At this time, the virtual spinal image is only displayed in the doctor's field of vision and does not overlap with the actual patient's spine. This virtual spinal image is the same as the virtual skeletal image in this invention.

[0205] During the procedure, the doctor inserts three Kirschner wires, numbered 1, 2, and 3, percutaneously into the affected area until the tips of the wires touch the patient's spine. Since there are no special requirements for the contact point between the Kirschner wires and the patient's spine, this step is relatively simple for the doctor, does not require precise manipulation, and causes very little trauma to the patient.

[0206] The NDI optical locator captures the optical positioning target ball on the Kirschner wires and calculates the coordinates of the three Kirschner wires using the optical positioning data processor. The coordinates of these three Kirschner wires are then sent to the mixed reality glasses.

[0207] After receiving the coordinates of the Kirschner wires, the mixed reality glasses display virtual images of the three Kirschner wires based on their coordinate positions and the relative coordinate positions of the three wires. These virtual images are the first virtual images described in this invention. The relative positional relationship between the virtual images of the Kirschner wires is derived from a high-precision optical positioning device, and therefore matches the relative positional relationship of the Kirschner wires already inserted into the patient's body.

[0208] The doctor uses a mobile C-arm X-ray machine in the operating room to create images of the patient's area in two orthogonal directions. Based on these images, the doctor can determine the positional relationship between the three Kirschner wires and the patient's bone, particularly the location of the contact points between the Kirschner wires and the bone surface. The C-arm X-ray machine used is the actual imaging device described in this invention.

[0209] Based on the positional relationship between the obtained Kirschner wires and the patient's bones, the doctor adjusts the position between the virtual image of the patient's spinal skeleton and the virtual images of the three Kirschner wires in the mixed reality pin placement guidance program until the positional relationship between the virtual image of the patient's spinal skeleton and the virtual images of the three Kirschner wires is consistent with the positional relationship presented by the C-arm X-ray machine.

[0210] The function of adjusting the position between the virtual image of the patient's spine and the virtual images of the three Kirschner wires ensures that the position between the three Kirschner wires remains constant throughout the adjustment process. Furthermore, the program includes auxiliary adjustment methods such as adsorption, position prediction, and fine-tuning.

[0211] The aforementioned adsorption function is achieved by ensuring that, when adjusting the position of Kirschner wire number 1, its endpoint P1 remains adsorbed onto the surface of the virtual image of the patient's spinal skeleton. This mirrors the real-world state of a Kirschner wire tip abutting against the bones of a real patient, and allows the doctor to easily adjust the contact point to the position imaged by the C-arm X-ray machine.

[0212] The predicted position function, once the position of Kirschner wire tip P1 (number 1) is determined, is such that, since the relative positions of Kirschner wires 1 and 2 are fixed, the movable range of tip P2 (number 2) is equivalent to a sphere formed with the position of tip P1 as its center and the distance between the tips of Kirschner wires 1 and 2 as its radius (|P2-P1|). Furthermore, to satisfy the adsorption function, this range is defined by the secant line formed by the intersection of this sphere and the bone surface. Movement within this range significantly reduces the workload for the physician.

[0213] The aforementioned fine-tuning function, when working with Kirschner wire number 3, ensures that only one point matches the position of Kirschner wire number 3's endpoint P3, based on the fixed positional relationship between the three Kirschner wire endpoints P3 and under the condition of satisfying the adsorption. However, to avoid over-constraint, inaccurate virtual CT images, and misalignment of the first two Kirschner wires, the fine-tuning function allows doctors to fine-tune the position between the virtual image of the spinal skeleton and the virtual image of the three Kirschner wires according to actual needs, without being subject to other constraints.

[0214] After completing the above operations, the position between the virtual image of the patient's spine and the virtual images of the three Kirschner wires is determined. The mixed reality pin placement guidance program forms a fixed-position combination of the virtual image of the patient's spine and the virtual images of the three Kirschner wires. Then, based on the actual coordinate positions of the three Kirschner wires captured by the optical positioning device, the virtual image of the combination is moved to those actual coordinate positions. At this point, the virtual image of the patient's spine is aligned with the patient's spine, thus completing the image registration.

[0215] Doctors can determine the location of the actual spinal bones under the skin based on the registered virtual images of the spine, so as to perform related surgical procedures, which greatly ensures the accuracy and safety of the surgical procedures.

[0216] An exemplary embodiment of the present invention provides an electronic device, see below. Figure 4 The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the image processing method in orthopedic surgery according to the embodiments of the present invention. Figure 4 The electronic device 30 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0217] The electronic device 30 may be in the form of a general-purpose computing device, such as a server device. The components of the electronic device 30 may include, but are not limited to: at least one processor 31, at least one memory 32, and a bus 33 connecting different system components (including memory 32 and processor 31).

[0218] Bus 33 includes a data bus, an address bus, and a control bus.

[0219] The memory 32 may include volatile memory, such as random access memory (RAM) 321 and / or cache memory 322, and may further include read-only memory (ROM) 323.

[0220] The memory 32 may also include a program / utility 325 having a set (at least one) of program modules 324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0221] The processor 31 executes various functional applications and data processing by running computer programs stored in the memory 32, such as the image processing method in orthopedic surgery in the embodiments of the present invention.

[0222] Electronic device 30 can also communicate with one or more external devices 34 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 35. Furthermore, the model-generated device 30 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 36. As shown, network adapter 26 communicates with other modules of the model-generated device 30 via bus 33. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the model-generated device 30, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0223] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0224] This embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the image processing method in orthopedic surgery according to this embodiment of the invention.

[0225] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0226] In a possible implementation, the present invention can also be implemented as a program product comprising program code, which, when the program product is run on a terminal device, is used to cause the terminal device to execute the image processing method in orthopedic surgery as described in the embodiments of the present invention.

[0227] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0228] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. An image processing system for orthopedic surgery, characterized in that, Several positioning target balls are fixed to the target bone at the lesion site of the patient. The image processing system includes: an optical positioning device, a mixed reality display device, a real image device, and a data processing server. The mixed reality display device is used to acquire a virtual image of the skeleton corresponding to the patient's target skeleton; The optical positioning device is used to obtain target coordinate information of each of the set components based on the positioning target ball; The mixed reality display device is communicatively connected to the optical positioning device and is used to obtain a first virtual image corresponding to the set component based on the target coordinate information; The real-image device is used to acquire real-images of the patient's target bone and the set component; The data processing server is communicatively connected to the real-image device and is used to obtain the actual relative position information between each of the set components and the target skeleton based on the real-image. The data processing server is also communicatively connected to the mixed reality display device, and is used to adjust the relative position between the first virtual image and the skeletal virtual image based on the actual relative position information.

2. The image processing system for orthopedic surgery as described in claim 1, characterized in that, The data processing server includes: A component numbering module is used to pre-set the numbering information corresponding to different specified components; A sequential image adjustment module is used to adjust the position of each designated component in the first virtual image sequentially according to the numbering information, based on the actual relative position information, until the relative position between the first virtual image and the skeletal virtual image reaches the actual relative position corresponding to the actual relative position information; or... The random image adjustment module is used to randomly adjust the position of any set component in the first virtual image based on the actual relative position information, and to iterate through all set components until the relative position between the first virtual image and the skeletal virtual image reaches the actual relative position corresponding to the actual relative position information.

3. The image processing system for orthopedic surgery as described in claim 2, characterized in that, The sequential image adjustment module includes: A reference adjustment point determination unit is used to arbitrarily select one of the set components as a reference adjustment point; The first relative position acquisition unit is used to acquire the first relative position information between the reference adjustment point and the target bone in the real image. The first relative position adjustment unit is used to adjust the relative position of the reference adjustment point in the first virtual image and the skeleton virtual image based on the first relative position information, until the relative position of the reference adjustment point reaches the first relative position corresponding to the first relative position information. An adjustment range determination unit is used to determine the adjustment range based on the reference adjustment point; The actual relative position adjustment unit is used to adjust the relative positions of all remaining set components in the first virtual image and the skeletal virtual image within the adjustment range, until the relative positions of all set components reach the actual relative positions corresponding to the actual relative position information.

4. The image processing system for orthopedic surgery as described in claim 3, characterized in that, The adjustment range determination unit is also used for: Using the reference adjustment point as the center of the sphere, the distance between the reference adjustment point and any one of the remaining setting components is selected as the radius of the sphere to form a sphere; Obtain the secant range formed by the sphere and the surface of the target bone, and obtain the adjustment range based on the secant range.

5. The image processing system for orthopedic surgery as described in claim 1, characterized in that, The optical positioning device includes an optical locator and an optical positioning data processor connected in communication; The optical locator is used to acquire the optical coordinate information of the positioning target ball; The optical positioning data processor is used to calculate the target coordinate information of each of the set components based on the optical coordinate information.

6. The image processing system for orthopedic surgery as described in claim 1, characterized in that, The data processing server is also used for: The first virtual image and the skeletal virtual image are combined to obtain the combined target virtual image.

7. The image processing system for orthopedic surgery as described in claim 6, characterized in that, The mixed reality display device is also used to receive the target virtual image for display.

8. The image processing system for orthopedic surgery as described in claim 1, characterized in that, The mixed reality display device also includes: The image acquisition module is used to acquire the skeletal model image corresponding to the target skeleton; An image conversion module is used to convert the skeleton model image into the skeleton virtual image; And / or, The mixed reality display device is also used to convert the target coordinate information into a first virtual image corresponding to the set component.

9. The image processing system for orthopedic surgery as described in claim 1, characterized in that, The real-world imaging device includes: The reality image acquisition module is used to acquire several initial reality images in multiple directions and dimensions; The reality image conversion module is used to acquire the target reality image corresponding to the target skeleton and the set component based on several initial reality images.

Images