Surgical space registration method and surgical navigation system

Abstract

This invention provides a surgical space registration method, a surgical navigation system, an electronic device, a computer program product, and a storage medium. The space registration method includes: acquiring preoperative three-dimensional data of the skin surface and identifying preoperative target region three-dimensional data and preoperative stable region three-dimensional data; acquiring three-dimensional data of a first surgical scene and identifying intraoperative target region three-dimensional data and intraoperative stable region three-dimensional data; performing coarse registration on the preoperative target region three-dimensional data and the intraoperative target region three-dimensional data to obtain a first spatial transformation matrix; performing fine registration on the preoperative stable region three-dimensional data and the intraoperative stable region three-dimensional data based on the first spatial transformation matrix to obtain a second spatial transformation matrix; and obtaining a spatial mapping relationship based on the first and second spatial transformation matrices. This invention eliminates the need for detecting marker points, thereby reducing the spatial registration time in surgical navigation and improving the efficiency of surgical operations.

Description

Technical Field

[0001] This invention relates to the field of computer-assisted surgery technology, and in particular to a surgical space registration method, a surgical navigation system, an electronic device, a computer program product, and a storage medium. Background Technology

[0002] The process of precisely registering the patient's actual position in space with their virtual position in a 3D model is called registration in surgical navigation. The accuracy of spatial registration directly affects the accuracy of the entire navigation system and is the most important step in the preoperative preparation stage.

[0003] Currently, most spatial registration methods in surgical navigation systems employ marker-based registration, including: bone implant screw markers, anatomical landmarks, and markers pasted on the skin surface. Spatial registration is completed by registering markers identified in a 3D model obtained from preoperative imaging (MRI / CT) with intraoperative markers identified by infrared tracking devices capturing light emitted or reflected from probes. Bone implant markers offer the highest accuracy but require preoperative instrument placement, causing additional trauma and pain to the patient. Anatomical landmarks utilize prominent anatomical features, requiring surgeons to extract them preoperatively from the image space, introducing human error. Furthermore, anatomical landmarks may shift or deform, leading to significant registration errors. Surgeons need to check multiple landmarks, increasing intraoperative registration time and reducing surgical efficiency. Skin-surface markers offer high accuracy and are simple to use, making them the most commonly used method. However, they suffer from issues such as marker light being easily blocked, markers falling off, and difficulties in pasting markers at certain surgical sites, as well as low accuracy in manual extraction.

[0004] Furthermore, existing augmented reality navigation systems require the use of augmented reality glasses to fuse a 3D model of the preoperative image with the patient's real-world surgical scene during intraoperative guidance, providing intraoperative guidance and navigation functions. However, due to the large size of the augmented reality glasses, prolonged wear by doctors leads to fatigue and discomfort, limiting the accuracy of surgical registration and navigation, posing a significant challenge for neurosurgical procedures with high precision requirements.

[0005] It should be noted that the information disclosed in the background section of this invention is intended only to enhance the understanding of the general background of this invention, and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this invention is to provide a surgical space registration method, a surgical navigation system, an electronic device, a computer program product, and a storage medium, which can realize a markerless spatial registration process in surgical navigation, thereby improving spatial registration accuracy and surgical efficiency.

[0007] To achieve the above objectives, the present invention provides a surgical space registration method, comprising:

[0008] The acquired preoperative medical images are segmented to obtain three-dimensional data of the preoperative skin surface;

[0009] The three-dimensional data of the preoperative skin surface are identified to identify the three-dimensional data of the preoperative target area, and the three-dimensional data of the preoperative target area are identified to identify the three-dimensional data of the preoperative stable area.

[0010] The acquired three-dimensional data of the first surgical scene are identified to identify the three-dimensional data of the target area during the operation, and the three-dimensional data of the target area during the operation are identified to identify the three-dimensional data of the stable area during the operation.

[0011] Coarse registration is performed on the preoperative target region 3D data and the intraoperative target region 3D data to obtain a first spatial transformation matrix;

[0012] Based on the first spatial transformation matrix, the three-dimensional data of the preoperative stable region and the three-dimensional data of the intraoperative stable region are precisely registered to obtain the second spatial transformation matrix.

[0013] Based on the first spatial transformation matrix and the second spatial transformation matrix, the spatial mapping relationship between the preoperative medical image space and the intraoperative real space is obtained.

[0014] Optionally, the segmentation of the acquired preoperative medical images to obtain preoperative three-dimensional skin surface data includes:

[0015] The acquired preoperative medical images are preprocessed to obtain preprocessed preoperative medical images;

[0016] A pre-trained segmentation model is used to segment the preprocessed preoperative medical images to obtain preoperative target tissue segmentation images;

[0017] Preoperative three-dimensional data of the skin surface are obtained based on the preoperative target tissue segmentation image.

[0018] Optionally, obtaining preoperative three-dimensional skin surface data based on the preoperative target tissue segmentation image includes:

[0019] Connectivity analysis is performed on the skin region in the preoperative target tissue segmentation image to extract the largest connected component, and the preoperative three-dimensional data of the skin surface is obtained based on the extracted largest connected component.

[0020] Optionally, the step of identifying the preoperative three-dimensional data of the skin surface to identify the preoperative three-dimensional data of the target area includes:

[0021] A pre-trained first target detection model is used to identify the three-dimensional data of the preoperative skin surface to identify the three-dimensional data of the preoperative target area;

[0022] The process of identifying the three-dimensional data of the preoperative target region to identify the three-dimensional data of the preoperative stable region includes:

[0023] A pre-trained second target detection model is used to identify the three-dimensional data of the preoperative target region in order to identify the three-dimensional data of the preoperative stable region.

[0024] The step of identifying the acquired three-dimensional data of the first surgical scene to identify the three-dimensional data of the target area during surgery includes:

[0025] A pre-trained third target detection model is used to identify the three-dimensional data of the first surgical scene in order to identify the three-dimensional data of the target area during the operation.

[0026] The process of identifying the three-dimensional data of the intraoperative target region to identify the three-dimensional data of the intraoperative stable region includes:

[0027] A pre-trained fourth target detection model is used to identify the three-dimensional data of the intraoperative target area in order to identify the three-dimensional data of the stable intraoperative area.

[0028] Optionally, the preoperative three-dimensional data of the skin surface, the preoperative three-dimensional data of the target area, and the preoperative three-dimensional data of the stable area are all three-dimensional image data, and the first surgical scene three-dimensional data, the intraoperative target area three-dimensional data, and the intraoperative stable area three-dimensional data are all three-dimensional point cloud data.

[0029] The coarse registration of the preoperative target region 3D data and the intraoperative target region 3D data includes:

[0030] Obtain 3D point cloud data of the target area based on the 3D image data of the target area before surgery;

[0031] Coarse registration is performed on the preoperative target region 3D point cloud data and the intraoperative target region 3D point cloud data.

[0032] The precise registration of the preoperative stable region 3D data and the intraoperative stable region 3D data based on the first spatial transformation matrix includes:

[0033] Based on the three-dimensional image data of the preoperative stable region, obtain the three-dimensional point cloud data of the preoperative stable region;

[0034] Based on the first spatial transformation matrix, the three-dimensional point cloud data of the preoperative stable region and the three-dimensional point cloud data of the intraoperative stable region are precisely registered.

[0035] Optionally, the preoperative medical image is a brain medical image;

[0036] The process of identifying the preoperative three-dimensional data of the skin surface to identify the preoperative target area three-dimensional data, and identifying the preoperative target area three-dimensional data to identify the preoperative stable area three-dimensional data, includes:

[0037] The three-dimensional data of the preoperative skin surface are identified to identify the three-dimensional data of the preoperative facial region, and the three-dimensional data of the preoperative facial region are identified to identify the three-dimensional data of the preoperative stable facial region.

[0038] The step of identifying the three-dimensional data of the first surgical scene to identify the three-dimensional data of the target area during surgery, and identifying the three-dimensional data of the target area during surgery to identify the three-dimensional data of the stable area during surgery, includes:

[0039] The three-dimensional data of the first surgical scene are identified to identify the three-dimensional data of the facial region during surgery, and the three-dimensional data of the facial region during surgery are identified to identify the three-dimensional data of the stable facial region during surgery.

[0040] The step of coarsely registering the preoperative target region 3D data and the intraoperative target region 3D data to obtain the first spatial transformation matrix includes:

[0041] Coarse registration is performed on the preoperative facial region 3D data and the intraoperative facial region 3D data to obtain a first spatial transformation matrix;

[0042] The step of performing fine registration of the preoperative stable region 3D data and the intraoperative stable region 3D data based on the first spatial transformation matrix to obtain the second spatial transformation matrix includes:

[0043] Based on the first spatial transformation matrix, the three-dimensional data of the preoperative facial stable region and the three-dimensional data of the intraoperative facial stable region are precisely registered to obtain the second spatial transformation matrix.

[0044] Optionally, the facial stabilization region includes at least one of the brow bone region, the frontal bone region, and the nasal region.

[0045] To achieve the above objectives, the present invention also provides a surgical navigation system, including a 3D vision module and a control terminal connected in communication. The 3D vision module is configured to scan the surgical scene during surgery to collect first surgical scene scan data. The control terminal is configured to obtain three-dimensional data of the first surgical scene based on the first surgical scene scan data and implement the surgical space registration method described above.

[0046] Optionally, the 3D vision module includes multiple optical cameras for scanning the surgical scene to acquire images of the first surgical scene from different angles. The control terminal is configured to perform three-dimensional reconstruction based on the images of the first surgical scene from different angles acquired by the multiple optical cameras to obtain a three-dimensional model of the surgical scene, and to obtain three-dimensional point cloud data of the first surgical scene based on the three-dimensional model of the surgical scene.

[0047] Optionally, the control terminal is further configured to perform three-dimensional reconstruction based on the segmentation results of the preoperative medical image to obtain a preoperative virtual three-dimensional model, and to plan the surgical path based on the preoperative virtual three-dimensional model to plan the surgical path.

[0048] Optionally, the control terminal is further configured to map the preoperative virtual 3D model and the surgical path to the intraoperative real space according to the spatial mapping relationship between the preoperative medical image space and the intraoperative real space, and to fuse the mapped preoperative virtual 3D model and the surgical path with the surgical scene 3D model for augmented reality display.

[0049] Optionally, the 3D vision module is further configured to scan surgical instruments during movement to acquire second surgical scene scan data including surgical instruments; the control terminal is further configured to acquire second surgical scene three-dimensional data based on the second surgical scene scan data, identify identifiers in the second surgical scene three-dimensional data, and acquire the pose information of the end of the corresponding surgical instrument based on the position information of the identified identifiers.

[0050] Optionally, the control terminal is also configured to display the movement trajectory of the end of the surgical instrument in real time.

[0051] To achieve the above objectives, the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the surgical space registration method described above.

[0052] To achieve the above objectives, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the surgical space registration method described above.

[0053] To achieve the above objectives, the present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the surgical space registration method described above.

[0054] Compared with existing technologies, the surgical space registration method, surgical navigation system, electronic device, computer program product, and storage medium provided by this invention have the following advantages:

[0055] The surgical space registration method provided by this invention segments the acquired preoperative medical image to obtain preoperative three-dimensional data of the skin surface; identifies the preoperative target region three-dimensional data based on the preoperative skin surface three-dimensional data, and identifies the preoperative stable region three-dimensional data based on the preoperative target region three-dimensional data; identifies the acquired first surgical scene three-dimensional data to identify the intraoperative target region three-dimensional data, and identifies the intraoperative stable region three-dimensional data based on the intraoperative target region three-dimensional data; performs coarse registration on the preoperative target region three-dimensional data and the intraoperative target region three-dimensional data to obtain a first spatial transformation matrix; finally, performs fine registration on the preoperative stable region three-dimensional data and the intraoperative stable region three-dimensional data based on the first spatial transformation matrix to obtain a second spatial transformation matrix; and finally, obtains the spatial mapping relationship between the preoperative medical image space and the intraoperative real space based on the first spatial transformation matrix and the second spatial transformation matrix, thereby completing the spatial registration. Therefore, the surgical space registration method provided by this invention does not require the detection of marker points, thus reducing the spatial registration time in surgical navigation and improving the efficiency of surgical operations. Furthermore, this invention first performs coarse registration on the preoperative target region 3D data and the intraoperative target region 3D data to obtain a first spatial transformation matrix, and then performs fine registration on the preoperative stable region 3D data and the intraoperative stable region 3D data based on the first spatial transformation matrix to obtain a second spatial transformation matrix. This can improve the accuracy of spatial registration in surgical navigation, lay a good foundation for achieving precise surgical navigation, and meet the accuracy requirements of neurosurgery.

[0056] Since the surgical navigation system, electronic device, computer program product, and storage medium provided by this invention belong to the same inventive concept as the surgical space registration method provided by this invention, the surgical navigation system, electronic device, computer program product, and storage medium provided by this invention have all the advantages of the surgical space registration method provided by this invention. Furthermore, the surgical navigation system provided by this invention uses a 3D vision module to scan the surgical scene in real time. The control terminal reconstructs a three-dimensional model of the surgical scene based on the scanning results of the 3D vision module, and fuses the preoperative virtual three-dimensional model reconstructed based on the segmentation results of preoperative medical images with the surgical scene three-dimensional model for display. This allows doctors to complete surgical operations without wearing augmented reality glasses. Moreover, since the 3D vision module is more stable than augmented reality glasses, it can further meet the precision requirements of neurosurgery. Attached Figure Description

[0057] Figure 1 A schematic flowchart of a surgical space registration method provided in one embodiment of the present invention;

[0058] Figure 2 A schematic diagram illustrating the acquisition of three-dimensional point cloud data of the preoperative facial region and three-dimensional point cloud data of the preoperative stable facial region according to an embodiment of the present invention;

[0059] Figure 3 This is a schematic diagram illustrating the acquisition of intraoperative facial stable region three-dimensional point cloud data from intraoperative facial region three-dimensional point cloud data according to an embodiment of the present invention.

[0060] Figure 4 This is a schematic diagram of a label-free registration process based on three-dimensional point cloud data of a stable facial region, provided by an embodiment of the present invention.

[0061] Figure 5 A schematic diagram illustrating the implementation process of a registration algorithm based on facial stable regions according to an embodiment of the present invention;

[0062] Figure 6 A schematic diagram of the training process of a segmentation model provided in one embodiment of the present invention;

[0063] Figure 7 This is a schematic diagram of an automatic segmentation and reasoning process for a head medical image provided in one embodiment of the present invention;

[0064] Figure 8 A schematic diagram illustrating the process of acquiring three-dimensional point cloud data of the preoperative facial region and three-dimensional point cloud data of the preoperative stable facial region, as provided in an embodiment of the present invention.

[0065] Figure 9 A schematic diagram of the training process of a first target detection model provided in one embodiment of the present invention;

[0066] Figure 10 A schematic diagram of the automatic identification and reasoning process for the preoperative facial region and the preoperative stable facial region provided in one embodiment of the present invention;

[0067] Figure 11 This is a schematic diagram illustrating an application scenario of the surgical navigation system according to an embodiment of the present invention;

[0068] Figure 12 This is a schematic diagram of the data transmission process of a surgical navigation system provided in one embodiment of the present invention;

[0069] Figure 13 A flowchart illustrating the automatic detection and extraction of three-dimensional point cloud data of a stable facial region based on three-dimensional point cloud data of a first surgical scene, as provided in an embodiment of the present invention.

[0070] Figure 14 A flowchart illustrating the preoperative virtual three-dimensional model reconstruction and surgical path planning provided by one embodiment of the present invention;

[0071] Figure 15 This is a schematic diagram of a scenario for identifying surgical instruments according to an embodiment of the present invention;

[0072] Figure 16 A block diagram of a server provided according to an embodiment of the present invention;

[0073] Figure 17 This is a block diagram of an electronic device provided according to an embodiment of the present invention. Detailed Implementation

[0074] The surgical space registration method, surgical navigation system, electronic device, and storage medium proposed in this invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The advantages and features of this invention will become clearer from the following description. It should be noted that the drawings are in a very simplified form and use non-precise proportions, only for the purpose of conveniently and clearly illustrating the embodiments of this invention. Please refer to the drawings to make the objectives, features, and advantages of this invention more apparent and understandable. It should be understood that the structures, proportions, sizes, etc., depicted in the accompanying drawings are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed in the specification, and are not intended to limit the implementation conditions of this invention. Any modifications to the structure, changes in the proportional relationships, or adjustments to the size, if they are the same as or similar to the effects and objectives achieved by this invention, should still fall within the scope of the technical content disclosed in this invention.

[0075] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0076] Furthermore, in the description of this specification, the reference to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., means that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0077] The core idea of ​​this invention lies in providing a surgical space registration method, a surgical navigation system, an electronic device, a computer program product, and a storage medium, which can realize a markerless spatial registration process in surgical navigation, improving spatial registration accuracy and surgical efficiency. It should be noted that the surgical space registration method provided by this invention can be applied to the electronic device provided by this invention, which can be configured on the surgical navigation system provided by this invention. The electronic device provided by this invention can be a personal computer, a mobile terminal, etc., and the mobile terminal can be a mobile phone, tablet computer, or other hardware device with various operating systems. Furthermore, it should be noted that, as those skilled in the art will understand, the term "proximal end" in this invention refers to the end closer to the operator, and the term "distal end" refers to the end farther from the operator, i.e., the end closer to the patient.

[0078] This invention provides a surgical space registration method, please refer to... Figure 1 The diagram illustrates a flowchart of a surgical space registration method according to an embodiment of the present invention. Figure 1 As shown, the surgical space registration method provided by the present invention includes the following steps:

[0079] Step S100: Segment the acquired preoperative medical images to obtain three-dimensional data of the preoperative skin surface.

[0080] Step S200: Identify the three-dimensional data of the preoperative skin surface to identify the three-dimensional data of the preoperative target area, and identify the three-dimensional data of the preoperative stable area.

[0081] Step S300: Identify the acquired three-dimensional data of the first surgical scene to identify the three-dimensional data of the target area during the operation, and identify the three-dimensional data of the stable area during the operation.

[0082] Step S400: Perform coarse registration on the preoperative target region 3D data and the intraoperative target region 3D data to obtain the first spatial transformation matrix.

[0083] Step S500: Based on the first spatial transformation matrix, perform fine registration on the three-dimensional data of the preoperative stable region and the three-dimensional data of the intraoperative stable region to obtain the second spatial transformation matrix.

[0084] Step S600: Obtain the spatial mapping relationship between the preoperative medical image space and the intraoperative real space based on the first spatial transformation matrix and the second spatial transformation matrix.

[0085] Therefore, the surgical space registration method provided by this invention eliminates the need for marker point detection, thereby reducing spatial registration time in surgical navigation and improving surgical efficiency. Furthermore, this invention improves the accuracy of spatial registration in surgical navigation by first performing coarse registration on the preoperative and intraoperative 3D data of the target region to obtain a first spatial transformation matrix, and then performing fine registration on the preoperative and intraoperative stable region 3D data based on the first spatial transformation matrix to obtain a second spatial transformation matrix. This lays a solid foundation for achieving precise surgical navigation and meets the accuracy requirements of neurosurgery. It should be noted that, as those skilled in the art will understand, the spatial registration method provided by this invention is applicable not only to neurosurgery but also to other surgical procedures, such as maxillofacial surgery and spinal surgery. Additionally, it should be noted that, as those skilled in the art will understand, the acquired preoperative medical images are 3D medical images comprising multiple 2D medical images. These preoperative medical images can be CT images, MRI images, or images acquired by other medical imaging equipment; this invention does not limit the scope of these images.

[0086] In one exemplary implementation, the preoperative medical image is a brain medical image;

[0087] The process of identifying the preoperative three-dimensional data of the skin surface to identify the preoperative target area three-dimensional data, and identifying the preoperative target area three-dimensional data to identify the preoperative stable area three-dimensional data, includes:

[0088] The three-dimensional data of the preoperative skin surface are identified to identify the three-dimensional data of the preoperative facial region, and the three-dimensional data of the preoperative facial region are identified to identify the three-dimensional data of the preoperative stable facial region.

[0089] The step of identifying the three-dimensional data of the first surgical scene to identify the three-dimensional data of the target area during surgery, and identifying the three-dimensional data of the target area during surgery to identify the three-dimensional data of the stable area during surgery, includes:

[0090] The three-dimensional data of the first surgical scene are identified to identify the three-dimensional data of the facial region during surgery, and the three-dimensional data of the facial region during surgery are identified to identify the three-dimensional data of the stable facial region during surgery.

[0091] The step of coarsely registering the preoperative target region 3D data and the intraoperative target region 3D data to obtain the first spatial transformation matrix includes:

[0092] Coarse registration is performed on the preoperative facial region 3D data and the intraoperative facial region 3D data to obtain a first spatial transformation matrix;

[0093] The step of performing fine registration of the preoperative stable region 3D data and the intraoperative stable region 3D data based on the first spatial transformation matrix to obtain the second spatial transformation matrix includes:

[0094] Based on the first spatial transformation matrix, the three-dimensional data of the preoperative facial stable region and the three-dimensional data of the intraoperative facial stable region are precisely registered to obtain the second spatial transformation matrix.

[0095] Therefore, for neurosurgical procedures, by first performing coarse registration of the preoperative and intraoperative three-dimensional facial region data to obtain a first spatial transformation matrix, and then performing fine registration of the preoperative and intraoperative stable facial region data based on the first spatial transformation matrix, the spatial registration process for markerless neurosurgical procedures can be completed, effectively improving the spatial registration accuracy and surgical efficiency of neurosurgical procedures.

[0096] Furthermore, the facial stabilization region includes at least one of the brow bone region, frontal bone region, and nose region. Since facial areas such as the cheeks are easily deformed, while the brow bone region, frontal bone region, and nose region are less prone to deformation, selecting at least one of these regions as the facial stabilization region ensures a more accurate second spatial transformation matrix, further improving the precision of neurosurgical spatial registration.

[0097] In one exemplary embodiment, the preoperative three-dimensional data of the skin surface, the preoperative three-dimensional data of the target area, and the preoperative three-dimensional data of the stable area are all three-dimensional image data, and the first surgical scene three-dimensional data, the intraoperative target area three-dimensional data, and the intraoperative stable area three-dimensional data are all three-dimensional point cloud data.

[0098] Correspondingly, the coarse registration of the preoperative target region 3D data and the intraoperative target region 3D data includes:

[0099] Obtain 3D point cloud data of the target area based on the 3D image data of the target area before surgery;

[0100] Coarse registration is performed on the preoperative target region 3D point cloud data and the intraoperative target region 3D point cloud data.

[0101] The precise registration of the preoperative stable region 3D data and the intraoperative stable region 3D data based on the first spatial transformation matrix includes:

[0102] Based on the three-dimensional image data of the preoperative stable region, obtain the three-dimensional point cloud data of the preoperative stable region;

[0103] Based on the first spatial transformation matrix, the three-dimensional point cloud data of the preoperative stable region and the three-dimensional point cloud data of the intraoperative stable region are precisely registered.

[0104] Therefore, by performing coarse registration based on the preoperative target region 3D point cloud data (e.g., preoperative facial region 3D point cloud data) and the intraoperative target region 3D point cloud data (e.g., intraoperative facial region 3D point cloud data), the efficiency of coarse registration can be effectively improved; by performing fine registration based on the preoperative stable region 3D point cloud data (e.g., preoperative facial stable region 3D point cloud data) and the intraoperative stable region 3D point cloud data (e.g., intraoperative facial stable region 3D point cloud data) and based on the first spatial transformation matrix, the efficiency of fine registration can be effectively improved.

[0105] Please refer to Figure 2 and Figure 3 ,in, Figure 2 A schematic diagram illustrating the acquisition of three-dimensional point cloud data of the preoperative facial region and three-dimensional point cloud data of the preoperative stable facial region provided by an embodiment of the present invention is shown. Figure 3 A schematic diagram illustrating the acquisition of intraoperative facial stable region three-dimensional point cloud data from intraoperative facial region three-dimensional point cloud data according to an embodiment of the present invention is shown. For example... Figure 2 As shown, by identifying preoperative three-dimensional image data of the skin surface, three-dimensional image data of the preoperative facial region can be identified. By identifying the preoperative three-dimensional image data of the facial region, three-dimensional image data of the stable preoperative facial region can be identified. By converting the format of the preoperative three-dimensional image data of the facial region, three-dimensional point cloud data of the preoperative facial region can be obtained. Similarly, by converting the format of the preoperative three-dimensional image data of the stable preoperative facial region, three-dimensional point cloud data of the stable preoperative facial region can be obtained. Figure 3 As shown, by identifying the three-dimensional point cloud data of the first surgical scene, the three-dimensional point cloud data of the facial region during surgery can be identified. By identifying the three-dimensional point cloud data of the facial region during surgery, the three-dimensional point cloud data of the stable facial region during surgery can be identified.

[0106] In one exemplary embodiment, the coarse registration of the preoperative target region 3D point cloud data (e.g., preoperative facial region 3D point cloud data) and the intraoperative target region 3D point cloud data (e.g., intraoperative facial region 3D point cloud data) includes:

[0107] The preoperative target region 3D point cloud data (e.g., preoperative facial region 3D point cloud data) and the intraoperative target region 3D point cloud data (e.g., intraoperative facial region 3D point cloud data) are coarsely registered using either a brute-force registration algorithm or a 4-point congruent set registration algorithm (4PCS registration algorithm).

[0108] Specifically, the detailed algorithm flow for the brute-force registration algorithm and the 4-point congruent set registration algorithm (4PCS registration algorithm) can be found in existing technologies and will not be elaborated here.

[0109] In one exemplary embodiment, the fine registration of the preoperative stable region 3D point cloud data (e.g., preoperative facial stable region 3D point cloud data) and the intraoperative stable region 3D point cloud data (e.g., intraoperative facial stable region 3D point cloud data) based on the first spatial transformation matrix includes:

[0110] The three-dimensional point cloud data of the preoperative stable region (e.g., the three-dimensional point cloud data of the preoperative facial stable region) is spatially transformed according to the first spatial transformation matrix to obtain the transformed three-dimensional point cloud data of the preoperative stable region (e.g., the three-dimensional point cloud data of the preoperative facial stable region).

[0111] The Iterative Closest Point Algorithm (ICP Registration Algorithm) is used to perform fine registration on the transformed preoperative stable region 3D point cloud data (e.g., preoperative facial stable region 3D point cloud data) and the intraoperative stable region 3D point cloud data (e.g., intraoperative facial stable region 3D point cloud data).

[0112] Specifically, the detailed algorithm flow of the Iterative Closest Point Algorithm (ICP Registration Algorithm) can be found in existing technologies and will not be elaborated here.

[0113] Please continue to refer to this. Figure 4 The diagram illustrates a markerless registration process based on three-dimensional point cloud data of a stable facial region, according to an embodiment of the present invention. Figure 4 As shown, based on the automatically extracted 3D point cloud data of the facial region before and during surgery, an initial spatial mapping matrix (i.e., the first spatial transformation matrix) is obtained by using a point cloud registration algorithm to map the preoperative point cloud data to the intraoperative real world. Then, based on the initial spatial mapping matrix (i.e., the first spatial transformation matrix), the preoperative virtual 3D model reconstructed based on the segmentation results of the preoperative medical images and the surgical path planned based on the preoperative virtual 3D model are mapped to the intraoperative surgical space (i.e., the intraoperative real world), completing the coarse registration. Next, based on the initial spatial mapping matrix (i.e., the first spatial transformation matrix), the acquired preoperative stable facial region 3D point cloud data is spatially transformed. Then, the spatial mapping matrix (i.e., the second spatial transformation matrix) between the transformed preoperative stable facial region 3D point cloud data and the intraoperative stable facial region point cloud data is obtained. Finally, the spatial mapping relationship is updated by combining the first and second spatial transformation matrices, and the surgical space registration result is checked to confirm successful registration, completing the surgical space registration process. Finally, based on the registration results (i.e., the spatial mapping relationship between the preoperative medical image space and the intraoperative real space), the preoperative 3D virtual model and surgical path are mapped and fused into the intraoperative surgical scene to complete the augmented reality display.

[0114] Please continue to refer to this. Figure 5 The diagram illustrates the implementation process of a registration algorithm based on facial stable regions according to an embodiment of the present invention. Figure 5As shown, the coarse registration process uses preoperative facial region 3D point cloud data P1 and intraoperative facial region 3D point cloud data P2 to obtain an initial rigid spatial transformation matrix (i.e., the first spatial transformation matrix) T1 for rotation and translation using a coarse registration algorithm (brute force registration algorithm, 4PCS registration algorithm, etc.). Based on the first spatial transformation matrix T1, a coarse registration result P1' of the preoperative facial region 3D point cloud data is obtained. The first spatial transformation matrix T1 can provide initial spatial transformation values ​​for fine registration. Fine registration performs spatial transformation on the acquired preoperative facial stable region 3D point cloud data based on the first spatial transformation matrix T1 of coarse registration, and then uses the spatially transformed preoperative facial stable region 3D point cloud data P3. ’ A point cloud registration algorithm (such as the ICP algorithm) is used to register the data to the acquired intraoperative stable facial 3D point cloud data P4, resulting in a more accurate spatial transformation matrix (i.e., the second spatial transformation matrix) T2. Combining the first spatial transformation matrix T1 and the second spatial transformation matrix T2, the preoperative image space can be transformed to the intraoperative real space, completing the surgical registration process. It should be noted that, as those skilled in the art will understand, based on the first spatial transformation matrix T1 and the second spatial transformation matrix T2, the preoperative facial 3D point cloud data P1 can be transformed to the intraoperative real space, thereby obtaining the spatially transformed preoperative facial 3D point cloud data P. T .

[0115] In one exemplary embodiment, segmenting the acquired preoperative medical images to obtain preoperative three-dimensional data of the skin surface includes:

[0116] The acquired preoperative medical images are preprocessed to obtain preprocessed preoperative medical images;

[0117] A pre-trained segmentation model is used to segment the preprocessed preoperative medical images to obtain preoperative target tissue segmentation images;

[0118] Preoperative three-dimensional data of the skin surface are obtained based on the preoperative target tissue segmentation image.

[0119] Therefore, by first preprocessing the acquired preoperative medical images (e.g., brain medical images) and then using a pre-trained segmentation model to segment the preprocessed preoperative medical images, segmentation efficiency can be effectively improved. It should be noted that, as those skilled in the art will understand, the segmentation result of the preoperative medical image (i.e., the preoperative target tissue segmentation image) includes not only the preoperative skin region but also the preoperative target lesion region, the preoperative blood vessel region, and other preoperative related tissue regions. Taking a brain medical image as an example, by segmenting the preprocessed preoperative brain medical image, the obtained preoperative target tissue segmentation image includes the preoperative target lesion region, the preoperative skull region, the preoperative intracranial blood vessel region, the preoperative skin region, and the preoperative brain functional region. It should be noted that, as those skilled in the art will understand, different tissue regions are represented by different pixel values ​​or different colors in the preoperative target tissue segmentation image. Furthermore, it should be noted that, as those skilled in the art will understand, the present invention does not limit the specific network structure of the segmentation model; the segmentation model can be a deep learning neural network model in the prior art.

[0120] Please continue to refer to this. Figure 6 The diagram illustrates the training process of a segmentation model provided in one embodiment of the present invention. Figure 6 As shown, taking the training of a segmentation model for segmenting brain medical images as an example, the process begins with collecting brain (skull) medical image data from patients and constructing a training dataset for the segmentation model. Then, target tissue data is labeled and delineated to obtain a dataset with the gold standard for target tissue segmentation results. Specific labels are assigned to multiple different target tissues. Due to the inconsistency in medical image acquisition equipment and methods, data consistency is poor. Therefore, preprocessing steps such as pixel value cropping and normalization are required before training. A multi-label segmentation model is then built. The model parameters are randomly initialized, and iterative training is performed using the labeled target tissue segmentation dataset. The model parameters are continuously updated until the loss function curve converges, resulting in a trained multi-label segmentation model for subsequent automatic segmentation of brain (skull) medical images.

[0121] Please continue to refer to this. Figure 7 The diagram illustrates an automatic segmentation and reasoning process for head medical images according to an embodiment of the present invention. Figure 7 As shown, the input three-dimensional medical image of the head is used for inference through a trained segmentation model, and the segmented head tissue (including target lesions, skull, intracranial blood vessels, skin and brain functional areas, etc.) can be output.

[0122] In one exemplary embodiment, after segmenting the preprocessed preoperative medical image using a pre-trained segmentation model, the surgical space registration method further includes:

[0123] The operation of receiving confirmation of the segmentation result from the user.

[0124] Specifically, if a user believes that the segmentation results of the segmentation model contain abnormal regions, manual intervention and adjustments can be made until the requirements are met.

[0125] In one exemplary embodiment, the preprocessing of the acquired preoperative medical images to obtain preprocessed preoperative medical images includes:

[0126] The pixel values ​​of the pixels in the acquired preoperative medical images are cropped to adjust the pixel value of each pixel in the preoperative medical images to a preset range;

[0127] The pixel values ​​of the truncated preoperative medical images are normalized to obtain preprocessed preoperative medical images.

[0128] Specifically, assuming the preset range is [a, b], for pixels in the preoperative medical image whose pixel value is greater than or equal to b, the pixel value of that pixel is adjusted to b; for pixels in the preoperative medical image whose pixel value is less than or equal to a, the pixel value of that pixel is adjusted to a; and for pixels in the preoperative medical image whose pixel value is greater than a and less than b, the pixel value of that pixel remains unchanged. This completes the pixel value truncation process in the preoperative medical image, adjusting the pixel value of each pixel in the preoperative medical image to [a, b]. By normalizing the pixel values ​​of the truncated preoperative medical image, the pixel values ​​of the truncated preoperative medical image can be adjusted to between 0 and 1. The specific process of normalization can be found in existing technologies and will not be elaborated here.

[0129] In one exemplary embodiment, acquiring preoperative three-dimensional data of the skin surface based on the preoperative target tissue segmentation image includes:

[0130] Connectivity analysis is performed on the skin region in the preoperative target tissue segmentation image to extract the largest connected component, and the preoperative three-dimensional data of the skin surface is obtained based on the extracted largest connected component.

[0131] Since preoperative medical images may have inaccurate segmentation, resulting in discrete points (i.e., noise) in the preoperative target tissue segmentation image, the accuracy of the obtained preoperative skin surface three-dimensional data can be guaranteed by performing connected component analysis on the skin region in the preoperative target tissue segmentation image to extract the largest connected component, thereby further improving the accuracy of the surgical space registration method provided by the present invention.

[0132] In one exemplary embodiment, the identification of the preoperative three-dimensional data of the skin surface to identify the preoperative three-dimensional data of the target area includes:

[0133] A pre-trained first target detection model is used to identify the three-dimensional data of the preoperative skin surface to identify the three-dimensional data of the preoperative target area;

[0134] The process of identifying the three-dimensional data of the preoperative target region to identify the three-dimensional data of the preoperative stable region includes:

[0135] A pre-trained second target detection model is used to identify the three-dimensional data of the preoperative target region in order to identify the three-dimensional data of the preoperative stable region.

[0136] The step of identifying the acquired three-dimensional data of the first surgical scene to identify the three-dimensional data of the target area during surgery includes:

[0137] A pre-trained third target detection model is used to identify the three-dimensional data of the first surgical scene in order to identify the three-dimensional data of the target area during the operation.

[0138] The process of identifying the three-dimensional data of the intraoperative target region to identify the three-dimensional data of the intraoperative stable region includes:

[0139] A pre-trained fourth target detection model is used to identify the three-dimensional data of the intraoperative target area in order to identify the three-dimensional data of the stable intraoperative area.

[0140] Therefore, by employing a pre-trained first object detection model to identify preoperative target regions (e.g., preoperative facial regions), both the identification efficiency and accuracy of preoperative target regions (e.g., preoperative facial regions) can be improved. Similarly, by employing a pre-trained second object detection model to identify preoperative stable regions (e.g., preoperative facial stable regions), both the identification efficiency and accuracy of preoperative stable regions (e.g., preoperative facial stable regions) can be improved. Furthermore, by employing a third object detection model to identify intraoperative target regions (e.g., intraoperative facial regions), both the identification efficiency and accuracy of intraoperative target regions (e.g., intraoperative facial regions) can be improved. Finally, by employing a pre-trained fourth object detection model to identify intraoperative stable regions (e.g., intraoperative facial stable regions), both the identification efficiency and accuracy of intraoperative stable regions (e.g., intraoperative facial stable regions) can be improved. It should be noted that, as those skilled in the art will understand, the present invention does not limit the specific network structure of the first target detection model, the second target detection model, the third target detection model, and the fourth target detection model. The first target detection model, the second target detection model, the third target detection model, and the fourth target detection model can be deep learning neural network models in the prior art.

[0141] Please refer to Figure 8 The diagram illustrates the process of acquiring three-dimensional point cloud data of the preoperative facial region and three-dimensional point cloud data of the preoperative stable facial region according to an embodiment of the present invention. Figure 8 As shown, by using a pre-trained first target detection model to identify the three-dimensional image data of the preoperative skin surface, the three-dimensional image data of the preoperative facial region can be automatically identified; by using a pre-trained second target detection model to identify the three-dimensional image data of the preoperative facial region, the three-dimensional image data of the preoperative stable facial region can be automatically identified; by converting the format of the preoperative facial region three-dimensional image data, the three-dimensional point cloud data of the preoperative facial region can be obtained; by converting the format of the preoperative facial stable region three-dimensional image data, the three-dimensional point cloud data of the preoperative stable facial region can be obtained.

[0142] Please continue to refer to this. Figure 9 The diagram illustrates the training process of a first target detection model provided by an embodiment of the present invention. Figure 9As shown, taking the training of a first target detection model for recognizing preoperative facial regions as an example, firstly, based on the skin results of tissue segmentation from the brain (skull) medical image data mentioned earlier, a three-dimensional image dataset of the skin surface is obtained for training the first target detection model (since the three-dimensional image data of the skin surface obtained based on the segmentation model has discrete points when the segmentation is inaccurate, connected component analysis is required to retain the largest connected component). Then, facial region data is labeled and delineated to obtain a dataset with the gold standard for facial region detection results. Next, the first target detection model is built, and the model parameters of the first target detection model are randomly initialized. Iterative training of the first target detection model is then performed using the labeled facial region detection result dataset, continuously updating the model parameters until the loss function curve converges, thus obtaining the trained first target detection model for subsequent automatic recognition of preoperative facial regions. It should be noted that, as those skilled in the art will understand, the training process of the second, third, and fourth target detection models is similar to that of the first target detection model, and will not be described in detail here. The difference lies in the fact that the dataset used for training the second object detection model is a three-dimensional image dataset of facial regions with at least one of the stable facial regions (including the frontal bone region, brow bone region, and nose region); the dataset used for training the third object detection model is a three-dimensional point cloud data of a first surgical scene with the facial regions labeled; and the dataset used for training the fourth object detection model is a three-dimensional point cloud dataset of facial regions with at least one of the stable facial regions (including the frontal bone region, brow bone region, and nose region).

[0143] Please continue to refer to this. Figure 10 This diagram illustrates the automatic identification and reasoning process of the preoperative facial region and the preoperative stable facial region provided by an embodiment of the present invention. Figure 10 As shown, the input of the first target detection model is the preoperative three-dimensional image data of the skin surface obtained based on the segmentation results of preoperative medical images. By performing feature extraction and target filtering through the trained first target detection model, the three-dimensional image data of the preoperative facial region can be automatically identified and extracted. The input of the second target detection model is the preoperative three-dimensional image data of the facial region output by the first target detection model. By performing feature extraction and target filtering through the trained second target detection model, the three-dimensional image data of the stable preoperative facial region can be automatically identified and extracted. Finally, the corresponding three-dimensional point cloud data of the preoperative facial region is generated based on the extracted three-dimensional image data of the stable preoperative facial region, and the corresponding three-dimensional point cloud data of the stable preoperative facial region is generated based on the extracted three-dimensional image data of the stable preoperative facial region for subsequent registration.

[0144] This invention also provides a surgical navigation system, please refer to... Figure 11 The diagram illustrates an application scenario of a surgical navigation system according to an embodiment of the present invention. Figure 11 As shown, the surgical navigation system provided by this invention includes a 3D vision module 100 and a control terminal 200 connected in communication. The 3D vision module 100 is configured to scan the surgical scene during surgery to collect first surgical scene scan data. The control terminal 200 is configured to acquire first surgical scene three-dimensional data based on the first surgical scene scan data and implement the surgical space registration method described above. Therefore, the surgical navigation system provided by this invention does not require the detection of marker points, thereby reducing the spatial registration time in surgical navigation and improving the efficiency of surgical operations. Furthermore, this invention improves the accuracy of spatial registration in surgical navigation by first performing coarse registration on the preoperative target region three-dimensional data and the intraoperative target region three-dimensional data to obtain a first spatial transformation matrix, and then performing fine registration on the preoperative stable region three-dimensional data and the intraoperative stable region three-dimensional data based on the first spatial transformation matrix to obtain a second spatial transformation matrix. This lays a good foundation for achieving precise surgical navigation and can meet the accuracy requirements of neurosurgery. Furthermore, the surgical navigation system provided by the present invention uses a 3D vision module 100 to scan the surgical scene. Since the 3D vision module 100 is more stable than augmented reality glasses, it can further meet the precision requirements of neurosurgery.

[0145] Please continue to refer to this. Figure 11 ,like Figure 11 As shown, in one exemplary embodiment, the control terminal 200 includes a server 210 and a display 220 connected in communication. The 3D vision module 100 is connected in communication with the server 210, which is configured to implement the spatial registration method described above. Please continue to refer to... Figure 12 The diagram illustrates the data transmission flow of a surgical navigation system according to an embodiment of the present invention. Figure 12 As shown, in one exemplary embodiment, the 3D vision module 100 includes a first data interface 110 and a 3D vision scanner 120 connected together, and the server 210 includes a data processing module 211 and a second data interface 212. The surgical scene captured by the 3D vision scanner 120 is transmitted to the data processing module 211 in the server 210 via the first data interface 110 and the second data interface 212. The data processing module 211 processes the preoperative medical images and intraoperative scan data, and after processing, guides the doctor to complete the surgical operation through the display on the monitor 220.

[0146] In one exemplary embodiment, the 3D vision module 100 (3D vision scanner 120) includes multiple optical cameras to acquire images of a first surgical scene from different angles. The control terminal 200 is configured to perform 3D reconstruction based on the first surgical scene images acquired by the multiple optical cameras from different angles to obtain a 3D model of the surgical scene, and to obtain 3D point cloud data of the first surgical scene based on the 3D model of the surgical scene. Specifically, when the 3D vision module 100 (3D vision scanner 120) includes multiple optical cameras, since the positions of the first surgical scene images captured by different optical cameras will have a certain positional difference, the 3D position information of each pixel in the first surgical scene image in the camera coordinate system can be obtained by calculating the positional deviation. Based on the pre-acquired camera coordinate system and world coordinate system (i.e., the coordinate system corresponding to the actual space during surgery), and based on the 3D position information of each pixel in the first surgical scene image, 3D reconstruction is performed to obtain a 3D model of the surgical scene. The 3D model of the surgical scene is then converted into a format to obtain the 3D point cloud data of the first surgical scene. It should be noted that, as those skilled in the art will understand, the 3D vision scanner 120 can also be other types of 3D cameras in the prior art, such as structured light 3D cameras, time-of-flight (TOF) 3D cameras, laser triangulation 3D cameras, etc. The specific working principle can be referred to the prior art, and will not be repeated here.

[0147] Please continue to refer to this. Figure 13 The diagram illustrates a process for automatically detecting and extracting three-dimensional point cloud data of a stable facial region based on three-dimensional point cloud data of a first surgical scene, according to an embodiment of the present invention. Figure 13 As shown, the 3D vision module 100 (3D vision scanner 120) scans the surgical scene during surgery to obtain first surgical scene scan data. Based on this data, 3D reconstruction is performed to reconstruct a 3D model of the surgical scene. The 3D model is then format-converted to generate corresponding first surgical scene 3D point cloud data. By recognizing this point cloud data, 3D point cloud data of the intraoperative facial region can be automatically extracted. Furthermore, by recognizing this 3D point cloud data of the intraoperative facial region, 3D point cloud data of the intraoperative stable facial region can be automatically extracted. The extracted 3D point cloud data of the intraoperative facial region and the 3D point cloud data of the intraoperative stable facial region can be used in subsequent registration processes.

[0148] In one exemplary embodiment, the control terminal 200 is further configured to perform three-dimensional reconstruction based on the segmentation results of the preoperative medical image to obtain a preoperative virtual three-dimensional model, and to plan the surgical path based on the preoperative virtual three-dimensional model to plan the surgical path.

[0149] For details, please refer to Figure 14 The diagram illustrates a flowchart of a preoperative virtual three-dimensional model reconstruction and surgical path planning provided by an embodiment of the present invention. Figure 14 As shown, taking a preoperative medical image of the brain (skull) as an example, the preoperative medical image (brain (skull) medical image) is segmented using a trained segmentation model. The target lesion, skull, intracranial blood vessels, skin, and brain functional areas can be automatically segmented. After the automatic segmentation is completed, the doctor confirms the segmentation results. If there are any abnormal segmentation areas, manual intervention and adjustment are performed until the requirements are met. After completion, a three-dimensional reconstruction is performed on the segmented head tissues such as the target lesion, skull, intracranial blood vessels, skin, and brain functional areas to obtain a preoperative virtual three-dimensional model. Based on the preoperative virtual three-dimensional model, the surgical approach point (i.e., the punching point) at the skin location and the surgical path to the target lesion can be determined. The planning of the surgical path needs to avoid intracranial blood vessels and brain functional areas (such as the motor area or language area).

[0150] In one exemplary embodiment, the control terminal 200 is further configured to map the preoperative virtual 3D model and the surgical path to the intraoperative real space according to the spatial mapping relationship between the preoperative medical image space and the intraoperative real space, and to fuse the mapped preoperative virtual 3D model and the surgical path with the surgical scene 3D model for augmented reality display. Thus, by fusing the mapped preoperative virtual 3D model and the surgical path with the surgical scene 3D model, augmented reality surgical navigation can be achieved, ensuring that the surgeon can operate the surgical instruments 300 according to the planned surgical path to reach the target lesion, thereby facilitating a safer and more precise completion of the surgical procedure.

[0151] In one exemplary embodiment, the 3D vision module 100 is further configured to scan the surgical instrument 300 during movement to acquire second surgical scene scan data including the surgical instrument 300; the control terminal 200 is further configured to acquire second surgical scene three-dimensional data based on the second surgical scene scan data, identify an identifier 310 in the second surgical scene three-dimensional data, and acquire the pose information of the end effector of the corresponding surgical instrument 300 based on the position information of the identified identifier 310. Thus, the surgical navigation system provided by this invention can track and identify the surgical instrument 300 during movement in real time. By acquiring the pose information of the end effector of the surgical instrument 300 during movement in real time, it can further effectively ensure that the surgeon can operate the surgical instrument 300 to reach the target lesion according to the planned surgical path, thereby further facilitating the surgeon to complete the surgical operation more safely and accurately. It should be noted that, as those skilled in the art will understand, the specific content regarding how to acquire the second surgical scene three-dimensional data based on the second surgical scene scan data can be found in the above-mentioned content regarding acquiring the first surgical scene three-dimensional data based on the first surgical scene scan data, and will not be repeated here. Furthermore, it should be noted that, as those skilled in the art can understand, the identifiers 310 on different surgical instruments 300 are different. Therefore, based on the pre-obtained mapping relationship between the identifiers 310 and the surgical instruments 300, the surgical instrument 300 corresponding to the identified identifier 310 can be determined.

[0152] Please continue to refer to this. Figure 15 The diagram illustrates a scenario of identifying a surgical instrument 300 according to an embodiment of the present invention. Figure 15 As shown, the surgical instrument 300 has an identifier 310 at its proximal end. The 3D vision module 100 can acquire second surgical scene scan data, including the surgical instrument 300, through the 3D vision scanner 120. The control terminal 200 (specifically, the data processing module 211 in the server 210) can obtain second surgical scene three-dimensional point cloud data based on the second surgical scene scan data. By identifying the second surgical scene three-dimensional point cloud data, the identifier 310 can be identified. Then, based on the location information of the identified identifier 310, the positioning and orientation estimation of the surgical instrument 300 where the identifier 310 is located can be realized. This information is then fused and displayed with the mapped preoperative virtual three-dimensional model and the fusion result of the surgical path and the surgical scene three-dimensional model, thereby facilitating the guidance of the surgical instrument 300 to reach the target lesion along the planned surgical path.

[0153] Specifically, multiple different identifiers 310 can be engraved on the proximal end of the surgical instrument 300, or they can be set on the surgical instrument 300 in other ways. Furthermore, this invention does not limit the number of identifiers 310 set on the surgical instrument 300 or the size of each identifier 310; these can be set according to actual conditions. It should be noted that, as those skilled in the art will understand, this invention does not limit the specific type of the identifiers 310; the identifiers 310 can be characters (including text, numbers, letters, etc.), symbols, graphics, etc.

[0154] Furthermore, the plurality of identifiers 310 on the same surgical instrument 300 are all different from each other. Thus, by setting a ring of multiple different identifiers 310 on the surgical instrument 300, not only can the position information of the surgical instrument 300 be obtained based on the position information of the identified identifiers 310, but also the rotation angle of the surgical instrument 300 can be determined based on the angle at which the identified identifiers 310 are set on the surgical instrument 300.

[0155] In one exemplary embodiment, the control terminal 200 is further configured to display the movement trajectory of the end effector of the surgical instrument 300 in real time. Specifically, the movement trajectory of the end effector of the surgical instrument 300 can be displayed on the fusion result of the mapped preoperative virtual 3D model and the surgical path with the surgical scene 3D model, thereby further ensuring that the surgeon can operate the surgical instrument 300 to reach the target lesion according to the planned surgical path.

[0156] Please continue to refer to this. Figure 16 The diagram illustrates a block structure of a server 210 according to an embodiment of the present invention. Figure 16As shown, the data processing module 211 includes a medical image processing unit 2111, an intraoperative registration unit 2112, and an augmented reality navigation unit 2113. The second data interface 212 receives preoperative medical images and intraoperative surgical scene scan data (including first surgical scene scan data and second surgical scene scan data). The preoperative medical images are transmitted to the medical image processing unit 2111 for processing such as target tissue segmentation, reconstruction of the preoperative virtual three-dimensional model, and surgical path planning. The processed preoperative skin surface three-dimensional data and the first surgical scene scan data are transmitted to the intraoperative registration unit 2112. The intraoperative registration unit 2112 automatically identifies the three-dimensional data of the target area (e.g., facial area) and the three-dimensional data of the stable area (e.g., stable facial area) before and during the operation to complete the surgical space registration. After registration, the data (including the preoperative virtual three-dimensional model, surgical path, surgical scene three-dimensional model, second surgical scene scan data, etc.) are transmitted to the augmented reality navigation unit 2113 to realize the fusion display of preoperative data and real surgical scene, and the fusion result is transmitted to the display 220 through the second data interface 212.

[0157] The present invention also provides an electronic device, please refer to... Figure 17 The diagram illustrates a block structure of an electronic device according to an embodiment of the present invention. Figure 17 As shown, the electronic device includes a processor 410 and a memory 430. The memory 430 stores a computer program, which, when executed by the processor 410, implements the surgical space registration method described above. Since the electronic device provided by this invention and the surgical space registration method described above belong to the same inventive concept, the electronic device provided by this invention possesses all the advantages of the surgical space registration method described above. For details, please refer to the relevant descriptions above; further elaboration will not be repeated here.

[0158] In one exemplary embodiment, when the computer program is executed by the processor 410, the following steps are also performed:

[0159] The first surgical scene images from different angles are reconstructed using the multiple optical cameras in the 3D vision module 100 to obtain a 3D model of the surgical scene, and 3D point cloud data of the first surgical scene is obtained based on the 3D model of the surgical scene.

[0160] In one exemplary embodiment, when the computer program is executed by the processor 410, the following steps are also performed:

[0161] Three-dimensional reconstruction is performed based on the segmentation results of the preoperative medical images to obtain a preoperative virtual three-dimensional model, and the surgical path is planned based on the preoperative virtual three-dimensional model to plan the surgical path.

[0162] In one exemplary embodiment, when the computer program is executed by the processor 410, the following steps are also performed:

[0163] Based on the spatial mapping relationship between the preoperative medical image space and the intraoperative real space, the preoperative virtual 3D model and the surgical path are mapped to the intraoperative real space, and the mapped preoperative virtual 3D model and the surgical path are fused with the surgical scene 3D model for augmented reality display.

[0164] In one exemplary embodiment, when the computer program is executed by the processor 410, the following steps are also performed:

[0165] Based on the second surgical scene scan data including the surgical instrument 300 collected by the 3D vision module 100, the second surgical scene three-dimensional data is obtained, and the identifier 310 in the second surgical scene three-dimensional data is identified. Based on the position information of the identified identifier 310, the pose information of the end of the corresponding surgical instrument 300 is obtained.

[0166] In one exemplary embodiment, when the computer program is executed by the processor 410, the following steps are also performed:

[0167] The movement trajectory of the end of the surgical instrument 300 is displayed in real time.

[0168] Please continue to refer to this. Figure 17 ,like Figure 17 As shown, the electronic device also includes a communication interface 420 and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The communication bus 440 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 440 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface 420 is used for communication between the aforementioned electronic device and other devices.

[0169] The processor 410 referred to in this invention can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 410 is the control center of the electronic device, connecting various parts of the entire electronic device through various interfaces and lines.

[0170] Furthermore, the memory 430 can be used to store the computer program, and the processor 410 implements various functions of the electronic device by running or executing the computer program stored in the memory 430 and calling the data stored in the memory 430.

[0171] The memory 430 may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0172] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the surgical space registration method described above. Since the computer program product provided by this invention and the surgical space registration method described above belong to the same inventive concept, the computer program product provided by this invention possesses all the advantages of the surgical space registration method described above. For details, please refer to the relevant descriptions above, which will not be repeated here.

[0173] In one exemplary implementation, when the computer program is executed by the processor, it also performs the following steps:

[0174] The first surgical scene images from different angles are reconstructed using the multiple optical cameras in the 3D vision module 100 to obtain a 3D model of the surgical scene, and 3D point cloud data of the first surgical scene is obtained based on the 3D model of the surgical scene.

[0175] In one exemplary embodiment, when the computer program is executed by the processor, it further performs the following steps:

[0176] Three-dimensional reconstruction is performed based on the segmentation results of the preoperative medical images to obtain a preoperative virtual three-dimensional model, and the surgical path is planned based on the preoperative virtual three-dimensional model to plan the surgical path.

[0177] In one exemplary embodiment, when the computer program is executed by the processor, it further performs the following steps:

[0178] Based on the spatial mapping relationship between the preoperative medical image space and the intraoperative real space, the preoperative virtual 3D model and the surgical path are mapped to the intraoperative real space, and the mapped preoperative virtual 3D model and the surgical path are fused with the surgical scene 3D model for augmented reality display.

[0179] In one exemplary embodiment, when the computer program is executed by the processor, it further performs the following steps:

[0180] Based on the second surgical scene scan data including the surgical instrument 300 collected by the 3D vision module 100, the second surgical scene three-dimensional data is obtained, and the identifier 310 in the second surgical scene three-dimensional data is identified. Based on the position information of the identified identifier 310, the pose information of the end of the corresponding surgical instrument 300 is obtained.

[0181] In one exemplary embodiment, when the computer program is executed by the processor, it further performs the following steps:

[0182] The movement trajectory of the end of the surgical instrument 300 is displayed in real time.

[0183] This invention also provides a readable storage medium storing a computer program that, when executed by a processor, can implement the surgical space registration method described above. Since the readable storage medium provided by this invention and the surgical space registration method described above belong to the same inventive concept, the readable storage medium provided by this invention possesses all the advantages of the surgical space registration method described above. For details, please refer to the relevant descriptions above, which will not be repeated here.

[0184] In one exemplary embodiment, when the computer program is executed by the processor, it further performs the following steps:

[0185] The first surgical scene images from different angles are reconstructed using the multiple optical cameras in the 3D vision module 100 to obtain a 3D model of the surgical scene, and 3D point cloud data of the first surgical scene is obtained based on the 3D model of the surgical scene.

[0186] In one exemplary embodiment, when the computer program is executed by the processor, the following steps are also performed:

[0187] Three-dimensional reconstruction is performed based on the segmentation results of the preoperative medical images to obtain a preoperative virtual three-dimensional model, and the surgical path is planned based on the preoperative virtual three-dimensional model to plan the surgical path.

[0188] In one exemplary embodiment, when the computer program is executed by the processor, the following steps are also performed:

[0189] Based on the spatial mapping relationship between the preoperative medical image space and the intraoperative real space, the preoperative virtual 3D model and the surgical path are mapped to the intraoperative real space, and the mapped preoperative virtual 3D model and the surgical path are fused with the surgical scene 3D model for augmented reality display.

[0190] In one exemplary embodiment, when the computer program is executed by the processor, the following steps are also performed:

[0191] Based on the second surgical scene scan data including the surgical instrument 300 collected by the 3D vision module 100, the second surgical scene three-dimensional data is obtained, and the identifier 310 in the second surgical scene three-dimensional data is identified. Based on the position information of the identified identifier 310, the pose information of the end of the corresponding surgical instrument 300 is obtained.

[0192] In one exemplary embodiment, when the computer program is executed by the processor, the following steps are also performed:

[0193] The movement trajectory of the end of the surgical instrument 300 is displayed in real time.

[0194] It should be noted that the readable storage medium provided by this invention can be any combination of one or more computer-readable media. The readable medium can be a computer-readable signal medium or a computer-readable storage medium. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable computer hard disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device.

[0195] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0196] In summary, compared with the prior art, the surgical space registration method, surgical navigation system, electronic device, computer program product, and storage medium provided by the present invention have the following advantages:

[0197] The surgical space registration method provided by this invention segments the acquired preoperative medical image to obtain preoperative three-dimensional data of the skin surface; identifies the preoperative target region three-dimensional data based on the preoperative skin surface three-dimensional data, and identifies the preoperative stable region three-dimensional data based on the preoperative target region three-dimensional data; identifies the acquired first surgical scene three-dimensional data to identify the intraoperative target region three-dimensional data, and identifies the intraoperative stable region three-dimensional data based on the intraoperative target region three-dimensional data; performs coarse registration on the preoperative target region three-dimensional data and the intraoperative target region three-dimensional data to obtain a first spatial transformation matrix; finally, performs fine registration on the preoperative stable region three-dimensional data and the intraoperative stable region three-dimensional data based on the first spatial transformation matrix to obtain a second spatial transformation matrix; and finally, obtains the spatial mapping relationship between the preoperative medical image space and the intraoperative real space based on the first spatial transformation matrix and the second spatial transformation matrix, thereby completing the spatial registration. Therefore, the surgical space registration method provided by this invention does not require the detection of marker points, thus reducing the spatial registration time in surgical navigation and improving the efficiency of surgical operations. Furthermore, this invention first performs coarse registration on the preoperative target region 3D data and the intraoperative target region 3D data to obtain a first spatial transformation matrix, and then performs fine registration on the preoperative stable region 3D data and the intraoperative stable region 3D data based on the first spatial transformation matrix to obtain a second spatial transformation matrix. This can improve the accuracy of spatial registration in surgical navigation, lay a good foundation for achieving precise surgical navigation, and meet the accuracy requirements of neurosurgery.

[0198] Since the surgical navigation system, electronic device, computer program product, and storage medium provided by this invention belong to the same inventive concept as the surgical space registration method provided by this invention, the surgical navigation system, electronic device, computer program product, and storage medium provided by this invention have all the advantages of the surgical space registration method provided by this invention. Furthermore, the surgical navigation system provided by this invention uses a 3D vision module 100 to scan the surgical scene in real time. The control terminal reconstructs a three-dimensional model of the surgical scene based on the scanning results of the 3D vision module 100, and fuses the preoperative virtual three-dimensional model reconstructed based on the segmentation results of preoperative medical images with the surgical scene three-dimensional model for display. This allows doctors to complete surgical operations without wearing augmented reality glasses. Moreover, since the 3D vision module 100 is more stable than augmented reality glasses, it can further meet the precision requirements of neurosurgery.

[0199] It should be noted that computer program code for performing the operations of this embodiment can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0200] It should also be noted that the apparatus and methods disclosed in the embodiments herein can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments herein. In this regard, each block in a flowchart or block diagram may represent a module, program, or part of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system to perform the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0201] In addition, the functional modules in the various embodiments of this article can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0202] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure are within the protection scope of the present invention. Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A surgical space registration method, characterized in that, include: The acquired preoperative medical images are segmented to obtain three-dimensional data of the preoperative skin surface; The three-dimensional data of the preoperative skin surface are identified to identify the three-dimensional data of the preoperative target area, and the three-dimensional data of the preoperative target area are identified to identify the three-dimensional data of the preoperative stable area. The acquired three-dimensional data of the first surgical scene are identified to identify the three-dimensional data of the target area during the operation, and the three-dimensional data of the target area during the operation are identified to identify the three-dimensional data of the stable area during the operation. Coarse registration is performed on the preoperative target region 3D data and the intraoperative target region 3D data to obtain a first spatial transformation matrix; Based on the first spatial transformation matrix, the three-dimensional data of the preoperative stable region and the three-dimensional data of the intraoperative stable region are precisely registered to obtain the second spatial transformation matrix. Based on the first spatial transformation matrix and the second spatial transformation matrix, obtain the spatial mapping relationship between the preoperative medical image space and the intraoperative real space; The target region is the facial region, and the stable region is the facial stable region, which includes at least one of the brow bone region, frontal bone region, and nose region.

2. The surgical space registration method according to claim 1, characterized in that, The segmentation of the acquired preoperative medical images to obtain preoperative three-dimensional skin surface data includes: The acquired preoperative medical images are preprocessed to obtain preprocessed preoperative medical images; A pre-trained segmentation model is used to segment the preprocessed preoperative medical images to obtain preoperative target tissue segmentation images; Preoperative three-dimensional data of the skin surface are obtained based on the preoperative target tissue segmentation image.

3. The surgical space registration method according to claim 2, characterized in that, The step of obtaining preoperative three-dimensional skin surface data based on the preoperative target tissue segmentation image includes: Connectivity analysis is performed on the skin region in the preoperative target tissue segmentation image to extract the largest connected component, and the preoperative three-dimensional data of the skin surface is obtained based on the extracted largest connected component.

4. The surgical space registration method according to claim 1, characterized in that, The process of identifying the preoperative three-dimensional data of the skin surface to determine the preoperative target area includes: A pre-trained first target detection model is used to identify the three-dimensional data of the preoperative skin surface to identify the three-dimensional data of the preoperative target area; The process of identifying the three-dimensional data of the preoperative target region to identify the three-dimensional data of the preoperative stable region includes: A pre-trained second target detection model is used to identify the three-dimensional data of the preoperative target region in order to identify the three-dimensional data of the preoperative stable region. The step of identifying the acquired three-dimensional data of the first surgical scene to identify the three-dimensional data of the target area during surgery includes: A pre-trained third target detection model is used to identify the three-dimensional data of the first surgical scene in order to identify the three-dimensional data of the target area during the operation. The process of identifying the three-dimensional data of the intraoperative target region to identify the three-dimensional data of the intraoperative stable region includes: A pre-trained fourth target detection model is used to identify the three-dimensional data of the intraoperative target area in order to identify the three-dimensional data of the stable intraoperative area.

5. The surgical space registration method according to claim 1, characterized in that, The preoperative three-dimensional data of the skin surface, the preoperative three-dimensional data of the target area, and the preoperative three-dimensional data of the stable area are all three-dimensional image data, and the first surgical scene three-dimensional data, the intraoperative target area three-dimensional data, and the intraoperative stable area three-dimensional data are all three-dimensional point cloud data. The coarse registration of the preoperative target region 3D data and the intraoperative target region 3D data includes: Obtain 3D point cloud data of the target area based on the 3D image data of the target area before surgery; Coarse registration is performed on the preoperative target region 3D point cloud data and the intraoperative target region 3D point cloud data. The precise registration of the preoperative stable region 3D data and the intraoperative stable region 3D data based on the first spatial transformation matrix includes: Based on the three-dimensional image data of the preoperative stable region, obtain the three-dimensional point cloud data of the preoperative stable region; Based on the first spatial transformation matrix, the three-dimensional point cloud data of the preoperative stable region and the three-dimensional point cloud data of the intraoperative stable region are precisely registered.

6. The surgical space registration method according to claim 1, characterized in that, The preoperative medical images are brain medical images.

7. A surgical navigation system, characterized in that, The device includes a 3D vision module and a control terminal with communication connection. The 3D vision module is configured to scan the surgical scene during surgery to collect first surgical scene scan data. The control terminal is configured to obtain three-dimensional data of the first surgical scene based on the first surgical scene scan data and implement the surgical space registration method as described in any one of claims 1 to 6.

8. The surgical navigation system according to claim 7, characterized in that, The 3D vision module includes multiple optical cameras for scanning the surgical scene to acquire images of the first surgical scene from different angles. The control terminal is configured to perform three-dimensional reconstruction based on the images of the first surgical scene from different angles acquired by the multiple optical cameras to obtain a three-dimensional model of the surgical scene, and to obtain three-dimensional point cloud data of the first surgical scene based on the three-dimensional model of the surgical scene.

9. The surgical navigation system according to claim 8, characterized in that, The control terminal is also configured to perform three-dimensional reconstruction based on the segmentation results of preoperative medical images to obtain a preoperative virtual three-dimensional model, and to plan the surgical path based on the preoperative virtual three-dimensional model.

10. The surgical navigation system according to claim 9, characterized in that, The control terminal is further configured to map the preoperative virtual 3D model and the surgical path to the intraoperative real space according to the spatial mapping relationship between the preoperative medical image space and the intraoperative real space, and to fuse the mapped preoperative virtual 3D model and the surgical path with the surgical scene 3D model for augmented reality display.

11. The surgical navigation system according to claim 10, characterized in that, The 3D vision module is also configured to scan surgical instruments during movement to collect second surgical scene scan data including surgical instruments; the control terminal is also configured to acquire second surgical scene three-dimensional data based on the second surgical scene scan data, identify identifiers in the second surgical scene three-dimensional data, and acquire the pose information of the end of the corresponding surgical instrument based on the position information of the identified identifiers.

12. The surgical navigation system according to claim 11, characterized in that, The control terminal is also configured to display the movement trajectory of the end of the surgical instrument in real time.

13. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, implements the surgical space registration method according to any one of claims 1 to 6.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the surgical space registration method according to any one of claims 1 to 6.

15. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which, when executed by a processor, implements the surgical space registration method according to any one of claims 1 to 6.

Images