Multi-modal image surgical navigation method and system, electronic equipment and medium

Abstract

The invention discloses a multi-modal image surgical navigation method and system, electronic equipment and a medium, and solves the problems that the capability of integrating multi-modal data is insufficient, a clear mechanism for synchronously displaying different modal information is lacked, and the comprehensive understanding and accurate operation of a surgeon on a tissue structure in a surgery are limited. By integrating a deep learning organ segmentation model, an esophageal model registration algorithm and a manual adjustment mechanism, the limitation of the prior art in esophageal surgery application is solved. The system can effectively process dynamic changes of the esophagus in the operation process, continuous and accurate navigation information is provided, an intuitive and interactive operation environment is created for surgeons, and the accuracy and safety of the operation are further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of surgical navigation, and specifically, to a multi-modal image surgical navigation method, system, electronic device and medium. Background Art

[0002] Surgical navigation technology has brought a revolutionary change to the implementation of modern medical surgeries. By integrating various pre-operative and intra-operative information, such systems aim to provide a more comprehensive virtual-reality fusion operation view, assisting surgeons in precisely positioning surgical instruments and key anatomical structures including lesions within complex human structures. This "perspective" auxiliary means for internal structures reduces patient pain while greatly enhancing surgical safety, precision, effectively shortening the surgical time, improving the overall surgical efficiency, and accelerating the patient's postoperative recovery; registration is a key technology in surgical navigation systems, and the registration accuracy and efficiency largely determine the quality of the navigation system and the success or failure of the surgery.

[0003] Related technologies are insufficient in integrating multi-modal data and lack a clear mechanism to synchronously display different modal information, which limits surgeons' comprehensive understanding and precise operation of tissue structures during surgery. This series of limitations highlights the market's demand for a surgical system that integrates highly automated pre-operative image processing and provides three-dimensional, multi-modal real-time navigation. Summary of the Invention

[0004] The purpose of the present invention is to solve the above problems, and provide a multi-modal image surgical navigation method, system, electronic device and medium, so as to solve the problems of insufficient ability in integrating multi-modal data, lack of a clear mechanism to synchronously display different modal information, and limitation of surgeons' comprehensive understanding and precise operation of tissue structures during surgery.

[0005] To solve the above problems, the present invention provides the following technical solutions: On the one hand, a multi-modal image surgical navigation method includes: Obtain target medical image data, input it into a trained medical image segmentation model, and obtain an image center curve; Obtain endoscopic trajectory data, input it into a trained medical image segmentation model, and obtain a trajectory center curve; Training the medical image segmentation model includes training a single-fold model and then performing multi-fold model fusion to convert the dynamic graph model into a static graph model; Automatically register the image center curve and the trajectory center curve to generate a spatial transformation matrix for surgical navigation.

[0006] In related embodiments, after obtaining the image center curve and the trajectory center curve, correct the segmentation results according to manually edited labels.

[0007] In related embodiments, after the single-fold model is trained, multi-fold model fusion includes: Preprocessing the window width and window level of the training data and the test data; Parallelly training models with different numbers of folds, and finally fusing and predicting the data to obtain the final result.

[0008] In related embodiments, after generating the spatial transformation matrix for surgical navigation, it further includes generating synchronous display of CT images, NDI magnetic navigation trajectories, endoscopic videos, and three-dimensional reconstruction models; the target medical image data and endoscopic trajectory data are in image or video format.

[0009] In related embodiments, obtaining target medical image data and inputting it into the trained medical image segmentation model to obtain the image center curve includes: Obtaining the esophageal image data of the patient, and constructing a continuous spatial point set according to the center points of the esophageal cross-sections; Extracting the starting point, ending point, and image feature points of the esophagus from the spatial point set, and connecting them to form an image center curve including the key morphological feature positions of the esophagus; The extraction of image feature points includes determining the curvature values according to the spatial point set, and arranging the curvature values in descending order to obtain a curvature list; Setting a threshold, and according to the curvature list and adjacent differences, obtaining multiple sub-intervals of the spatial point set, and selecting the points that best match the preset esophageal empirical model from the sub-intervals of the spatial point set as the image feature points.

[0010] In related embodiments, obtaining endoscopic trajectory data and inputting it into the trained medical image segmentation model to obtain the trajectory center curve includes: Calculating vectors according to the pose of the endoscopic trajectory, rotating the three-dimensional trajectory data to the image center line coordinate system, and then performing average translational transformation on the feature points of the trajectory data and the feature point set of the center curve to complete the rough registration transformation and obtain the trajectory center curve.

[0011] In related embodiments, automatically registering the image center curve and the trajectory center curve to generate a spatial transformation matrix for surgical navigation includes: The first refined registration includes: obtaining the center curve of the trajectory data, and initially registering it with the image center curve according to the iterative closest point algorithm to obtain a spatial transformation relationship; The second refined registration includes: re-segmenting the endoscopic trajectory data, performing high-precision scanning with a smaller step size, fitting a more accurate image center curve, and initially registering it with the image center curve according to the iterative closest point algorithm to obtain a precise transformation relationship; When the registration is completed, the obtained spatial transformation matrix accurately maps the endoscopic position collected by the real-time spatial pose to the corresponding medical image and the pre-established three-dimensional model, thereby realizing real-time navigation of esophageal endoscopic surgery.

[0012] In a second aspect, a surgical navigation system based on multimodal images is used to implement the multimodal image surgical navigation method; it includes: Medical image segmentation module: used to train a medical image segmentation model and a 3D reconstruction model; obtained by training an existing image semantic segmentation network based on manually annotated data.

[0013] Registration algorithm module: used to input a target image or image to obtain the image center curve and the trajectory center curve and perform automatic registration to generate a spatial transformation matrix for surgical navigation; Manual editing module: used to manually edit the segmentation result; Multimodal information display module: used to display the generated CT image, NDI magnetic navigation trajectory, endoscopic video, and 3D reconstruction model synchronously.

[0014] In a third aspect, an electronic device includes a memory and a processor. When a computer-readable instruction stored in the memory is executed by the processor, the processor executes the multimodal image surgical navigation method.

[0015] In a fourth aspect, a computer-readable storage medium stores a computer program. When the computer program runs on a computer, the computer executes the multimodal image surgical navigation method.

[0016] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention effectively utilizes the complementary information of multimodal images to overcome the technical challenges faced by single-image guidance. The multimodal image-guided surgical navigation system can provide real-time positioning information of instruments for surgeons, helping doctors avoid important tissue structures while removing diseased tissues and directly reach the target position, effectively protecting important tissues, blood vessels, and organs around the lesion, avoiding unnecessary injuries, and reducing the probability of surgical complications.

[0017] (2) The present invention can break the information barrier between multiple modalities, achieve linked display, increase the information directly obtained by doctors, and reduce the difficulty of doctors in matching multimodal information one by one; register the 3D model and the NDI magnetic navigation trajectory to facilitate observing the relative positions of the trajectory and the model in the same coordinate system.

[0018] (3) The present invention utilizes the characteristics of the NDI magnetic navigation device to match the endoscopic video image sequence with the NDI magnetic navigation trajectory sequence one by one, achieving the effect that one NDI trajectory point corresponds to one endoscopic image; by using the corresponding relationship between the three-dimensional model and the CT image, the two are established in the same coordinate system to facilitate observing the relative positions of the tissues and organs around the CT; the matching of the above-mentioned multiple modality information is established in one interface, and synchronous display and synchronous adjustment can be realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings, where: Figure 1 is the system architecture diagram of the medical image segmentation model; Figure 2 is the processing flow chart of the medical image segmentation algorithm based on deep learning; Figure 3 is the flow chart of the 3D reconstruction algorithm based on VTK; Figure 4 is the unadjusted effect diagram of the window width and window level adjustment algorithm; Figure 5 is the adjusted effect diagram of the window width and window level adjustment algorithm; Figure 6 is the modeling effect of the marching cubes method; Figure 7 is the schematic diagram of the multi-modal data synchronous display system; Figure 8 is the multi-modal registration synchronous display interface diagram. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] In order to make the objectives, technical solutions and advantages of the present invention clearer, the following will further describe the present invention in detail in conjunction with Figures 1 to 8 The described embodiments should not be regarded as limitations of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0021] 1. Method Overview

[0022] The present invention proposes an innovative multi-modal surgical navigation software system specifically designed for esophageal surgery. By integrating a deep learning organ segmentation model, an esophageal model registration algorithm, and a manual adjustment mechanism, the present invention addresses the limitations of existing technologies in the application of esophageal surgery. The system can effectively handle the dynamic changes of the esophagus during surgery, provide continuous and accurate navigation information, create an intuitive and interactive surgical environment for surgeons, and further improve the precision and safety of surgery. In addition, our system also focuses on user-friendliness and ease of operation, aiming to lower the technical threshold and promote the wide popularization and application of multi-modal navigation technology in the field of esophageal surgery.

[0023] This method can achieve all-round navigation for endoscopic esophageal surgery operations, mainly including three core functions: medical image segmentation and 3D modeling based on deep learning, automatic registration of magnetic navigation trajectories, and multi-modal data fusion display.

[0024] 2. Specific methods

[0025] As Figure 1 shown, medical image segmentation and 3D modeling based on deep learning: Use deep learning technology to automatically segment the uploaded original medical image data, and import the segmentation results into a 3D modeling algorithm for processing to obtain a 3D organ model.

[0026] Automatic registration of magnetic navigation trajectories: Use an automatic registration algorithm based on point cloud registration to register the uploaded magnetic navigation trajectory data with the 3D esophageal model for data calibration for the multi-modal information fusion function. Multi-modal data fusion display: Real-time registration of the original CT image with the segmentation result, magnetic navigation trajectory data, endoscopic video data, and 3D model data to achieve synchronous display of multi-modal data on the same interface.

[0027] 2.1 Medical image segmentation and 3D reconstruction based on deep learning

[0028] As Figure 2 shown, the medical image segmentation algorithm based on deep learning mainly includes two parts: model training and deployment. When deploying, considering the diverse segmentation needs of users, the system provides a function for manually editing the segmentation results, allowing users to modify and correct the segmentation results according to their own needs.

[0029] The core of the 3D reconstruction algorithm based on VTK is to use the discrete marching cubes method to perform surface reconstruction on the segmentation results and perform a series of post-processing to obtain a 3D reconstruction model of the segmentation results.

[0030] 2.1.1 Medical image segmentation and 3D reconstruction based on deep learning

[0031] As Figure 2 and3 As shown in the figure, for the training process of the nnU-Net network model, the system mainly adopts the following improvement strategies: (1) Explanation of the model training algorithm Based on the existing open-source nnU-Net network, the window width and window level of the training data and test data are preprocessed (window width range 50 - 350), so that each data highlights the eleven organs around the esophagus to be segmented; the models with different folds are trained in parallel, and finally the data is aggregated to the same server for ensemble prediction to obtain the final result.

[0032] (2) Inference prediction of the static graph model Since the dynamic graph model performs inference based on the Python interpretation and execution method, the inference speed is slow, while the static graph model performs inference based on C++ in a way of compiling first and then running, with high efficiency and fast speed. Therefore, we adopt the method of converting the dynamic graph model into a static graph for inference, combined with the fast processing mode we designed, which further speeds up the inference speed.

[0033] 2.1.2 3D reconstruction algorithm based on VTK

[0034] As Figure 3 shown, aiming at the characteristics of large data dispersion degree and difficult data smoothing of the segmentation result data, the system adopts a 3D reconstruction algorithm with the discrete marching cubes method as the core. The marching cubes method is a common algorithm for 3D reconstruction of medical image data, and the discrete marching cubes method is an improved algorithm for the segmentation results of discontinuous medical image data. After the reconstruction processing by the marching cubes algorithm, the model is further processed by the window smoothing and vertex simplification algorithms to optimize the overall display effect of the model.

[0035] 2.2 Registration algorithm module 2.2.1 Medical image processing

[0036] As Figures 4 to 8 shown, using the patient's CT scan data, by averaging the pixel coordinates of the esophagus region in the cross-sectional image sequence, the center of the esophagus in each cross-section is determined, and a continuous spatial point set is constructed in combination with the Z-axis coordinate information. . In the two-dimensional CT cross-section, for the mask image , where indicates that the pixel point belongs to the esophagus region, indicates that it does not belong, then the center point of the esophagus can be determined by calculating the coordinate mean of all pixel points in the esophagus region through formula (1-1) and formula (1-2): (1-1) (1 - 2) Based on the spatial point set, extract the central curve of the esophagus and the positions of its key morphological features, including the starting point, ending point of the esophagus, and a feature point near the third stenosis of the esophagus with significant curvature changes.

[0037] To improve the data quality, perform sliding window average filtering on the point set to reduce the influence of noise. For each point in the point set , consider a window and calculate the coordinate mean of all adjacent points within it using formula (1 - 3): (1 - 3) The selection of the curvature feature point follows the following steps: (1) For three consecutive points in the point set in the form of , the curvature can be estimated using formula (1 - 4) : , , (1 - 4) (2) Sort the curvature values of all points to obtain an ordered curvature list , where represents the th largest curvature after sorting.

[0038] (3) Set a threshold and calculate the adjacent difference of the indices corresponding to the sorted curvatures. If , then it is considered that has a spatial discontinuity between the point with its curvature value and the adjacent point, and it is used as the grouping boundary. Group according to the difference value to form multiple point set sub - intervals, and the curvature within each sub - interval is relatively stable and spatially adjacent.

[0039] (4) For each sub - interval, calculate the degree of conformity between its representative point and a preset empirical model (such as the position of the third stenosis of the esophagus). Select the representative point of the sub - interval closest to the empirical model as the feature point.

[0040] 2.2.2 Endoscopic Trajectory Analysis and Registration Preparation

[0041] Pre - process the three - dimensional trajectory data of the endoscopic insertion and withdrawal obtained by the spatial pose acquisition device. The data is a point set in the form of . The distance from point to the starting point is calculated using formula (1 - 5): (1 - 5) Set the threshold of consecutive decreasing times , if the condition is met: , it is considered that the endoscope withdrawal stage starts from , and the trajectory point set is segmented accordingly.

[0042] Associate the pre - processed trajectory data with the esophageal center curve in the 3D reconstruction model. First, perform rough registration: Use formula (1 - 3) to screen valid trajectory points for two segments of trajectory data respectively, apply similar data smoothing means, and use formula (1 - 4) to find the corresponding feature points of the two - segment trajectories through similar steps and average them.

[0043] For a set of basic feature point pairs, namely the starting point , curvature feature point , and ending point , the pose vector is formed as formula (1 - 6): , , , (1 - 6) In 3D space, the rough transformation from the endoscope trajectory to the model center line can be defined by the rotation matrix , and the translation vector . For the model pose , , and the trajectory describing the endoscope pose , , , describe the transformation that makes rotate to be parallel to , describe the transformation that makes rotate to be parallel to . Define the basic rotation operation as shown in formula (1 - 7). Describe the average translation transformation from the feature point set in the endoscope trajectory to the feature point set in the model center line (starting point , curvature feature point , ending point and more specified corresponding points) after the first two rotation transformations. Finally, calculate the rough registration transformation by the formula group (1 - 8):

[0044] where

[0045]

[0046]

[0047]

[0048]

[0049] (1 - 7)

[0050]

[0051]

[0052]

[0053]

[0054] (1 - 8) 2.2.3 Central Curve Fitting and Refined Registration

[0055] Based on the rough registration , the trajectory point set is transformed to develop along the -z direction, and then a step-by-step refinement strategy is adopted to fit the esophageal central curve in the actual operation. First, based on formula (1 - 3), a large step size is used to preliminarily scan the trajectory data along the -z direction to obtain a rough central curve ; subsequently, with the help of the Iterative Closest Point (ICP) algorithm, that is, formula (1 - 9), the roughly fitted central curve is initially registered with the esophageal central curve extracted from the medical image to obtain a preliminary spatial transformation relationship .

[0056] (1 - 9) Further refine the registration process, re-segment the endoscopic trajectory data, and perform high-precision scanning with a small step size to fit a more accurate esophageal central curve . Finally, apply (1 - 9) again for fine registration until the preset registration accuracy is achieved to obtain the fine registration transformation .

[0057] 2.2.4 Completion and Application of Registration

[0058] When the registration is completed, the obtained spatial transformation matrix can accurately map the endoscopic position collected by the real-time spatial pose to the corresponding medical image and the pre-established three-dimensional model, thereby realizing the real-time navigation of esophageal endoscopic surgery.

[0059] 2.3 Multimodal Information Display Module

[0060] Multimodal image-guided surgical navigation effectively utilizes the complementary information of multimodal images to overcome the technical challenges faced by single-image guidance. The multimodal image-guided surgical navigation system provides surgeons with real-time instrument positioning information, helping them to reach the target location while removing lesions while avoiding critical tissue structures. This effectively protects vital tissues, blood vessels, and organs surrounding the lesion, avoids unnecessary damage, and reduces the likelihood of surgical complications. The primary multimodal information used in this system includes CT images, NDI magnetic navigation trajectories, endoscopic videos, and 3D reconstructed models.

[0061] In order to break down the information barriers between multiple modalities, realize linkage display, increase the information doctors can obtain intuitively, and reduce the difficulty of doctors in matching multi-modal information one by one, the design of this system is as follows: (1) Align the 3D model with the NDI magnetic navigation trajectory to facilitate observation of the relative position of the trajectory and model in the same coordinate system.

[0062] (2) By utilizing the characteristics of the NDI magnetic navigation device, the endoscopic video image sequence is matched one-to-one with the NDI magnetic navigation trajectory sequence, so that one NDI trajectory point corresponds to one endoscopic image.

[0063] (3) By using the correspondence between the three-dimensional model and the CT image, the two are established in the same coordinate system, making it easier to observe the relative positions of tissues and organs around the CT.

[0064] (4) The matching of the multiple modal information involved above is established in a single interface, and synchronous display and synchronous adjustment can be achieved. An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, the steps of a multimodal imaging surgical navigation method are implemented.

[0065] A multimodal imaging-based surgical navigation system is used to implement a multimodal imaging surgical navigation method; the system comprises: Medical image segmentation module: used to train medical image segmentation models and 3D reconstruction models; Registration algorithm module: used to input the target image or image to obtain the image center curve and trajectory center curve and automatically register to generate the space transformation matrix for surgical navigation; Manual editing module: used to manually edit segmentation results; Multimodal information display module: used to display the generated CT images, NDI magnetic navigation trajectories, endoscopic videos and 3D reconstructed models simultaneously.

[0066] The electronic device can be a desktop computer, a notebook, a handheld computer, a cloud server, or other electronic devices. The electronic device can include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the figure is only an example of the electronic device and does not constitute a limitation on the electronic device. It can include more or fewer components than those shown in the figure, or different components.

[0067] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0068] The memory can be an internal storage unit of the electronic device, for example, the hard disk or memory of the electronic device. The memory can also be an external storage device of the electronic device, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the electronic device. The memory can also include both the internal storage unit and the external storage device of the electronic device. The memory is used to store computer programs and other programs and data required by the electronic device.

[0069] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are only illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of devices, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code, and the module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks can occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0070] In addition, each functional module in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist alone, or two or more modules may be integrated to form an independent part.

[0071] If the above-mentioned function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprises", "comprising" or any other variation thereof is intended to cover a non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the element.

[0072] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention. It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0073] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or replacements, which should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A multimodal imaging surgical navigation method, characterized in that obtain target medical imaging data, input it into a trained medical image segmentation model, and obtain an image center curve; obtain endoscopic trajectory data, input it into a trained medical image segmentation model, and obtain a trajectory center curve; Training the medical image segmentation model includes training a single-fold model and then performing multi-fold model fusion to convert the dynamic graph model into a static graph model; Automatically register the image center curve and the trajectory center curve to generate a spatial transformation matrix for surgical navigation.

2. The multi-modal imaging surgical navigation method according to claim 1, wherein After obtaining the image center curve and the trajectory center curve, correct the segmentation result according to the manually edited label.

3. A multimodal imaging surgical navigation method according to claim 1, characterized in that, Multi-fold model fusion after single-fold model training includes: Preprocess the window width and window level of the training data and the test data; Parallel train models with different numbers of folds, and finally fuse and predict the data to obtain the final result.

4. The multimodal imaging surgical navigation method according to claim 1, wherein After generating the spatial transformation matrix for surgical navigation, it also includes generating synchronous display of CT images, NDI magnetic navigation trajectories, endoscopic videos, and three-dimensional reconstruction models; the target medical imaging data and endoscopic trajectory data are in image or video format.

5. The multi-modal imaging surgical navigation method according to claim 1, characterized in that, Obtain target medical imaging data, input it into a trained medical image segmentation model, and obtain an image center curve, including: Obtain the esophageal imaging data of the patient, and construct a continuous spatial point set according to the center points of the esophageal cross-sections; Extract the starting point, ending point, and image feature points of the esophagus from the spatial point set, and connect them to form an image center curve containing the key morphological feature positions of the esophagus; Image feature point extraction includes determining the curvature value according to the spatial point set, and arranging the curvature values in descending order to obtain a curvature list; Set a threshold, and according to the curvature list and adjacent differences, obtain multiple sub-intervals of the spatial point set, and select the points that best match the preset esophageal empirical model from the sub-intervals of the spatial point set as image feature points.

6. The multimodal image surgical navigation method according to claim 1, wherein Obtain endoscopic trajectory data, input it into a trained medical image segmentation model, and obtain a trajectory center curve, including: Calculate the vector according to the pose of the endoscopic trajectory. After rotating the three-dimensional trajectory data to the image center line coordinate system, average the translational changes of the feature points of the trajectory data and the feature point set of the center curve to complete the rough registration transformation and obtain the trajectory center curve.

7. The multimodal imaging surgical navigation method according to claim 1, wherein Automatically registering the image center curve and the trajectory center curve to generate a spatial transformation matrix for surgical navigation includes: The initial fine registration includes: obtaining the center curve of the trajectory data, and performing the initial registration of it and the image center curve according to the iterative closest point algorithm to obtain the spatial transformation relationship; The secondary fine registration includes: re-segmenting the endoscopic trajectory data, performing high-precision scanning with a smaller step size, fitting a more accurate image center curve, and performing the initial registration of it and the image center curve according to the iterative closest point algorithm to obtain the precise transformation relationship; When the registration is completed, the obtained spatial transformation matrix accurately maps the endoscopic position collected by the real-time spatial pose to the corresponding medical image and the pre-established three-dimensional model, so as to realize the real-time navigation of the esophageal endoscopic surgery.

8. A surgical navigation system based on multimodal imaging, characterized in that, For implementing the multimodal imaging surgical navigation method according to any one of claims 1-7; It includes: Medical image segmentation module: used to train the medical image segmentation model and the three-dimensional reconstruction model; Registration algorithm module: It is used to input the target image or video to obtain the image center curve and the trajectory center curve, and perform automatic registration to generate a spatial transformation matrix for surgical navigation; Manual editing module: It is used to manually edit the segmentation results; Multi-modal information display module: It is used to display and synchronously show the generated CT images, NDI magnetic navigation trajectories, endoscopic videos, and three-dimensional reconstruction models.

9. An electronic device includes a memory and a processor, and computer-readable instructions are stored in the memory. When the computer-readable instructions are executed by the processor, it is characterized in that Cause the processor to execute the multi-modal image surgical navigation method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, A computer program is stored in the readable storage medium. When the computer program runs on a computer, it causes the computer to execute the multi-modal image surgical navigation method according to any one of claims 1-7.

Images