Arthroscopic surgery real-time navigation method and system based on multi-modal image fusion

Abstract

The present application relates to the technical field of medical image processing and surgical navigation, and particularly relates to a real-time navigation method and system for arthroscopic surgery based on multi-modal image fusion, comprising the following steps: S1, collecting preoperative CT or MRI data of a patient to generate a three-dimensional model, and simultaneously collecting intraoperative arthroscopic real-time video to provide input data for subsequent registration; S2, performing image preprocessing on the three-dimensional model and the arthroscopic video, including denoising, enhancement, and extracting key anatomical features and lesion boundaries to obtain processing results that can be used for registration. In the present application, the preoperative three-dimensional model is fused with the intraoperative arthroscopic real-time video, and multi-strategy registration, closed-loop correction and dynamic updating are combined, thereby realizing continuous registration and real-time navigation of multi-modal data, so as to improve the problem that traditional arthroscopic surgical navigation mostly relies on single modal image, and the intraoperative registration accuracy is insufficient due to the lack of multi-source information fusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing and surgical navigation, and particularly relates to a real-time navigation method and system for arthroscopic surgery based on multi-modal image fusion. BACKGROUND

[0002] With the development of minimally invasive surgical technology, arthroscopic surgery has become an important means for the treatment of joint diseases. Intraoperative doctors mainly rely on two-dimensional arthroscopic video for operation, but it is difficult to accurately determine the position of the lesion and the spatial relationship of the surrounding anatomical structure with only single video information. Preoperative CT or MRI three-dimensional images can provide rich structural information, but in traditional surgical navigation, these preoperative images and intraoperative operation field lack effective fusion, which is difficult to guide the operation of doctors in real time.

[0003] At present, traditional arthroscopic surgery navigation mostly relies on single modal images, and the lack of multi-source information fusion leads to insufficient intraoperative registration accuracy. SUMMARY

[0004] In order to make up for the above shortcomings, the present application provides a real-time navigation method and system for arthroscopic surgery based on multi-modal image fusion, which aims to improve the problem that traditional arthroscopic surgery navigation mostly relies on single modal images, and the lack of multi-source information fusion leads to insufficient intraoperative registration accuracy.

[0005] In the first aspect, the present application provides a real-time navigation method for arthroscopic surgery based on multi-modal image fusion, comprising the following steps:

[0006] S1, collecting preoperative CT or MRI data of the patient to generate a three-dimensional model, and collecting intraoperative arthroscopic real-time video at the same time, providing input data for subsequent registration;

[0007] S2, image preprocessing is performed on the three-dimensional model and arthroscopic video, including denoising, enhancement, and extracting key anatomical features and lesion boundaries, to obtain processed results that can be used for registration;

[0008] S3, based on the preprocessed results, rigid registration, non-rigid registration and semantic key point matching are simultaneously performed to generate candidate registration results, and the confidence index of each candidate result is calculated to provide options for optimal registration selection;

[0009] S4, the optimal registration result is selected according to the candidate result confidence and the stability of the time sequence of consecutive frames, and closed-loop correction is performed using continuous frame error prediction to update the optimal result;

[0010] S5, the optimal registration result is applied to the three-dimensional model and superimposed to the real-time arthroscopic video to generate a navigation view, and the lesion boundary, key anatomical points and implant path are displayed simultaneously for intraoperative navigation;

[0011] S6, the steps of generating a candidate registration, optimal registration selection and navigation display are cyclically performed during the operation process, ensuring that each frame of video obtains an updated optimal registration result and forms a continuous closed loop.

[0012] By the above technical solution, the preoperative three-dimensional model is fused with the real-time arthroscopic video during the operation, combined with multi-strategy registration, closed-loop correction and dynamic updating, and then the continuous registration and real-time navigation of multi-modal data are realized, thereby improving the problem that the traditional arthroscopic surgery navigation mostly relies on single modal image, and the in-situ registration accuracy is insufficient due to the lack of multi-source information fusion.

[0013] Preferably, the arthroscopic video includes:

[0014] The collected video is adaptively filtered and denoised;

[0015] The filtered video is image enhanced;

[0016] The key anatomical features of bone and soft tissue are extracted to provide input data for multi-strategy registration.

[0017] Preferably, the three-dimensional model includes:

[0018] The surface grid of the three-dimensional model is optimized;

[0019] The lesion boundary and key anatomical points are labeled to provide an accurate model basis for candidate registration result generation.

[0020] Preferably, the generation of the candidate registration result includes:

[0021] Simultaneous rigid registration is performed to handle overall displacement and rotation;

[0022] Non-rigid registration is performed to cope with tissue deformation;

[0023] The semantic key point matching based on deep learning generates a registration result, forming a multi-strategy candidate set.

[0024] Preferably, each candidate registration result includes:

[0025] The matching degree of key features is calculated;

[0026] The lesion boundary coincidence degree is calculated;

[0027] The influence of tissue deformation is evaluated to form a confidence index, which provides a reference for optimal registration selection.

[0028] Preferably, the optimal registration result includes:

[0029] The optimal registration result is selected according to the candidate result confidence and the stability of the time sequence of consecutive frames;

[0030] The optimal registration result is selected and closed-loop correction is performed to update the optimal registration result.

[0031] Preferably, the selecting the optimal registration result comprises:

[0032] spatial registration accuracy;

[0033] temporal stability to ensure registration continuity.

[0034] Preferably, the navigation view comprises:

[0035] applying the optimal registration result to the three-dimensional model and superimposing it on the real-time arthroscopic video;

[0036] displaying lesion boundaries, key anatomical points and implant paths;

[0037] providing interactive operations, including view rotation and transparency adjustment.

[0038] Preferably, the cyclic execution comprises:

[0039] generating candidate registration results;

[0040] selecting the optimal registration result and performing closed-loop correction;

[0041] updating the navigation view so that each frame of video during the operation is updated for registration.

[0042] In a second aspect, the present application provides the following technical solution: an arthroscopic surgery real-time navigation system based on multi-modal image fusion, comprising the following modules:

[0043] An image acquisition module is used to acquire preoperative CT or MRI data of a patient to generate a three-dimensional model, and simultaneously acquire intraoperative arthroscopic real-time video to provide input data for subsequent registration;

[0044] An image preprocessing module is used to perform image preprocessing on the three-dimensional model and the arthroscopic video, including denoising, enhancement, and extraction of key anatomical features and lesion boundaries, to obtain processed results that can be used for registration;

[0045] A multi-strategy candidate registration module is used to generate candidate registration results based on the preprocessed results, simultaneously performing rigid registration, non-rigid registration and semantic key point matching, and calculating a confidence index for each candidate result to provide alternatives for optimal registration selection;

[0046] An adaptive optimal registration module is used to select the optimal registration result according to the candidate result confidence and temporal sequence stability of consecutive frames, and to perform closed-loop correction using consecutive frame error prediction to update the optimal result;

[0047] A navigation view generation module is configured to apply the optimal registration result to the three-dimensional model and superimpose the navigation view to the real-time arthroscopic video, and display the lesion boundary, key anatomical points and implant path for intraoperative navigation;

[0048] A closed-loop control and update module is configured to cyclically perform the candidate registration generation, optimal registration selection and navigation display steps during the surgery to ensure that each frame of video obtains an updated optimal registration result and forms a continuous closed loop.

[0049] The present application has the following beneficial effects:

[0050] 1、In the present application, by fusing the preoperative three-dimensional model with the intraoperative arthroscopic real-time video, and combining multi-strategy registration, closed-loop correction and dynamic updating, continuous registration and real-time navigation of multi-modal data are realized, thereby improving the problem that traditional arthroscopic surgery navigation mostly relies on single modal image, and the intraoperative registration accuracy is insufficient due to lack of multi-source information fusion.

[0051] 2、In the present application, by preprocessing the three-dimensional model and arthroscopic video and extracting key anatomical features and lesion boundaries, standardized data that can be used for registration are obtained, thereby improving the problem that traditional systems rely on manual recognition of feature points, and the subsequent registration is unstable due to non-standardized processing results.

[0052] 3、In the present application, by performing rigid registration, non-rigid registration and semantic key point matching in parallel to generate candidate registration results and calculate confidence indicators, a multi-strategy candidate set is formed to provide a reference for optimal registration selection, thereby improving the problem that traditional registration methods mostly use a single strategy, and the registration reliability is insufficient due to lack of processing of complex deformation and semantic features.

[0053] 4、In the present application, by cyclically performing candidate registration generation, optimal registration selection and navigation view update during the surgery, each frame of video obtains an updated optimal registration result and forms a closed loop, thereby improving the problem that traditional navigation systems mostly have registration result update lag in long-time surgery, and the navigation view continuity is poor due to inability to update in real time. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 The method flowchart of the arthroscopic surgery real-time navigation method based on multi-modal image fusion proposed in the present application;

[0055] Figure 2 The data preprocessing flowchart of the arthroscopic surgery real-time navigation method based on multi-modal image fusion proposed in the present application;

[0056] Figure 3A multi-strategy registration schematic diagram of the arthroscopic surgery real-time navigation method based on multi-modal image fusion provided by the present application;

[0057] Figure 4 An optimal registration selection flowchart of the arthroscopic surgery real-time navigation method based on multi-modal image fusion provided by the present application;

[0058] Figure 5 A navigation view generation diagram of the arthroscopic surgery real-time navigation method based on multi-modal image fusion provided by the present application;

[0059] Figure 6 A module architecture diagram of the arthroscopic surgery real-time navigation system based on multi-modal image fusion provided by the present application. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0061] Embodiment one:

[0062] In the first embodiment of the present application, the present application provides an arthroscopic surgery real-time navigation method based on multi-modal image fusion, as shown in Figures 1-5 , which comprises the following steps:

[0063] S1, collecting preoperative CT or MRI data of a patient to generate a three-dimensional model, and collecting intraoperative arthroscopic real-time video to provide input data for subsequent registration.

[0064] Specifically, when collecting preoperative image data of a patient, the system calls a medical image acquisition module to output original volume data from a CT scanner or an MRI scanner. The original volume data can be represented as a three-dimensional voxel matrix: ; wherein is a spatial coordinate index, respectively represent the resolution of the volume data in three dimensions. The volume data is sent as input into a three-dimensional reconstruction module to generate a three-dimensional model by interpolation and surface reconstruction algorithm , wherein ; the function represents a three-dimensional reconstruction process, and the output is a surface or volume rendering model corresponding to the preoperative anatomy of the patient.

[0065] Intraoperative video stream is collected through an arthroscope, and the video frame sequence can be represented as:

[0066] ; wherein is the two-dimensional pixel coordinate, is the image resolution, is the time series index, is the total frame number. Video data is input into the real-time image buffer module, and the output is a single arthroscopic image .

[0067] In step S1, two types of output data are obtained: three-dimensional model data and real-time video frame data . Both constitute the input of the subsequent registration stage, and the data flow relationship is:

[0068] ; where the left side is the original input data obtained by acquisition, and the right side is the processed output data.

[0069] Through the above data acquisition and modeling, a dual-input data basis of preoperative structured model and intraoperative real-time video frame is established, so that the subsequent registration module can simultaneously receive model and video data on a unified input interface.

[0070] S2, image preprocessing is performed on the three-dimensional model and arthroscopic video, including denoising, enhancement, and extraction of key anatomical features and lesion boundaries, to obtain processed results that can be used for registration.

[0071] Further, the arthroscopic video includes:

[0072] Adaptive filtering denoising is performed on the collected video;

[0073] Image enhancement is performed on the filtered video;

[0074] Key anatomical features of bones and soft tissues are extracted to provide input data for multi-strategy registration.

[0075] The three-dimensional model includes:

[0076] Surface mesh optimization is performed on the three-dimensional model;

[0077] Lesion boundaries and key anatomical points are labeled to provide an accurate model basis for candidate registration result generation.

[0078] Specifically, image preprocessing needs to be performed on the three-dimensional model and arthroscopic video respectively to ensure the effectiveness and stability of the input data in subsequent registration calculations.

[0079] The arthroscopic video part, input: the original arthroscopic video sequence obtained by acquisition, denoted as ; where represents the video frame. Processing steps: apply an adaptive filter to the original video sequence to reduce imaging noise. The filtered result can be represented as: ; where The filter kernel is adaptively adjusted according to the local variance and noise level. Image enhancement is performed on the basis of the filtered results, including histogram equalization or adaptive histogram equalization based on contrast limited (CLAHE). The enhanced video frame is represented as: ; wherein is the enhancement operator. Based on the enhanced video, key anatomical features and soft tissue boundaries are extracted. The output is a feature set: ; wherein represents the parametric representation of feature points or boundary curves. The output: the pre-processed video feature set is input to the multi-strategy registration.

[0080] The three-dimensional model part, input: the preoperatively obtained joint three-dimensional model, denoted as ; wherein are the vertex, edge and face sets respectively. The processing steps, the model is subjected to surface mesh optimization to reduce redundant triangular facets and improve surface smoothness. The optimized model is represented as: ; wherein is the mesh optimization operator. The lesion boundary and key anatomical points are labeled on the model, the boundary can be parameterized as: ; the key point set is: . The output: the mesh-optimized and labeled three-dimensional model data is used for the generation of candidate registration results.

[0081] Input-output relationship and role, input: joint video sequence , three-dimensional model . Output: video feature set , optimized model data . Technical role: the video preprocessing step reduces noise interference and enhances tissue boundaries, making the features extracted in a dynamic environment more stable. The three-dimensional model optimization reduces redundant geometric errors, ensures the accuracy of boundary and key point labeling, and improves the matching accuracy in the subsequent registration process. The output data is a unified feature and labeling result, which is directly input to the multi-strategy registration algorithm, ensuring that the subsequent calculation process has a clear data basis.

[0082] S3, based on the pre-processing results, simultaneously perform rigid registration, non-rigid registration and semantic key point matching to generate candidate registration results, and calculate confidence indicators for each candidate result to provide options for optimal registration selection.

[0083] Further, generating candidate registration results includes:

[0084] Simultaneously performing rigid registration to handle overall displacement and rotation;

[0085] Non-rigid registration is performed to cope with tissue deformation.

[0086] The registration result is generated based on semantic key point matching of deep learning, forming a multi-strategy candidate set.

[0087] Each candidate registration result includes:

[0088] Calculate the key feature matching degree;

[0089] Calculate the lesion boundary coincidence degree;

[0090] Evaluate the impact of tissue deformation and form a confidence index to provide a reference for the optimal registration selection.

[0091] Specifically, based on the pre-processing result obtained in step S2, a plurality of candidate registration results are generated, and a confidence index is calculated for each candidate result.

[0092] Input data, video feature set: ; three-dimensional model data: ; wherein is the three-dimensional model after grid optimization, is the lesion boundary set, is the key anatomical point set.

[0093] Candidate registration generation process, the generation of candidate registration results is completed by three strategies in parallel:

[0094] Rigid registration, target: handle overall translation and rotation. Calculate the rigid transformation matrix between the model point set and the video feature point set . The formula is: ; wherein is the video feature point, is the corresponding point on the model.

[0095] Non-rigid registration, target: handle the elastic deformation of joint soft tissue. The transformation model is set as ; wherein represents the non-rigid displacement field. The optimization function is: ; wherein represents the deformation smoothness constraint term, is the weight coefficient.

[0096] Semantic key point matching, based on deep learning network to extract semantic key point feature vectors from video and three-dimensional model: , match by nearest neighbor or similarity measure function: , output semantic matching relationship, form a candidate registration result set.

[0097] Index calculation of candidate results, for each candidate registration result , a confidence index needs to be calculated The index is composed of the following parts: key feature matching degree ; wherein is the feature matching correctness index.

[0098] Lesion boundary coincidence degree ; wherein is the set of lesion boundaries extracted from the video, is the set of three-dimensional model boundaries.

[0099] Deformation evaluation ; wherein represents the norm of the deformation displacement field, is the adjustment parameter. The final confidence index is: ; wherein is the weighting coefficient.

[0100] Output data and effects, output: candidate registration result set .

[0101] Effect: Multi-strategy candidate results provide registration solutions at different levels of rigidity, non-rigidity and semantic matching, avoiding the failure of a single method in complex scenes. The confidence index calculated for each candidate result provides quantitative basis for the selection of the optimal registration in the subsequent step. The output results form a registration candidate set and its evaluation index, laying a data foundation for the optimal solution selection in step S4.

[0102] S4, according to the confidence of the candidate result and the stability of the time sequence of consecutive frames, select the optimal registration result, and use the continuous frame error prediction for closed-loop correction to update the optimal result.

[0103] Further, the optimal registration result includes:

[0104] According to the confidence of the candidate result and the stability of the time sequence of consecutive frames, select the optimal registration result;

[0105] Use the continuous frame error sequence for closed-loop correction to update the optimal registration result for dynamic registration.

[0106] Selecting the optimal registration result includes:

[0107] Spatial registration accuracy;

[0108] Time sequence stability to ensure registration continuity.

[0109] Specifically, according to the candidate registration result set and its confidence index output by step S3, combined with the time sequence characteristics of consecutive frames, the optimal registration result is selected, and is constantly updated through the closed-loop correction mechanism to obtain a dynamically stable registration output.

[0110] Input data, candidate registration result set: ; wherein represents the th candidate registration result, represents its confidence. Time series data: ; wherein represents the registration error at the th frame.

[0111] Optimal registration selection, for each candidate result, construct a comprehensive evaluation function: ; wherein is the confidence of the candidate result; is the time series stability index; is the weighting coefficient. The time series stability index is defined as: ; wherein represents the registration error at the th frame, is the adjustment parameter. The final selection of the comprehensive score largest result as the optimal registration: .

[0112] Closed-loop correction mechanism, after selecting the optimal result, based on the continuous frame error sequence for prediction and correction. Let the error of the th frame be: ; wherein represents the model projection point set, represents the video feature point set, represents the point set error function. The error is modeled by a sliding window: ; wherein is the predicted next frame error, is the error prediction function. If , trigger correction, update the registration parameters: ; wherein represents the registration parameter adjustment amount according to the predicted error correction.

[0113] Output data and effect, output: optimal registration result sequence .

[0114] Effect: comprehensive confidence and time stability, ensure that the registration result selected in the candidate set has spatial accuracy and time continuity. Through the closed-loop correction mechanism, the error is constantly corrected, maintaining the stability and robustness of the dynamic registration result. The output result provides an accurate registration basis for the subsequent surgery navigation superimposition display.

[0115] S5, apply the optimal registration result to the three-dimensional model, and superimpose it to the real-time arthroscopic video to generate a navigation view, simultaneously display the lesion boundary, key anatomical points and implant path, for intraoperative navigation.

[0116] Further, the navigation view includes:

[0117] Apply the optimal registration result to the 3D model and overlay to the live arthroscopic video;

[0118] Display lesion boundary, key anatomical points and implant path;

[0119] Provide interactive operations, including view rotation and transparency adjustment.

[0120] Specifically, based on the optimal registration result output in step S4, the 3D model is overlaid with the live arthroscopic video to form a navigation view. The navigation view should include lesion boundary, key anatomical points and implant path, and provide necessary interactive functions.

[0121] Input data, optimal registration result , including the mapping relationship between the 3D model and the video coordinate system. The 3D model , wherein represents a three-dimensional point. The arthroscopic video frame , wherein represents a two-dimensional pixel point in the video. The lesion boundary set . The key anatomical point set . The implant path .

[0122] The 3D model is overlaid with the video, and the optimal registration result is used to project the 3D point to the video coordinate system: ; wherein is a projection function. The resulting projection point set is overlaid on the arthroscopic video frame to form an enhanced image.

[0123] Feature display, lesion boundary : Obtain the mapping of the boundary curve in the video coordinate system through the projection transformation: and draw the boundary in the navigation view. Key anatomical points : After projection, display a marker on the video for each point to identify the joint structure. Implant path : Project the path point set to form a trajectory segment, and display it as an implant guide reference in the navigation view.

[0124] Interactive operation, define an interaction matrix for rotation and transparency adjustment: ; wherein represents the adjusted 3D model point set, and the updated projection result replaces the display in the navigation view.

[0125] Output and action, output: navigation view , which is composed of video frames and superimposed three-dimensional models and feature information after projection: .

[0126] Action: By displaying the lesion boundary on the video screen, it provides positioning reference for the range of tissue lesions. By displaying key anatomical points on the video screen, it provides structural identification support. By displaying the implant path on the video screen, it provides intraoperative operation guidance. Through the interaction matrix, rotation and transparency adjustment are realized to meet the operation needs of the operator for different viewing angles and display levels.

[0127] S6, the steps of generating candidate registration, optimal registration selection and navigation display are cyclically executed during the operation process, ensuring that each frame of video obtains updated optimal registration results and forms a continuous closed loop.

[0128] Further, the cycle includes:

[0129] Generating candidate registration results;

[0130] Selecting the optimal registration result and performing closed loop correction;

[0131] Updating the navigation view so that each frame of video during the operation is updated.

[0132] Specifically, the entire operation process takes video frames as time sequence input, and the process of candidate registration generation, optimal registration selection and closed loop correction, and navigation view update is cyclically executed to form a dynamic closed loop.

[0133] Input data, real-time arthroscopic video frame stream: ; wherein is the frame video. Three-dimensional model data . The last frame optimal registration result .

[0134] Candidate registration result generation, for each frame , based on the preprocessing result, multi-strategy registration is performed: ; wherein represents the candidate registration parameter, represents the corresponding confidence.

[0135] Optimal registration selection and closed loop correction, by synthesizing the score function: , the optimal registration is selected: ; wherein is the time sequence stability, defined as: ; wherein is the error value of the past frames.

[0136] Navigation view updated, utilizing the updated optimal registration. Calculate the projection: and in video frames Overlay: ;in The boundary of the lesion, Key anatomical points, For implantation pathways.

[0137] Output and Function: Output: Time Series Navigation View .

[0138] Function: Generates updated optimal registration results for each frame of video. The registration results are temporally continuous, reducing abrupt changes caused by single-frame errors. The navigation view maintains closed-loop updates, ensuring continuous provision of navigation information for model-video fusion throughout the procedure.

[0139] Example 2:

[0140] In minimally invasive surgeries such as knee meniscus or ligament reconstruction, surgeons need to perform precise operations under arthroscopy, while simultaneously referring to 3D models generated by preoperative CT or MRI to plan the cutting and implantation paths. Traditional navigation systems mostly rely on single-modal images or manual reference to preoperative images, lacking a fusion and dynamic registration mechanism between preoperative 3D images and intraoperative real-time video. This makes it difficult to obtain continuous and accurate spatial positioning information during surgery, affecting the accuracy of lesion identification and implant path planning, and increasing surgical risks and operational difficulty. To solve these problems, this invention provides a real-time arthroscopic surgical navigation system based on multimodal image fusion, the structure of which is as follows: Figure 6 As shown. The specific implementation process of this system is as follows:

[0141] The image acquisition module is used to acquire preoperative CT or MRI data of patients to generate a three-dimensional model, and at the same time acquire real-time arthroscopic video during the operation to provide input data for subsequent registration.

[0142] The image preprocessing module is used to preprocess the 3D model and arthroscopic video, including denoising, enhancement, and extraction of key anatomical features and lesion boundaries to obtain processing results that can be used for registration.

[0143] The multi-strategy candidate registration module is used to generate candidate registration results by simultaneously performing rigid registration, non-rigid registration and semantic key point matching based on the preprocessing results, and calculate the confidence index for each candidate result to provide alternatives for the optimal registration selection.

[0144] An adaptive optimal registration module is configured to select an optimal registration result according to a candidate result confidence and a continuous frame time sequence stability, and to perform a closed-loop correction using a continuous frame error prediction to update the optimal result.

[0145] A navigation view generation module is configured to apply the optimal registration result to the three-dimensional model, and to superimpose the navigation view to the real-time arthroscopic video to generate a navigation view, while displaying a lesion boundary, a key anatomical point and an implant path for intraoperative navigation.

[0146] A closed-loop control and update module is configured to cyclically perform the steps of generating a candidate registration, selecting an optimal registration and displaying a navigation during a surgery to ensure that each frame of video obtains an updated optimal registration result and forms a continuous closed loop.

[0147] Specifically, the image acquisition module has an input of preoperative CT or MRI data and intraoperative arthroscopic video data of a patient, and an output of a three-dimensional model and a real-time video sequence. The image acquisition module provides a basis data source for subsequent registration processing, and ensures that the three-dimensional model and the intraoperative video are in a state of being synchronously processable.

[0148] The image preprocessing module has an input of the three-dimensional model and the arthroscopic video frame output by the image acquisition module, and an output of a processing result after denoising, enhancement and feature extraction. The image preprocessing module removes noise in the collected data, enhances tissue boundaries, and extracts a lesion boundary and key anatomical feature points, to provide input data directly usable for registration.

[0149] The multi-strategy candidate registration module has an input of the image preprocessing result, and an output of a plurality of candidate registration results and corresponding confidence indicators. The multi-strategy candidate registration module generates a plurality of feasible solutions by performing rigid registration, non-rigid registration and registration based on semantic key points in parallel, and calculates a confidence indicator for each candidate result to provide a candidate result set for subsequent screening.

[0150] The adaptive optimal registration module has an input of the candidate registration result set, and an output of an optimal registration result. The adaptive optimal registration module selects an optimal registration result according to a confidence of the candidate result and a time sequence stability of continuous frames, and eliminates accumulated errors through a closed-loop correction to ensure accuracy and continuity of the registration result in a dynamic process.

[0151] The navigation view generation module has an input of the optimal registration result and a real-time arthroscopic video frame, and an output of a navigation video stream superimposed with the three-dimensional model. The navigation view generation module maps the registered three-dimensional model to the arthroscopic video, and displays a lesion boundary, a key anatomical point and an implant path to provide intuitive image support for intraoperative navigation.

[0152] The input of the closed-loop control and update module is the current registration state and navigation result, and the output is the updated candidate registration result and navigation view. The module cyclically performs candidate registration generation, optimal registration selection and navigation view update during the surgery process, ensures that each frame of video is updated to the latest optimal registration result, and maintains the continuity and stability of the registration process.

[0153] Finally, it should be noted that: the above only for the preferred embodiments of the present application, and not for the purpose of limiting the present application, although the foregoing detailed description of the present application is made with reference to the foregoing embodiments, for those skilled in the art, it still can be modified, or part of the technical features of the equivalent replacement, within the spirit and principles of the present application, any modification, equivalent replacement, improvement, etc., should be included within the scope of the present application.

Claims

1. An arthroscopic surgery real-time navigation system based on multi-modal image fusion, characterized in that, The application relates to a navigation system for arthroscopic surgery, comprising the following modules: An image acquisition module is used for acquiring preoperative CT or MRI data of a patient to generate a three-dimensional model, and simultaneously acquiring arthroscopic real-time video to provide input data for subsequent registration; An image preprocessing module is used for pre-processing the three-dimensional model and the arthroscopic video to obtain processing results that can be used for registration; The preprocessing of the arthroscopic video comprises the following steps: performing adaptive filtering denoising on the acquired video, performing image enhancement on the filtered video, and extracting bone and soft tissue key anatomical features; The preprocessing of the three-dimensional model comprises the following steps: performing surface grid optimization on the three-dimensional model, and marking lesion boundaries and key anatomical points; A multi-strategy candidate registration module is used for simultaneously performing rigid registration, non-rigid registration and semantic key point matching based on the preprocessing results to generate candidate registration results, and calculating a confidence index for each candidate result to provide alternatives for optimal registration selection; The calculation of the confidence index comprises the following steps: calculating key feature matching degree, calculating lesion boundary coincidence degree, and evaluating tissue deformation influence to form the confidence index; An adaptive optimal registration module is used for selecting an optimal registration result according to the candidate result confidence and the stability of a continuous frame time sequence, and performing closed-loop correction by using continuous frame error prediction to update the optimal result; A navigation view generation module is used for applying the optimal registration result to the three-dimensional model, and superimposing the three-dimensional model to the real-time arthroscopic video to generate a navigation view, and simultaneously displaying lesion boundaries, key anatomical points and implant paths for intraoperative navigation; A closed-loop control and update module is used for cyclically performing the steps of generating candidate registration, optimal registration selection and navigation display during the surgery to ensure that each frame of video obtains an updated optimal registration result and forms a continuous closed loop.

Images