MRI image and arthroscope video data processing method and ACL damage diagnosis system

Abstract

The invention relates to the technical field of diagnosis software, and discloses a joint injury diagnosis system based on MRI (Magnetic Resonance Imaging) images and arthroscope video data. Comprising a data acquisition step, an MRI image segmentation step, an MRI three-dimensional reconstruction step, an arthroscope video feature extraction step, an arthroscope camera pose calculation step, an arthroscope three-dimensional reconstruction step, a multi-modal registration step and a fusion display processing step. Performing two-dimensional projection rendering on the MRI three-dimensional Gaussian scene model based on the visual angle parameters; and displaying the two-dimensional projection rendering result and the corresponding arthroscope video frame in an overlapping manner according to a layer mode. Through the combined use of the MRI image processing subsystem and the arthroscope video processing subsystem, the universality of the knee joint anterior cruciate ligament injury diagnosis system is improved, and the diagnosis time and the missed diagnosis possibility are reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of diagnostic software, in particular to an MRI image and arthroscopic video data processing method and an ACL injury diagnosis system. BACKGROUND

[0002] Anterior cruciate ligament (ACL) injury is one of the most common sports injuries of the knee joint, and its early and accurate diagnosis is crucial for developing a reasonable treatment plan and improving patient outcomes. Currently, the diagnosis of ACL injury mainly relies on clinical physical examination, magnetic resonance imaging (MRI) and arthroscopic examination. Among them, MRI, as a non-invasive examination method, can clearly show the morphology, signal changes and associated injuries of ACL, and is the main basis for preoperative diagnosis; while arthroscopic examination is considered as the "gold standard" for diagnosing ACL injury, which can directly observe the ligament condition, but it is a invasive operation. However, the existing diagnostic methods still have obvious limitations: MRI examination can provide three-dimensional anatomical information, but it has insufficient diagnostic specificity for partial tears and mucoid degeneration, and the interpretation of the results is easily affected by the experience level of the radiologists; arthroscopic examination has high accuracy, but its field of view is limited to the surface of the joint, making it difficult to assess internal ligament damage and bone marrow edema, and it cannot be routinely performed before surgery.

[0003] The existing technologies either focus on static intelligent diagnosis based on single modality images, or focus on surgical planning and navigation based on preoperative images, or focus on providing multi-modal fusion navigation for surgery itself. However, there is no technical solution to systematically solve the core challenge of accurately matching and fusing the three-dimensional anatomical structure information provided by preoperative high-resolution MRI with the two-dimensional dynamic surface and functional information provided by intraoperative / preoperative arthroscopic video, and to build an artificial intelligence model that can achieve early and accurate automatic diagnosis of ACL injury. Therefore, developing an ACL injury AI diagnosis model that can realize three-dimensional matching and multi-modal fusion learning of MRI and arthroscopic video data has important clinical significance and urgent application needs for addressing the shortcomings of existing diagnostic methods, bridging the diagnosis and treatment links, and improving the level of diagnosis and treatment integration. SUMMARY

[0004] The purpose of the present application is to provide a ligament injury diagnosis system based on knee MRI images and arthroscopic video images, to solve the problems of image interference and picture occlusion in the actual anterior cruciate ligament injury clinical diagnosis, as well as the problems of traditional arthroscopic narrow field of view, limited two-dimensional field of view, and easy to miss small lesions. A MRI image and arthroscopic video data processing method, comprising the following steps: S1, data acquisition step: Obtain knee MRI image data and corresponding arthroscopic video data of the same subject; S2, MRI image segmentation step: After standardizing preprocessing of the MRI image data, the processed MRI image data is input into a preset MRI image segmentation model for segmentation, and a corresponding segmentation result is output, obtaining segmentation data; S3, MRI three-dimensional reconstruction step: Based on the segmentation data obtained in step S2, corresponding three-dimensional spatial coordinate information is extracted, and the three-dimensional spatial coordinate information is represented as a plurality of three-dimensional Gaussian distributions containing center position parameters and covariance parameters; a plurality of three-dimensional Gaussian distributions are combined to form an MRI three-dimensional Gaussian scene model, and the MRI three-dimensional Gaussian scene model is stored in a preset three-dimensional coordinate system; S4, arthroscopic video feature extraction step: The continuous video frames in the arthroscopic video data are processed frame by frame, and in each video frame, a feature point detection algorithm is used to extract local feature points, and a corresponding feature description information is generated based on a binary feature descriptor. The feature points in adjacent video frames are matched, and a random sample consensus operation is performed on the matching result to obtain a filtered feature point matching relationship; S5, arthroscopic camera pose calculation step: Based on the feature point matching relationship obtained in step S4, a simultaneous localization and mapping operation is performed on the arthroscopic video data to calculate the arthroscopic camera pose parameters corresponding to each video frame, the camera pose parameters including position parameters and attitude parameters, and the camera pose parameters are recorded in time sequence of the video frames; S6, arthroscopic three-dimensional reconstruction step: Based on the initialized three-dimensional point information and the camera pose parameters obtained in step S5, the feature points in the arthroscopic video frames are back-projected to the three-dimensional space, and the three-dimensional Gaussian distribution parameters corresponding to the back-projected three-dimensional points are assigned; by iteratively updating the three-dimensional Gaussian distribution parameters, an arthroscopic three-dimensional Gaussian scene model is constructed, and the arthroscopic three-dimensional Gaussian scene model and the MRI three-dimensional Gaussian scene model are represented in the same data structure form; S7, multi-modal registration step: Two-dimensional rendering operations are respectively performed on the MRI three-dimensional Gaussian scene model and the arthroscopic three-dimensional Gaussian scene model to generate corresponding two-dimensional rendered images; the two-dimensional rendered images are input into a pre-trained cross-modal image matching model to obtain a pixel-level correspondence relationship between the two-dimensional rendered images; a three-dimensional spatial transformation matrix containing translation parameters, rotation parameters and scale parameters is calculated based on the pixel-level correspondence relationship, and the three-dimensional spatial transformation matrix is applied to at least one three-dimensional Gaussian scene model; S8, fusion display processing step: Obtain the view angle parameter corresponding to the current arthroscopic video frame, and perform two-dimensional projection rendering on the MRI three-dimensional Gaussian scene model processed in step S7 based on the view angle parameter; the two-dimensional projection rendering result is superimposed and displayed with the corresponding arthroscopic video frame in a layer manner, wherein the two-dimensional projection rendering result is displayed in a semi-transparent form, and visual identification information is attached to the pre-marked area.

[0005] The second aspect of the application provides an arthroscopic video data-based MRI image arthroscopic injury diagnosis system, comprising: a data processing server, wherein the data processing server is in communication connection with an MRI image database, an arthroscopic device and a display; The data processing server comprises: The data acquisition module is configured to: Obtain the knee MRI image data and the corresponding arthroscopic video data of the same subject; The MRI image segmentation module is configured to: After standardizing and preprocessing the MRI image data, input the processed MRI image data into a preset MRI image segmentation model for segmentation, and output the corresponding segmentation result; after obtaining the segmentation result, perform an up-sampling operation based on a diffusion model on the segmentation result to generate high-resolution segmentation data corresponding to the original segmentation result; The MRI three-dimensional reconstruction module is configured to: Based on the obtained segmentation data, extract the corresponding three-dimensional spatial coordinate information, and represent the three-dimensional spatial coordinate information as a plurality of three-dimensional Gaussian distributions containing center position parameters and covariance parameters; combine the plurality of three-dimensional Gaussian distributions to form an MRI three-dimensional Gaussian scene model, and store the MRI three-dimensional Gaussian scene model in a preset three-dimensional coordinate system; The arthroscopic video feature extraction module is configured to: Process the continuous video frames in the arthroscopic video data frame by frame, extract local feature points in each video frame using a feature point detection algorithm, and generate corresponding feature description information based on a binary feature descriptor; match the feature points in adjacent video frames, and perform a random sample consensus operation on the matching result to obtain the filtered feature point matching relationship; The arthroscopic camera pose calculation module is configured to: Based on the obtained feature point matching relationship, perform a simultaneous localization and mapping operation on the arthroscopic video data to calculate the arthroscopic camera pose parameters corresponding to each video frame, wherein the camera pose parameters include position parameters and attitude parameters, and are recorded in chronological order of the video frames; The arthroscopic three-dimensional reconstruction module is configured to: Based on the camera pose parameters and the initialized three-dimensional point information, the feature points in the arthroscopic video frame are back projected to the three-dimensional space, and the corresponding three-dimensional Gaussian distribution parameters are assigned to the back projected three-dimensional points; by iteratively updating the three-dimensional Gaussian distribution parameters, an arthroscopic three-dimensional Gaussian scene model is constructed, and the arthroscopic three-dimensional Gaussian scene model and the MRI three-dimensional Gaussian scene model are represented in a consistent data structure form; The multi-modal registration module is configured to: The MRI three-dimensional Gaussian scene model and the arthroscopic three-dimensional Gaussian scene model are respectively subjected to two-dimensional rendering operation to generate corresponding two-dimensional rendering images; the two-dimensional rendering images are input into a pre-trained cross-modal image matching model to obtain pixel-level correspondence between the two-dimensional rendering images; a three-dimensional space transformation matrix containing translation parameters, rotation parameters and scale parameters is calculated based on the pixel-level correspondence, and the three-dimensional space transformation matrix is applied to at least one three-dimensional Gaussian scene model; The fusion display processing module is configured to: Obtain the view angle parameters corresponding to the current arthroscopic video frame, and perform two-dimensional projection rendering on the MRI three-dimensional Gaussian scene model based on the view angle parameters; the two-dimensional projection rendering result and the corresponding arthroscopic video frame are superimposed and displayed in a layer manner, wherein the two-dimensional projection rendering result is displayed in a semi-transparent form, and visual identification information is attached to the pre-marked area.

[0006] The third aspect of the present application provides an electronic device, comprising: a memory and at least one processor, the memory having instructions stored therein, and the memory and the at least one processor being interconnected by a circuit; the at least one processor invokes the instructions in the memory to enable the electronic device to perform the MRI image and arthroscopic video data processing method as described above.

[0007] The fourth aspect of the present application provides a computer-readable storage medium having instructions stored therein, which, when executed on a computer, causes the computer to perform the MRI image and arthroscopic video data processing method as described above.

[0008] The present application has the following beneficial effects: The present application improves the versatility of the knee anterior cruciate ligament injury diagnosis system by combining the MRI image processing subsystem and the arthroscopic video processing subsystem, and reduces the diagnosis time and the possibility of missed diagnosis.

[0009] The present application introduces the innovative method of multi-modal three-dimensional registration technology, which reduces the problems of visual field limitation, subjective bias and incomplete diagnosis in actual clinical application. BRIEF DESCRIPTION OF DRAWINGS

[0010] Fig. 1 A structural diagram of an MRI image processing subsystem; Fig. 2 A structural diagram of an arthroscopic video processing subsystem; Fig. 3 A structural diagram of a multi-modal registration module. DETAILED DESCRIPTION

[0011] The terms "first", "second", "third", "fourth" and the like in the description and in the claims of the present application, if any, are used for distinguishing between similar objects talking about the exemplary embodiments and do not necessarily have to appear in the description or claims of the present application. It is to be understood that the data so designated are interchangeable under appropriate circumstances. The embodiments herein described can be implemented in other than the order shown and described above without departing from the scope of the present application. Additionally, the terms "comprise" or "have" and variations thereof in the description and in the claims of the present application are intended to cover both a complete process, method, system, product or apparatus that consists of the recited steps or units elements, and a partial process, method, system, product or apparatus that consists of the recited steps or unit elements.

[0012] For the purpose of facilitating understanding, the specific flow of the embodiments of the present application is described as follows. The first embodiment of the MRI image and arthroscopic video data processing method in the embodiments of the present application comprises: S1, a data acquisition step: Acquire knee MRI image data and corresponding arthroscopic video data of the same subject; S2, an MRI image segmentation step: After standardizing and preprocessing the MRI image data, input the processed MRI image data into a preset MRI image segmentation model for segmentation, output the corresponding segmentation result, and obtain segmentation data; Specifically comprising: adopting a deep learning-based convolutional neural network to perform segmentation processing on the MRI image data, the convolutional neural network being an improved form of the nnU-Net network structure; the network structure comprising an encoding path and a decoding path, wherein the encoding path is composed of multiple levels of down-sampling layers, each level of down-sampling layer comprising a convolution operation unit and a nonlinear activation unit, for performing step-by-step feature encoding on the input MRI image; the decoding path is composed of multiple levels of up-sampling layers corresponding to the levels of the encoding path, each up-sampling layer receiving a feature map from the corresponding level of the encoding path through a jump connection and performing splicing processing; in the MRI image segmentation step, the standardized MRI image data is taken as the network input, and the pixel-level or voxel-level segmentation label is taken as the network output; after obtaining the segmentation output, a diffusion model-based up-sampling operation is performed on the segmentation output, wherein the diffusion model takes the segmentation result as the initial state, generates segmentation data with higher resolution than the initial segmentation result through multiple steps of reverse diffusion operation, and the high-resolution segmentation data is taken as the input data of the subsequent three-dimensional reconstruction step.

[0013] S3, MRI three-dimensional reconstruction step: Based on the segmentation data obtained in step S2, the corresponding three-dimensional spatial coordinate information is extracted, and the three-dimensional spatial coordinate information is represented as a plurality of three-dimensional Gaussian distributions containing center position parameters and covariance parameters; a plurality of three-dimensional Gaussian distributions are combined to form an MRI three-dimensional Gaussian scene model, and the MRI three-dimensional Gaussian scene model is stored in a preset three-dimensional coordinate system; Specifically comprising: based on the segmentation data output in step S2, performing three-dimensional coordinate mapping on the spatial position corresponding to each segmentation voxel or pixel to obtain a three-dimensional spatial coordinate set; representing each spatial point in the three-dimensional spatial coordinate set as a three-dimensional Gaussian distribution containing a center position parameter and a covariance matrix parameter; combining a plurality of three-dimensional Gaussian distributions to form an MRI three-dimensional Gaussian scene model; during the construction of the MRI three-dimensional Gaussian scene model, joint optimization operations are performed on the spatial position parameters, covariance parameters and weight parameters of the three-dimensional Gaussian distribution; the MRI three-dimensional Gaussian scene model is stored in a unified three-dimensional coordinate description manner and is configured to support two-dimensional rendering operation based on the view angle parameter, to serve as the input data of the MRI side in the multi-modal registration step.

[0014] S4, arthroscopic video feature extraction step: The continuous video frames in the arthroscopic video data are processed frame by frame, local feature points are extracted in each video frame by using a feature point detection algorithm, and corresponding feature description information is generated for the feature points based on a binary feature descriptor; the feature points in adjacent video frames are matched, and a random sample consensus operation is performed on the matching result to obtain a filtered feature point matching relationship; Specifically comprising: performing feature point detection processing on each video frame in the arthroscopic video data, the feature point detection processing adopts a FAST corner detection algorithm to extract local feature point coordinates in the video frame; for the feature point coordinates, a BRIEF binary descriptor is used to generate a corresponding feature description vector; performing matching operation on the feature description vectors in adjacent video frames to obtain an initial feature point matching set; performing a random sample consensus operation on the initial feature point matching set to eliminate matching pairs that do not satisfy the geometric consistency constraint, to obtain a filtered feature point matching relationship; the feature point matching relationship is stored in association with the video frame index in chronological order, and serves as input data for the subsequent camera pose calculation step.

[0015] S5, arthroscopic camera pose calculation step: Based on the feature point matching relationship obtained in step S4, performing simultaneous localization and mapping operation on the arthroscopic video data to calculate the arthroscopic camera pose parameters corresponding to each video frame, the camera pose parameters including position parameters and attitude parameters, and recording in chronological order of the video frames; S6, arthroscopic three-dimensional reconstruction step: Based on the initialized three-dimensional point information and the camera pose parameters obtained in step S5, the feature points in the arthroscopic video frames are back-projected to the three-dimensional space, and the corresponding three-dimensional Gaussian distribution parameters are assigned to the back-projected three-dimensional points; by iteratively updating the three-dimensional Gaussian distribution parameters, an arthroscopic three-dimensional Gaussian scene model is constructed, which is represented in the same data structure form as the MRI three-dimensional Gaussian scene model; Specifically comprising: based on the arthroscopic camera pose parameters calculated in step S5, the feature points extracted and matched in step S4 are back-projected to the three-dimensional space according to the corresponding video frame index to generate an initialized three-dimensional point set; for each three-dimensional point in the initialized three-dimensional point set, a corresponding three-dimensional Gaussian distribution parameter is assigned, including a center position parameter and a covariance parameter; by performing multiple rounds of iterative update operation on the three-dimensional Gaussian distribution parameters, an arthroscopic three-dimensional Gaussian scene model is constructed; the arthroscopic three-dimensional Gaussian scene model and the MRI three-dimensional Gaussian scene model are represented in the same data organization form and parameter description method, and stored in the same three-dimensional coordinate description framework.

[0016] S7, multi-modal registration step: performing a two-dimensional rendering operation on the MRI three-dimensional Gaussian scene model and the arthroscopic three-dimensional Gaussian scene model respectively to generate corresponding two-dimensional rendering images; inputting the two-dimensional rendering images into a pre-trained cross-modal image matching model to obtain a pixel-level correspondence relationship between the two-dimensional rendering images; calculating a three-dimensional space transformation matrix containing translation parameters, rotation parameters and scale parameters based on the pixel-level correspondence relationship, and applying the three-dimensional space transformation matrix to at least one of the three-dimensional Gaussian scene models; Specifically, it comprises: performing a two-dimensional rendering operation on an MRI three-dimensional Gaussian scene model and an arthroscopic three-dimensional Gaussian scene model respectively to generate a corresponding two-dimensional rendering image sequence; inputting the two-dimensional rendering image sequence into a pre-trained cross-modal image matching model, the cross-modal image matching model comprising a convolutional neural network encoder for extracting local features, a visual Transformer encoder for extracting global features, and a feature decoder for predicting image correspondence; based on the two-dimensional pixel-level correspondence relationship output by the cross-modal image matching model, a three-dimensional space transformation matrix containing translation parameters, rotation parameters and scale parameters is calculated; and the three-dimensional space transformation matrix is applied to at least one of the MRI three-dimensional Gaussian scene model or the arthroscopic three-dimensional Gaussian scene model to complete the unification of the three-dimensional space coordinates.

[0017] S8, fusion display processing step: Obtain the view angle parameters corresponding to the current arthroscopic video frame, and perform two-dimensional projection rendering on the MRI three-dimensional Gaussian scene model processed in step S7 based on the view angle parameters; the two-dimensional projection rendering result and the corresponding arthroscopic video frame are superimposed and displayed in a layer manner, wherein the two-dimensional projection rendering result is displayed in a semi-transparent form, and visual identification information is added to the pre-marked area.

[0018] Specifically, it comprises: obtaining the camera view angle parameters corresponding to the current arthroscopic video frame, and performing a two-dimensional projection rendering operation on the MRI three-dimensional Gaussian scene model after multi-modal registration based on the camera view angle parameters to generate a corresponding projection image; the projection image and the arthroscopic video frame are superimposed and displayed in a layer manner, wherein the projection image is set to a semi-transparent display state; visual identification information is added to the pre-marked area of the MRI three-dimensional Gaussian scene model, and the visual identification information includes color attributes, boundary line attributes or shape attributes; the fusion display processing step synchronously updates the display content in the time sequence of the arthroscopic video frame.

[0019] See Figs. 1-3 The joint injury diagnosis system based on MRI images and arthroscopic video data in the embodiment of the application is described as follows, comprising: A data processing server is connected in communication with an MRI image database, an arthroscopic device, and a display; The data processing server comprises: A data acquisition module configured to: acquire knee MRI image data and corresponding arthroscopic video data of the same subject; An MRI image segmentation module configured to: after standardizing and preprocessing the MRI image data, input the processed MRI image data into a preset MRI image segmentation model for segmentation, output a corresponding segmentation result; after obtaining the segmentation result, perform an up-sampling operation based on a diffusion model on the segmentation result to generate high-resolution segmentation data corresponding to the original segmentation result; Specifically, the segmentation module is configured to automatically segment the knee MRI image data of a patient. The module uses an improved nnU-Net (U-shaped neural network) segmentation network, which includes an encoder, a decoder, and a skip connection structure. The encoder is composed of 5 down-sampling layers, each layer including two 3x3 convolution kernel convolution layers and a ReLU activation function, followed by a 2x2 max-pooling layer; the decoder is composed of 5 up-sampling layers, each layer including a transposed convolution layer and a feature fusion layer; the skip connection splices the low-level features of the encoder with the corresponding layer features of the decoder. The network input is a DICOM (Digital Imaging and Communications in Medicine) format image of the knee MRI, and the output is a pixel-level segmentation result of cartilage, meniscus, ligament, and other tissue structures. After segmentation, the diffusion model is used to up-sample the segmentation result, increase the image resolution from 512x512 to 1024x1024, enhance the spatial resolution, and obtain a high-precision segmentation result.

[0020] An MRI three-dimensional reconstruction module configured to: based on the obtained segmentation data, extract corresponding three-dimensional spatial coordinate information, and represent the three-dimensional spatial coordinate information as a plurality of three-dimensional Gaussian distributions containing center position parameters and covariance parameters; combine the plurality of three-dimensional Gaussian distributions to form an MRI three-dimensional Gaussian scene model, and store the MRI three-dimensional Gaussian scene model in a preset three-dimensional coordinate system; Specifically, the MRI reconstruction module is connected to the segmentation module and is configured to construct a three-dimensional model based on the segmentation result. The module first extracts a three-dimensional point cloud from the high-resolution segmentation result; then uses the 3D Gaussian Splatting technology to reconstruct the MRI three-dimensional Gaussian scene model of the knee joint, which completely retains the spatial position information of the joint anatomical details and suspected lesions.

[0021] An arthroscopic video feature extraction module is configured to: Process the arthroscopic video data frame by frame, extract local feature points in each video frame using a feature point detection algorithm, and generate corresponding feature description information based on binary feature descriptors for the feature points; match the feature points in adjacent video frames, and perform a random sample consensus operation on the matching results to obtain filtered feature point matching relationships; Specifically, the feature extraction module is configured to process real-time video streams in arthroscopic surgery. This module uses the FAST (Features from Accelerated Segment Test) algorithm and the BRIEF (Binary Robust Independent Elementary Features) descriptor to extract and match feature points in video frames; through the SLAM (Simultaneous Localization and Mapping) technology and the RANSAC (Random Sample Consensus) optimization algorithm, the camera pose matrix between adjacent frames is calculated, and a complete camera motion trajectory sequence is constructed.

[0022] An arthroscopic camera pose calculation module is configured to: Based on the obtained feature point matching relationships, perform a simultaneous localization and mapping operation on the arthroscopic video data, calculate the corresponding arthroscopic camera pose parameters of each video frame, the camera pose parameters include position parameters and attitude parameters, and record them in the time order of the video frames; An arthroscopic three-dimensional reconstruction module is configured to: Based on the camera pose parameters and the initialized three-dimensional point information, the feature points in the arthroscopic video frames are back-projected to the three-dimensional space, and the corresponding three-dimensional Gaussian distribution parameters are assigned to the back-projected three-dimensional points; by iteratively updating the three-dimensional Gaussian distribution parameters, an arthroscopic three-dimensional Gaussian scene model is constructed, and the arthroscopic three-dimensional Gaussian scene model and the MRI three-dimensional Gaussian scene model are represented in the same data structure form; Connected with the feature extraction module, used for constructing a three-dimensional scene model in the joint cavity. This module is based on the obtained camera pose and the initialized 3D point cloud; uses 3D Gaussian splashing technology for three-dimensional scene reconstruction and optimization to obtain a 3D Gaussian scene model of the arthroscope, which accurately reflects the real-time anatomical structure in the joint cavity.

[0023] A multi-modal registration module is configured to: performing a two-dimensional rendering operation on the MRI three-dimensional Gaussian scene model and the arthroscopic three-dimensional Gaussian scene model respectively to generate corresponding two-dimensional rendering images; inputting the two-dimensional rendering images into a pre-trained cross-modal image matching model to obtain a pixel-level correspondence relationship between the two-dimensional rendering images; calculating a three-dimensional space transformation matrix containing translation parameters, rotation parameters and scale parameters based on the pixel-level correspondence relationship, and applying the three-dimensional space transformation matrix to at least one three-dimensional Gaussian scene model; Specifically, the multi-modal registration module is used to align the coordinates of the MRI three-dimensional Gaussian scene model and the arthroscopic three-dimensional Gaussian scene model. This module is based on the MatchAnything pre-trained cross-modal image matching large model, which contains a CNN Fine Encoder (convolutional neural network fine encoder), a DINO Coarse Encoder (self-supervised visual Transformer coarse encoder) and a Transformer Flow Decoder (Transformer flow decoder). The registration process is as follows: the MRI scene and the arthroscopic scene are respectively rendered into two-dimensional images with a resolution of 1024x1024, and are input into the MatchAnything model; the arthroscopic image is extracted by the CNN Fine Encoder to extract detailed features; the MRI image is extracted by the DINO Coarse Encoder and the Transformer Flow Decoder to extract global features; the model outputs the pixel-level matching results between the two-dimensional images, and the matching accuracy reaches the sub-pixel level; through iterative optimization of the transformation matrix composed of scale, position translation and rotation parameters, the two 3D Gaussian scenes are geometrically aligned in the world coordinate system, so that an accurate image pair can be generated under any viewing angle.

[0024] The fusion display processing module is configured to: Obtain the viewing angle parameters corresponding to the current arthroscopic video frame, and perform two-dimensional projection rendering on the MRI three-dimensional Gaussian scene model based on the viewing angle parameters; superimpose and display the two-dimensional projection rendering result and the corresponding arthroscopic video frame in a layer manner, wherein the two-dimensional projection rendering result is displayed in a semi-transparent form, and the pre-marked area is additionally marked with visual identification information.

[0025] Specifically, the fusion display processing module is connected with the registration module, and is used for presenting the registration result to the doctor in real time. The module uses image layer display technology to superimpose the MRI three-dimensional Gaussian rendering image (in which the lesion position detected by the model is marked with a red highlight contour) after coordinate alignment on the original arthroscopic video screen to form a fusion diagnosis image. The superimposed image is output to the operating room display through HDMI (high-definition multimedia interface) or DisplayPort (DisplayPort) to intuitively provide the doctor with diagnosis information such as the position and range of the lesion.

[0026] The present patent seamlessly integrates the rich lesion diagnosis information of preoperative MRI into the dynamic field of view of intraoperative arthroscopic video by respectively processing, three-dimensional reconstruction and accurate registration of MRI images and arthroscopic video data, so that the doctor can directly "see" the projection of deep or small lesions in arthroscopic operation, thereby greatly improving the comprehensiveness and accuracy of joint injury diagnosis and reducing the risk of missed diagnosis and excessive dependence on the experience of the doctor. At the same time, the system has clear modular design and strong universality, and is suitable for various joint injury diagnosis scenarios.

[0027] The electronic device can be quite different in configuration or performance, and can include one or more processors (central processing units, CPUs) (for example, one or more processors) and memories, one or more storage media (for example, one or more mass storage devices) that store applications or data. The memories and the storage media can be temporary storage or persistent storage. The programs stored in the storage medium can include one or more modules, and each module can include a series of instruction operations in the electronic device. Further, the processor can be configured to communicate with the storage medium and execute the series of instruction operations in the storage medium on the electronic device.

[0028] The electronic device can also include one or more power supplies, one or more wired or wireless network interfaces, one or more input / output interfaces, and / or one or more operating systems, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art can understand that the structure of the electronic device does not constitute a limitation based on the electronic device, and can include more or fewer components than the foregoing, or combine certain components, or different component arrangements.

[0029] The application further provides a computer readable storage medium, which can be a nonvolatile computer readable storage medium or a volatile computer readable storage medium, and the computer readable storage medium stores instructions, and the instructions make a computer execute the steps of the foregoing method when the instructions are run on the computer.

[0030] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the system or device, unit described above can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0031] The integrated unit, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the application essentially or the part of the prior art that makes a contribution or the whole or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0032] The foregoing and the foregoing embodiments are only used to illustrate the technical solutions of the application, rather than limit the same; although the application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application.

Claims

1. A method for processing MRI images and arthroscopic video data, characterized in that, include: S1. Data Acquisition Steps: Acquire knee MRI images and corresponding arthroscopic video data of the same subject; S2, MRI image segmentation steps: After standardizing and preprocessing the MRI image data, the processed MRI image data is input into a preset MRI image segmentation model for segmentation, and the corresponding segmentation results are output to obtain segmentation data. S3, MRI 3D Reconstruction Steps: Based on the segmented data obtained in step S2, the corresponding three-dimensional spatial coordinate information is extracted, and the three-dimensional spatial coordinate information is represented as multiple three-dimensional Gaussian distributions containing center position parameters and covariance parameters; the multiple three-dimensional Gaussian distributions are combined to form an MRI three-dimensional Gaussian scene model, and the MRI three-dimensional Gaussian scene model is stored in a preset three-dimensional coordinate system; S4. Arthroscopic video feature extraction steps: The continuous video frames in the arthroscopic video data are processed frame by frame. In each video frame, a feature point detection algorithm is used to extract local feature points, and corresponding feature description information is generated for the feature points based on binary feature descriptors. Feature points in adjacent video frames are matched, and random sampling consistency operation is performed on the matching results to obtain the filtered feature point matching relationship. S5. Arthroscopic camera pose calculation steps: Based on the feature point matching relationship obtained in step S4, synchronous localization and mapping operations are performed on the arthroscopic video data to calculate the arthroscopic camera pose parameters corresponding to each video frame. The camera pose parameters include position parameters and attitude parameters, and are recorded in the time order of the video frames. S6. Arthroscopic 3D Reconstruction Steps: Based on the initialized 3D point information and the camera pose parameters obtained in step S5, the feature points in the arthroscopic video frame are back-projected into 3D space, and corresponding 3D Gaussian distribution parameters are assigned to the back-projected 3D points. By iteratively updating the 3D Gaussian distribution parameters, an arthroscopic 3D Gaussian scene model is constructed. The arthroscopic 3D Gaussian scene model and the MRI 3D Gaussian scene model are represented using the same data structure. S7. Multimodal registration steps: Two-dimensional rendering operations are performed on the MRI three-dimensional Gaussian scene model and the arthroscopic three-dimensional Gaussian scene model respectively to generate corresponding two-dimensional rendered images; the two-dimensional rendered images are input into a pre-trained cross-modal image matching model to obtain the pixel-level correspondence between the two-dimensional rendered images; Based on the pixel-level correspondence, a three-dimensional spatial transformation matrix containing translation parameters, rotation parameters, and scale parameters is calculated, and the three-dimensional spatial transformation matrix is ​​applied to at least one three-dimensional Gaussian scene model; S8. Fusion display processing steps: Obtain the viewpoint parameters corresponding to the current arthroscopic video frame, and perform two-dimensional projection rendering on the MRI three-dimensional Gaussian scene model processed in step S7 based on the viewpoint parameters; overlay the two-dimensional projection rendering result with the corresponding arthroscopic video frame in a layer manner, wherein the two-dimensional projection rendering result is displayed in a semi-transparent form, and visual identification information is added to the pre-marked areas.

2. The method for processing MRI images and arthroscopic video data according to claim 1, characterized in that: The MRI image segmentation step in step S2 specifically includes: A deep learning-based convolutional neural network (CNN) is used to segment MRI image data. The CNN is an improved version of the nnU-Net network structure. The network structure includes an encoding path and a decoding path. The encoding path consists of multiple downsampling layers, each including a convolutional operation unit and a nonlinear activation unit, used for progressive feature encoding of the input MRI image. The decoding path consists of multiple upsampling layers corresponding to the encoding path layers. Each upsampling layer receives feature maps from the corresponding layers of the encoding path via skip connections and performs concatenation. In the MRI image segmentation step, standardized MRI image data is used as network input, and pixel-level or voxel-level segmentation labels are used as network output. After obtaining the segmentation output, an upsampling operation based on a diffusion model is performed on the segmentation output. The diffusion model uses the segmentation result as the initial state and generates segmentation data with a resolution higher than the initial segmentation result through multiple back-diffusion operations. This high-resolution segmentation data serves as the input data for subsequent 3D reconstruction steps.

3. The method for processing MRI images and arthroscopic video data according to claim 1, characterized in that: The MRI three-dimensional reconstruction step described in step S3 specifically includes: Based on the segmentation data output in step S2, three-dimensional coordinate mapping is performed on the spatial location corresponding to each segmented voxel or pixel to obtain a set of three-dimensional spatial coordinates. Each spatial point in the set of three-dimensional spatial coordinates is represented as a three-dimensional Gaussian distribution containing a center position parameter and a covariance matrix parameter. Multiple three-dimensional Gaussian distributions are combined to form an MRI three-dimensional Gaussian scene model. During the construction of the MRI three-dimensional Gaussian scene model, joint optimization operations are performed on the spatial position parameter, covariance parameter, and weight parameter of the three-dimensional Gaussian distribution. The MRI three-dimensional Gaussian scene model is stored in a unified three-dimensional coordinate description method and configured to support two-dimensional rendering operations based on viewpoint parameters, as input data for the MRI side in the multimodal registration step.

4. The method for processing MRI images and arthroscopic video data according to claim 1, characterized in that: The arthroscopic video feature extraction step in step S4 specifically includes: Feature point detection processing is performed on each video frame in the arthroscopic video data. The feature point detection processing adopts the FAST corner detection algorithm to extract local feature point coordinates in the video frame. For the feature point coordinates, the BRIEF binary descriptor is used to generate the corresponding feature description vector. The feature description vectors in adjacent video frames are matched to obtain an initial feature point matching set. The initial feature point matching set is subjected to a random sampling consistency operation to remove matching pairs that do not meet the geometric consistency constraint, and the filtered feature point matching relationship is obtained. The feature point matching relationship is stored in chronological order with the video frame index and is used as input data for subsequent camera pose calculation steps.

5. The method for processing MRI images and arthroscopic video data according to claim 1, characterized in that: The arthroscopic three-dimensional reconstruction step described in step S6 specifically includes: Based on the arthroscopic camera pose parameters calculated in step S5, the feature points extracted and matched in step S4 are back-projected into three-dimensional space according to the corresponding video frame index to generate an initial three-dimensional point set; each three-dimensional point in the initial three-dimensional point set is assigned a corresponding three-dimensional Gaussian distribution parameter, which includes a center position parameter and a covariance parameter; by performing multiple rounds of iterative update operations on the three-dimensional Gaussian distribution parameter, an arthroscopic three-dimensional Gaussian scene model is constructed; the arthroscopic three-dimensional Gaussian scene model and the MRI three-dimensional Gaussian scene model are represented using the same data organization form and parameter description method, and stored under the same three-dimensional coordinate description framework.

6. The method for processing MRI images and arthroscopic video data according to claim 1, characterized in that: The multimodal registration step in step S7 specifically includes: Two-dimensional rendering operations are performed on the MRI 3D Gaussian scene model and the arthroscopic 3D Gaussian scene model respectively to generate corresponding 2D rendered image sequences. The 2D rendered image sequences are input into a pre-trained cross-modal image matching model, which includes a convolutional neural network encoder for extracting local features, a visual Transformer encoder for extracting global features, and a feature decoder for predicting image correspondences. Based on the 2D pixel-level correspondences output by the cross-modal image matching model, a 3D spatial transformation matrix including translation parameters, rotation parameters, and scale parameters is calculated. The 3D spatial transformation matrix is ​​applied to at least one of the MRI 3D Gaussian scene model or the arthroscopic 3D Gaussian scene model to achieve the unification of 3D spatial coordinates.

7. The method for processing MRI images and arthroscopic video data according to claim 1, characterized in that: The fusion display processing step described in step S8 specifically includes: The camera viewpoint parameters corresponding to the current arthroscopic video frame are obtained, and a two-dimensional projection rendering operation is performed on the MRI three-dimensional Gaussian scene model after multimodal registration based on the camera viewpoint parameters to generate a corresponding projection image; the projection image and the arthroscopic video frame are overlaid and displayed in a layer manner, wherein the projection image is set to a semi-transparent display state; visual identification information is added to the pre-marked areas in the MRI three-dimensional Gaussian scene model, and the visual identification information includes color attributes, boundary line attributes, or shape attributes; the fusion display processing step synchronously updates the display content according to the time sequence of the arthroscopic video frames.

8. An ACL diagnostic system based on MRI images and arthroscopic video data, characterized in that, It includes a data processing server, which is communicatively connected to an MRI image database, arthroscopic equipment, and a display. The data processing server includes: The data acquisition module is configured as follows: Acquire knee MRI images and corresponding arthroscopic video data of the same subject; The MRI image segmentation module is configured as follows: After standardizing and preprocessing the MRI image data, the processed MRI image data is input into a preset MRI image segmentation model for segmentation, and the corresponding segmentation results are output. After obtaining the segmentation results, an upsampling operation based on a diffusion model is performed on the segmentation results to generate high-resolution segmentation data corresponding to the original segmentation results. The MRI 3D reconstruction module is configured as follows: Based on the obtained segmentation data, the corresponding three-dimensional spatial coordinate information is extracted, and the three-dimensional spatial coordinate information is represented as multiple three-dimensional Gaussian distributions containing center position parameters and covariance parameters; the multiple three-dimensional Gaussian distributions are combined to form an MRI three-dimensional Gaussian scene model, and the MRI three-dimensional Gaussian scene model is stored in a preset three-dimensional coordinate system; The arthroscopic video feature extraction module is configured as follows: The continuous video frames in the arthroscopic video data are processed frame by frame. In each video frame, a feature point detection algorithm is used to extract local feature points, and corresponding feature description information is generated for the feature points based on binary feature descriptors. Feature points in adjacent video frames are matched, and random sampling consistency operation is performed on the matching results to obtain the filtered feature point matching relationship. The arthroscopic camera pose calculation module is configured as follows: Based on the obtained feature point matching relationship, synchronous localization and mapping operations are performed on the arthroscopic video data to calculate the arthroscopic camera pose parameters corresponding to each video frame. The camera pose parameters include position parameters and attitude parameters, and are recorded in the time order of the video frames. The arthroscopic 3D reconstruction module is configured as follows: Based on the camera pose parameters and the initialized 3D point information, the feature points in the arthroscopic video frame are back-projected into 3D space, and corresponding 3D Gaussian distribution parameters are assigned to the back-projected 3D points. By iteratively updating the 3D Gaussian distribution parameters, an arthroscopic 3D Gaussian scene model is constructed. The arthroscopic 3D Gaussian scene model and the MRI 3D Gaussian scene model are represented using the same data structure. The multimodal registration module is configured as follows: Two-dimensional rendering operations are performed on the MRI three-dimensional Gaussian scene model and the arthroscopic three-dimensional Gaussian scene model respectively to generate corresponding two-dimensional rendered images; the two-dimensional rendered images are input into a pre-trained cross-modal image matching model to obtain the pixel-level correspondence between the two-dimensional rendered images; a three-dimensional spatial transformation matrix containing translation parameters, rotation parameters and scale parameters is calculated based on the pixel-level correspondence, and the three-dimensional spatial transformation matrix is ​​applied to at least one three-dimensional Gaussian scene model; The integrated display processing module is configured as follows: Obtain the viewpoint parameters corresponding to the current arthroscopic video frame, and perform two-dimensional projection rendering on the MRI three-dimensional Gaussian scene model based on the viewpoint parameters; overlay the two-dimensional projection rendering result with the corresponding arthroscopic video frame in a layer manner, wherein the two-dimensional projection rendering result is displayed in a semi-transparent form, and visual identification information is added to the pre-marked areas.

9. An electronic device comprising a memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the electronic device to perform the steps of the data processing method as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the steps of the data processing method as described in any one of claims 1-7.

Images