Three-dimensional organ reconstruction registration fusion method based on multi-modal medical image

Abstract

The invention provides a three-dimensional organ reconstruction registration fusion method based on a multi-modal medical image, and belongs to the technical field of medical images. Comprising the following steps: S1, collecting a multi-modal medical image of an ith organ of a target patient in real time, and constructing a preoperative data set; s2, after the first data set is constructed, continuously constructing an intra-operative data set through the intra-operative ultrasonic image; s3, constructing an AI model by using a convolutional neural network; s4, based on the preoperative data set and the intraoperative data set, constructing a stability coefficient and a crowding degree coefficient of the ith organ; and S5, associating the stability coefficient of the ith organ with the crowding degree coefficient of the ith organ, and constructing a comprehensive dynamic risk coefficient. According to the method, CT, MRI and intraoperative ultrasonic images are collected at the same time, the preoperative data set is constructed, the organ volume change rate and the body position adjustment displacement are updated in real time in the operation, the optimal operation path can be automatically predicted, and the subjectivity of depending on doctor experience in traditional operation path planning is reduced.

Description

Technical Field

[0001] This invention relates to the field of medical imaging technology, and more specifically, to a three-dimensional organ reconstruction, registration, and fusion method based on multimodal medical images. Background Technology

[0002] With the rapid development of minimally invasive surgery, accurate understanding of the position, shape, and spatial relationship of the patient's organs during surgery has become a key factor in improving surgical safety and precision. In complex surgical areas such as the abdomen, different organs have closely intertwined structures and significant dynamic changes. Affected by various factors such as respiratory movements, organ traction, negative pressure suction, surgical instrument intervention, and changes in patient position, organs will undergo real-time volume changes, spatial displacement, and local deformation. This may cause deviations between the pre-planned surgical path and the actual organ condition, increasing operational risks. For example, in hepatobiliary surgery and pancreatic surgery, even slight displacement of adjacent organs may lead to insufficient safe distance between the surgical instruments and the target structure, resulting in intraoperative bleeding and accidental injury to adjacent organs.

[0003] Currently, preoperative 3D models are generally generated from static images such as CT and MRI for preoperative planning and pathway determination. However, these images only reflect the static state of the patient at a certain moment and cannot characterize the dynamic behavior of organs changing over time during surgery. In actual surgical scenarios, it is difficult to keep up with the rapid changes of organs. Therefore, a 3D organ reconstruction registration and fusion method based on multimodal medical images is proposed to solve this problem. Summary of the Invention

[0004] To overcome the above deficiencies, the present invention provides a method for three-dimensional organ reconstruction, registration and fusion based on multimodal medical images that overcomes or at least partially solves the above technical problems.

[0005] This invention is implemented as follows:

[0006] This invention provides a three-dimensional organ reconstruction registration and fusion method based on multimodal medical images, comprising:

[0007] S1. Real-time acquisition of multimodal medical images of the i-th organ of the target patient, including CT images and MRI images, and extraction of the volume of the i-th organ. Three-dimensional spatial coordinates Distance to the adjacent organ of the (i+1)th organ and local displacement vector Construct a preoperative dataset;

[0008] S2. After constructing the first dataset, continue to collect dynamic and morphological data of the i-th organ in real time using intraoperative ultrasound images, including the local volume change rate of the i-th organ caused by intraoperative manipulation. And the relative displacement of the i-th organ caused by intraoperative positioning adjustments Construct an intraoperative dataset;

[0009] S3. Using a convolutional neural network, construct an AI model and input the preoperative and intraoperative datasets into the AI ​​model to predict and output the surgical path;

[0010] S4. Construct the stability coefficient of the i-th organ based on the preoperative and intraoperative datasets. and crowding coefficient And the stability coefficient of the i-th organ Compared with the stability threshold A, when the stability coefficient of the i-th organ... When the stability threshold A is less than or equal to the value of the i-th organ crowding coefficient, a first alarm command is generated. Compared with the crowding threshold Q, when the crowding coefficient of the i-th organ... If the congestion level exceeds the congestion threshold Q, a second alarm command is generated.

[0011] S5, Set the stability coefficient of the i-th organ With the crowding coefficient of the i-th organ Correlation, constructing a comprehensive dynamic risk coefficient And will integrate dynamic risk coefficient Compared with the comprehensive risk threshold W, when the comprehensive dynamic risk coefficient If the risk exceeds the comprehensive risk threshold W, an assessment is conducted, and optimization instructions are generated.

[0012] In a preferred embodiment, S1 includes:

[0013] S11. By taking CT and MRI images of the target patient, preoperative CT and MRI image data are obtained, and a three-dimensional coordinate system (x, y, z) is established. The CT and MRI images are then segmented in three dimensions. Edge detection is performed on the CT and MRI images to separate the i-th organ from the CT and MRI images, obtaining the three-dimensional segmented region of the i-th organ. Based on the number of voxels and the physical volume of the voxels in the segmented region, the volume of the i-th organ is calculated. ;

[0014] S12. Based on the three-dimensional segmentation region of preoperative CT and MRI images, calculate the centroid of the target organ in the established three-dimensional coordinate system to obtain the three-dimensional spatial coordinates of the i-th organ. ;

[0015] S13. Perform three-dimensional segmentation on the i-th organ and the (i+1)-th organ in the CT and MRI images respectively, extract the surface point sets, and obtain the adjacent distance between the i-th organ and the (i+1)-th organ by calculating the shortest Euclidean distance between the surface point sets of the target organ and the surface point sets of adjacent organs. ;

[0016] S14. Take CT and MRI images at time points n and (n+1) respectively, and perform rigid registration between the CT and MRI images taken at time point n and time point (n+1). Calculate the difference in three-dimensional spatial coordinates of the i-th organ between time points n and (n+1) to obtain the local displacement vector of the i-th organ. This is used to construct a preoperative dataset.

[0017] In a preferred embodiment, S2 includes:

[0018] S21. During the operation, images of the i-th organ are continuously acquired using intraoperative color Doppler ultrasound on the target patient. Each frame of the i-th organ image is then segmented in three dimensions, and the number of segmented voxels is counted. The rate of change in local volume of the i-th organ caused by the intraoperative manipulation is calculated. ;

[0019] S22. Adjust the patient's position during the operation and use color Doppler ultrasound to acquire images of the i-th organ in real time. Based on a three-dimensional coordinate system, extract the three-dimensional spatial coordinates of the i-th organ from the images and register them with the preoperative three-dimensional spatial coordinates of the i-th organ. Calculate the difference in local coordinates of the i-th organ before and after the position adjustment to obtain the relative displacement of the i-th organ caused by the intraoperative position adjustment. .

[0020] In a preferred embodiment, S3 includes:

[0021] S31. Construct an AI model using a convolutional neural network, and train and test the AI ​​model using preoperative and intraoperative datasets. Use the trained AI model as a three-dimensional organ reconstruction, registration, and fusion evaluation model for multimodal medical images. Simultaneously, use the output of the AI ​​model running on the device as a feature vector to identify feature information, and use the trained AI model as data for prediction.

[0022] In a preferred embodiment, S4 includes:

[0023] S41. The volume of the i-th organ based on the preoperative dataset. 3D coordinates Distance to the adjacent organ of the (i+1)th organ and local displacement vector The stability coefficient of the i-th organ is calculated in the following way;

[0024] First, consider the volume of the i-th organ. 3D coordinates Distance to the adjacent organ of the (i+1)th organ and local displacement vector Normalization is performed to obtain the normalized volume of the i-th organ. 3D coordinates Distance to the adjacent organ of the (i+1)th organ and local displacement vector ;

[0025] In the formula, Represented as the minimum volume of an organ. Represented as the maximum volume of the organ;

[0026] For the three-dimensional coordinates of the i-th organ To perform normalization, first calculate the deviation vector of the three-dimensional coordinates of the i-th organ. ;

[0027] In the formula, Let the coordinates of the i-th organ be 3D spatial coordinates. Represented as the reference three-dimensional spatial coordinates of the i-th organ;

[0028] Then the deviation modulus was calculated. ;

[0029] In the formula, Represented as the x-axis coordinate value of the i-th organ in three-dimensional space. This represents the reference coordinate value of the x-axis in the three-dimensional space of the i-th organ. This is represented by the y-axis coordinate value of the i-th organ in three-dimensional space. This represents the reference coordinate value of the y-axis in the three-dimensional space of the i-th organ. Represented as the z-axis coordinate value of the i-th organ in three-dimensional space. This represents the reference coordinate value of the z-axis in the three-dimensional space of the i-th organ;

[0030] Finally, the three-dimensional coordinates of the i-th organ Perform normalization processing;

[0031] In the formula, This is represented as the magnitude of the deviation between the current coordinates and the reference coordinates of the i-th organ. This is expressed as the maximum deviation modulus of the organ;

[0032] In the formula, Represented as the minimum distance between adjacent organs. This represents the maximum distance between adjacent organs;

[0033] Local displacement vector of the i-th organ Perform normalization processing;

[0034] First, calculate the magnitude of the local displacement vector of the i-th organ. ;

[0035] In the formula, Let be the local displacement component of the i-th organ in the x-axis direction. This represents the local displacement component of the i-th organ along the y-axis. This is represented as the local displacement component of the i-th organ in the z-axis direction;

[0036] Then, the local displacement vector of the i-th organ... The result is obtained after normalization.

[0037] In the formula, This is expressed as the maximum value of the magnitude of the local displacement vector of the organ;

[0038] Based on the volume of the i-th organ 3D coordinates Distance to the adjacent organ of the (i+1)th organ and local displacement vector The stability coefficient of the i-th organ is calculated using the following formula. ;

[0039] ;

[0040] In the formula, , , and These are the weighting coefficients.

[0041] In a preferred embodiment, S4 further includes:

[0042] S42, the stability coefficient of the i-th organ Compared with the stability threshold A;

[0043] when When >A, it means that the i-th organ in the target patient is in normal position and shape during the operation, and is used as a reference for surgical path planning;

[0044] when When ≤A, it indicates that the i-th organ in the target patient has an abnormal position and shape during the operation, and generates the first alarm command and adopts optimization strategies, including adjusting the surgical path incision angle based on the offset direction and magnitude of the i-th organ, correcting the incision direction by 10%-30% of the offset, and increasing the image acquisition frequency of the i-th organ by 20%-70%.

[0045] In a preferred embodiment, S4 further includes:

[0046] S43. Local volume change rate of the i-th organ caused by intraoperative manipulation based on intraoperative data. And the relative displacement of the i-th organ caused by intraoperative positioning adjustments The crowding coefficient of the i-th organ is obtained in the following way. ;

[0047] First, the rate of change in local volume of the i-th organ caused by intraoperative manipulation was analyzed. And the relative displacement of the i-th organ caused by intraoperative positioning adjustments Normalization was performed to obtain the normalized rate of change in local volume of the i-th organ caused by intraoperative manipulation. And the relative displacement of the i-th organ caused by intraoperative positioning adjustments ;

[0048] In the formula, This is expressed as the maximum rate of change of organ volume;

[0049] In the formula, This is expressed as the maximum displacement value of the organ;

[0050] Based on the rate of change in local volume of the i-th organ caused by intraoperative manipulation And the relative displacement of the i-th organ caused by intraoperative positioning adjustments The crowding coefficient of the i-th organ is calculated using the following formula. ;

[0051] In the formula, This indicates a decimal excluding zero, with a value of 0.001.

[0052] In a preferred embodiment, S4 further includes:

[0053] S44, the crowding coefficient of the i-th organ Compare with the crowding threshold Q;

[0054] when When >Q, it indicates that the volume change rate and relative displacement of the i-th organ of the target patient are abnormal, and a second alarm command is generated to take corrective measures, including correcting the direction of the surgical path incident angle by 12%–33% based on the relative displacement direction of the i-th organ, reducing the traction force and negative pressure suction by 5%–20%, and adding 5%–15% spatial offset compensation to the surgical path.

[0055] when When ≤Q, it indicates that the rate of change of volume and relative displacement of the i-th organ in the target patient are normal.

[0056] In a preferred embodiment, S5 includes:

[0057] S51, the stability coefficient of the i-th organ With the crowding coefficient of the i-th organ The comprehensive dynamic risk coefficient is obtained through the following methods. ;

[0058] .

[0059] In a preferred embodiment, S5 further includes:

[0060] S52, Incorporate dynamic risk coefficient Compare with the comprehensive risk threshold W;

[0061] when When >W, it indicates that the i-th organ of the target patient is in an abnormal position during the operation. Optimization instructions are generated, including adjusting the surgical path incision angle by 10%-30% according to the displacement direction of the i-th organ, and expanding the safety boundary by 5%-15% based on the change in crowding.

[0062] when When W ≤ W, it indicates that the i-th organ of the target patient is in a normal position during the operation.

[0063] This invention provides a three-dimensional organ reconstruction registration and fusion method based on multimodal medical images, the beneficial effects of which include:

[0064] 1. This invention constructs a preoperative dataset by simultaneously acquiring CT, MRI and intraoperative ultrasound images, and updates the organ volume change rate and body position adjustment displacement in real time during the operation. It can effectively reflect the dynamic deformation of organs during real surgery. Compared with traditional surgical planning methods that rely on a single modality or static images, the three-dimensional organ model generated by this invention is closer to the real anatomical state and intraoperative morphological changes, significantly improving the accuracy of organ modeling and registration.

[0065] 2. By using calculable parameters such as the volume of the i-th organ, its three-dimensional coordinates, distance to neighboring organs, local displacement vector, intraoperative volume change rate, and relative displacement, an organ stability coefficient and a crowding coefficient are constructed. This enables a quantitative assessment of organ position stability, local compression degree, and anatomical risk. Warning commands are automatically generated through threshold comparison, allowing for real-time monitoring of risks during surgery. By correlating the organ stability coefficient and the crowding coefficient, a comprehensive dynamic risk coefficient is constructed and compared with a comprehensive risk threshold. This allows for automatic determination of whether there is a risk in the current surgical path. An AI model based on convolutional neural networks is invented, which uses preoperative and intraoperative data as input to automatically predict the optimal surgical path, reducing the subjectivity of traditional surgical path planning that relies on the doctor's experience. Attached Figure Description

[0066] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0067] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] Example 1, referring to Figure 1 This invention provides a technical solution: a three-dimensional organ reconstruction registration and fusion method based on multimodal medical images, comprising:

[0070] S1. Real-time acquisition of multimodal medical images of the i-th organ of the target patient, including CT images and MRI images, and extraction of the volume of the i-th organ. Three-dimensional spatial coordinates Distance to the adjacent organ of the (i+1)th organ and local displacement vector Construct a preoperative dataset;

[0071] S2. After constructing the first dataset, continue to collect dynamic and morphological data of the i-th organ in real time using intraoperative ultrasound images, including the local volume change rate of the i-th organ caused by intraoperative manipulation. And the relative displacement of the i-th organ caused by intraoperative positioning adjustments Construct an intraoperative dataset;

[0072] S3. Using a convolutional neural network, construct an AI model and input the preoperative and intraoperative datasets into the AI ​​model to predict and output the surgical path;

[0073] S4. Construct the stability coefficient of the i-th organ based on the preoperative and intraoperative datasets. and crowding coefficient And the stability coefficient of the i-th organ Compared with the stability threshold A, when the stability coefficient of the i-th organ... When the stability threshold A is less than or equal to the value of the i-th organ crowding coefficient, a first alarm command is generated. Compared with the crowding threshold Q, when the crowding coefficient of the i-th organ... If the congestion level exceeds the congestion threshold Q, a second alarm command is generated.

[0074] S5, Set the stability coefficient of the i-th organ With the crowding coefficient of the i-th organ Correlation, constructing a comprehensive dynamic risk coefficient And will integrate dynamic risk coefficient Compared with the comprehensive risk threshold W, when the comprehensive dynamic risk coefficient If the risk exceeds the comprehensive risk threshold W, an assessment is conducted, and optimization instructions are generated.

[0075] In this embodiment, the present invention constructs a preoperative dataset by simultaneously acquiring CT, MRI and intraoperative ultrasound images, and updates the organ volume change rate and body position adjustment displacement in real time during the operation. This can effectively reflect the dynamic deformation of organs during real surgery. Compared with traditional surgical planning methods that rely on a single modality or static images, the three-dimensional organ model generated by the present invention is closer to the real anatomical state and intraoperative morphological changes, significantly improving the accuracy of organ modeling and registration.

[0076] By using calculable parameters such as the volume of the i-th organ, its three-dimensional coordinates, distance to neighboring organs, local displacement vector, intraoperative volume change rate, and relative displacement, organ stability coefficient and crowding coefficient are constructed. This enables a quantitative assessment of organ position stability, local compression degree, and anatomical risk. Furthermore, by comparing thresholds, early warning commands are automatically generated, allowing for real-time monitoring of risks during the surgical procedure.

[0077] By correlating organ stability coefficient and crowding coefficient, a comprehensive dynamic risk coefficient is constructed and compared with a comprehensive risk threshold. This allows for the automatic determination of whether the current surgical path is risky. An AI model based on convolutional neural networks has been invented that uses both preoperative and intraoperative data as input to automatically predict the optimal surgical path, reducing the subjectivity of traditional surgical path planning that relies on the doctor's experience.

[0078] Example 2 is an explanation of Example 1; please refer to it. Figure 1 Specifically, S1 includes:

[0079] S11. By taking CT and MRI images of the target patient, preoperative CT and MRI image data are obtained, and a three-dimensional coordinate system (x, y, z) is established. The CT and MRI images are then segmented in three dimensions. Edge detection is performed on the CT and MRI images to separate the i-th organ from the CT and MRI images, obtaining the three-dimensional segmented region of the i-th organ. Based on the number of voxels and the physical volume of the voxels in the segmented region, the volume of the i-th organ is calculated. ;

[0080] First, calculate the voxel volume. ;

[0081] In the formula, Represented as the pixel physical size of CT and MRI images along the x-axis. Represented as the pixel physical size of CT and MRI images along the y-axis. It is expressed as slice thickness and the data is obtained from CT and MRI images;

[0082] In the formula, This represents the number of segmented voxels of the i-th organ;

[0083] S12. Based on the three-dimensional segmentation region of preoperative CT and MRI images, calculate the centroid of the target organ in the established three-dimensional coordinate system to obtain the three-dimensional spatial coordinates of the i-th organ. ;

[0084] S13. Perform three-dimensional segmentation on the i-th organ and the (i+1)-th organ in the CT and MRI images respectively, extract the surface point sets, and obtain the adjacent distance between the i-th organ and the (i+1)-th organ by calculating the shortest Euclidean distance between the surface point sets of the target organ and the surface point sets of adjacent organs. ;

[0085] S14. Take CT and MRI images at time points n and (n+1) respectively, and perform rigid registration between the CT and MRI images taken at time point n and time point (n+1). Calculate the difference in three-dimensional spatial coordinates of the i-th organ between time points n and (n+1) to obtain the local displacement vector of the i-th organ. This allows for the construction of a preoperative dataset. In this embodiment, by performing three-dimensional segmentation on CT and MRI images respectively, and combining edge detection to extract organ boundaries, the present invention can fully utilize the high-resolution bony structural features of CT and the soft tissue contrast advantage of MRI, significantly improving the accuracy of organ segmentation. By statistically analyzing the number of voxels within the three-dimensional segmentation region and combining it with the physical volume of the voxels, the present invention can directly and accurately obtain the true volume of the target organ. Based on the three-dimensional segmentation region, the organ centroid is directly calculated, and the obtained three-dimensional coordinates have the following advantages: they are not affected by local noise points, the position description is uniform and repeatable, and they have better registration between multimodal images.

[0086] This invention extracts a set of points on the surface of organs and calculates the shortest Euclidean distance between adjacent organs to obtain the proximity distance between organs. It can reflect the true three-dimensional geometric relationship of organs, is more sensitive to potential compression and collision risks between organs, and has high quantification accuracy and strong mathematical interpretability.

[0087] This invention achieves accurate reflection of organ displacement caused by respiratory movement, pulsation, and posture changes by rigidly registering CT / MRI images at time points n and n+1 and calculating the difference in three-dimensional coordinates of the organ. This eliminates errors caused by non-anatomical factors such as image shooting angle and patient position, and improves the reliability of displacement features for subsequent model input.

[0088] Example 3 is an explanation of Example 1; please refer to the provided text. Figure 1 Specifically, S2 includes:

[0089] S21. During the operation, images of the i-th organ are continuously acquired using intraoperative color Doppler ultrasound on the target patient. Each frame of the i-th organ image is then segmented in three dimensions, and the number of segmented voxels is counted. The rate of change in local volume of the i-th organ caused by the intraoperative manipulation is calculated. ;

[0090] In the formula Represented as the first The volume of the i-th organ during the time period. Represented as the first The volume of the i-th organ during the time period;

[0091] S22. Adjust the patient's position during the operation and use color Doppler ultrasound to acquire images of the i-th organ in real time. Based on a three-dimensional coordinate system, extract the three-dimensional spatial coordinates of the i-th organ from the images and register them with the preoperative three-dimensional spatial coordinates of the i-th organ. Calculate the difference in local coordinates of the i-th organ before and after the position adjustment to obtain the relative displacement of the i-th organ caused by the intraoperative position adjustment. Construct an intraoperative dataset.

[0092] In this embodiment, the present invention continuously acquires organ images using a color Doppler ultrasound machine and performs three-dimensional segmentation and voxel statistics on each frame. It can capture rapid volume changes caused by operations such as breathing, heartbeat, traction, and electrocoagulation. The three-dimensional coordinates of the organs are extracted through real-time ultrasound images and registered with the preoperative three-dimensional coordinate system. The present invention can calculate the relative displacement of organs after the body position is adjusted, and can accurately reflect the direction and amplitude of organ movement caused by bed lifting, tilting, and rotation.

[0093] By combining intraoperative dynamic data with preoperative static data, convolutional neural networks can learn the dynamic changes of organs during surgery.

[0094] Example 4 is an explanation of Example 1; please refer to the provided text. Figure 1 Specifically, S3 includes:

[0095] S31. Construct an AI model using a convolutional neural network, and train and test the AI ​​model using preoperative and intraoperative datasets. Use the trained AI model as a three-dimensional organ reconstruction, registration, and fusion evaluation model for multimodal medical images. Simultaneously, use the output of the AI ​​model running on the device as a feature vector to identify feature information, and use the trained AI model as data for prediction.

[0096] S4 includes:

[0097] S41. The volume of the i-th organ based on the preoperative dataset. 3D coordinates Distance to the adjacent organ of the (i+1)th organ and local displacement vector The stability coefficient of the i-th organ is calculated in the following way;

[0098] First, consider the volume of the i-th organ. 3D coordinates Distance to the adjacent organ of the (i+1)th organ and local displacement vector Normalization is performed to obtain the normalized volume of the i-th organ. 3D coordinates Distance to the adjacent organ of the (i+1)th organ and local displacement vector ;

[0099] In the formula, Represented as the minimum volume of an organ. Represented as the maximum volume of the organ;

[0100] For the three-dimensional coordinates of the i-th organ To perform normalization, first calculate the deviation vector of the three-dimensional coordinates of the i-th organ. ;

[0101] In the formula, Let the coordinates of the i-th organ be 3D spatial coordinates. Represented as the reference three-dimensional spatial coordinates of the i-th organ;

[0102] Then the deviation modulus was calculated. ;

[0103] In the formula, Represented as the x-axis coordinate value of the i-th organ in three-dimensional space. This represents the reference coordinate value of the x-axis in the three-dimensional space of the i-th organ. This is represented by the y-axis coordinate value of the i-th organ in three-dimensional space. This represents the reference coordinate value of the y-axis in the three-dimensional space of the i-th organ. Represented as the z-axis coordinate value of the i-th organ in three-dimensional space. This represents the reference coordinate value of the z-axis in the three-dimensional space of the i-th organ;

[0104] Finally, the three-dimensional coordinates of the i-th organ Perform normalization processing;

[0105] In the formula, This is represented as the magnitude of the deviation between the current coordinates and the reference coordinates of the i-th organ. This is expressed as the maximum deviation modulus of the organ;

[0106] In the formula, Represented as the minimum distance between adjacent organs. This represents the maximum distance between adjacent organs;

[0107] Local displacement vector of the i-th organ Perform normalization processing;

[0108] First, calculate the magnitude of the local displacement vector of the i-th organ. ;

[0109] In the formula, Let be the local displacement component of the i-th organ in the x-axis direction. This represents the local displacement component of the i-th organ along the y-axis. This is represented as the local displacement component of the i-th organ in the z-axis direction;

[0110] Then, the local displacement vector of the i-th organ... The result is obtained after normalization.

[0111] In the formula, This is expressed as the maximum value of the magnitude of the local displacement vector of the organ;

[0112] Based on the volume of the i-th organ 3D coordinates Distance to the adjacent organ of the (i+1)th organ and local displacement vector The stability coefficient of the i-th organ is calculated using the following formula. ;

[0113] ;

[0114] In the formula, , , and The weighting coefficients are derived based on the influence of the volume, three-dimensional coordinates, adjacent distance to the (i+1)th organ, and local displacement vector of the i-th organ on the surgery, combined with experience. , , and The value.

[0115] In this embodiment, an AI model is constructed using a convolutional neural network. Combined with preoperative and intraoperative datasets, multimodal medical images are processed uniformly and deep feature extracted, achieving a fusion analysis of organ morphology, spatial location, and dynamic changes. Compared to traditional methods relying on single-modal images and human experience, this invention can automatically extract multidimensional information from CT, MRI, and intraoperative ultrasound images, enabling the identification and modeling of organ structural features, displacement features, and relationships with neighboring organs with higher precision. This significantly improves the accuracy and stability of 3D organ reconstruction, registration, and surgical path prediction. Furthermore, the AI ​​model can automatically output multidimensional feature vectors reflecting organ volume, centroid location, distance between adjacent organs, and local displacement changes. This allows the system to more sensitively capture dynamic changes during surgery, improving the response speed and reliability of surgical navigation and early warning.

[0116] This invention normalizes different physical quantities such as volume, spatial coordinates, distances between adjacent organs, and local displacement vectors, giving each parameter a unified dimension, thereby improving the effectiveness and comparability of feature fusion. By constructing the stability coefficient of the i-th organ, the potential displacement, deformation, and stress on the organ during surgery can be quantitatively reflected, achieving an objective evaluation of organ stability. Furthermore, by introducing the magnitude of three-dimensional coordinate deviation and the magnitude of local displacement vector, the spatial drift and instantaneous displacement of the organ are accurately quantified, improving sensitivity to potential risk changes. By calculating the shortest distance between adjacent organs, the system can identify potential risks of crowding, compression, or structural interference in the surgical area in advance, improving the safety margin of intraoperative navigation.

[0117] Example 5 is an explanation of Example 1; please refer to it. Figure 1 Specifically, S4 also includes:

[0118] S42, the stability coefficient of the i-th organ Compared with a stability threshold A, preoperative CT / MRI and intraoperative ultrasound imaging data were first collected from a large number of successfully completed similar surgical cases. The volume, three-dimensional coordinate deviation, adjacent distance, and local displacement vector of each organ in historical patients were calculated to obtain the stability coefficient. Subsequently, these historical data were categorized into two types based on surgical outcomes: "normal intraoperative performance" and "abnormal intraoperative displacement / deformation." A stability coefficient distribution model was constructed for all samples with normal intraoperative performance, and its mean and standard deviation were calculated. A stability threshold A was then set based on empirical risk control strategies.

[0119] when When >A, it means that the i-th organ in the target patient is in normal position and shape during the operation, and is used as a reference for surgical path planning;

[0120] when When ≤A, it indicates that the i-th organ in the target patient has an abnormal position and shape during the operation, and generates the first alarm command and adopts optimization strategies, including adjusting the surgical path incision angle based on the offset direction and magnitude of the i-th organ, correcting the incision direction by 10%-30% of the offset, and increasing the image acquisition frequency of the i-th organ by 20%-70%.

[0121] In this embodiment, the stability coefficient of the i-th organ is compared with a stability threshold. This invention enables real-time, objective, and quantitative assessment of the spatial position and morphological changes of organs during surgery. When the stability coefficient exceeds the threshold, the system automatically determines that the organ is in a normal state and can be used for surgical reference, thereby ensuring that the surgical path planning is based on reliable tissue parameters and improving the reliability and accuracy of intraoperative navigation. When the stability coefficient is lower than or equal to the threshold, the system can immediately identify risks such as abnormal organ displacement, rotation, or deformation, and automatically generate alarm commands, allowing the surgeon to be aware of potential hazards and intervene immediately.

[0122] Example 6 is an explanation of Example 1; please refer to the provided text. Figure 1 Specifically, S4 also includes:

[0123] S43. Local volume change rate of the i-th organ caused by intraoperative manipulation based on intraoperative data. And the relative displacement of the i-th organ caused by intraoperative positioning adjustments The crowding coefficient of the i-th organ is obtained in the following way. ;

[0124] First, the rate of change in local volume of the i-th organ caused by intraoperative manipulation was analyzed. And the relative displacement of the i-th organ caused by intraoperative positioning adjustments Normalization was performed to obtain the normalized rate of change in local volume of the i-th organ caused by intraoperative manipulation. And the relative displacement of the i-th organ caused by intraoperative positioning adjustments ;

[0125] In the formula, This represents the maximum rate of change in organ volume, obtained based on historical data.

[0126] In the formula, This is represented as the maximum displacement value of the organ, obtained based on historical data;

[0127] Based on the rate of change in local volume of the i-th organ caused by intraoperative manipulation And the relative displacement of the i-th organ caused by intraoperative positioning adjustments The crowding coefficient of the i-th organ is calculated using the following formula. ;

[0128] In the formula, This indicates a decimal excluding zero, with a value of 0.001.

[0129] In this embodiment, by introducing the crowding coefficient of the i-th organ, the present invention achieves real-time quantitative assessment of the degree of organ compression, spatial competition, and the mutual influence between organs during surgery. The system first normalizes the local volume change rate and relative displacement to eliminate dimensional inconsistencies caused by different organ volumes, movement amplitudes, and individual differences, ensuring the comparability and stability of crowding calculation results across different patients. Subsequently, the crowding coefficient is calculated using a bivariate nonlinear expression, fully reflecting the coupling effect of organ volume increase and spatial displacement: when volume expansion and displacement increase simultaneously, the crowding coefficient rises significantly, effectively reflecting that the organ is under pressure, crowded, or in a high-risk, narrow spatial state.

[0130] Example 7 is an explanation of Example 1; please refer to it. Figure 1 Specifically, S4 also includes:

[0131] S44, the crowding coefficient of the i-th organ Compared with the crowding threshold Q; the crowding coefficient of the i-th organ in each case was extracted from a large amount of historical data of similar surgeries and labeled with clinical outcome labels, and normal and abnormal samples were compared. The distribution is statistically and discriminantly analyzed. ROC curves and empirical percentiles are used to generate candidate thresholds. The optimal threshold is selected through cross-validation and independent validation sets, thereby determining the crowding threshold Q.

[0132] when When >Q, it indicates that the volume change rate and relative displacement of the i-th organ of the target patient are abnormal, and a second alarm command is generated to take corrective measures, including correcting the direction of the surgical path incident angle by 12%–33% based on the relative displacement direction of the i-th organ, reducing the traction force and negative pressure suction by 5%–20%, and adding 5%–15% spatial offset compensation to the surgical path.

[0133] when When ≤Q, it indicates that the rate of change of volume and relative displacement of the i-th organ in the target patient are normal.

[0134] In this embodiment, by comparing the crowding coefficient of the i-th organ with a preset crowding threshold, the present invention can identify in real time during surgery the volume changes and relative displacement abnormalities caused by instrument traction, tissue displacement, or abnormal local pressure, thereby constructing a timely and accurate risk warning mechanism. When the crowding coefficient exceeds the threshold Q, the system will immediately trigger a second alarm command and automatically provide a path correction scheme based on the spatial displacement direction.

[0135] Example 8 is an explanation of Example 1; please refer to it. Figure 1 Specifically, S5 includes:

[0136] S51, the stability coefficient of the i-th organ With the crowding coefficient of the i-th organ The comprehensive dynamic risk coefficient is obtained through the following methods. ;

[0137] .

[0138] In this embodiment, by correlating the stability coefficient and crowding coefficient of the i-th organ and multiplying them to form a comprehensive dynamic risk coefficient, this invention achieves a leap from single-indicator judgment to multi-dimensional coupled analysis in risk assessment. The stability coefficient reflects the overall deviation of the organ in terms of volume, position, and local displacement, while the crowding coefficient characterizes the immediate spatial risk caused by intraoperative pressure changes and relative motion. The combined effect of the two can amplify the sensitivity to abnormal states, enabling the system to strongly reflect any deviation in any dimension through a significant change in the comprehensive dynamic risk coefficient, thereby constructing a more accurate and robust intraoperative risk identification mechanism.

[0139] Example 9, this example is an explanation of Example 1, please refer to it. Figure 1 Specifically, S5 also includes:

[0140] S52, Incorporate dynamic risk coefficient The risk coefficient was compared with the comprehensive risk threshold W; the comprehensive dynamic risk coefficient of each case at each time point was calculated for a large number of historical cases. Furthermore, using manually labeled abnormal events as tags, the optimal cut-off point was determined using ROC curves and the Youden index as the comprehensive risk threshold W.

[0141] when When >W, it indicates that the i-th organ of the target patient is in an abnormal position during the operation. Optimization instructions are generated, including adjusting the surgical path incision angle by 10%-30% according to the displacement direction of the i-th organ, and expanding the safety boundary by 5%-15% based on the change in crowding.

[0142] when When W ≤ W, it indicates that the i-th organ of the target patient is in a normal position during the operation.

[0143] In this embodiment, by comparing the comprehensive dynamic risk coefficient with the comprehensive risk threshold, the present invention achieves real-time, quantitative, and executable risk decision-making capabilities for intraoperative organ positional abnormalities. When the comprehensive dynamic risk coefficient exceeds the threshold, the system can instantly identify the combined risks of organ positional displacement and spatial compression, and automatically generate targeted optimization instructions.

[0144] The threshold is set to facilitate comparison. The size of the threshold depends on the amount of sample data and the number of bases set by those skilled in the art for each set of sample data; as long as it does not affect the ratio between the parameter and the quantized value, it is acceptable.

[0145] The above formulas are all derived from software simulation using a large amount of data and are selected to be close to the actual values. The coefficients in the formulas are set by those skilled in the art according to the actual situation. The above description is only a preferred embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any equivalent substitutions or changes made by those skilled in the art within the technical scope disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the protection scope of the present invention.

Claims

1. A three-dimensional organ reconstruction registration and fusion method based on multimodal medical images, characterized in that, include: S1. Real-time acquisition of multimodal medical images of the i-th organ of the target patient, including CT images and MRI images, and extraction of the volume of the i-th organ. Three-dimensional spatial coordinates Distance to the adjacent organ of the (i+1)th organ and local displacement vector Construct a preoperative dataset; S2. After constructing the first dataset, continue to collect dynamic and morphological data of the i-th organ in real time using intraoperative ultrasound images, including the local volume change rate of the i-th organ caused by intraoperative manipulation. And the relative displacement of the i-th organ caused by intraoperative positioning adjustments Construct an intraoperative dataset; S3. Using a convolutional neural network, construct an AI model and input the preoperative and intraoperative datasets into the AI ​​model to predict and output the surgical path; S4. Construct the stability coefficient of the i-th organ based on the preoperative and intraoperative datasets. and crowding coefficient And the stability coefficient of the i-th organ Compared with the stability threshold A, when the stability coefficient of the i-th organ... When the stability threshold A is less than or equal to the value of the i-th organ crowding coefficient, a first alarm command is generated. Compared with the crowding threshold Q, when the crowding coefficient of the i-th organ... If the congestion level exceeds the congestion threshold Q, a second alarm command is generated. S5, Set the stability coefficient of the i-th organ With the crowding coefficient of the i-th organ Correlation, constructing a comprehensive dynamic risk coefficient And will integrate dynamic risk coefficient Compared with the comprehensive risk threshold W, when the comprehensive dynamic risk coefficient If the risk exceeds the comprehensive risk threshold W, an assessment is conducted, and optimization instructions are generated.

2. The method for three-dimensional organ reconstruction, registration, and fusion based on multimodal medical images according to claim 1, characterized in that, S1 includes: S11. By taking CT and MRI images of the target patient, preoperative CT and MRI image data are obtained, and a three-dimensional coordinate system (x, y, z) is established. The CT and MRI images are then segmented in three dimensions. Edge detection is performed on the CT and MRI images to separate the i-th organ from the CT and MRI images, obtaining the three-dimensional segmented region of the i-th organ. Based on the number of voxels and the physical volume of the voxels in the segmented region, the volume of the i-th organ is calculated. ; S12. Based on the three-dimensional segmentation region of preoperative CT and MRI images, calculate the centroid of the target organ in the established three-dimensional coordinate system to obtain the three-dimensional spatial coordinates of the i-th organ. ; S13. Perform three-dimensional segmentation on the i-th organ and the (i+1)-th organ in the CT and MRI images respectively, extract the surface point sets, and obtain the adjacent distance between the i-th organ and the (i+1)-th organ by calculating the shortest Euclidean distance between the surface point sets of the target organ and the surface point sets of adjacent organs. ; S14. Take CT and MRI images at time points n and (n+1) respectively, and perform rigid registration between the CT and MRI images taken at time point n and time point (n+1). Calculate the difference in three-dimensional spatial coordinates of the i-th organ between time points n and (n+1) to obtain the local displacement vector of the i-th organ. This is used to construct a preoperative dataset.

3. The method for three-dimensional organ reconstruction registration and fusion based on multimodal medical images according to claim 2, characterized in that, S2 includes: S21. During the operation, images of the i-th organ are continuously acquired using intraoperative color Doppler ultrasound on the target patient. Each frame of the i-th organ image is then segmented in three dimensions, and the number of segmented voxels is counted. The rate of change in local volume of the i-th organ caused by the intraoperative manipulation is calculated. ; S22. Adjust the patient's position during the operation and use color Doppler ultrasound to acquire images of the i-th organ in real time. Based on a three-dimensional coordinate system, extract the three-dimensional spatial coordinates of the i-th organ from the images and register them with the preoperative three-dimensional spatial coordinates of the i-th organ. Calculate the difference in local coordinates of the i-th organ before and after the position adjustment to obtain the relative displacement of the i-th organ caused by the intraoperative position adjustment. .

4. The method for three-dimensional organ reconstruction registration and fusion based on multimodal medical images according to claim 3, characterized in that, S3 includes: S31. Construct an AI model using a convolutional neural network, and train and test the AI ​​model using preoperative and intraoperative datasets. Use the trained AI model as a three-dimensional organ reconstruction, registration, and fusion evaluation model for multimodal medical images. Simultaneously, use the output of the AI ​​model running on the device as a feature vector to identify feature information, and use the trained AI model as data for prediction.

5. The method for three-dimensional organ reconstruction registration and fusion based on multimodal medical images according to claim 4, characterized in that, S4 includes: S41. The volume of the i-th organ based on the preoperative dataset. 3D coordinates Distance to the adjacent organ of the (i+1)th organ and local displacement vector The stability coefficient of the i-th organ is calculated in the following way; First, consider the volume of the i-th organ. 3D coordinates Distance to the adjacent organ of the (i+1)th organ and local displacement vector Normalization is performed to obtain the normalized volume of the i-th organ. 3D coordinates Distance to the adjacent organ of the (i+1)th organ and local displacement vector ; In the formula, Represented as the minimum volume of an organ. Represented as the maximum volume of the organ; For the three-dimensional coordinates of the i-th organ To perform normalization, first calculate the deviation vector of the three-dimensional coordinates of the i-th organ. ; In the formula, Let the coordinates of the i-th organ be 3D spatial coordinates. Represented as the reference three-dimensional spatial coordinates of the i-th organ; Then the deviation modulus was calculated. ; In the formula, Represented as the x-axis coordinate value of the i-th organ in three-dimensional space. This represents the reference coordinate value of the x-axis in the three-dimensional space of the i-th organ. This is represented by the y-axis coordinate value of the i-th organ in three-dimensional space. This represents the reference coordinate value of the y-axis in the three-dimensional space of the i-th organ. Represented as the z-axis coordinate value of the i-th organ in three-dimensional space. This represents the reference coordinate value of the z-axis in the three-dimensional space of the i-th organ; Finally, the three-dimensional coordinates of the i-th organ Perform normalization processing; In the formula, This is represented as the magnitude of the deviation between the current coordinates and the reference coordinates of the i-th organ. This is expressed as the maximum deviation modulus of the organ; In the formula, Represented as the minimum distance between adjacent organs. This represents the maximum distance between adjacent organs; Local displacement vector of the i-th organ Perform normalization processing; First, calculate the magnitude of the local displacement vector of the i-th organ. ; In the formula, Let be the local displacement component of the i-th organ in the x-axis direction. This represents the local displacement component of the i-th organ along the y-axis. This is represented as the local displacement component of the i-th organ in the z-axis direction; Then, the local displacement vector of the i-th organ... The result is obtained after normalization. In the formula, This is expressed as the maximum value of the magnitude of the local displacement vector of the organ; Based on the volume of the i-th organ 3D coordinates Distance to the adjacent organ of the (i+1)th organ and local displacement vector The stability coefficient of the i-th organ is calculated using the formula. .

6. The method for three-dimensional organ reconstruction registration and fusion based on multimodal medical images according to claim 5, characterized in that, S4 also includes: S42, the stability coefficient of the i-th organ Compare with the stability threshold A; combine historical data and analyze the value of the stability threshold A from the historical data, and combine with experience to determine the stability threshold A; when When >A, it means that the i-th organ in the target patient is in normal position and shape during the operation, and is used as a reference for surgical path planning; when When ≤A, it indicates that the i-th organ in the target patient has an abnormal position and shape during the operation, and generates the first alarm command and adopts optimization strategies, including adjusting the surgical path incision angle based on the offset direction and magnitude of the i-th organ, correcting the incision direction by 10%-30% of the offset, and increasing the image acquisition frequency of the i-th organ by 20%-70%.

7. The method for three-dimensional organ reconstruction registration and fusion based on multimodal medical images according to claim 6, characterized in that, S4 also includes: S43. Local volume change rate of the i-th organ caused by intraoperative manipulation based on intraoperative data. And the relative displacement of the i-th organ caused by intraoperative positioning adjustments The crowding coefficient of the i-th organ is obtained in the following way. ; First, the rate of change in local volume of the i-th organ caused by intraoperative manipulation was analyzed. And the relative displacement of the i-th organ caused by intraoperative positioning adjustments Normalization was performed to obtain the normalized rate of change in local volume of the i-th organ caused by intraoperative manipulation. And the relative displacement of the i-th organ caused by intraoperative positioning adjustments ; In the formula, This is expressed as the maximum rate of change of organ volume; In the formula, This is expressed as the maximum displacement value of the organ; Based on the rate of change in local volume of the i-th organ caused by intraoperative manipulation And the relative displacement of the i-th organ caused by intraoperative positioning adjustments The crowding coefficient of the i-th organ is calculated using the formula. .

8. The method for three-dimensional organ reconstruction registration and fusion based on multimodal medical images according to claim 7, characterized in that, S4 also includes: S44, the crowding coefficient of the i-th organ Compare with the crowding threshold Q; when When >Q, it indicates that the volume change rate and relative displacement of the i-th organ of the target patient are abnormal, and a second alarm command is generated to take corrective measures, including correcting the direction of the surgical path incident angle by 12%–33% based on the relative displacement direction of the i-th organ, reducing the traction force and negative pressure suction by 5%–20%, and adding 5%–15% spatial offset compensation to the surgical path. when When ≤Q, it indicates that the rate of change of volume and relative displacement of the i-th organ in the target patient are normal.

9. A method for three-dimensional organ reconstruction registration and fusion based on multimodal medical images according to claim 8, characterized in that, S5 includes: S51, the stability coefficient of the i-th organ With the crowding coefficient of the i-th organ The comprehensive dynamic risk coefficient is obtained through the following methods. ; 。 10. A method for three-dimensional organ reconstruction, registration, and fusion based on multimodal medical images according to claim 9, characterized in that, S5 also includes: S52, Incorporate dynamic risk coefficient Compare with the comprehensive risk threshold W; when When >W, it indicates that the i-th organ of the target patient is in an abnormal position during the operation. Optimization instructions are generated, including adjusting the surgical path incision angle by 10%-30% according to the displacement direction of the i-th organ, and expanding the safety boundary by 5%-15% based on the change in crowding. when When W ≤ W, it indicates that the i-th organ of the target patient is in a normal position during the operation.

Images