Intraoperative guiding method and system based on deep learning

Abstract

The invention discloses an intraoperative guidance method and system based on deep learning. The method comprises the following steps: pre-operative planning and model establishment; carrying out intraoperative multi-modal image acquisition and preprocessing; constructing and training a deep learning model; designing a double-flow multi-mode deep network to fuse high-frequency surface motion of a microscope video and low-frequency and scaled depth information of an intraoperative OCT image; remapping the preoperative retina model to the current anatomical form in the operation; performing intraoperative registration and real-time compensation control; a three-dimensional space deformation field output by the deep learning model is mapped to a preoperative three-dimensional model coordinate system, a real-time registration matrix is calculated and compared with the tail end position of the surgical instrument, and instrument movement is adjusted through a servo control system to achieve dynamic error compensation. According to the method, deep learning is combined with an intraoperative image registration technology, so that high fusion and accurate matching of preoperative planning and intraoperative guidance are realized, and the intelligent level and clinical application value of a surgical navigation system are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of surgical navigation technology, and in particular to an intraoperative guidance method and system based on deep learning. Background Technology

[0002] With the development of ophthalmic microsurgery, microscopy and OCT (optical coherence tomography) imaging technology have become the main means of intraoperative assessment of lesions and tissue structures. Especially in procedures such as corneal transplantation, glaucoma filtration surgery, and minimally invasive retinal surgery, doctors need to operate on micron-level structures at high magnification, requiring extremely high precision in registration and instrument guidance.

[0003] Currently, surgical navigation systems commonly used in clinical practice are usually based on three-dimensional reconstruction and path planning using preoperative images. However, during the surgical phase, due to tissue deformation, changes in microscope viewing angle, and interference from lighting and liquid environment, there are significant deviations between the intraoperative images and the preoperative models, resulting in the accumulation of registration errors and a decrease in navigation accuracy.

[0004] Traditional registration algorithms based on geometric features or template matching struggle to achieve real-time, high-precision registration in microscopic settings. Meanwhile, the real-time tracking and guidance of surgical instruments depends on the accuracy of image localization; errors in image registration reduce instrument localization and control precision, impacting surgical safety.

[0005] As deep learning continues to be applied in computer vision and medical image analysis, structures based on convolutional neural networks (CNNs) and attention mechanisms have achieved significant results in multimodal image registration and target recognition. However, how to combine these algorithms with microscopic and OCT fusion images to achieve refined dynamic registration of intraoperative lesions with preoperative models and instrument guidance remains a challenge and key area of ​​current research. Summary of the Invention

[0006] In view of the above problems, the present invention provides an intraoperative guidance method and system based on deep learning, which aims to achieve precise identification and real-time registration of lesions during surgery by fusing microscope and OCT images with a preoperative planning model, and automatically guide surgical instruments to the planned position, thereby significantly improving the accuracy and safety of microsurgery.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a deep learning-based intraoperative guidance method, comprising the following steps: Step 1: Preoperative planning and model establishment; acquiring multi-frame OCT B-Scan slice sequence data from the patient's fundus. I OCT ( x, zThe layer extraction algorithm of the three-dimensional U-Net network was used to segment the retinal layers of the OCT data, and the depth location functions of the main tissue interfaces in the retina were obtained: internal limiting membrane. S ILM ( x, z ), inner tufted layer S IPL ( x, z ), outer plexiform layer S OPL ( x, z ), external membrane S ELM ( x, z ), retinal pigment epithelium S PRE ( x, z The pixel coordinates of a certain segmented contour in the k-th B-scan are ( i, j Convert it to real-world three-dimensional coordinates: P xyz ( idx, kdy, jdz ); obtain the 3D point cloud data of the corresponding layer. P ILM , P IPL A three-dimensional voxel mesh is constructed, and point cloud data from a single segmentation layer is extracted and set as the boundary. A fast nearest neighbor search algorithm is used to calculate the nearest distance from the mesh point to the point cloud. d Compare grid points with their neighbors. z The value determines the distance sign, and a three-dimensional distance scalar field (Volume) is constructed to the point cloud surface. x, y, z Using the Marching Cubes algorithm, a threshold isoValue = 0 is set to represent the critical surface. The voxel space is scanned with unit cubes, and the intersection points are solved by linear interpolation at the boundary positions to form triangular patches. After traversing all voxels, the three-dimensional scalar field is converted into an isosurface triangular mesh. The above operation is repeated to obtain the {ILM, IPL, OPL, ELM, PRE} mesh model, which is then combined to generate a three-dimensional mesh model of the multilayer retinal tissue. Step 2: Intraoperative multimodal image acquisition and preprocessing; real-time acquisition of intraoperative microscope videos and OCT B-scan images; establishment of a three-dimensional volumetric data model from the intraoperative OCT images using the method in Step 1. V intra ; Step 3: Deep learning model construction and training; Design a dual-stream multimodal deep network to fuse high-frequency surface motion from microscope videos with low-frequency but scaled depth information from intraoperative OCT images, regressing a dense 3D deformation field; Remap the preoperative retinal model to the current anatomical morphology during surgery, thereby reflecting and compensating for retinal deformation in real time. Step 4: Intraoperative registration and real-time compensation control; The three-dimensional spatial deformation field output by the deep learning model is mapped to the preoperative three-dimensional model coordinate system, the real-time registration matrix is ​​calculated and compared with the position of the surgical instrument end, and the instrument movement is adjusted through the servo control system to achieve dynamic error compensation.

[0008] In one embodiment of the present invention, the deep learning model construction and training in step 3 includes designing an architecture consisting of a high frame rate tracking subnetwork and a low frame rate correction subnetwork; step 3 specifically includes: Step 31: 2D-3D registration and feature projection network; associating high frame rate 2D microscope images with 3D space; firstly, based on the predicted deformation field from the previous time step. Φ(t - 1) Acting on preoperative three-dimensional spatial voxels V pre Obtain the current estimated 3D structure V deformed ( t-1 And using a differentiable renderer, learn the illumination and tissue optical properties of the microscope, and... V deformed ( t-1 Rendered into a virtual 2D microscope image I rendered (t) ; Using a shared-weight CNN backbone, respectively from I micro (t) and I rendered (t) Extracting high-dimensional feature maps F micro (t) and F rendered (t) Calculate the residual between the two feature maps. ΔF 2D (t) As the module output, it encodes the motion differences observed on the 2D image plane from the previous moment to the current moment; Step 32: Deformation update network; using a 3D recurrent convolutional network ConvLSTM as the core, with... Φ(t - 1) and the output of the previous module ΔF 2D (t) As input, the hidden state H(t-1) of the RNN encodes the dynamic history of retinal deformation, effectively modeling spatiotemporal sequence dependencies, and maintaining a hidden state to remember the deformation history of the retina; this module determines the deformation field Φ of the previous frame based on the new 2D driving features. pred(t-1) How needs to be updated to match the observations of the current frame; ultimately predict the current deformation field update Φ. pred (t)= Φ pred (t-1)+ ΔΦ(t); Whenever new intraoperative OCT data is available intra When (τ) arrives, a true deformation field Φ is calculated as the source of the monitoring signal. gt (t), calculate the predicted deformation field Φ pred (t) and the actual deformation field Φ gt Similarity loss between (t); Step 33: Define the loss function; Deformation field supervision loss is used to calculate the difference between the predicted deformation field and the actual deformation field calculated based on OCT during intraoperative OCT data updates. Deformation field smoothness loss, used to constrain the spatial smoothness of the predicted deformation field: Photometric consistency loss, used to constrain the photometric consistency between optical flow calculated from the deformation field and the sequence of microscope images: Deformation cycle consistency loss is used to constrain the reversibility of the deformation field; Step 34: Staged model training to avoid convergence difficulties; 2D-3D registration and feature projection network renderer pre-trained, with fixed Φ pred (t)=0, using the preoperative microscopic OCT dataset, it is possible to obtain from V pre Generate realistic microscope images; train the deformation update network on sequences containing only OCT data; then, input ΔF 2D (t) is set as random noise; finally, using complete microscopic and OCT surgical data, at the OCT update time τ, a strong supervisory signal is provided to correct the cumulative error of the entire system and teach the 2D flow how to contribute to accurate 3D deformation estimation.

[0009] In one embodiment of the present invention, the deformation field monitoring loss in step 33 is: .

[0010] In one embodiment of the present invention, the deformation field smoothness loss in step 33 is: .

[0011] In one embodiment of the present invention, the photometric uniformity loss in step 33 is: .

[0012] In one embodiment of the present invention, the deformation cycle consistency loss in step 33 is: .

[0013] In one embodiment of the present invention, the total loss function in step 33 is defined as:

[0014] Configure weights based on task importance. .

[0015] Secondly, the present invention provides a deep learning-based intraoperative guidance system for implementing the aforementioned deep learning-based intraoperative guidance method, the system comprising: Preoperative planning and model building module; used to acquire multi-frame OCT B-Scan slice sequence data of the patient's fundus. I OCT ( x, z The layer extraction algorithm of the three-dimensional U-Net network was used to segment the retinal layers of the OCT data, and the depth location functions of the main tissue interfaces in the retina were obtained: internal limiting membrane. S ILM ( x, z ), inner tufted layer S IPL ( x, z ), outer plexiform layer S OPL ( x, z ), external membrane S ELM ( x, z ), retinal pigment epithelium S PRE ( x, z The pixel coordinates of a certain segmented contour in the k-th B-scan are ( i, j Convert it to real-world three-dimensional coordinates: P xyz ( idx, kdy, jdz ); obtain the 3D point cloud data of the corresponding layer. P ILM , P IPL A three-dimensional voxel mesh is constructed, and point cloud data from a single segmentation layer is extracted and set as the boundary. A fast nearest neighbor search algorithm is used to calculate the nearest distance from the mesh point to the point cloud. d Compare grid points with their neighbors. z The value determines the distance sign, and a three-dimensional distance scalar field (Volume) is constructed to the point cloud surface. x, y, zUsing the Marching Cubes algorithm, a threshold isoValue = 0 is set to represent the critical surface. The voxel space is scanned with unit cubes, and the intersection points are solved by linear interpolation at the boundary positions to form triangular patches. After traversing all voxels, the three-dimensional scalar field is converted into an isosurface triangular mesh. The above operation is repeated to obtain the {ILM, IPL, OPL, ELM, PRE} mesh model, which is then combined to generate a three-dimensional mesh model of the multilayer retinal tissue. Intraoperative multimodal image acquisition and preprocessing module; used for real-time acquisition of intraoperative microscope videos and OCT B-scan images; and for establishing a three-dimensional volumetric data model of the intraoperative OCT images according to the method in step 1. V intra ; The deep learning model building and training module is used to design a dual-stream multimodal deep network that fuses high-frequency surface motion from microscope videos with low-frequency but scaled depth information from intraoperative OCT images to regress a dense 3D deformation field; and remaps the preoperative retinal model to the current anatomical morphology during surgery, thereby reflecting and compensating for retinal deformation in real time. Intraoperative registration and real-time compensation control module: used to map the three-dimensional spatial deformation field output by the deep learning model to the preoperative three-dimensional model coordinate system, calculate the real-time registration matrix and compare it with the position of the surgical instrument end, and adjust the instrument movement through the servo control system to achieve dynamic error compensation.

[0016] Thirdly, the present invention provides a computer-readable storage medium storing computer instructions which are executed by a processor using the method described above.

[0017] Fourthly, the present invention provides a computer program product, the computer program product storing computer instructions, the computer instructions being executed by a processor using the method described above.

[0018] The beneficial effects achieved by this invention are as follows: Efficient use of multi-rate data: High-frequency tracking is achieved using a 60FPS microscope, and low-frequency correction is achieved using a 1FPS OCT, perfectly combining the advantages of both.

[0019] Refined registration: By fusing microscopic and OCT multimodal features through deep learning, subpixel-level alignment between intraoperative images and preoperative models is achieved. ConvLSTM enables the network to learn the biomechanical properties of the retina, generating temporally smooth and physically plausible deformation sequences.

[0020] High-precision guidance: Automatically generates instrument compensation paths, with guidance errors controlled within 0.1 mm, improving accuracy by approximately 40% compared to traditional template registration algorithms.

[0021] High real-time performance: The system can achieve a real-time processing speed of 60 frames per second in a GPU environment, which meets the needs of microsurgical operations.

[0022] High robustness: It can maintain stable recognition and guidance performance under different lighting conditions, tissue deformation and blood obstruction conditions.

[0023] In summary, the intraoperative guidance method and system based on deep learning provided by this invention, by combining deep learning with intraoperative image registration technology, achieves a high degree of integration and precise matching between preoperative planning and intraoperative guidance, significantly improving the intelligence level and clinical application value of the surgical navigation system. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the drawings without creative effort.

[0025] Figure 1 A flowchart of a deep learning-based intraoperative guidance method provided by the present invention.

[0026] Figure 2 A schematic diagram of the microscope and OCT fusion encoding and decoding network structure provided by the present invention. Detailed Implementation

[0027] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are merely illustrative of the technical solution of the present invention and are therefore intended to limit the scope of protection of the present invention.

[0028] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used in this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention; the terms “comprising” and “having”, and any variations thereof, in the specification, claims and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0029] In the description of the embodiments of this invention, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this invention, "multiple" means two or more, unless otherwise explicitly defined.

[0030] In this invention, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least some embodiments of the invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this invention can be combined with other embodiments.

[0031] In the description of the embodiments of this invention, the term "and / or" is merely a description of the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, in this invention, the character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0032] like Figure 1 As shown, this invention provides a deep learning-based intraoperative guidance method, comprising the following steps: Step 1: Preoperative planning and model establishment; acquiring multi-frame OCT B-Scan slice sequence data from the patient's fundus. I OCT ( x, z The layer extraction algorithm of the three-dimensional U-Net network was used to segment the retinal layers of the OCT data, and the depth location functions of the main tissue interfaces in the retina were obtained: internal limiting membrane. S ILM ( x, z ), inner tufted layer S IPL ( x, z ), outer plexiform layer S OPL ( x, z ), external membrane S ELM ( x, z ), retinal pigment epithelium S PRE ( x, z The pixel coordinates of a certain segmented contour in the k-th B-scan are ( i, j Convert it to real-world three-dimensional coordinates: P xyz ( idx, kdy, jdz ); obtain the 3D point cloud data of the corresponding layer. P ILM , P IPL ; A 3D voxel mesh is constructed, and point cloud data from a single segmentation layer is extracted and set as the boundary. A fast nearest neighbor search algorithm is used to calculate the nearest distance from the mesh point to the point cloud. d Compare grid points with their neighbors. zThe value determines the distance sign, and a three-dimensional distance scalar field (Volume) is constructed to the point cloud surface. x, y, z ); Using the Marching Cubes algorithm, a threshold isoValue = 0 is set to represent the critical surface. The voxel space is scanned with unit cubes, and the intersection points are solved by linear interpolation at the boundary positions to form triangular patches. After traversing all voxels, the three-dimensional scalar field is converted into an isosurface triangular mesh. The above operation is repeated to obtain the {ILM, IPL, OPL, ELM, PRE} mesh model, which is then combined to generate a three-dimensional mesh model of the multilayer retinal tissue. Step 2: Intraoperative multimodal image acquisition and preprocessing; real-time acquisition of intraoperative microscope video (60FPS) and OCTB-scan images (1FPS); establishment of a three-dimensional volumetric data model from the intraoperative OCT images using the method in Step 1. V intra ; Step 3: Deep learning model construction and training; Design a dual-stream multimodal deep network to fuse high-frequency surface motion (optical flow / texture) from microscope videos with low-frequency but scaled depth information from intraoperative OCT images, regressing a dense 3D deformation field; Remap the preoperative retinal model to the current anatomical morphology during surgery, thereby reflecting and compensating for retinal deformation in real time. Step 4: Intraoperative registration and real-time compensation control; The three-dimensional spatial deformation field output by the deep learning model is mapped to the preoperative three-dimensional model coordinate system, the real-time registration matrix is ​​calculated and compared with the position of the surgical instrument end, and the instrument movement is adjusted through the servo control system to achieve dynamic error compensation.

[0033] In some embodiments, the deep learning model construction and training in step 3 includes designing an architecture consisting of a high frame rate tracking subnet and a low frame rate correction subnet; the overall system operation flow is shown in Figure 2, and step 3 specifically includes: Step 31: 2D-3D registration and feature projection network; associating high frame rate 2D microscope images with 3D space; firstly, based on the predicted deformation field from the previous time step. Φ(t - 1) Acting on preoperative three-dimensional spatial voxels V pre Obtain the current estimated 3D structure V deformed ( t-1 And using a differentiable renderer, learn the illumination and tissue optical properties of the microscope, and... V deformed ( t-1 Rendered into a virtual 2D microscope image I rendered (t) ; Using a shared-weight CNN backbone, respectively from I micro (t) and I rendered (t) Extracting high-dimensional feature maps F micro (t) and F rendered (t) Calculate the residual between the two feature maps. ΔF 2D (t) As the module output, it encodes the motion differences observed on the 2D image plane from the previous moment to the current moment; Step 32: Deformation update network; using a 3D recurrent convolutional network ConvLSTM as the core, with... Φ(t - 1) and the output of the previous module ΔF 2D (t) As input, the hidden state H(t-1) of the RNN encodes the dynamic history of retinal deformation, effectively modeling spatiotemporal sequence dependencies, and maintaining a hidden state to remember the deformation history of the retina; this module determines the deformation field Φ of the previous frame based on the new 2D driving features. pred (t-1) How needs to be updated to match the observations of the current frame; ultimately predict the current deformation field update Φ. pred (t)= Φ pred (t-1)+ ΔΦ(t); Whenever new intraoperative OCT data is available intra When (τ) arrives, it can be used as a monitoring signal source to calculate a true deformation field Φ. gt (t), calculate the predicted deformation field Φ pred (t) and the actual deformation field Φ gt Similarity loss between (t); Step 33: Define the loss function; Deformation field supervision loss (only effective when OCT data is updated) is used to calculate the difference between the predicted deformation field and the true deformation field calculated based on OCT data during intraoperative OCT data updates:

[0034] Deformation field smoothness loss, used to constrain the spatial smoothness of the predicted deformation field:

[0035] Photometric consistency loss, used to constrain the photometric consistency between optical flow calculated from the deformation field and the sequence of microscope images:

[0036] Deformation cycle consistency loss is used to constrain the invertibility of the deformation field:

[0037] The total loss function is defined as:

[0038] Configure weights based on task importance. ; Step 34: Staged model training to avoid convergence difficulties; 2D-3D registration and feature projection network renderer pre-trained, with fixed Φ pred (t)=0, using the preoperative microscopic OCT dataset, it is possible to obtain from V pre Generate realistic microscope images; train the deformation update network on sequences containing only OCT data; then, input ΔF 2D (t) can be set to random noise; finally, using complete microscopic and OCT surgical data, at the OCT update time τ, a strong supervisory signal is provided to correct the cumulative error of the entire system and teach the 2D flow how to contribute to accurate 3D deformation estimation.

[0039] This invention provides a deep learning-based intraoperative guidance method. In the preoperative stage, OCT image data is acquired, OCT layer structure features are extracted based on U-Net, and a three-dimensional model of the patient's retina is reconstructed. The doctor marks the drug injection boundary and the predetermined operation area in the three-dimensional model and generates the desired instrument path.

[0040] During the intraoperative phase, the microscope and OCT acquire image streams at different frequencies, which are then preprocessed and input into a deep learning network. The global encoder extracts features from the microscope images and combines them with the predicted deformation from the previous frame (initially zero, while the OCT update uses the actual deformation field) to predict the three-dimensional spatial deformation field, which is then applied to the preoperative three-dimensional spatial voxels of the retina to create a real-time retinal three-dimensional model. The local encoder, based on U-Net, extracts the updated OCT layered structure features to forcibly correct the three-dimensional model.

[0041] Based on the difference between this result and the preoperative planned path, the instrument guide outputs displacement compensation amounts Δx, Δy, and Δz. The control system updates the desired trajectory and adjusts the end effector motion of the robotic arm in real time through a force-position hybrid control mode to ensure that the instrument trajectory accurately follows the target.

[0042] In the experiment, image data from 20 retinal drug injection surgeries were selected for validation, and the method of the present invention was compared with that of the traditional rigid template registration algorithm. The results showed that the method of the present invention had an average registration error of 0.087 mm (46.2% lower than the control group), an instrument guidance error in the drug injection area of ​​less than 0.12 mm, and a model inference speed of 60 FPS.

[0043] Furthermore, the method of this invention can dynamically update the deep model parameters based on intraoperative feedback, achieving dynamic online learning. When new intraoperative OCT images arrive, the system uses them as strong supervisory signals, freezing the entire massive main network through a lightweight "fast adaptation layer," updating only the weights of this small portion of the "fast adaptation layer" to instantly fine-tune the pre-trained model. Simultaneously, a memory playback buffer is used to retain historical data to prevent forgetting. This mechanism enables the network to continuously self-optimize during surgery, adjusting its deformation field predictions in real time to adapt to the specific retinal characteristics of the patient and the unique dynamics of the current surgery, significantly improving accuracy and robustness in unknown scenarios.

[0044] In summary, the intraoperative guidance method based on deep learning provided by this invention can achieve real-time and precise registration of preoperative planning and intraoperative images in minimally invasive retinal microsurgery, and provide high-precision instrument guidance function, significantly improving surgical safety and efficiency, and has broad clinical application prospects.

[0045] Furthermore, the present invention also provides a deep learning-based intraoperative guidance system for implementing the aforementioned deep learning-based intraoperative guidance method, the system comprising: Preoperative planning and model building module; used to acquire multi-frame OCT B-Scan slice sequence data of the patient's fundus. I OCT ( x, z The layer extraction algorithm of the three-dimensional U-Net network was used to segment the retinal layers of the OCT data, and the depth location functions of the main tissue interfaces in the retina were obtained: internal limiting membrane. S ILM ( x, z ), inner tufted layer S IPL ( x, z ), outer plexiform layer S OPL ( x, z ), external membrane S ELM ( x, z ), retinal pigment epithelium S PRE ( x, z The pixel coordinates of a certain segmented contour in the k-th B-scan are ( i, j Convert it to real-world three-dimensional coordinates: P xyz ( idx, kdy, jdz ); obtain the 3D point cloud data of the corresponding layer. P ILM , P IPLA three-dimensional voxel mesh is constructed, and point cloud data from a single segmentation layer is extracted and set as the boundary. A fast nearest neighbor search algorithm is used to calculate the nearest distance from the mesh point to the point cloud. d Compare grid points with their neighbors. z The value determines the distance sign, and a three-dimensional distance scalar field (Volume) is constructed to the point cloud surface. x, y, z Using the Marching Cubes algorithm, a threshold isoValue = 0 is set to represent the critical surface. The voxel space is scanned with unit cubes, and the intersection points are solved by linear interpolation at the boundary positions to form triangular patches. After traversing all voxels, the three-dimensional scalar field is converted into an isosurface triangular mesh. The above operation is repeated to obtain the {ILM, IPL, OPL, ELM, PRE} mesh model, which is then combined to generate a three-dimensional mesh model of the multilayer retinal tissue. Intraoperative multimodal image acquisition and preprocessing module; used for real-time acquisition of intraoperative microscope videos and OCT B-scan images; and for establishing a three-dimensional volumetric data model of the intraoperative OCT images according to the method in step 1. V intra ; The deep learning model building and training module is used to design a dual-stream multimodal deep network that fuses high-frequency surface motion from microscope videos with low-frequency but scaled depth information from intraoperative OCT images to regress a dense 3D deformation field; and remaps the preoperative retinal model to the current anatomical morphology during surgery, thereby reflecting and compensating for retinal deformation in real time. Intraoperative registration and real-time compensation control module: used to map the three-dimensional spatial deformation field output by the deep learning model to the preoperative three-dimensional model coordinate system, calculate the real-time registration matrix and compare it with the position of the surgical instrument end, and adjust the instrument movement through the servo control system to achieve dynamic error compensation.

[0046] In some embodiments, the present invention provides a computer-readable storage medium storing computer instructions that are executed by a processor as in any of the above embodiments of a deep learning-based intraoperative guidance method.

[0047] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples (not an exhaustive list) of readable storage media may include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0048] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0049] Embodiments of the present invention may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in a deep learning-based intraoperative guidance method according to various embodiments of the present invention as described in the "Exemplary Methods" section above.

[0050] The steps of the method of the present invention are not limited to the specific order described above, unless otherwise specifically stated. Furthermore, in some embodiments, the invention may also be implemented as a program recorded on a recording medium, the program comprising machine-readable instructions for implementing the method according to the invention. Therefore, the invention also covers recording media storing programs for performing the method according to the invention.

[0051] Although the invention has been described with reference to preferred embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, the technical features mentioned in the various embodiments can be combined in any manner as long as there is no structural conflict. The invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A deep learning-based intraoperative guidance method, characterized in that, Includes the following steps: Step 1: Preoperative planning and model establishment; acquiring multi-frame OCT B-Scan slice sequence data from the patient's fundus. I OCT ( x,z The layer extraction algorithm of the three-dimensional U-Net network was used to segment the retinal layers of the OCT data, and the depth location functions of the main tissue interfaces in the retina were obtained: internal limiting membrane. S ILM ( x,z ), inner tufted layer S IPL ( x,z ), outer plexiform layer S OPL ( x,z ), external membrane S ELM ( x,z ), retinal pigment epithelium S PRE ( x,z The pixel coordinates of a certain segmented contour in the k-th B-scan are ( ij Convert it to real-world three-dimensional coordinates: P xyz ( idx, kdy, jdz ); obtain the 3D point cloud data of the corresponding layer. P ILM , P IPL A three-dimensional voxel mesh is constructed, and point cloud data from a single segmentation layer is extracted and set as the boundary. A fast nearest neighbor search algorithm is used to calculate the nearest distance from the mesh point to the point cloud. d Compare grid points with their neighbors. z The value determines the distance sign, and a three-dimensional distance scalar field (Volume) is constructed to the point cloud surface. x,y,z Using the Marching Cubes algorithm, a threshold isoValue = 0 is set to represent the critical surface. The voxel space is scanned with unit cubes, and the intersection points are solved by linear interpolation at the boundary positions to form triangular patches. After traversing all voxels, the three-dimensional scalar field is converted into an isosurface triangular mesh. The above operation is repeated to obtain the {ILM, IPL, OPL, ELM, PRE} mesh model, which is then combined to generate a three-dimensional mesh model of the multilayer retinal tissue. Step 2: Intraoperative multimodal image acquisition and preprocessing; real-time acquisition of intraoperative microscope videos and OCT B-scan images; establishment of a three-dimensional volumetric data model from the intraoperative OCT images using the method in Step 1. V intra ; Step 3: Deep learning model construction and training; Design a dual-stream multimodal deep network to fuse high-frequency surface motion from microscope videos with low-frequency but scaled depth information from intraoperative OCT images, regressing a dense 3D deformation field; Remap the preoperative retinal model to the current anatomical morphology during surgery, thereby reflecting and compensating for retinal deformation in real time. Step 4: Intraoperative registration and real-time compensation control; The three-dimensional spatial deformation field output by the deep learning model is mapped to the preoperative three-dimensional model coordinate system, the real-time registration matrix is ​​calculated and compared with the position of the surgical instrument end, and the instrument movement is adjusted through the servo control system to achieve dynamic error compensation.

2. The intraoperative guidance method based on deep learning according to claim 1, characterized in that, The deep learning model construction and training in step 3 includes designing an architecture consisting of a high frame rate tracking subnet and a low frame rate correction subnet. Step 3 specifically includes: Step 31: 2D-3D registration and feature projection network; associating high frame rate 2D microscope images with 3D space; firstly, based on the predicted deformation field from the previous time step. Φ(t-1) Acting on preoperative three-dimensional spatial voxels V pre Obtain the current estimated 3D structure V deformed ( t-1 And using a differentiable renderer, learn the illumination and tissue optical properties of the microscope, and... V deformed ( t-1 Rendered into a virtual 2D microscope image I rendered (t) ; Using a shared-weight CNN backbone, respectively from I micro (t) and I rendered (t) Extracting high-dimensional feature maps F micro (t) and F rendered (t) Calculate the residual between the two feature maps. ΔF 2D (t) As the module output, it encodes the motion differences observed on the 2D image plane from the previous moment to the current moment; Step 32: Deformation update network; using a 3D recurrent convolutional network ConvLSTM as the core, with... Φ(t-1) and the output of the previous module ΔF 2D (t) As input, the hidden state H(t-1) of the RNN encodes the dynamic history of retinal deformation, effectively modeling spatiotemporal sequence dependencies, and maintaining a hidden state to remember the deformation history of the retina; this module determines the deformation field Φ of the previous frame based on the new 2D driving features. pred (t-1) How needs to be updated to match the observations of the current frame; ultimately predict the current deformation field update Φ. pred (t)= Φ pred (t-1)+ ΔΦ(t); Whenever new intraoperative OCT data is available intra When (τ) arrives, a true deformation field Φ is calculated as the source of the monitoring signal. gt (t), calculate the predicted deformation field Φ pred (t) and the actual deformation field Φ gt Similarity loss between (t); Step 33: Define the loss function; Deformation field supervision loss is used to calculate the difference between the predicted deformation field and the actual deformation field calculated based on OCT during intraoperative OCT data updates. Deformation field smoothness loss, used to constrain the spatial smoothness of the predicted deformation field: Photometric consistency loss, used to constrain the photometric consistency between optical flow calculated from the deformation field and the sequence of microscope images: Deformation cycle consistency loss is used to constrain the reversibility of the deformation field; Step 34: Staged model training to avoid convergence difficulties; 2D-3D registration and feature projection network renderer pre-trained, with fixed Φ pred (t)=0, using the preoperative microscopic OCT dataset, it is possible to obtain from V pre Generate realistic microscope images; train the deformation update network on sequences containing only OCT data; then, input ΔF 2D (t) is set as random noise; finally, using complete microscopic and OCT surgical data, at the OCT update time τ, a strong supervisory signal is provided to correct the cumulative error of the entire system and teach the 2D flow how to contribute to accurate 3D deformation estimation.

3. The intraoperative guidance method based on deep learning according to claim 2, characterized in that, The deformation field monitoring loss in step 33 is: .

4. The intraoperative guidance method based on deep learning according to claim 3, characterized in that, The deformation field smoothness loss in step 33 is: .

5. The intraoperative guidance method based on deep learning according to claim 4, characterized in that, The photometric consistency loss in step 33 is: .

6. The intraoperative guidance method based on deep learning according to claim 5, characterized in that, The deformation cycle consistency loss in step 33 is: .

7. The intraoperative guidance method based on deep learning according to claim 6, characterized in that, The total loss function in step 33 is defined as follows: Configure weights based on task importance. .

8. A deep learning-based intraoperative guidance system, characterized in that, The system for implementing the deep learning-based intraoperative guidance method according to any one of claims 1 to 7, the system comprising: Preoperative planning and model building module; used to acquire multi-frame OCT B-Scan slice sequence data of the patient's fundus. I OCT ( x,z The layer extraction algorithm of the three-dimensional U-Net network was used to segment the retinal layers of the OCT data, and the depth location functions of the main tissue interfaces in the retina were obtained: internal limiting membrane. S ILM ( x,z ), inner tufted layer S IPL ( x,z ), outer plexiform layer S OPL ( x,z ), external membrane S ELM ( x,z ), retinal pigment epithelium S PRE ( x,z The pixel coordinates of a certain segmented contour in the k-th B-scan are ( ij Convert it to real-world three-dimensional coordinates: P xyz ( idx, kdy, jdz ); obtain the 3D point cloud data of the corresponding layer. P ILM , P IPL A three-dimensional voxel mesh is constructed, and point cloud data from a single segmentation layer is extracted and set as the boundary. A fast nearest neighbor search algorithm is used to calculate the nearest distance from the mesh point to the point cloud. d Compare grid points with their neighbors. z The value determines the distance sign, and a three-dimensional distance scalar field (Volume) is constructed to the point cloud surface. x,y,z Using the Marching Cubes algorithm, a threshold isoValue = 0 is set to represent the critical surface. The voxel space is scanned with unit cubes, and the intersection points are solved by linear interpolation at the boundary positions to form triangular patches. After traversing all voxels, the three-dimensional scalar field is converted into an isosurface triangular mesh. The above operation is repeated to obtain the {ILM, IPL, OPL, ELM, PRE} mesh model, which is then combined to generate a three-dimensional mesh model of the multilayer retinal tissue. Intraoperative multimodal image acquisition and preprocessing module; used for real-time acquisition of intraoperative microscope videos and OCT B-scan images; and for establishing a three-dimensional volumetric data model of the intraoperative OCT images according to the method in step 1. V intra ; The deep learning model building and training module is used to design a dual-stream multimodal deep network that fuses high-frequency surface motion from microscope videos with low-frequency but scaled depth information from intraoperative OCT images to regress a dense 3D deformation field; and remaps the preoperative retinal model to the current anatomical morphology during surgery, thereby reflecting and compensating for retinal deformation in real time. Intraoperative registration and real-time compensation control module: used to map the three-dimensional spatial deformation field output by the deep learning model to the preoperative three-dimensional model coordinate system, calculate the real-time registration matrix and compare it with the position of the surgical instrument end, and adjust the instrument movement through the servo control system to achieve dynamic error compensation.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are executed by a processor according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product stores computer instructions, which are executed by a processor according to any one of claims 1 to 7.

Images