Thoracoscopic surgery image display method, system and equipment and storage medium
By constructing a three-dimensional model based on medical image data, combining laparoscopic image matching to identify blood vessels and lung clefts, the problem of invisibility of internal lung images in laparoscopic surgery is solved, and visual navigation of internal lung information is achieved.
Patent Information
- Application Number
- CN202510590631.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-08
- Publication Date
- 2025-08-15
AI Technical Summary
In existing thoracoscopic surgery, laparoscopic cameras cannot obtain internal images of the lungs, especially when bleeding causes blurred local areas and cannot clearly observe tissue structure.
The surface images of the lungs are segmented through medical imaging data, and a three-dimensional model containing blood vessels and lung fissures is constructed. The matching identification of laparoscopic images and medical imaging data is used to display the matching blood vessels and lung fissure areas in the three-dimensional model, providing internal information of the lungs that cannot be detected by laparoscopic.
When the lung surface is bleeding, image features of local areas can be obtained from the three-dimensional model to provide doctors with navigation and ensure the accuracy and visibility of the surgery.
Smart Images
Figure CN120496760A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of surgical navigation, and more specifically, to a method, system, electronic device, and storage medium for displaying thoracoscopic surgical images. Background Art
[0002] Video-Assisted Thororacic Surgery (VATS) is a minimally invasive surgical technique that uses tiny incisions and thoracic equipment to diagnose and treat chest diseases. Compared to traditional open thoracotomy, VATS is less invasive and has a faster recovery time, making it an important technique in the field of thoracic surgery. In lung surgery, VATS is suitable for biopsy of lung nodules or masses, resection of early lung cancer (lobectomy, segmentectomy), removal of bullae (for pneumothorax), and lung volume reduction surgery (for emphysema).
[0003] A laparoscope is a medical device that integrates optical imaging, lighting, and an operating channel. During surgery, doctors can use an optical imaging system (such as a camera) to observe the lung surface in real time and locate the site of lesions. However, since the laparoscope's camera can only capture images of the lung surface and not the interior, bleeding during surgery can cause blurry images of certain areas, making it difficult to clearly see the tissue structure. Summary of the Invention
[0004] The purpose of this application is to provide a thoracoscopic surgery image display method, system, electronic device and storage medium to solve the above-mentioned problems existing in the prior art.
[0005] In a first aspect, a method for displaying thoracoscopic surgery images is provided, comprising the following steps:
[0006] S1: Based on medical imaging data, the lung surface image is segmented according to the lung surface blood vessels or lung fissures;
[0007] S2: constructing a 3D lung model including blood vessels or lung fissures based on the segmented image;
[0008] S3: Acquire laparoscopic images of the lungs through the laparoscope camera during surgery;
[0009] S4: Identify the laparoscopic image, extract the blood vessels or pulmonary fissures in the laparoscopic image, match and identify the blood vessels or pulmonary fissures in the laparoscopic image with the blood vessels or pulmonary fissures in the medical imaging data, and extract the matched blood vessels or pulmonary fissures;
[0010] S5: Based on the transformation relationship between the medical image data and the three-dimensional model, the area of the three-dimensional model containing the matching blood vessels or lung fissures is displayed on the display device.
[0011] Optionally, the medical imaging data is a CT or MRI image, and a method for matching blood vessels in the laparoscopic image with blood vessels in the medical imaging data is as follows:
[0012] Extract blood vessels in laparoscopic images and blood vessel bifurcation points in medical imaging data respectively;
[0013] Extracting geometric features of vascular bifurcation points, including branch angle, branch length, and vessel diameter ratio;
[0014] The vascular bifurcation points in the laparoscopic image are screened for geometric features that match those in the medical imaging data.
[0015] Optionally, a method for screening vascular bifurcation points in the laparoscopic image that match the geometric features of vascular bifurcation points in the medical imaging data is as follows:
[0016] Extract vascular models from medical imaging data;
[0017] According to the viewing angle parameters of the laparoscopic camera, the vascular model of the medical imaging data is projected onto a two-dimensional plane to generate a virtual laparoscopic image;
[0018] The RANSAC algorithm is used to screen the vascular bifurcation points in the laparoscopic image whose geometric features match those in the virtual laparoscopic image.
[0019] Optionally, a method for matching the lung fissure in the laparoscopic image with the lung fissure in the medical imaging data is as follows: extracting local curvature extreme points of the lung fissure in the medical imaging data;
[0020] Perform 3D point cloud reconstruction on the laparoscopic image to generate lung surface point cloud and propose local curvature extreme points of lung fissures;
[0021] The extreme points in the laparoscopic image whose similarity with the local curvature extreme points of the medical imaging data reaches a preset value are screened out. Based on the screened extreme points, the distance values between the extreme point and the surrounding N extreme points and the distance values between the corresponding extreme point in the medical imaging data and the surrounding N extreme points are further calculated, and it is judged whether the similarity between the two sets of distance values reaches the preset value. If the preset value is reached, it is determined that the lung fissure indicated by the extreme point matches the lung fissure corresponding to the extreme point in the medical imaging data.
[0022] Optionally, the local curvature extreme point similarity refers to the ratio of the curvatures of the extreme points. The similarity between distance values refers to (d1 / D1+d2 / D2+d3 / D3+…+d N / D N ) / N; where d1, d2, d3, ..., d N They refer to the distances from the local curvature extreme point of the lung fissure in the laparoscopic image to the surrounding N local curvature extreme points in ascending order, D1, D2, D3, ..., D NThey refer to the distances from the corresponding extreme point of local curvature of the pulmonary fissure in the medical image data to the surrounding N extreme points of local curvature in ascending order.
[0023] Optionally, the display device is AR glasses or a display screen.
[0024] In the second aspect, a thoracoscopic surgery image display system is provided, including a medical imaging device, a display device, a laparoscope, a three-dimensional model construction module and a data processing module. The medical imaging device, the display device, the laparoscope, the three-dimensional model construction module and the data processing module are used to implement the thoracoscopic surgery image display method of the first aspect.
[0025] In a third aspect, an electronic device is provided, comprising a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of implementing any one of the methods of the first aspect are performed.
[0026] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any method of the first aspect are implemented.
[0027] Beneficial effects: By combining the three-dimensional model with the laparoscopic image, it can provide internal lung information that cannot be detected by the laparoscope. When bleeding occurs on the surface of the lung and causes unclear images in a local area, the image features of the area can also be obtained from the constructed three-dimensional model, providing navigation for doctors' laparoscopic surgery. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.
[0029] Figure 1 A flowchart of a method for displaying thoracoscopic surgery images provided in an embodiment of the present application;
[0030] Figure 2 A surgical scenario in which the thoracoscopic surgical image display system according to an embodiment of the present application is applied;
[0031] Figure 3 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0032] The following, in conjunction with the accompanying drawings, provides a clear and complete description of the technical solutions in the embodiments of this application. Obviously, the described embodiments represent only a portion of the embodiments of this application and do not constitute a complete set of embodiments. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of this application without inventive effort are intended to fall within the scope of protection of this application. Unless otherwise defined, technical or scientific terms used in this application should have the same ordinary meanings as those understood by persons of ordinary skill in the art. The terms "first," "second," and similar expressions used in this application do not denote any order, quantity, or importance; they are merely used to distinguish between different components. Terms such as "include" or "comprising" mean that the element or object preceding the term includes the elements or objects listed after the term, and their equivalents, without excluding other elements or objects. Terms such as "connect," "couple," or "connected" are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used solely to indicate relative positional relationships. When the absolute position of the described objects changes, the relative positional relationships may also change accordingly.
[0033] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application can be combined with each other if there is no conflict.
[0034] The thoracoscopic surgery image display method provided in the embodiment of the present application is as follows: Figure 1 As shown, the following steps are included:
[0035] S1: Based on medical imaging data, segment the lung surface image according to the lung surface blood vessels or fissures. In medical imaging, segmenting the lung surface image based on the lung surface blood vessels or fissures is a key step in accurately locating lung lobes, lung segments, and lesions. The following is a detailed segmentation method and technical process for CT, MRI, and other images:
[0036] Fissure segmentation (for pulmonary lobe segmentation): Fissures are anatomical structures that separate the pulmonary lobes. They appear as linear high-density shadows in CT images and as low-signal bands in MRI. The segmentation process is as follows:
[0037] Data preprocessing: Anisotropic diffusion filtering was used to reduce noise while preserving the edges of the lung fissures, and CLAHE (contrast-limited adaptive histogram equalization) was used to enhance the contrast between the lung fissures and surrounding lung tissue.
[0038] Pulmonary fissure segmentation method: In one embodiment, threshold segmentation can be performed by extracting high-density areas of the pulmonary fissures (HU values of approximately -500 to -200) from the CT scan. The broken edges of the pulmonary fissures are connected using a morphological closing operation. The pulmonary fissures are manually or semi-automatically traced in the coronal and sagittal planes to synthesize a 3D model. In another embodiment, segmentation can also be performed using deep learning-based methods, such as 3DU-Net and nnUNet, which are used to process 3D volume data (such as CT and MRI) and can achieve accurate segmentation of fine structures such as pulmonary fissures and blood vessels.
[0039] Pulmonary vascular segmentation (for lung segmentation): Pulmonary vascular images show a high-density tree-like structure in CT. The segmentation goal is to extract the pulmonary artery and vein, and then further extract the skeleton model of the vascular structure. The specific segmentation method is as follows:
[0040] Data preprocessing: Tubular structures are enhanced and background noise is suppressed through Frangi filters. Frangi filters are an image processing technology based on the Hessian matrix for enhancing tubular structures (such as blood vessels, bronchi, etc.) in images. By analyzing local geometric features, slender tubular areas are highlighted while suppressing background noise and other anatomical structures. The Hessian matrix is a matrix used to describe the local curvature of a function. In the embodiment of this application, it is mainly used for detecting tubular structures (such as blood vessels), edge enhancement, and feature extraction.
[0041] Blood vessel segmentation method: In one embodiment, a blood vessel starting point can be selected at the hilum of the lung, and the blood vessel region can be expanded based on grayscale and gradient constraints to perform region growing. In another embodiment, a deep learning method, such as the VesselNet deep learning model, can also be used. The VesselNet deep learning model is a deep learning model designed for three-dimensional data and is used for blood vessel segmentation tasks. After segmentation, the blood vessel centerline is extracted to form a blood vessel skeleton, and the lung segments (such as the anterior and posterior segments of the right upper lobe) are divided according to the topological structure of the blood vessel tree.
[0042] In addition, in the embodiment of the present application, the pulmonary fissure and blood vessel segmentation can be combined, and the pulmonary fissure segmentation results can be combined with the blood vessel skeleton to mark the lung segment boundaries.
[0043] S2: Construct a 3D model of the lung containing the vascular fissure based on the segmented image. In medical image processing, constructing a 3D model containing identification landmarks (such as vascular bifurcation points and pulmonary fissure curvature extremes) based on segmented lung images (such as CT or MRI segmentation results) is a key step in surgical navigation, lesion localization, and dynamic deformation analysis. The following is a detailed technical process and implementation method:
[0044] Data preprocessing: The input data includes segmented binary masks (such as 3D binary images of lungs, blood vessels, and lung fissures) and original grayscale images for extracting intensity features (such as HU values of CT). Ensure that the resolution of all modal data is consistent, such as 1×1×1mm 3 , the segmentation result is aligned through rigid registration to unify the coordinate system with the original image.
[0045] 3D reconstruction and mesh generation:
[0046] 1. Lung surface reconstruction: The MarchingCubes algorithm is used to extract isosurfaces from binary masks and generate triangular mesh models for lung surface reconstruction. The MarchingCubes algorithm is a classic method for extracting isosurfaces (such as organ surfaces) from three-dimensional volume data (such as CT and MRI) and generating triangular mesh models. It is widely used in three-dimensional reconstruction of medical images. Given three-dimensional volume data and a target threshold (such as the HU value of lung tissue in CT), the algorithm traverses each voxel (cube unit), determines the relationship between its vertex value and the threshold, and generates an isosurface (a set of triangular facets); the three-dimensional data is divided into multiple small cubes (voxels), each voxel consisting of 8 vertices. Laplace smoothing is used to reduce surface jaggedness and optimize mesh quality.
[0047] 2. Vascular and pulmonary fissure reconstruction: Extract the vascular segmentation mask and generate the vascular centerline skeleton; generate the vascular surface using the Ball-Pivoting algorithm or Poisson reconstruction; extract the pulmonary fissure mask, generate a smooth surface, and mark the extreme points of curvature (the deepest concave areas).
[0048] S3: Laparoscopic images of the lungs are obtained using the laparoscope's camera during surgery. A laparoscope is a medical device that integrates optical imaging, illumination, and an operating channel. It enters the body through natural orifices or tiny incisions and is used for direct inspection, diagnosis, and minimally invasive surgery. Images are directly captured by a front-end CMOS / CCD sensor, which converts them into electrical signals and transmits them to a display.
[0049] S4: Identify the laparoscopic image, extract the blood vessels or pulmonary fissures in the laparoscopic image, match and identify the blood vessels or pulmonary fissures in the laparoscopic image with the blood vessels or pulmonary fissures in the medical imaging data, and extract the matched blood vessels or pulmonary fissures. In this embodiment of the application, the medical imaging data is a CT or MRI image, and the matching method of the blood vessels in the laparoscopic image and the blood vessels in the medical imaging data is as follows:
[0050] The blood vessels in the laparoscopic images and the vascular bifurcation points in the medical imaging data are extracted respectively; in the laparoscopic images, the vascular bifurcation is located by Harris corner detection or HRNet deep learning model. Harris corner detection is used to identify corner points in the image to achieve feature point detection. The HRNet deep learning model, full name of which is High-ResolutionNet (high-resolution network), is used to extract topological nodes from the segmented vascular model, traverse the skeleton branches, and count the number of connections of each node. The bifurcation points must meet the topological rules of "one in and two out" or "one in and three out". The centerline and branch attributes (length, diameter) are generated by the skeletonization algorithm.
[0051] After the vascular bifurcation point is determined, the geometric features of the vascular bifurcation point are further extracted, including branch angle, branch length and vascular diameter ratio.
[0052] The laparoscopic image is screened for vascular bifurcation points whose geometric features match those in the medical imaging data. The specific screening method is as follows: the vascular model in the medical imaging data is extracted and, based on the viewing angle parameters of the laparoscopic camera, the vascular model from the medical imaging data is projected onto a two-dimensional plane to generate a virtual laparoscopic image. The RANSAC algorithm is then used to screen for vascular bifurcation points in the laparoscopic image whose geometric features match those in the virtual laparoscopic image. RANSAC (Random Sample Consensus) is an iterative method for estimating mathematical model parameters from a dataset containing outliers.
[0053] The matching method of the lung fissures in the laparoscopic image and the lung fissures in the medical imaging data is as follows: extract the local curvature extreme points of the lung fissures in the medical imaging data; perform 3D point cloud reconstruction on the laparoscopic image to generate a lung surface point cloud and propose the local curvature extreme points of the lung fissures; screen out the extreme points in the laparoscopic image whose similarity with the local curvature extreme points of the medical imaging data reaches a preset value, and further calculate the distance value between the extreme point and the surrounding N extreme points and the distance value between the corresponding extreme point in the medical imaging data and the surrounding N extreme points based on the screened extreme points, and judge whether the approximation between the two sets of distance values reaches the preset value. If the preset value is reached, the lung fissure of the extreme point is determined to match the lung fissure of the corresponding extreme point in the medical imaging data. The local curvature extreme point similarity refers to the ratio between the curvatures of the extreme points. If the ratio is between 0.95 and 1.05, the two are considered to be similar. The approximation between the distance values refers to (d1 / D1+d2 / D2+d3 / D3+…+d N / D N ) / N, where d1, d2, d3…d N They refer to the distances from the local curvature extreme point of the pulmonary fissure in the laparoscopic image to the surrounding N local curvature extreme points in ascending order, D1, D2, D3…D NThese values refer to the distances from the corresponding local curvature extreme point of the lung fissure in the medical imaging data to the N surrounding local curvature extreme points, arranged in ascending order. The selection logic is as follows: First, points with similar local curvature extremes are selected in both modalities. The distance values are then used to determine whether the spatial distribution of these points is similar, i.e., the arrangement pattern of the curvature extreme points in three-dimensional space. Finally, the matching of the lung fissures in both modalities is determined.
[0054] S5: Based on the transformation relationship between the medical image data and the 3D model, the area of the 3D model containing the matching blood vessels or lung fissures is displayed on a display device. The display device can be a display screen or AR glasses. The doctor can perform operations such as zooming in, out, rotating, and creating sections on the 3D model.
[0055] In an embodiment of the present application, by combining a three-dimensional model with a laparoscope, internal information of the lungs that cannot be detected by the laparoscope can be provided. When bleeding occurs on the surface of the lungs, resulting in unclear images of local areas, the image features of the area can also be obtained from the constructed three-dimensional model, providing navigation for the doctor's laparoscopic surgery.
[0056] Based on the same inventive concept, the present application also provides a thoracoscopic surgery image display system in an embodiment, including a medical imaging device, a display device, a laparoscope, a three-dimensional model construction module and a data processing module. The three-dimensional model construction module can be a physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The three-dimensional model construction module converts the image of the medical imaging device into a three-dimensional model. Similarly, the data processing module can also be a physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The data processing module is mainly used for image processing, recognition, 3D point cloud modeling, etc. The three-dimensional model building module and the data processing module can exist independently and can be merged into one physical server. The medical imaging device can be a computed tomography (CT): which generates cross-sectional images of the body by using X-rays and complex computer processing. CT scans can provide more detailed image information than ordinary X-rays. Magnetic resonance imaging (MRI): uses powerful magnetic fields and radio waves to create detailed images of organs and structures in the body. Ultrasound imaging equipment: uses high-frequency sound waves to produce real-time images of structures in the body. Positron emission tomography (PET): A nuclear medicine imaging technology that generates three-dimensional images by detecting the distribution of radioactive substances in the body. The embodiments of the present application use CT equipment or MRI. Figure 2 This is a surgical scenario in which the thoracoscopic surgery image display system of the embodiment of the present application is applied.
[0057] The smart head-mounted display device in the embodiment of the present application uses AR glasses. The AR glasses can have a built-in data processing module or an external data processing module (connected to the data processing module on the server through the network). The data processing module is used to calculate the position of the patch, perform image processing, etc. In the embodiment of the present application, whether the AR glasses use a built-in or external data processing module, they all fall into the category of smart head-mounted display devices.
[0058] In addition, the present application also provides an electronic device, such as Figure 3As shown, it is a schematic diagram of the structure of the electronic device 400 provided in an embodiment of the present application, including: a processor 41, a memory 42, and a bus 43; the memory 42 is used to store execution instructions, including a memory 421 and an external memory 422; the memory 421 here is also called an internal memory, which is used to temporarily store the operation data in the processor 41 and the data exchanged with the external memory 422 such as a hard disk. The processor 41 exchanges data with the external memory 422 through the memory 421. When the electronic device 400 is running, the processor 41 and the memory 42 communicate through the bus 43, so that the processor 41 executes Figure 1 Steps of the thoracoscopic surgery image display method.
[0059] The present application also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program executes the steps of the thoracoscopic surgery image display method in the above-mentioned method embodiment. The storage medium can be a volatile or non-volatile computer-readable storage medium.
[0060] An embodiment of the present application also provides a computer program product, which includes computer instructions. When the computer instructions are executed by a processor, the steps of the thoracoscopic surgery image display method in the above method embodiment can be executed. For details, please refer to the above method embodiment and will not be repeated here.
[0061] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).
[0062] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.
[0063] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.
[0064] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The communication bus mentioned above may be a Peripheral Component Interconnect standard bus or an Extended Industry Standard Architecture bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.
[0065] The communication interface is used for communication between the above electronic device and other devices.
[0066] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk memory. Alternatively, the memory may be at least one storage device located away from the processor.
[0067] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0068] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods and systems according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.
[0069] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.
[0070] The computer program for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The computer program can be executed entirely on the target object computing device, partially on the target object device, as a separate software package, partially on the target object computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the target object computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).
[0071] It should be noted that although several units or subunits of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the application, the features and functions of two or more units described above can be embodied in a single unit. Conversely, the features and functions of a single unit described above can be further divided and embodied by multiple units.
[0072] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.
[0073] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods or systems. Therefore, the present application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware.
[0074] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.
[0075] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.
Claims
1. A method for displaying thoracoscopic surgery images, comprising the steps of: S1: Segment the lung surface image according to the lung surface blood vessels or lung fissures based on medical imaging data; S2: Construct a 3D lung model including blood vessels or lung fissures based on the segmented image; S3: Acquire laparoscopic images of the lungs through the laparoscope camera during surgery; S4: Identify the laparoscopic image, extract the blood vessels or lung fissures in the laparoscopic image, match and identify the blood vessels or lung fissures in the laparoscopic image with the blood vessels or lung fissures in the medical imaging data, and extract the matched blood vessels or lung fissures; S5: Based on the transformation relationship between the medical image data and the three-dimensional lung model, the region of the three-dimensional model containing the matching blood vessels or lung fissures is displayed on a display device.
2. The thoracoscopic surgery image display method according to claim 1, wherein: Matching and identifying blood vessels in laparoscopic images with those in medical imaging data includes: Extract blood vessels in laparoscopic images and blood vessel bifurcation points in medical imaging data respectively; Extracting geometric features of a blood vessel bifurcation point, wherein the geometric features include a branch angle, a branch length, and a blood vessel diameter ratio; The vascular bifurcation points in the laparoscopic image are screened for geometric features that match those in the medical imaging data.
3. The thoracoscopic surgery image display method according to claim 2, wherein: Screening of vascular bifurcation points in laparoscopic images that match the geometric features of vascular bifurcation points in medical imaging data, including: Extract vascular models from medical imaging data, According to the viewing angle parameters of the laparoscopic camera, the vascular model of the medical imaging data is projected onto a two-dimensional plane to generate a virtual laparoscopic image; The RANSAC algorithm is used to screen the vascular bifurcation points in the laparoscopic image whose geometric features match those in the virtual laparoscopic image.
4. The thoracoscopic surgery image display method according to claim 1, characterized in that: Matching and identifying lung fissures in laparoscopic images with lung fissures in medical imaging data includes: Extract local curvature extreme points of lung fissures in medical imaging data; Perform 3D point cloud reconstruction on the laparoscopic image to generate lung surface point cloud and propose local curvature extreme points of lung fissures; Filter out the extreme points in the laparoscopic image whose similarity with the local curvature extreme points of the medical imaging data reaches a preset value; Based on the screened extreme point, calculate the distance between the extreme point and the surrounding N extreme points, as well as the distance between the corresponding extreme point in the medical imaging data and the surrounding N extreme points; It is determined whether the approximation between the two sets of distance values reaches a preset value. If so, it is determined that the lung fissure indicated by the extreme value point matches the lung fissure corresponding to the extreme value point in the medical imaging data.
5. The thoracoscopic surgery image display method according to claim 4, characterized in that: The local curvature extreme point similarity refers to the ratio between the curvatures of the extreme points.
6. The thoracoscopic surgery image display method according to claim 5, characterized in that: The similarity between the distance values is (d1 / D1+d2 / D2+d3 / D3+…+d N / D N ) / N; where d1, d2, d3, ..., d N They refer to the distances from the local curvature extreme point of the lung fissure in the laparoscopic image to the surrounding N local curvature extreme points in ascending order, D1, D2, D3, ..., D N They refer to the distances from the corresponding extreme point of local curvature of the pulmonary fissure in the medical image data to the surrounding N extreme points of local curvature in ascending order.
7. The thoracoscopic surgery image display method according to claim 2, wherein: Its characteristics are: The display device is AR glasses or a display screen.
8. A thoracoscopic surgery image display system, comprising a medical imaging device, a display device, a laparoscope, a three-dimensional model building module, and a data processing module, characterized in that: The medical imaging equipment, display device, laparoscope, three-dimensional model construction module and data processing module are used to implement the thoracoscopic surgery image display method according to any one of claims 1-6.
9. An electronic device, characterized in that: include: A processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the method according to any one of claims 1 to 6 are performed.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.