Implicit neural network construction method for three-dimensional lung CT tubular structure segmentation based on topology prior
The implicit neural network addresses memory and resolution challenges in lung CT tubular structure segmentation by integrating topological priors and centerline guidance, achieving high-resolution, accurate, and efficient segmentation of complex lung structures.
Patent Information
- Application Number
- CN202411259330.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-09
- Publication Date
- 2024-12-13
AI Technical Summary
Existing methods for three-dimensional lung CT tubular structure segmentation face challenges such as memory limitations, imbalance between resolution and accuracy, category imbalance, and structural complexity, leading to discontinuous segmentation results and inefficiencies in handling complex tubular structures.
An implicit neural network construction method that integrates topological prior information, uses a feature pyramid encoder and dual-branch implicit decoder, and employs a centerline prediction auxiliary task to optimize segmentation results, enabling high-resolution output while reducing memory usage and addressing scale and foreground-background imbalance.
The method generates continuous and accurate tubular structure segmentations of arbitrary resolution, effectively handling complex lung structures with reduced memory consumption and improved accuracy by focusing on foreground information and maintaining spatial integrity.
Smart Images
Figure CN119131062A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of medical image processing, and in particular relates to an implicit neural network construction method for three-dimensional lung CT tubular structure segmentation based on topological prior. Background Art
[0002] In clinical practice, lung diseases such as lung cancer, emphysema, and lung infection have a high incidence rate. Accurate diagnosis and assessment of disease progression are essential for developing effective treatment plans. Three-dimensional reconstruction of tubular structures in the lungs, including trachea and blood vessels, is an important basis for automated diagnostic procedures and quantitative analysis and evaluation. With the advent of deep learning, medical image segmentation has become a prominent research field. Many studies have used deep learning methods to segment tubular structures such as arteries, veins, and trachea in lung CT scans.
[0003] 1. CNN-based solutions (such as 3DU-Net) for tubular structure segmentation. Improved versions of this method focus on voxel connectivity or topology, such as emphasizing accurate airway segmentation through 26-voxel connection analysis. Although these methods can achieve certain results, they still have the following disadvantages:
[0004] (1) These methods need to strike a balance between segmentation accuracy and memory usage. Using larger resolution images as input can lead to increased memory usage, while using downsampling or sliding window pruning methods may result in a reduction in segmentation resolution or loss of global spatial information.
[0005] (2) The segmentation of pulmonary duct structures has challenges related to imbalanced categories, complex structures, and imbalanced scales. Commonly used segmentation methods often produce discontinuous segmentation results in smaller anatomical regions.
[0006] 2. In the segmentation of medical images, some methods have been used that use implicit neural representation (INR). However, there are still few technical solutions for the tubular structure of lung 3D CT, which cannot handle complex tubular structure segmentation.
[0007] In the field of medical image processing, especially the segmentation of tubular structures such as blood vessels and airways in lung CT images, it is a key step in clinical diagnosis and treatment planning. However, existing technologies have several limitations and challenges in handling such tasks:
[0008] 1. Memory limitations: Most existing segmentation methods are limited by memory capacity when processing high-resolution images, which restricts their ability to handle large-scale datasets.
[0009] 2. Balance between resolution and accuracy: To reduce memory usage, existing methods may need to downsample images or adopt sliding window cropping, which may result in reduced segmentation resolution or loss of global spatial information.
[0010] 3. Category imbalance: The segmentation task of lung tubular structures often faces the problem of category imbalance, that is, tubular structures of different scales are unevenly distributed in the image, making it difficult for the segmentation algorithm to accurately identify all structures.
[0011] 4. Structural complexity: Lung tubular structures such as blood vessels and airways have a high degree of structural complexity, making it difficult for existing methods to maintain the continuity and integrity of the segmentation results. Summary of the invention
[0012] The purpose of the present invention is to provide an implicit neural network construction method for three-dimensional lung CT tubular structure segmentation based on topological prior, which uses an implicit neural network to generate continuous coordinate points, reduces the amount of calculation and parameters, and can reduce memory usage while generating high-resolution segmentation results; introduces a topological sampling strategy and centerline guidance, which helps to solve the problem of foreground-background imbalance and alleviate the problem of scale imbalance.
[0013] To achieve the above-mentioned purpose, the technical solution of the present invention is: a method for constructing an implicit neural network for three-dimensional lung CT tubular structure segmentation based on topological prior, constructing an implicit neural network for three-dimensional lung CT tubular structure segmentation based on topological prior, using implicit neural network to learn the continuous representation of three-dimensional lung CT tubular structure segmentation results, and reconstructing three-dimensional lung CT tubular structure segmentation results of arbitrary resolution; in addition, by integrating the topological structure information of the three-dimensional lung CT tubular structure, dense sampling of small branches is performed; in addition, a centerline prediction auxiliary task is used to optimize the three-dimensional lung CT tubular structure segmentation results.
[0014] In one embodiment of the present invention, the implicit neural network includes a feature pyramid encoder, a point feature encoder and a dual-branch implicit decoder.
[0015] In one embodiment of the present invention, the feature pyramid encoder is composed of a network structure consisting of four convolution modules, each of which includes a 3×3×3 convolution layer, a batch normalization layer, a linear rectification layer and a downsampling layer.
[0016] In one embodiment of the present invention, the implicit neural network uses the three-dimensional CT image as input, is first processed by the feature pyramid encoder, and generates a series of feature matrices F1, F2, ..., F at different scales. n ,Each feature matrix carries information about different dimensions and levels of the image;
[0017] After the image feature extraction is completed, the point feature encoder is used to sample and encode the feature matrices of different scales to form the point feature code F(p);
[0018] The point feature code F(p) is input into the dual-branch implicit decoder for decoding. The decoding process is implemented by a multi-layer perceptron, whose main task is to predict the occupancy rate of the lung trachea category at each position and reconstruct the three-dimensional lung CT tubular structure segmentation result of arbitrary resolution.
[0019] In one embodiment of the present invention, the point feature encoder is used to sample and encode feature matrices of different scales to form point feature codes F(p), which are specifically as follows:
[0020] The feature matrix of the three-dimensional CT image is obtained by the feature pyramid encoder in the form of H×W×D discrete form; then the discrete data is implicitly mapped, and each coordinate of the feature matrix of size H×W×D is mapped to a continuous spatial interval h×ω×d (h, ω, d∈R[-1,1]); then, continuous sampling coordinate points p(x, y, z)(x, y, z∈R[-1,1]) are input, and a series of features corresponding to corresponding points at different scales are generated by performing trilinear interpolation queries, namely F1(p), F2(p)...F n (p); Then, each sampling coordinate point p and its features F at different scales are n (p) to form the point feature code F(p).
[0021] In one embodiment of the present invention, the point feature code F(p) is input into a dual-branch implicit decoder for decoding, as follows:
[0022] Two multilayer perceptrons are used as dual-branch implicit decoders to obtain the segmentation result s′ and the centerline extraction result c′, respectively, which can be expressed as:
[0023] s′=D s (F(p))
[0024] c′=D c (F(p))
[0025] In the formula, D s is the vascular implicit decoding module, D c It is the centerline decoding module.
[0026] In one embodiment of the present invention, the centerline prediction auxiliary task is used to optimize the segmentation result of the tubular structure of the three-dimensional lung CT, specifically as follows:
[0027] The segmentation loss function is:
[0028]
[0029] The centerline loss function is:
[0030]
[0031] In the formula, s and c represent the true labels of the segmentation and centerline, respectively, and T prec( s, c′) is the intersection of the predicted center line and the segmentation mark, T sens( s′, c) is the intersection of the centerline true label and the segmentation. The true label of the centerline is obtained by binarizing the centerline with radius information.
[0032] In one embodiment of the present invention, the topological structure information of the tubular structure of the three-dimensional lung CT is integrated to densely sample the small branches, that is, different strategies are used to select feature points in the training stage and the inference stage, as follows:
[0033] In the training phase, two methods, random sampling and topological point sampling, are used to select feature points. The random sampling method randomly selects feature points in the continuous space, and the topological point sampling method uses the information of the center line to sample the coordinate points. Specifically, the lung trachea center line with radius information is first extracted, and then the sampling coefficient of random point sampling near the center line is determined according to the radius size of the tubular structure of the three-dimensional lung CT. The points sampled by the topological point sampling method are expressed as follows:
[0034] P topo ~Correlation(p skkel , r)
[0035] Among them, p topo represents the coordinate points obtained by topological sampling, Correlation represents the sampling process of topological points, and p skel represents the coordinates of the skeleton point, and r represents the radius of the tubular structure of the 3D lung CT. There is a negative correlation between the radius and the number of sampling points. As the radius decreases, the sampling points generated for the corresponding tracheal branches become more dense.
[0036] During inference, uniform sampling is used instead of random sampling and topological point sampling to output the segmentation result at the original resolution of the image.
[0037] The present invention also provides an implicit neural network construction system for three-dimensional lung CT tubular structure segmentation based on topological prior, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.
[0038] The present invention also provides a computer-readable storage medium on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.
[0039] Compared with the prior art, the present invention has the following beneficial effects:
[0040] 1. Compared with the traditional CNN method, this application uses an implicit neural network to generate continuous coordinate points, reducing the amount of calculation and parameters. It can generate high-resolution segmentation results while reducing memory usage.
[0041] 2. Compared with the existing INR segmentation method, this proposal introduces a topological sampling strategy and centerline guidance. The topological sampling strategy helps to solve the problem of foreground-background imbalance, and the centerline guidance of the multi-implicit decoder helps to alleviate the problem of scale imbalance, which enables the implicit neural segmentation network to be applicable to complex tubular structures. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 This is a diagram of the architecture of the topology prior implicit tubular structure segmentation network of the present invention.
[0043] Figure 2 A schematic diagram is generated for the sampling points and center lines of the present invention. DETAILED DESCRIPTION
[0044] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.
[0045] The present invention provides a method for constructing an implicit neural network for three-dimensional lung CT tubular structure segmentation based on topological prior, constructs an implicit neural network for three-dimensional lung CT tubular structure segmentation based on topological prior, uses the implicit neural network to learn the continuous representation of the three-dimensional lung CT tubular structure segmentation results, and reconstructs the three-dimensional lung CT tubular structure segmentation results of arbitrary resolution; in addition, by integrating the topological structure information of the three-dimensional lung CT tubular structure, dense sampling of small branches is performed; in addition, a centerline prediction auxiliary task is used to optimize the three-dimensional lung CT tubular structure segmentation results.
[0046] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0047] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0048] This embodiment provides a method for constructing an implicit neural network for three-dimensional lung CT tubular structure segmentation based on topological prior. The implicit neural network for three-dimensional lung CT tubular structure segmentation based on topological prior includes a feature pyramid encoder, a point feature encoder and a dual-branch implicit decoder. The structure is as follows: Figure 1 As shown. The use of implicit neural networks can learn continuous representations of segmentation results, reconstruct segmentation results of arbitrary resolution, and effectively process large three-dimensional lung CT images covering the entire lung while ensuring memory efficiency. In addition, by integrating the three-dimensional tubular topological structure information to densely sample small branches, the network can better focus on foreground information and learn more relative spatial position information, thereby reducing confusion caused by feature similarity between categories. The centerline prediction auxiliary task is used to optimize the segmentation results of tubular structures, alleviate the discontinuity caused by small branches, and improve the accuracy and reliability of the segmentation results.
[0049] 1. Multi-task implicit neural network
[0050] The multi-task implicit neural network designed by the present invention is as follows Figure 1 As shown in the figure, a 3D CT image is used as input and is first processed by a feature pyramid encoder. The feature pyramid encoder consists of a network structure consisting of four convolution modules, each of which includes a 3×3×3 convolution layer, a batch normalization (BatchNorm, BN) layer, a linear rectification (Rectified Linear Unit, ReLU) layer, and a downsampling layer. This encoder structure can generate a series of feature matrices F1, F2...F at different scales. n , each feature matrix carries information about different dimensions and levels of the image.
[0051] After completing the image feature extraction, the point feature encoding method is used to sample and encode features of different scales. Specifically, the image obtains a feature matrix in the discrete form of H×W×D through the feature pyramid encoder. Then the discrete data is implicitly mapped, and each coordinate of the feature matrix of size H×W×D is mapped to a continuous spatial interval h×w×d (h, w, d∈R[-1, 1]). Subsequently, a continuous sampling coordinate point p(x, y, z)(x, y, z∈R[-1, 1]) is input, and a series of features corresponding to the point at different scales can be generated by performing trilinear interpolation queries, namely F1(p), F2(p)...F n (p). Then, each sampling coordinate point p and its features F at different scales are n (p) are connected to form a point feature encoding F(p). This point feature encoding mechanism not only samples features of different scales, but also incorporates the spatial position information of the coordinate points. This feature expression combined with spatial position is conducive to the network's learning of the positional relationship of 18 similar tubular lung and tracheal features in three-dimensional space. In addition, this point feature encoding provides a rich and continuous feature representation by combining continuous coordinate points and corresponding features, which helps prevent overfitting during training.
[0052] The decoding process is implemented by a multilayer perceptron (MLP), whose main task is to predict the occupancy rate of the lung and trachea categories at each position. The core advantage of this encoding and decoding strategy is that it does not need to output the entire image in the entire space at the same time, but can output the category probability at each position point by point. It has efficient memory utilization and allows the processing of complex lung CT images without being limited by computing resources. More importantly, query points p of arbitrary precision can be sampled in continuous space to generate outputs with arbitrary high resolution, thereby obtaining more detailed and accurate segmentation results.
[0053] 2. Sampling strategy based on topological information
[0054] In the training stage and the inference stage, the present invention adopts different strategies to select feature points. In the training stage, two methods, random sampling and topological point sampling, are used to select feature points. The random sampling method randomly selects feature points in a continuous space, which can ensure that feature information at different positions is covered during the training process. The topological point sampling method uses the information of the center line to sample the coordinate points. This method can more effectively select feature points related to the topological structure of the tubular structure. Figure 2As shown, the present invention first extracts the center line of the lung and trachea with radius information, and then determines the sampling coefficient of random point sampling near the center line according to the radius of the tubular structure, thereby more effectively selecting feature points related to the topological structure of the lung and trachea. The points of topological sampling can be expressed as follows:
[0055] P topo ~Correlation(p skel , r)
[0056] Among them, p skel represents the coordinates of the skeleton point, and r represents the radius of the tubular structure. There is a negative correlation between the radius and the number of sampling points. As the radius decreases, the sampling points generated for the tracheal branch become denser. Through this sampling method, the network can better focus on small tubular structures, solving the problem that it is difficult to obtain information about small structures using only random sampling, and thus pay more attention to foreground information.
[0057] During the reasoning process, the present invention adopts uniform sampling to replace random sampling and topological point information sampling, so as to output the segmentation result with the original resolution of the image.
[0058] 3. Centerline prediction auxiliary task
[0059] The present invention uses two multi-layer perceptrons (MLPs) as implicit decoders to obtain the segmentation result s′ and the centerline extraction result c′, which can be expressed as:
[0060] s′=D s (F(p))
[0061] c′=D c (F(p))
[0062] The corresponding segmentation loss function is:
[0063]
[0064] The centerline loss function is:
[0065]
[0066] In the above formula, s and c represent the true labels of the segmentation and centerline, respectively, and T prec (s, c′) is the intersection of the predicted centerline and the segmentation annotation, T sens (s′, c) is the intersection of the centerline true label and the segmentation. The centerline true label is obtained by binarizing the centerline with radius information, such as Figure 2 (d) as shown.
[0067] The present invention also provides an implicit neural network construction system for three-dimensional lung CT tubular structure segmentation based on topological prior, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.
[0068] The present invention also provides a computer-readable storage medium on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.
[0069] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0070] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0071] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0072] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0073] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.
Claims
1. A method for constructing an implicit neural network for segmenting tubular structures in three-dimensional lung CT based on topological priors, characterized in that: An implicit neural network for 3D lung CT tubular structure segmentation based on topological priors is constructed. The implicit neural network is used to learn the continuous representation of 3D lung CT tubular structure segmentation results and reconstruct 3D lung CT tubular structure segmentation results of arbitrary resolution. In addition, the topological structure information of 3D lung CT tubular structure is integrated to densely sample small branches. In addition, the centerline prediction auxiliary task is used to optimize the segmentation results of tubular structures in 3D lung CT.
2. According to claim 1, the implicit neural network construction method for three-dimensional lung CT tubular structure segmentation based on topological prior is characterized in that: The implicit neural network includes a feature pyramid encoder, a point feature encoder and a dual-branch implicit decoder.
3. The implicit neural network construction method for three-dimensional lung CT tubular structure segmentation based on topological prior according to claim 2, characterized in that: The feature pyramid encoder is composed of a network structure consisting of four convolutional modules, each of which includes a 3×3×3 convolutional layer, a batch normalization layer, a linear rectification layer, and a downsampling layer.
4. The implicit neural network construction method for three-dimensional lung CT tubular structure segmentation based on topological prior according to claim 1, characterized in that: The implicit neural network uses the three-dimensional CT image as input, and firstly processes it through the feature pyramid encoder to generate a series of feature matrices F1, F2, ..., F at different scales. n ,Each feature matrix carries information about different dimensions and levels of the image; After the image feature extraction is completed, the point feature encoder is used to sample and encode the feature matrices of different scales to form the point feature code F(p); The point feature code F(p) is input into a dual-branch implicit decoder for decoding. The decoding process is implemented by a multi-layer perceptron, whose main task is to predict the occupancy rate of the lung trachea category at each position and reconstruct the three-dimensional lung CT tubular structure segmentation result of arbitrary resolution.
5. The implicit neural network construction method for three-dimensional lung CT tubular structure segmentation based on topological prior according to claim 4, characterized in that: The point feature encoder is used to sample and encode feature matrices of different scales to form point feature encoding F(p), which is specifically as follows: The feature matrix of the three-dimensional CT image is obtained by the feature pyramid encoder in the form of H×W×D discrete form; then the discrete data is implicitly mapped, and each coordinate of the feature matrix of size H×W×D is mapped to a continuous spatial interval h×w×d (h, w, d∈[-1,1]); then, continuous sampling coordinate points p(x, y, z)(x, y, z∈R[-1,1]) are input, and a series of features corresponding to corresponding points at different scales are generated by performing trilinear interpolation queries, namely F1(p), F2(p)...F n (p); Then, each sampling coordinate point p and its features F at different scales are n (p) to form the point feature code F(p).
6. The implicit neural network construction method for 3D lung CT tubular structure segmentation based on topological prior according to claim 5, characterized in that: The point feature code F(p) is input into the dual-branch implicit decoder for decoding, as follows: Two multilayer perceptrons are used as dual-branch implicit decoders to obtain the segmentation result s′ and the centerline extraction result c′, respectively, which can be expressed as: s′=D s (F(p)) c′=D c (F(p)) In the formula, D s is the vascular implicit decoding module, D c It is the centerline decoding module.
7. The implicit neural network construction method for three-dimensional lung CT tubular structure segmentation based on topological prior according to claim 6, characterized in that: The centerline prediction auxiliary task is used to optimize the segmentation result of the tubular structure of the three-dimensional lung CT, as follows: The segmentation loss function is: The centerline loss function is: In the formula, s and c represent the true labels of the segmentation and centerline, respectively, and T prec (s, c′) is the intersection of the predicted centerline and the segmentation annotation, T sens (s′c) is the intersection of the centerline true label and the segmentation. The true label of the centerline is obtained by binarizing the centerline with radius information.
8. The method for constructing an implicit neural network for segmenting tubular structures in three-dimensional lung CT based on topological priors according to claim 1, characterized in that: The method integrates the topological structure information of the tubular structure of the three-dimensional lung CT to densely sample the small branches, that is, different strategies are used to select feature points in the training stage and the inference stage, as follows: In the training phase, two methods, random sampling and topological point sampling, are used to select feature points. The random sampling method randomly selects feature points in the continuous space, and the topological point sampling method uses the information of the center line to sample the coordinate points. Specifically, the lung trachea center line with radius information is first extracted, and then the sampling coefficient of random point sampling near the center line is determined according to the radius size of the tubular structure of the three-dimensional lung CT. The points sampled by the topological point sampling method are expressed as follows: P topo -Correlation(p skel ,r) Among them, p topo represents the coordinate points obtained by topological sampling, Correlation represents the sampling process of topological points, and p skel represents the coordinates of the skeleton point, and r represents the radius of the tubular structure of the 3D lung CT. There is a negative correlation between the radius and the number of sampling points. As the radius decreases, the sampling points generated for the corresponding tracheal branches become more dense. During inference, uniform sampling is used instead of random sampling and topological point sampling to output the segmentation result at the original resolution of the image.
9. An implicit neural network construction system for 3D lung CT tubular structure segmentation based on topological prior, characterized in that: The method comprises a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps as claimed in any one of claims 1 to 8 can be implemented.
10. A computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, and when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 8 can be implemented.