Image segmentation method based on grid deformation fitting and computer program product
Through a grid deformation fitting method, combined with deep learning and recurrent neural network, the accuracy and stability problems of image segmentation in complex boundaries and low-contrast areas are solved, and high-precision and high-stability image segmentation effect is achieved. Suitable for high-risk organ segmentation in three-dimensional medical images.
Patent Information
- Application Number
- CN202411186248.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-28
AI Technical Summary
When dealing with complex boundaries and low contrast areas, existing image segmentation methods have problems such as inaccurate segmentation, discontinuous segmentation lines, lack of multi-scale feature fusion, and insufficient segmentation accuracy in low contrast areas.
Using a method based on mesh deformation fitting, combined with deep learning prediction and mesh deformation optimization strategy, preliminary segmentation is performed through the TNT-MNet model, voxel and point cloud features are extracted, and vertex positions are used using recurrent neural networks and graph convolutional networks. Prediction and adjustment to achieve uniform grid density and fuse multi-scale features to improve segmentation accuracy and stability.
It improves the accuracy and stability of image segmentation, ensures the continuity and consistency of segmentation results, is suitable for high-precision segmentation of high-risk organs in three-dimensional medical images, and provides more reliable technical support.
Smart Images

Figure CN118691828B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image data processing, and in particular, relates to an image segmentation method based on grid deformation fitting and a computer program product. Background Art
[0002] Image segmentation refers to the process of dividing an image into several regions with similar properties. From a mathematical point of view, image segmentation is the process of dividing an image into non-overlapping regions.
[0003] In the field of medical image processing technology, high-precision segmentation of images of organs at risk is a key step in achieving accurate radiotherapy and surgical planning. Organs at risk refer to important organs in the patient's body. When they are threatened, damaged or have limited function, they may pose a serious threat to life. Common organs at risk include the heart, lungs, liver, kidneys, brain, pancreas, etc. For images of organs at risk, traditional manual segmentation methods rely on a large amount of manual drawing by radiologists, which is not only labor-intensive but also easily affected by human factors, resulting in insufficient repeatability and accuracy of the segmentation results. Existing machine segmentation methods are usually unable to accurately evolve to the target boundary when faced with situations such as uneven image grayscale, blurred boundaries, and complex shapes.
[0004] In recent years, image segmentation methods have been increasingly needed in clinical practice and have been widely studied in medical image computing. Technologies such as deep neural networks (DNNs) and fully convolutional neural networks (FCNs) have demonstrated good performance in medical image segmentation. By combining convolutional neural networks (CNNs) and recurrent neural networks (RNNs), automatic segmentation of three-dimensional medical images has been achieved. These methods extract multi-level features from images and generate fine-grained three-dimensional feature maps, thereby achieving preliminary segmentation of images. However, these methods still have the following problems when dealing with complex boundaries and low-contrast areas:
[0005] 1. When processing small target areas, there is often an inaccurate segmentation problem, especially in areas with low contrast and complex boundaries. Existing image segmentation methods are prone to discontinuity of segmentation lines at the boundaries, resulting in poor stability and consistency of segmentation results;
[0006] 2. Lack of effective fusion of features of different scales leads to unsatisfactory segmentation accuracy and robustness;
[0007] 3. When processing low-contrast areas, it is difficult to achieve the high precision required for clinical applications. Summary of the invention
[0008] In order to solve the shortcomings of existing image segmentation methods when dealing with complex boundaries and low-contrast areas, the present invention proposes an image segmentation method and a computer program product based on grid deformation fitting, which combines deep learning prediction with grid deformation optimization strategy to improve the accuracy and stability of image segmentation.
[0009] In order to solve the above technical problems, the present invention adopts the following technical solutions to achieve the above problems:
[0010] In one aspect, the present invention provides an image segmentation method based on grid deformation fitting, comprising:
[0011] The TNT-MNet model is used to perform preliminary segmentation on the input image to generate a rough segmentation result of the target object, and voxel features are extracted from the input image. Point cloud features are extracted from the global point cloud of the target object generated by the rough segmentation result.
[0012] The generated rough segmentation results and the extracted voxel features and point cloud features are input into the recurrent neural network, and the recurrent neural network is used to predict the position and movement distance of each vertex of the target object in each iteration process, and the position of the vertex is adjusted in combination with the normal direction of the vertex to align it with the target surface;
[0013] Use a multi-layer perceptron to perform nonlinear transformation on the feature vector of each vertex, predict the relative position change and movement distance of each vertex, and readjust the vertex position;
[0014] Use the graph convolutional network to update the feature vector of each vertex to obtain a new vertex feature matrix; use the new vertex feature matrix to further adjust the vertex position;
[0015] According to the length of each edge formed between the vertices, vertices are added or deleted to make the grid density uniform, forming a segmented refined grid image.
[0016] In some embodiments of the present application, the point cloud feature extraction process may include:
[0017] Convert the rough segmentation result into a point cloud, each of which contains three-dimensional coordinates and related feature values;
[0018] The graph convolutional neural network is used to process the point cloud data and extract the point cloud features.
[0019] In some embodiments of the present application, after extracting the point cloud features, the point cloud features can be fused with the voxel features and then input into a subsequent recurrent neural network to improve the accuracy of vertex position prediction. The specific feature fusion process may include:
[0020] The extracted voxel features and point cloud features are connected in the feature dimension to form a joint feature vector;
[0021] Input the joint feature vector into the multi-layer perceptron for feature fusion;
[0022] The multilayer perceptron performs nonlinear transformation on the input features through multiple fully connected layers and activation functions to generate a fused feature vector, which is input into the recurrent neural network for prediction and adjustment of vertex positions.
[0023] In some embodiments of the present application, the process of adjusting the vertex position may include:
[0024] Using the rough segmentation result of the target object to form an initial grid image of the target object, and using the initial grid image to determine the initial position of each vertex of the target object;
[0025] Input the initial position of each vertex and its corresponding fused feature vector into the recurrent neural network;
[0026] Update the position and movement distance of the vertex in each iteration through the recurrent neural network;
[0027] In each iteration, the normal direction of the vertex is updated by calculating the normals of the triangles around the current vertex, and the updated vertex position is obtained by adjusting the current vertex position along the normal direction by the updated moving distance. Specifically, if the current vertex is inside the target object, it moves along the positive direction of the normal; if the current vertex is outside the target object, it moves along the negative direction of the normal.
[0028] In some embodiments of the present application, in order to achieve feature sharing and effective knowledge transfer across tasks, a recurrent neural network may be configured to perform the following multi-scale feature extraction and fusion process in each iteration:
[0029] Use convolution kernels of different sizes to perform convolution operations on the grid image updated after the previous iteration to extract feature maps of different scales;
[0030] Through pooling operations, feature maps of different scales are reduced in dimension to retain key feature information;
[0031] The convolution operation is used to fuse the feature maps of different scales after dimensionality reduction to generate a fused feature map;
[0032] The fused feature map is normalized to ensure the consistency of the numerical range of features of different scales and avoid instability caused by differences in feature values in subsequent processing.
[0033] In some embodiments of the present application, the following method may be used to uniformize the mesh density of the mesh image formed after the vertex positions are adjusted:
[0034] Calculate the length of each edge of the grid and compare it with the preset maximum edge length threshold and minimum edge length threshold;
[0035] For edges whose length exceeds the maximum edge length threshold, a new vertex is inserted, for example, a new vertex is inserted at the midpoint of the edge to refine the mesh;
[0036] For an edge whose length is less than the minimum edge length threshold, the two vertices of the edge can be merged into one vertex to reduce the redundancy of the mesh.
[0037] In another aspect, the present invention further proposes a computer program product, including a computer program or instructions, which, when executed by a processor, can implement the steps of the above-mentioned image segmentation method based on grid deformation fitting.
[0038] Compared with the prior art, the advantages and positive effects of the present invention are mainly reflected in:
[0039] 1. The image segmentation method of the present invention combines voxel features and point cloud features, and uses a recurrent neural network (RNN) to predict the position and movement distance of each vertex of the target object, thereby achieving fine segmentation of small target areas. The continuity and consistency of the segmentation results are ensured by iterative optimization of the vertex position and adjustment of the normal direction. The uniformity and quality of the mesh are ensured by adaptive topological optimization of the adjusted vertex position, and the accuracy and stability of the segmentation results are improved.
[0040] 2. During the prediction process, the recurrent neural network (RNN) integrates feature information from different scales, realizes feature sharing and effective knowledge transfer across tasks through multi-scale feature fusion, ensures the consistency and accuracy of segmentation results at different scales, and improves the accuracy and robustness of segmentation.
[0041] 3. The image segmentation method of the present invention can not only handle complex boundaries and low-contrast areas, but also significantly improve the segmentation accuracy and stability of the segmentation results. It is particularly suitable for high-precision segmentation of high-risk organs in three-dimensional medical images, thereby providing more reliable technical support for radiotherapy and surgical planning.
[0042] Other features and advantages of the present invention will become more apparent after reading the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 It is a general architecture flow chart of an embodiment of the image segmentation method based on grid deformation fitting proposed by the present invention;
[0044] Figure 2 yes Figure 1 A specific flow chart of an embodiment of the preliminary segmentation and feature extraction process in FIG.
[0045] Figure 3 yes Figure 1 A specific flow chart of an embodiment of the iterative vertex position prediction and multi-scale feature fusion process in FIG.
[0046] Figure 4 yes Figure 1 A specific flow chart of an embodiment of the bias prediction and grid structure update process in FIG.
[0047] Figure 5 It is a model framework diagram of an embodiment of the image segmentation method based on grid deformation fitting proposed by the present invention;
[0048] Figure 6 is a comparison of the initial grid image and the refined grid image; where, Figure 6 (a) is the initial grid image. Figure 6 (b) is the refined grid image;
[0049] Figure 7 It is a schematic diagram of adaptive grid topology adjustment;
[0050] Figure 8 It is a schematic diagram of the offset process of the vertex position;
[0051] Fig. 9 This is a comparison chart of the visual segmentation results of the HaN dataset;
[0052] Fig.10 This is a comparison chart of the visual segmentation results of the HaNSeg dataset;
[0053] Fig.11 This is a comparison chart of the visual segmentation results of image segmentation on the SegRag dataset. DETAILED DESCRIPTION
[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0055] In view of the problems that existing manual image segmentation methods have large workload and low repeatability, and existing automatic image segmentation methods have insufficient segmentation accuracy and discontinuous segmentation lines when dealing with complex boundaries and low-contrast areas, as well as lack of multi-scale feature fusion and effective knowledge transfer mechanisms, this embodiment proposes an image segmentation method based on grid deformation fitting, which combines deep learning prediction with grid deformation optimization to solve the above technical problems.
[0056] The image segmentation method of this embodiment is described in detail below by taking the segmentation of a target organ in a three-dimensional medical image (target object in the image to be segmented) as an example.
[0057] Combination Figure 1 , Figure 5 As shown, the specific process includes the following:
[0058] S100: Preliminary segmentation and feature extraction.
[0059] This process uses the TNT-MNet model to perform preliminary segmentation on the input three-dimensional medical image, generate a rough segmentation result of the target organ, extract voxel features from the three-dimensional medical image, and extract point cloud features from the global point cloud of the target organ generated by the rough segmentation result.
[0060] See also Figure 2 , this process may include the following detailed steps:
[0061] S110: Image preprocessing.
[0062] The input 3D medical image is preprocessed, including but not limited to denoising, normalization and standardization. The purpose of these operations is to improve the quality and consistency of the image, thereby providing a reliable basis for subsequent image segmentation and feature extraction.
[0063] In some embodiments, the following specific operations may be set during image preprocessing:
[0064] S111, performing denoising processing on the three-dimensional medical image using a filter (such as a Gaussian filter or a median filter, etc.) to reduce noise in the image;
[0065] S112, normalizing the pixel values of the image to a range of [0, 1] to reduce the intensity difference between different images;
[0066] S113. Standardize the image to have zero mean and unit variance to improve the training effect of the model.
[0067] S120: Preliminary segmentation.
[0068] The TNT-MNet model is used to perform preliminary segmentation on the preprocessed 3D medical images.
[0069] The TNT-MNet model is a deep learning model based on a convolutional neural network (CNN), which extracts multi-level features from the input image and generates fine-grained three-dimensional feature maps. These feature maps contain rich spatial and contextual information and can effectively identify and segment the rough area of the target organ.
[0070] The encoder part of the TNT-MNet model processes the input image layer by layer through a series of convolutional layers and pooling layers to extract feature representations of different scales. The decoder part restores the feature representation to the original image size through a deconvolution layer and outputs a rough segmentation result of the target organ, that is, an initial grid image of the target organ is formed, such as Figure 6 As shown in part (a) of .
[0071] The rough segmentation result contains the approximate position and shape of the target organ, which can provide a reference for the subsequent fine segmentation. Assume that the input 3D medical image is I , whose size is H×W×D ( H Indicates height, W Indicates width, D represents depth), then the rough segmentation result output by the TNT-MNet model Scoarse The size of is the same as the input image.
[0072] S130: Feature extraction.
[0073] The process includes two parts: voxel feature extraction and point cloud feature extraction. The specific steps are as follows:
[0074] S131. Extract voxel features from three-dimensional medical images.
[0075] Voxel features represent the local properties of each voxel, such as intensity, texture, and edge information. The TNT-MNet model can be used to encode the input image to extract voxel features. V .
[0076] Voxel features V Using the convolution operation of the TNT-MNet model from the input image I Extracted from, the formula is as follows:
[0077] V=Conv(I) .
[0078] S132. From the rough segmentation result Scoarse Point cloud features are extracted from the generated global point cloud of the target organ.
[0079] Point cloud features represent the geometric shape and position of the target organ in three-dimensional space. ScoarseConvert to Point Cloud P , each point cloud contains three-dimensional coordinates (x,y,z) And related eigenvalues. Use graph neural network (GNN) to process point cloud data and extract point cloud features F point .
[0080] Point cloud features F point From point cloud through graph convolutional neural network (GCN) P Extracted from, the formula is as follows:
[0081] F point =GCN(P).
[0082] S140, feature fusion.
[0083] The extracted voxel features V and point cloud features F point Fusion is performed to generate comprehensive features, which can be expressed as F fused Specifically, the following processes may be included:
[0084] S141, voxel features V and point cloud features F point Connect on the feature dimension to form a joint feature vector F =[ V , F point ];
[0085] S142, the joint feature vector F Input into Multilayer Perceptron (MLP) for feature fusion;
[0086] S143, Multilayer Perceptron (MLP) performs nonlinear transformation on input features through multiple fully connected layers and activation functions (such as ReLU) to generate a fused feature vector F fused :
[0087] F fused = MLP ( F ).
[0088] After the above process S100, the rough segmentation of the input image is completed, and the rough segmentation result is output. Scoarse and the preliminary feature representation (i.e., the fused feature vector F fused ), which is used in the subsequent segmentation process.
[0089] S200, iterative vertex position prediction and multi-scale feature fusion.
[0090] A recurrent neural network (RNN) is used to predict the position and movement distance of each vertex of the target organ in each iteration step, and the vertex position is adjusted according to the current position and normal direction of the vertex to achieve accurate alignment with the target surface. At the same time, feature information from different scales is fused to achieve feature sharing and effective knowledge transfer across tasks through multi-scale feature fusion.
[0091] See also Figure 3 , this process may include the following detailed steps:
[0092] S210, iterate vertex position prediction.
[0093] Using the rough segmentation result output from process S100 Scoarse The initial mesh image of the target organ formed can determine the initial positions of the vertices of the target organ. These initial positions are based on the rough segmentation results of the target organ, and can ensure that the vertices are evenly distributed and cover the main area of the target organ.
[0094] In order to update the vertex position in each iteration and achieve accurate adjustment of the vertex position, this embodiment uses a recurrent neural network (RNN) to predict the vertex position. The architecture of the recurrent neural network (RNN) includes an input layer, a hidden layer, and an output layer, which can update the vertex position in each iteration. The specific steps are as follows:
[0095] Input layer: Input is the result of rough segmentation Scoarse Determined vertex position p t and its corresponding fused feature vector F fused ;
[0096] Hidden layer: hidden state of RNN h t Represents the feature representation of the current vertex, which includes convolutional layers and pooling layers, and can achieve multi-scale feature extraction and fusion;
[0097] Output layer: output updated vertex positions p t+1 and moving distance Δ p t .
[0098] The formula is as follows:
[0099] h t = RNN( p t , F fused )
[0100] Δ p t = W t h t +b h ;
[0101] Among them, W t and b h Represent the weight and bias parameters of the linear transformation respectively.
[0102] During each iteration, the normal direction of the vertex needs to be updated n t . The normal direction n t It can be determined by calculating the normal of the triangle surface around the vertex, the formula is as follows:
[0103] ;
[0104] in, n t,i Represents a vertex p t Location i The normal direction of each triangle; N is the number of connected triangles.
[0105] When adjusting the vertex position, the moving distance Δ can be predicted by the recurrent neural network (RNN) p t , the current vertex position p t Adjust the moving distance Δ along the normal direction p t , you can get the updated vertex position p t+1 .
[0106] Among them, the current vertex p t The direction of movement along the normal can be determined according to its position inside or outside the target organ: the vertex inside the target organ moves along the positive direction of the normal; the vertex outside the target organ moves along the negative direction of the normal.
[0107] S220, multi-scale feature fusion.
[0108] Extracting feature maps of different scales from segmented 3D medical images F ,The feature map of each scale can be obtained through convolution kernels and pooling operations of different sizes in RNN, which contains multi-level information of the image.
[0109] The specific process is as follows:
[0110] S221, use convolution kernels of different sizes to perform convolution operations on the grid image updated after the previous iteration to extract feature maps of different scales F ;
[0111] S222, feature maps of different scales through pooling operation F Perform dimensionality reduction to retain key feature information;
[0112] S223, use convolution operation to reduce the dimension of feature maps of different scales F Fusion is performed to generate a fusion feature map F fused ;
[0113] S224, fusion feature map F fused Normalization is performed to ensure the consistency of the numerical range of features at different scales and to avoid instability caused by differences in feature values in subsequent processing.
[0114] In order to improve the accuracy and robustness of the prediction, this embodiment introduces a feature enhancement and feedback mechanism. That is, the currently predicted vertex position and movement distance are used as the input of the recurrent neural network (RNN), and the original features are combined to perform feature extraction and fusion again, and the vertex position is continuously iterated and optimized.
[0115] Specifically, the current vertex position obtained by iterative recurrent neural network (RNN) p t+1 and moving distance Δ p t Feedback to the convolutional layer of the RNN for feature extraction and update of feature representation F updated Using the updated feature representation F updated In the next round of iteration, feature extraction and fusion are performed again to generate new fusion features. F fused,new The above process is repeated to iteratively optimize the vertex position and moving distance until the preset convergence condition is met or the maximum number of iterations is reached.
[0116] Through the above steps, the recurrent neural network (RNN) is used to iteratively predict vertex movement and update positions, and combined with multi-scale feature fusion, the target organ can be accurately segmented to form a refined grid image of the target organ, such as Figure 6 As shown in part (b) of Figure 3, multi-scale feature fusion ensures feature sharing and effective knowledge transfer across tasks, improving the accuracy and robustness of segmentation.
[0117] S300, bias prediction and grid structure update.
[0118] This process combines deep learning prediction with mesh deformation optimization to perform high-precision segmentation on finely segmented mesh images. Specifically, the relative position and movement distance of vertices are predicted through the bias prediction module, and the mesh structure information is updated using the graph convolutional network (GCN), ensuring that vertices maintain global structural consistency during the iteration process, thereby improving segmentation accuracy.
[0119] See also Figure 4 , this process may include the following detailed steps:
[0120] S310, bias prediction.
[0121] In order to further optimize the position of the vertices, this embodiment designs a bias prediction module, which predicts the relative position and movement distance of each vertex by performing nonlinear transformation on vertex features.
[0122] Specifically, the fusion feature map of each vertex F fused The input bias prediction module uses a multi-layer perceptron (MLP) to perform nonlinear transformation on the input features and predict the relative position change of each vertex Δ p and moving distance d The formula is as follows:
[0123] Δ p =MLPpos( F fused )
[0124] d =||Δ p ||;
[0125] Among them, MLPpos represents a multi-layer perceptron for predicting relative position changes.
[0126] According to the predicted relative position change Δ p and moving distance d , adjust the vertex position, the formula is as follows:
[0127] p t+1 = p t +Δ p t · n t ;
[0128] in, p t Indicates the current vertex position; p t+1 Indicates the adjusted vertex position; Δ pt Indicates the relative position change of the current vertex; n t Indicates the normal direction of the current vertex.
[0129] S320. Update the grid structure information using the graph convolutional network.
[0130] In order to ensure the global consistency and accuracy of the grid structure, this embodiment introduces a graph convolutional network (GCN) to update the connection relationship between the vertices and edges of the grid. The graph convolutional network (GCN) can effectively capture the local and global structural information between vertices and improve the accuracy of segmentation.
[0131] Specifically, the grid structure can be represented as a graph G = (V, E) ,in, V represents a set of vertices, E Represents a set of edges. Each vertex v ∈ V , with the corresponding eigenvector F ; Each edge ( v i ,v j )∈ E , indicating the connection relationship between two adjacent vertices.
[0132] The grid structure is updated using the graph convolutional network (GCN) through the adjacency matrix A and the feature matrix H The vertex features are propagated and updated through the GCN layer. The formula is as follows:
[0133] H(l+1)= σ( A·H(l)·W(l) );
[0134] in, H(l) Indicates l The vertex feature matrix of the layer, W(l) Indicates l The weight matrix of the layer, σ represents the activation function.
[0135] Through the propagation and update of the GCN layer, a new vertex feature matrix is obtained H(l+1) The updated features are used to further adjust the vertex positions to improve the accuracy and stability of the segmentation.
[0136] S330, mesh adaptive topology optimization.
[0137] During the iterative deformation of the mesh, the expansion and contraction of the vertices in each region may cause a change in the point density distribution. To solve this problem, this embodiment adopts an adaptive topology optimization strategy to ensure the uniformity and structural rationality of the mesh.
[0138] like Figure 7 As shown, the specific process includes the following:
[0139] S331. Calculate the length of each edge in the grid L , and with the preset maximum edge length threshold Lmax and the minimum edge length threshold Lmin Make a comparison.
[0140] S332, for length L Exceeding the maximum edge length threshold Lmax Insert new vertices to refine the mesh;
[0141] This step can be achieved by inserting a new vertex at the midpoint of the edge.
[0142] S333, for length L Below minimum edge length threshold Lmin The two vertices of the edge are merged into one vertex to reduce the redundancy of the mesh.
[0143] Thus, the final refined grid image is formed, completing the segmentation of the input image.
[0144] Figure 8 The offset process of the vertex position is shown.
[0145] Next, we use different data sets to verify the segmentation results of the above image segmentation method.
[0146] Fig. 9 This is a comparison chart of the visual segmentation results of image segmentation of the HaN dataset using multiple existing image segmentation models and the image segmentation model of this embodiment (defined as the RefineMesh model in this embodiment). Fig. 9 The rightmost column of figures in is the result of segmenting the brainstem, mandible, optic nerve, and optic chiasm using the image segmentation model of this embodiment.
[0147] Table 1 Quantitative comparison of segmentation performance for HaN dataset
[0148]
[0149] Combined with the results of the quantitative comparison of segmentation performance in Table 1, it can be seen that the image segmentation method of this embodiment has significantly improved segmentation performance on all organs compared with other models on the HaN dataset, especially large organs such as the brainstem and mandible, and small organs such as the optic nerve and chiasm, with a performance improvement of about 2.49%. Models based on pixel classification usually have discontinuous segmentation at fuzzy edges, while this embodiment uses the grid representation characteristics to independently determine the position movement of each vertex, which can obtain a higher Dice coefficient.
[0150] Fig.10 This is a comparison chart of the visual segmentation results of image segmentation of the HaNSeg dataset using multiple existing image segmentation models and the image segmentation model RefineMesh of this embodiment. Fig.10 The rightmost column of figures in is the result of segmenting the brainstem, mandible, optic nerve and other organs using the image segmentation model of this embodiment.
[0151] Table 2 Quantitative comparison of segmentation performance for HaNSeg dataset
[0152]
[0153] Combined with the quantitative comparison results of the segmentation performance in Table 2, it can be seen that the image segmentation method of this embodiment on the HaNSeg dataset has a certain improvement in the average Dice coefficient and HD95 index compared with other models, while the performance of the Voxel2Mesh model is significantly reduced compared with the TNT-MNet model. The main reason is that its smaller feature map information and joint prediction method make it difficult to handle small-volume organs at risk.
[0154] Table 3 Quantitative comparison of segmentation performance of small organs on the HaNSeg dataset
[0155]
[0156] Combined with the quantitative comparison results of small organ segmentation performance in Table 3, it can be seen that the image segmentation method of this embodiment on the HaNSeg dataset improves the average Dice coefficient of small organ segmentation by 2.13% compared with other models, which is significantly better than the models of Voxel2Mesh and Kong et al., demonstrating its robustness and fine control capabilities in dealing with complex anatomical structures.
[0157] Fig.11 This is a comparison chart of the visual segmentation results of the SegRag dataset using multiple existing image segmentation models and the image segmentation model RefineMesh of this embodiment. Fig.11The rightmost column of figures in is the result of segmenting the brainstem, mandible, optic nerve and other organs using the image segmentation model of this embodiment.
[0158] Table 4 Quantitative comparison of segmentation performance for SegRag dataset
[0159]
[0160] Combined with the segmentation performance quantitative comparison results in Table 4, it can be seen that the image segmentation method of this embodiment still maintains a stable performance improvement on the SegRag dataset. The model proposed by Kong et al. is also optimized to a certain extent, while the segmentation performance of the Voxel2Mesh model decreases.
[0161] Table 5 Quantitative comparison of segmentation performance of small organs on the SegRag dataset
[0162]
[0163] From the quantitative comparison results of the small organ segmentation performance in Table 5, it can be seen that the image segmentation method of this embodiment performs well on complex structures and small organs when used on the SegRag dataset, and both the Dice coefficient and the HD95 index are significantly improved.
[0164] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for a person skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to replace some of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions claimed to be protected by the present invention.
Claims
1. An image segmentation method based on grid deformation fitting, characterized in that: include: The TNT-MNet model is used to perform preliminary segmentation on the input image to generate a rough segmentation result of the target object, and voxel features are extracted from the input image. Point cloud features are extracted from the global point cloud of the target object generated by the rough segmentation result. The generated rough segmentation results and the extracted voxel features and point cloud features are input into the recurrent neural network, and the recurrent neural network is used to predict the position and movement distance of each vertex of the target object in each iteration process, and the position of the vertex is adjusted in combination with the normal direction of the vertex to align it with the target surface; Use a multi-layer perceptron to perform nonlinear transformation on the fused feature map of each vertex, predict the relative position change and movement distance of each vertex, and readjust the vertex position; Use the graph convolutional network to update the feature vector of each vertex to obtain a new vertex feature matrix; use the new vertex feature matrix to further adjust the vertex position; According to the length of each edge formed between the vertices, vertices are added or deleted to make the grid density uniform, thus forming a segmented and refined grid image; The recurrent neural network performs the following multi-scale feature extraction and fusion process in each iteration: Use convolution kernels of different sizes to perform convolution operations on the grid image updated after the previous iteration to extract feature maps of different scales; Through pooling operations, feature maps of different scales are reduced in dimension to retain key feature information; The convolution operation is used to fuse the feature maps of different scales after dimensionality reduction to generate a fused feature map; Normalize the fused feature map.
2. The image segmentation method based on grid deformation fitting according to claim 1, characterized in that: The point cloud feature extraction process includes: Convert the rough segmentation result into a point cloud, each of which contains three-dimensional coordinates and related feature values; The graph convolutional neural network is used to process the point cloud data and extract the point cloud features.
3. The image segmentation method based on grid deformation fitting according to claim 1, characterized in that: After extracting the point cloud features, the following feature fusion process is performed: The extracted voxel features and point cloud features are connected in the feature dimension to form a joint feature vector; Input the joint feature vector into the multi-layer perceptron for feature fusion; The multilayer perceptron performs nonlinear transformation on the input features through multiple fully connected layers and activation functions to generate a fused feature vector.
4. The image segmentation method based on grid deformation fitting according to claim 3, characterized in that: The process of adjusting the vertex position includes: Using the rough segmentation result of the target object to form an initial grid image of the target object, and using the initial grid image to determine the initial position of each vertex of the target object; Input the initial position of each vertex and its corresponding fused feature vector into the recurrent neural network; Update the position and movement distance of the vertex in each iteration through the recurrent neural network; In each iteration, the normal direction of the vertex is updated by calculating the normals of the triangles around the current vertex, and the current vertex position is adjusted along the normal direction by the updated moving distance to obtain the updated vertex position.
5. The image segmentation method based on grid deformation fitting according to claim 4, characterized in that: In each iteration of the recurrent neural network, the moving direction of the current vertex along the normal is determined according to the position of the current vertex inside and outside the target object: The vertices inside the target object move along the positive direction of the normal; Vertices outside the target object are moved in the negative direction of the normal.
6. The image segmentation method based on grid deformation fitting according to any one of claims 1 to 5, characterized in that: The process of uniformizing the grid density includes: Calculate the length of each edge and compare it with the preset maximum edge length threshold and minimum edge length threshold; For edges whose length exceeds the maximum edge length threshold, new vertices are inserted to refine the mesh; For an edge whose length is less than the minimum edge length threshold, the two vertices of the edge are merged into one vertex to reduce the redundancy of the mesh.
7. The image segmentation method based on grid deformation fitting according to claim 6, characterized in that: The process of inserting a new vertex on the edge whose length exceeds the maximum edge length threshold includes: Insert a new vertex at the midpoint of an edge whose length exceeds the maximum edge length threshold.
8. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
