Gridding and attention mechanism-based point cloud complementation tumor three-dimensional reconstruction system

Abstract

The invention discloses a point cloud complementation tumor three-dimensional reconstruction system based on gridding and an attention mechanism, and relates to the technical field of medical image processing, and the system comprises a data acquisition module, an image preprocessing module, and a model reconstruction module. The data acquisition module is used for collecting and acquiring two-dimensional CT image data; the image preprocessing module is used for preprocessing the two-dimensional CT image; and the model reconstruction module is used for performing three-dimensional reconstruction on the preprocessed two-dimensional CT image data to generate a three-dimensional medical image and a point cloud completion network comprising grids and geometric perception Transformer, extracting contour information from the two-dimensional segmentation mask, and predicting and reconstructing complete surface shapes of the liver and the tumor from the contour information. Therefore, in order to solve the problems existing in existing medical images, more accurate three-dimensional liver and tumor morphological reconstruction is provided for clinic, the accuracy of disease diagnosis and formulation of an auxiliary treatment scheme are improved, risks in an operation are reduced, and the success rate of the operation is improved.

Description

Technical Field

[0001] The present application relates to the field of medical image processing technology, and in particular to a point cloud completion tumor three-dimensional reconstruction system based on gridding and attention mechanism. Background Art

[0002] Liver tumors are among the most common malignancies worldwide, posing a serious threat to human health and life. With the rise of minimally invasive treatments, radiofrequency ablation (RFA), which targets localized ablation, has been gradually applied to the treatment of malignant liver tumors. This imaging-guided technique eradicates tumors, and its effectiveness relies heavily on the physician's ability to accurately identify and locate the tumor in CT images.

[0003] CT images can provide detailed information about human organs and lesions, but their two-dimensional representation lacks spatial depth. The two-dimensional CT images commonly used in clinical practice can only provide anatomical information of a specific plane of the liver, and cannot fully display the three-dimensional morphology, positional relationships, and surrounding tissue environment of the liver and tumor, making it difficult to intuitively reflect the three-dimensional anatomical structure. This means that doctors lack an intuitive spatial reference during operation, increasing the risk of inaccurate ablation areas, which may result in tumor residue or damage to healthy tissue. Therefore, accurate three-dimensional reconstruction technology is crucial to the successful implementation of radiofrequency ablation. Reconstructing high-precision three-dimensional shapes from two-dimensional medical images has become a key challenge in the fields of computer vision and medical image analysis. Summary of the Invention

[0004] To solve the above-mentioned problems, provide more accurate three-dimensional liver and tumor morphology reconstruction for clinicians, improve the accuracy of disease diagnosis and the formulation of auxiliary treatment plans, this application provides a point cloud completion tumor three-dimensional reconstruction system based on gridding and attention mechanism.

[0005] The point cloud completion tumor three-dimensional reconstruction system provided in this application based on gridding and attention mechanism adopts the following technical solutions.

[0006] A point cloud-completed tumor 3D reconstruction system based on gridding and attention mechanism, including: data acquisition module, image preprocessing module, and model reconstruction module;

[0007] The data acquisition module is used to collect and acquire two-dimensional CT image data;

[0008] The image preprocessing module includes: preprocessing two-dimensional CT images;

[0009] The model reconstruction module is used to perform three-dimensional reconstruction of preprocessed two-dimensional CT image data to generate three-dimensional medical images. It includes a point cloud completion network of grid and geometry-aware Transformer, extracts contour information from two-dimensional segmentation masks, and predicts and reconstructs the complete surface shape of the liver and tumor from the contour information.

[0010] By adopting the above technical solution, high-precision real-time three-dimensional reconstruction of the liver and tumor can be achieved. The system can accurately restore the three-dimensional shape, volume and spatial position relationship of the tumor with the surrounding liver tissue, which helps to accurately evaluate the characteristics of the lesion.

[0011] Optionally, the point cloud completion network consists of a feature extraction module and a multi-stage point cloud generation module;

[0012] The feature extraction module adopts a dual-branch structure to extract the shape code f from the input sparse, incomplete contour point cloud P. It includes the first branch: the 3D CNN-based branch converts the point cloud into a 3DGrid to extract local geometric features, and the second branch: the Transformer-based branch dynamically evaluates the importance of each point by establishing long-range dependencies, captures global context information, and extracts global semantic features.

[0013] Under the guidance of shape coding, the multi-stage point generation module uses point generators with different structures to gradually reconstruct a complete and clear 3D point cloud shape of the liver and tumor from coarse to fine.

[0014] By adopting the above technical solution, introducing 3D Grid as the intermediate representation of point cloud, combining it with the Transformer mechanism and the multi-stage point generation module, we can achieve the gradual reconstruction of the complete and clear 3D point cloud shape of the liver and tumor from coarse-grained to fine-grained.

[0015] Optionally, the feature extraction module takes an N × 3 contour point cloud P as input;

[0016] The first branch uses the grid layer to process the input data and convert it into 64 3 Dimensional 3DGrid, hierarchical feature extraction is performed through cascaded 3D CNN, a two-layer fully connected network is constructed, and a local geometric feature f1 of dimension 1×C1 is obtained through nonlinear transformation; the extraction formula of f1 is:

[0017] G=Gridding(P)

[0018]

[0019] The second branch is based on the Transformer architecture. The input contour point cloud P first generates a feature vector matrix X with a dimension of n×c through MLP. This feature matrix X is used as the input of the CTrans module. The global context information is extracted through the cascaded CTrans1 and CTrans2 modules, and finally the global semantic feature f2 is obtained through maximum pooling.

[0020] The local geometric features f1 obtained by the 3D CNN branch are concatenated with the global semantic features f2 obtained by the Transformer branch to form a comprehensive shape code f.

[0021] By employing this technical solution, the dual-branch feature extraction module extracts both local geometric details and global semantic features of the liver and tumor. The local feature branch extracts subtle structural information, while the global feature branch models overall morphological relationships. The two complement each other and provide multi-layered feature support for subsequent 3D reconstruction.

[0022] Optionally, perform hierarchical feature extraction through four cascaded 3D CNNs with feature dimensions of 32, 64, 128, and 256, respectively, to build a two-layer fully connected network with layer sizes of 2048 and 1024;

[0023] Use the gridding module to convert the input partial point cloud P into a voxel grid G∈R H×W×C ,The grid features are extracted through four stacked 3D convolutional layers, and the local features f1 of the point cloud are extracted through the fully connected layer;

[0024] According to the feature aggregation requirements, the corresponding query value is constructed, and two CTrans module structure variants are derived: X = MLP (P), Y = FPS (X, d);

[0025] The CTrans1 module uses the downsampled features as query input. By applying the farthest point sampling to the feature matrix X, the downsampled feature matrix Y of size n / d×c is obtained as the query of the CTrans1 module. The CTrans2 module uses the feature matrix X as query input and establishes the overall structural association of the point cloud through the global feature interaction mechanism.

[0026] X is the point cloud feature extracted by MLP, Y is the downsampled feature obtained by using the farthest point sampling method on X, where d is the downsampling ratio. The CTrans1 module uses Y as the query input, and W is the weight:

[0027]

[0028] f1=Norm(Q1+MultiHead(Q1,K1,V1))

[0029] CTrans1(X,d)=FFN(f1)+f1;

[0030] The CTrans2 module uses the feature matrix X as the query input, specifically:

[0031]

[0032] f2=Norm(Q2+MultiHead(Q2,K2,V2))

[0033] CTrans2(X)=FFN(f2)+f2;

[0034] The global semantic feature f2 is obtained by maximum pooling. The specific feature extraction formula is:

[0035] f2=Maxpool(CTrans2(CTrans1(X, d)));

[0036] By concatenating these two feature vectors f1 and f 2, Get the shape code f, which contains the overall contour information of the input point cloud, specifically:

[0037] f=Concat(f1, f2).

[0038] By employing these technical solutions, the 3D CNN branch excels at capturing the edge structures and local details of the liver and tumor, while the Transformer branch uses an attention mechanism to model global topological relationships and understand the overall morphology and spatial layout of the organ. Relying solely on local features makes it difficult to grasp the global structure, while purely global information can overlook key details. Therefore, the synergy between the two is particularly critical. During the feature fusion process, the 3D CNN branch, guided by the Transformer, optimizes the local geometric representation to conform to the overall morphology, while the Transformer branch leverages the detailed information supplemented by the CNN to increase sensitivity to boundary regions and morphological changes. This complementary interaction ensures that the shape encoding f is both locally accurate and globally consistent, providing a more complete and accurate shape prior for subsequent point cloud generation, improving reconstruction quality.

[0039] Optionally, the multi-stage point generation module includes a coarse point generator and two refined point generators.

[0040] Optionally, a method for generating a preliminary point cloud P0 of size N0×3 by a coarse point generator includes:

[0041] P c =Reshape(CTrans2(MLP(f)))

[0042] P0=FPS(Concat(P,P c ))

[0043] A1. Through MLP and Reshape, the shape code f is decoded into a feature vector matrix to obtain the decoded point cloud P c ;

[0044] A2, transform the features through the cascaded CTrans2 module;

[0045] A3. The point cloud features are converted into three-dimensional coordinates through MLP and Reshape, generating a size N c ×3 point cloud P c ;

[0046] A4. Point cloud P c Merge with the input contour point cloud P and sample it through the farthest point sampling algorithm to obtain the rough point cloud P0.

[0047] By adopting the above technical solution, the coarse point generator generates a preliminary point cloud P0 of size N0×3 to capture the overall three-dimensional shape of the liver and tumor, providing a rough but complete three-dimensional structural representation.

[0048] Optionally, the refined point generator includes MLP, CTrans2 and Skip Transformer modules, which are used to refine the point cloud P generated in the previous stage. i Realize fine-grained point cloud structure P i+1 The specific methods include:

[0049] B1. Shape code f and point cloud P i After processing by MLP and CTrans2, the generated point cloud features are used as the input of the STrans module;

[0050] f p =Concat(MLP(P i ), f)

[0051] f s =CTrans2(MLP(f p ))

[0052] The point cloud P i Perform feature fusion with shape code f to obtain enhanced point cloud features f p ; Further refine the features through MLP and CTrans2 operations to construct a high-order representation f s , P0 obtained by the coarse point generator and P1 obtained by the first refined point generator are collectively referred to as P i ;

[0053] B2. In the STrans module, q and k are concatenated and fed into the MLP to obtain the feature neighbor v. The relationship matrix between points is generated through the relationship calculation module. The KNN method is used to calculate the position code δ of the point. δ is added to the relationship matrix and fed into the AttenMLP to generate the attention weight distribution.

[0054] δ=KNN(P i )

[0055] v = MLP(Concat(q, k))

[0056] w A =AttenMLP(Relation(q,k)+δ)

[0057] δ is the point cloud P i Use K-nearest neighbor algorithm (KNN) to get the position code, w A is the point cloud attention weight,

[0058] B3. Multiply the generated attention weights by the feature vector v with position encoding element by element to obtain the weighted feature representation;

[0059] B4. Process weighted features through MLP to generate high-resolution point cloud features:

[0060]

[0061] f output It is the high-resolution point cloud feature generated by the STrans module;

[0062] B5. The features output by the STrans module are fed into the SharedMLP to calculate the point offset ΔP:

[0063] ΔP=SharedMLP(f output )

[0064] Offset and point cloud P after copying r times i Add together to get the refined fine-grained point cloud P i+1 ;

[0065] P i+1 =Tile(P i , r)+ΔP

[0066] Tile(P i , r) identifies the point cloud P i Copied r times.

[0067] By adopting the above technical solution, two refined point generator modules are used to gradually refine the point cloud structure of the liver tumor to reconstruct a dense 3D liver tumor point cloud.

[0068] In summary, this application has at least the following beneficial effects:

[0069] 1. This application proposes a point cloud completion network (GRFormer) for liver and tumor reconstruction. By introducing 3DGrid as an intermediate representation of point clouds and combining it with the Transformer mechanism, the network successfully reconstructs high-quality, complete three-dimensional shapes with precise geometric details, achieving more accurate and efficient liver and tumor reconstruction. It can provide clinicians with precise three-dimensional reconstruction tools to assist in disease assessment and surgical planning, helping to improve the accuracy of disease diagnosis and assist in the formulation of treatment plans.

[0070] 2. The dual-branch feature extraction architecture of this application is used to capture local and global information respectively, thereby providing comprehensive and accurate feature expression for subsequent point cloud generation.

[0071] 3. The multi-stage point generation module of this application ensures that the generated shapes are fully restored in terms of accuracy and details by fusing Transformer modules with different structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0073] Figure 1 This is a flowchart for reconstructing the 3D shape of the liver and tumor from CT images.

[0074] Figure 2 It is the overall structure of GR Former.

[0075] Figure 3 It is the feature extraction module.

[0076] Figure 4 is a coarse point generator.

[0077] Figure 5 is the refinement point generator.

[0078] Figure 6 It is the Cross Transformer (CTrans) module.

[0079] Figure 7 It is the Skip Transformer (STrans) module. DETAILED DESCRIPTION

[0080] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0081] The following is combined with Figures 1 to 7 This application is described in further detail.

[0082] The embodiments of the present application disclose a point cloud completion tumor three-dimensional reconstruction system based on gridding and attention mechanism.

[0083] Reference Figures 1 to 7 , a point cloud completion tumor 3D reconstruction system based on gridding and attention mechanism, including: data acquisition module, image preprocessing module, and model reconstruction module.

[0084] The data acquisition module is used to collect and acquire two-dimensional CT image data.

[0085] The image preprocessing module includes: preprocessing the two-dimensional CT images, extracting the contour point cloud data of the liver and tumor based on the accurately delineated liver and tumor areas, and normalizing them, thereby constructing a high-precision three-dimensional point cloud dataset of the liver and tumor.

[0086] The model reconstruction module is used to perform three-dimensional reconstruction of preprocessed two-dimensional CT image data to generate three-dimensional medical images. It includes a point cloud completion network of grid and geometry-aware Transformer, extracts contour information from two-dimensional segmentation masks, and predicts and reconstructs the complete surface shape of the liver and tumor from the contour information.

[0087] The point cloud completion network consists of a feature extraction module and a multi-stage point cloud generation module.

[0088] The feature extraction module adopts a dual-branch structure, including a first branch and a second branch, which is used to extract the shape code f from the input sparse and incomplete contour point cloud P.

[0089] The feature extraction module takes the contour point cloud P as input.

[0090] The first branch: The branch based on 3D CNN (three-dimensional convolution) converts the point cloud into a 3D grid to extract the local geometric features of the liver and tumor. The grid layer is used to process the input data and convert it into a 64 3Dimensional 3D Grid, hierarchical feature extraction is performed through four cascaded 3D CNNs (four is ideal, as too many convolutional modules will cause information loss), with feature dimensions of 32, 64, 128, and 256, respectively, gradually capturing feature representations from local geometric structure to global semantic information. To further enhance feature representation, a two-layer fully connected network is constructed with layer sizes of 2048 and 1024, respectively. Local geometric features f1 of dimension 1×C1 are obtained through nonlinear transformations. A gridding module is used to convert the input partial point cloud P into a voxel grid. The grid features are extracted through four stacked 3D convolutional layers, and the local features f1 of the point cloud are extracted through a fully connected layer.

[0091] Extraction formula of f1:

[0092] G=Gridding(P)

[0093]

[0094] The second branch: The Transformer-based branch dynamically evaluates the importance of each point by establishing long-distance dependencies, captures global context information, and extracts global semantic features f2.

[0095] Based on the Transformer architecture, the Cross Transformer (CTrans) module is employed. The input contour point cloud P is first processed through an MLP to generate a feature vector matrix X of dimension n×c, where each row represents the feature vector of a point. This feature matrix X serves as the input to the CTrans module. Global context information is extracted through the cascaded CTrans1 and CTrans2 modules, and global semantic features f2 are finally obtained through maximum pooling.

[0096] According to different feature aggregation requirements, the corresponding query value is constructed, and two CTrans module structure variants are derived, specifically: X = MLP (P), Y = FPS (X, f).

[0097] The CTrans1 module uses the downsampled features as query input and applies farthest point sampling to the feature matrix X to obtain a downsampled feature matrix Y of size n / d×c (n / d means that the input data is n points, and only n / d points are retained after farthest point sampling. Each point has c coordinate features, so it is (n / d)×c) as the query of the CTrans1 module; the CTrans2 module uses the feature matrix X as query input and establishes the overall structural association of the point cloud through the global feature interaction mechanism;

[0098] X is the point cloud feature extracted by MLP, Y is the downsampled feature obtained by using the farthest point sampling method on X, where d is the downsampling ratio. The CTrans1 module uses Y as the query input, and W is the weight:

[0099]

[0100] f1=Norm(Q1+MultiHead(Q1,K1,V1))

[0101] CTrans1(X,d)=FFN(f1)+f1;

[0102] The CTrans2 module uses the feature matrix X as the query input, specifically:

[0103]

[0104] f2=Norm(Q2+MultiHead(Q2,K2,V2))

[0105] CTrans2(X)=FFN(f2)+f2;

[0106] Q, K, and V are the inputs to the Transformer module, representing the query, key, and value, respectively. They are derived by transforming the input feature X through different weight matrices W. Q determines the location of interest, K calculates correlation with other locations, and V provides the actual feature content. The degree of match between Q and K determines the weighting of V, enabling the model to dynamically focus on important information.

[0107] The global semantic feature f2 is obtained by maximum pooling. The specific feature extraction formula is:

[0108] f2=MaxPool(CTrans2(CTrans1(X, d)));

[0109] The local geometric features f1 and global semantic features f2 of the input point cloud are obtained by respectively using the 3D CNN-based and Transformer-based branches. 2, Get the shape code f, which contains the overall contour information of the input point cloud, specifically:

[0110] f=Concat(f1, f2).

[0111] The Transformer-based branch uses an attention mechanism to build long-range dependencies and dynamically assess the importance of each location in the input data. This mechanism accurately quantifies the contribution of different regions to feature extraction by calculating correlation scores across all locations. This global interaction modeling approach enables the Transformer branch to fully capture the global context of the input data and extract semantically distinct global features, including key information such as tumor malignancy and lesion boundary characteristics.

[0112] The 3D CNN branch excels at capturing edge structures and local details of the liver and tumor, while the Transformer branch uses an attention mechanism to model global topological relationships and understand the overall morphology and spatial layout of the organs. Relying solely on local features makes it difficult to grasp global structure, while purely global information can overlook key details. Therefore, the synergistic effect of the two is crucial. During the feature fusion process, the 3D CNN branch, guided by the Transformer, optimizes local geometric representations to align with the overall morphology. The Transformer branch leverages the detailed information supplemented by the CNN to enhance sensitivity to boundary regions and morphological variations. The dual-branch feature extraction module extracts both local geometric details and global semantic features of the liver and tumor. The local feature branch extracts subtle structural information, while the global feature branch models overall morphological relationships. The two complement each other to provide multi-layered feature support for the subsequent 3D reconstruction process. This complementary interaction ensures that the shape encoding f is both locally accurate and globally consistent, providing a more complete and accurate shape prior for subsequent point cloud generation, improving reconstruction quality.

[0113] Under the guidance of shape coding, the multi-stage point generation module uses point generators with different structures to gradually reconstruct a complete and clear 3D point cloud shape of the liver and tumor from coarse to fine.

[0114] The multi-stage point generation module consists of a coarse point generator and two refined point generators.

[0115] Coarse point cloud generator: In this module, the shape code f is decoded through the MLP, Reshape and CTrans2 modules to obtain a coarse point cloud P0 that can contain global information, which will serve as the seed point cloud for the subsequent reconstruction and refinement process.

[0116] The method of generating a preliminary point cloud P0 of size N0×3 by a coarse point generator includes:

[0117] A1. Through MLP and Reshape, the shape code f is decoded into a feature vector matrix to obtain the decoded point cloud P c ;

[0118] A2, transforming features through three cascaded CTrans2 modules to capture more complex and detailed feature relationships;

[0119] A3. The point cloud features are processed by MLP and Reshape to convert the feature vector into three-dimensional coordinates, generating a size N c ×3 point cloud P c ;

[0120] A4. Point cloud P c Merge with the input contour point cloud P and sample it through the farthest point sampling algorithm to obtain the rough point cloud P0.

[0121] P c =Reshape(CTrans2(MLP(f)))

[0122] P0=FPS(Concat(P,P c ))

[0123] The coarse point generator generates a preliminary point cloud P0 of size N0×3 to capture the overall 3D shape of the liver and tumor, providing a rough but complete 3D structural representation.

[0124] The refined point generator includes MLP, CTrans2 and Skip Transformer modules. The refined point generator is used to generate the point cloud P generated in the previous stage. i Realize fine-grained point cloud structure P i+1 The specific methods include:

[0125] B1. Shape code f and point cloud P i After processing by MLP and CTrans2, the generated point cloud features are used as the input of the STrans module;

[0126] f p =Concat(MLP(P i ), f)

[0127] f s =CTrans2(MLP(f p ))

[0128] The point cloud P i Perform feature fusion with shape code f to obtain enhanced point cloud features f p ; Further refine the features through MLP and CTrans2 operations to construct a high-order representation f s , P0 obtained by the coarse point generator and P1 obtained by the first refined point generator are collectively referred to as P i ;

[0129] B2. In the STrans module, q and k are concatenated (splicing refers to the operation of connecting and merging multiple feature tensors along a certain dimension (usually the channel dimension), which is a form of feature fusion. If the size of q and k are both n×c, concatenating along the channel dimension is the same as concatenating along the c dimension, and the resulting feature size after concatenation is n×2c) and then fed into the MLP to obtain the feature adjacency v. On the other hand, the relationship matrix between points is generated through the relationship calculation module, and the point position code δ is calculated using the KNN method. δ is added to the relationship matrix and fed into the AttenMLP to generate the attention weight distribution.

[0130] δ=KNN(P i )

[0131] v = MLP(Concat(q, k))

[0132] w A =AttenMLP(Reelation(q,k)+δ)

[0133] δ is the point cloud P i Use K-nearest neighbor algorithm (KNN) to get the position code, w A is the point cloud attention weight,

[0134] B3. Multiply the generated attention weights by the feature vector v with position encoding element by element to obtain the weighted feature representation;

[0135] B4. Process weighted features through MLP to generate high-resolution point cloud features:

[0136]

[0137] f output It is the high-resolution point cloud feature generated by the STrans module;

[0138] B5. The features output by the STrans module are fed into the SharedMLP to calculate the point offset ΔP:

[0139] ΔP=SharedMLP(f output )

[0140] Offset and point cloud P after copying r times i Add together to get the refined fine-grained point cloud P i+1 ;

[0141] P i+1 =Tile(P i , r)+ΔP

[0142] Tile(P i , r) identifies the point cloud P iCopied r times.

[0143] By introducing 3D Grid as the intermediate representation of point cloud, and combining it with the Transformer mechanism and the multi-stage point generation module, we can achieve the progressive reconstruction of complete and clear 3D point cloud shapes of the liver and tumor from coarse-grained to fine-grained. Through the cascade of coarse point generators and two refined point generators, we have achieved coarse-to-fine point cloud modeling of the liver and tumor. The advantages of this method are: first, the overall morphology of the organ is quickly reconstructed through low-resolution point clouds, and then the details of the tumor are accurately portrayed through two-level progressive refinement. This hierarchical and progressive structure can not only ensure the reconstruction efficiency, but also gradually improve the geometric accuracy of key areas, and finally achieve the coordinated optimization representation of the anatomical structure and the lesion area.

[0144] The above are only preferred embodiments of the invention and are not intended to limit the invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the invention should be included in the scope of protection of the invention.

Claims

1. A point cloud completion tumor 3D reconstruction system based on gridding and attention mechanism, characterized by: include: Data acquisition module, image preprocessing module, model reconstruction module; The data acquisition module is used to collect and acquire two-dimensional CT image data; The image preprocessing module includes: preprocessing two-dimensional CT images; The model reconstruction module is used to perform three-dimensional reconstruction of preprocessed two-dimensional CT image data to generate three-dimensional medical images. It includes a point cloud completion network of grid and geometry-aware Transformer, extracts contour information from two-dimensional segmentation masks, and predicts and reconstructs the complete surface shape of the liver and tumor from the contour information.

2. The point cloud completion tumor 3D reconstruction system based on gridding and attention mechanism according to claim 1, characterized in that: The point cloud completion network consists of a feature extraction module and a multi-stage point cloud generation module; The feature extraction module adopts a dual-branch structure to extract the shape code f from the input sparse, incomplete contour point cloud P. It includes the first branch: the 3D CNN-based branch converts the point cloud into a 3DGrid to extract local geometric features, and the second branch: the Transformer-based branch dynamically evaluates the importance of each point by establishing long-range dependencies, captures global context information, and extracts global semantic features. Under the guidance of shape coding, the multi-stage point generation module uses point generators with different structures to gradually reconstruct a complete and clear 3D point cloud shape of the liver and tumor from coarse to fine.

3. The point cloud completion tumor 3D reconstruction system based on gridding and attention mechanism according to claim 2, characterized in that: The feature extraction module takes the N×3 contour point cloud P as input; The first branch uses the grid layer to process the input data and convert it into 64 3 Dimensional 3DGrid, hierarchical feature extraction is performed through cascaded 3D CNN, a two-layer fully connected network is constructed, and a local geometric feature f1 of dimension 1×C1 is obtained through nonlinear transformation; the extraction formula of f1 is: G=Gridding(P) The second branch is based on the Transformer architecture. The input contour point cloud P first generates a feature vector matrix X with a dimension of n×c through MLP. This feature matrix X is used as the input of the CTrans module. The global context information is extracted through the cascaded CTrans1 and CTrans2 modules, and finally the global semantic feature f2 is obtained through maximum pooling. The local geometric features f1 obtained by the 3D CNN branch are concatenated with the global semantic features f2 obtained by the Transformer branch to form a comprehensive shape code f.

4. The point cloud completion tumor 3D reconstruction system based on gridding and attention mechanism according to claim 3, characterized in that: Hierarchical feature extraction is performed through four cascaded 3D CNNs with feature dimensions of 32, 64, 128, and 256, respectively, to construct a two-layer fully connected network with layer sizes of 2048 and 1024; Use the gridding module to convert the input partial point cloud P into a voxel grid G∈R H×W×C ,The grid features are extracted through four stacked 3D convolutional layers, and the local features f1 of the point cloud are extracted through the fully connected layer; According to the feature aggregation requirements, the corresponding query value is constructed, and two CTrans module structure variants are derived: X = MLP (P), Y = FPS (X, d); The CTrans1 module uses the downsampled features as query input. By applying the farthest point sampling to the feature matrix X, the downsampled feature matrix Y of size n / d×c is obtained as the query of the CTrans1 module. The CTrans2 module uses the feature matrix X as query input and establishes the overall structural association of the point cloud through the global feature interaction mechanism. X is the point cloud feature extracted by MLP, Y is the downsampled feature obtained by using the farthest point sampling method on X, where d is the downsampling ratio. The CTrans1 module uses Y as the query input, and W is the weight: f1=Norm(Q1+MultiHead(Q1,K1,V1)) CTrans1(X,d)=FFN(f1)+f1; The CTrans2 module uses the feature matrix X as the query input, specifically: f2=Norm(Q2+MultiHead(Q2,K2,V2)) CTrans2(X)=FFN(f2)+f2; The global semantic feature f2 is obtained by maximum pooling. The specific feature extraction formula is: f2=Maxpool(CTrans2(CTrans1(C,d))); By concatenating these two feature vectors f1 and f 2, Get the shape code f, which contains the overall contour information of the input point cloud, specifically: f=Concat(f1, f2).

5. The point cloud completion tumor 3D reconstruction system based on gridding and attention mechanism according to claim 3, characterized in that: The multi-stage point generation module includes a coarse point generator and two refined point generators.

6. The point cloud completion tumor 3D reconstruction system based on gridding and attention mechanism according to claim 5, characterized in that: The method of generating a preliminary point cloud P0 of size N0×3 by a coarse point generator includes: P c =Reshape(CTrans2(MLP(f))) P0=FPS(Concat(P,P c )) A1. Through MLP and Reshape, the shape code f is decoded into a feature vector matrix to obtain the decoded point cloud P c ; A2, transform the features through the cascaded CTrans2 module; A3. The point cloud features are converted into three-dimensional coordinates through MLP and Reshape, generating a size N c ×3 point cloud P c ; A4. Point cloud P c Merge with the input contour point cloud P and sample it through the farthest point sampling algorithm to obtain the rough point cloud P0.

7. The point cloud completion tumor 3D reconstruction system based on gridding and attention mechanism according to claim 6, characterized in that: The refined point generator includes MLP, CTrans2 and Skip Transformer modules. The refined point generator is used to generate the point cloud P generated in the previous stage. i Realize fine-grained point cloud structure P i+1 The specific methods include: B1. Shape code f and point cloud P i After processing by MLP and CTrans2, the generated point cloud features are used as the input of the STrans module; f p =Concat(MLP(P i ),f) f s =CTrans2(MLP(f p )) The point cloud P i Perform feature fusion with shape code f to obtain enhanced point cloud features f p ; Further refine the features through MLP and CTrans2 operations to construct a high-order representation f s , P0 obtained by the coarse point generator and P1 obtained by the first refined point generator are collectively referred to as P i ; B2. In the STrans module, q and k are concatenated and fed into the MLP to obtain the feature neighbor v. The relationship matrix between points is generated through the relationship calculation module. The KNN method is used to calculate the position code δ of the point. δ is added to the relationship matrix and fed into the AttenMLP to generate the attention weight distribution. δ=KNN(P i ) v = MLP(Concat(q, k)) w A =AttenMLP(Relation(q,k)+d) δ is the point cloud P i Use K-nearest neighbor algorithm (KNN) to get the position code, w A is the point cloud attention weight, B3. Multiply the generated attention weights by the feature vector v with position encoding element by element to obtain the weighted feature representation; B4. Process weighted features through MLP to generate high-resolution point cloud features: f output It is the high-resolution point cloud feature generated by the STrans module; B5. The features output by the STrans module are fed into the SharedMLP to calculate the point offset ΔP: ΔP=SharedMLP(f output ) Offset and point cloud P after copying r times i Add together to get the refined fine-grained point cloud P i+1 ; P i+1 =Tile(P i ,r)+ΔP Tile(P i , r) identifies the point cloud P i Copied r times.

Images