Point cloud completion tumor three-dimensional reconstruction system based on gridding and attention mechanism

Abstract

The application discloses a grid-based and attention mechanism-based point cloud completion tumor three-dimensional reconstruction system, relates to the technical field of medical image processing, and comprises a data acquisition module, an image preprocessing module and a model reconstruction module.The data acquisition module is used for collecting and acquiring two-dimensional CT image data.The image preprocessing module comprises preprocessing of the two-dimensional CT image.The model reconstruction module is used for three-dimensional reconstruction of the preprocessed two-dimensional CT image data, generates three-dimensional medical images, comprises a point cloud completion network of a grid and a geometric perception Transformer, extracts contour information from a two-dimensional segmentation mask, and predicts and reconstructs the complete surface shape of a liver and a tumor from the contour information.Thus, the existing problems of medical images are solved, more accurate three-dimensional liver and tumor shape reconstruction is provided for clinic, the accuracy of disease diagnosis and the formulation of an auxiliary treatment scheme are improved, intraoperative risk is reduced, and the success rate of surgery is improved.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, and in particular to a point cloud-based three-dimensional tumor reconstruction system based on meshing and attention mechanisms. Background Technology

[0002] Liver cancer is one of the most common malignant tumors worldwide, posing a serious threat to human health and life. With the rise of minimally invasive treatments, radiofrequency ablation, targeting local ablation, has been increasingly applied to the treatment of malignant liver tumors. It is an image-guided technique for eradicating tumors, and its effectiveness largely depends on the surgeon's ability to accurately identify and locate the tumor in CT images.

[0003] CT images provide detailed information about human organs and lesions, but their two-dimensional representation lacks spatial depth. Clinically, commonly used two-dimensional CT images only provide anatomical information of a single plane of the liver, failing to fully display the three-dimensional morphology, positional relationships, and surrounding tissue environment of the liver and tumors, making it difficult to intuitively reflect the three-dimensional anatomical structure. This lack of intuitive spatial reference during operation increases the risk of inaccurate ablation areas, potentially leading to tumor residue or damage to healthy tissue. Therefore, accurate three-dimensional reconstruction technology is crucial for the successful implementation of radiofrequency ablation. Reconstructing high-precision three-dimensional shapes from two-dimensional medical images has become a key challenge in the fields of computer vision and medical image analysis. Summary of the Invention

[0004] To address the aforementioned issues and provide more accurate three-dimensional reconstructions of liver and tumor morphology for clinical use, thereby improving the accuracy of disease diagnosis and the formulation of adjuvant treatment plans, this application provides a point cloud-based three-dimensional tumor reconstruction system based on meshing and attention mechanisms.

[0005] The point cloud completion tumor 3D reconstruction system based on meshing and attention mechanisms provided in this application adopts the following technical solution.

[0006] A point cloud completion tumor 3D reconstruction system based on meshing and attention mechanisms includes: a data acquisition module, an image preprocessing module, and a model reconstruction module;

[0007] The data acquisition module is used to collect and acquire two-dimensional CT image data;

[0008] The image preprocessing module includes: preprocessing two-dimensional CT images;

[0009] The model reconstruction module is used to perform three-dimensional reconstruction on preprocessed two-dimensional CT image data to generate three-dimensional medical images, including a point cloud completion network of mesh and geometrically perceptive Transformer, extracting contour information from two-dimensional segmentation mask, and predicting and reconstructing the complete surface shape of the liver and tumor from the contour information.

[0010] By adopting the above technical solution, high-precision real-time three-dimensional reconstruction of the liver and tumors can be achieved. The system can accurately restore the three-dimensional shape, size and spatial relationship of the tumor with the surrounding liver tissue, which helps to accurately assess the characteristics of the lesion.

[0011] Optionally, the point cloud completion network consists of a feature extraction module and a multi-stage point cloud generation module;

[0012] The feature extraction module adopts a dual-branch structure to extract shape encoding f from the input sparse and incomplete contour point cloud P. It includes a first branch: a 3D CNN-based branch that extracts local geometric features by converting the point cloud into a 3D grid. It also includes a second branch: a Transformer-based branch that dynamically evaluates the importance of each point by establishing long-distance dependencies, captures global contextual information, and extracts global semantic features.

[0013] Guided by shape encoding, the multi-stage point generation module uses point generators with different structures to gradually reconstruct a complete and clear 3D point cloud shape of the liver and tumor from coarse to fine.

[0014] By adopting the above technical solution, introducing 3D Grid as the intermediate representation of point cloud, and combining it with the Transformer mechanism and a multi-stage point generation module, a complete and clear 3D point cloud shape of the liver and tumor can be reconstructed progressively from coarse to fine granular.

[0015] Optionally, the feature extraction module uses The contour point cloud P is used as input;

[0016] The first branch processes the input data using a grid layer, converting it to 64. 3 A 3D grid of dimension 1 is obtained through hierarchical feature extraction using cascaded 3D CNNs, constructing a two-layer fully connected network, and obtaining the dimension 1 through non-linear transformation. The local geometric feature f1; the extraction formula for f1:

[0017]

[0018]

[0019] ;

[0020] The second branch is based on the Transformer architecture. The input contour point cloud P is first generated using an MLP to obtain a dimension of... The feature vector matrix X is used as the input of the CTrans module. Global context information is extracted through the cascaded CTrans1 and CTrans2 modules, and finally global semantic features f2 are obtained through max pooling.

[0021] The local geometric features f1 obtained from the 3D CNN branch and the global semantic features f2 obtained from the Transformer branch are concatenated to form a comprehensive shape code f.

[0022] By employing the above technical solution, the dual-branch feature extraction module can extract local geometric details and global semantic features of the liver and tumor. The local feature branch extracts fine structural information, while the global feature branch models the overall morphological relationships. The two complement each other and are integrated to provide multi-level feature support for the subsequent 3D reconstruction process.

[0023] Optionally, hierarchical feature extraction is performed using four cascaded 3D CNNs with feature dimensions of 32, 64, 128 and 256 respectively, to construct a two-layer fully connected network with layer sizes of 2048 and 1024 respectively.

[0024] The input partial point cloud P is converted into a voxel mesh using the meshing module. The mesh features are extracted through four stacked 3D convolutional layers, and the local features f1 of the point cloud are extracted through a fully connected layer.

[0025] Based on the feature aggregation requirements, corresponding query values ​​are constructed, resulting in two variations of the CTrans module structure: , ;

[0026] The CTrans1 module uses downsampled features as query input. By applying farthest-point sampling to the feature matrix X, it obtains a sample of size X. The downsampled feature matrix Y is used as the query for the CTrans1 module; the CTrans2 module uses the feature matrix X as the query input and establishes the overall structural association of the point cloud through a global feature interaction mechanism.

[0027] X represents the point cloud features extracted by MLP, and Y represents the downsampled features of X obtained by using the farthest point sampling method, where d is the downsampling ratio. The CTrans1 module uses Y as the query input, and W is the weight.

[0028] , ,

[0029]

[0030] ;

[0031] The CTrans2 module uses the feature matrix X as the query input, specifically:

[0032] , ,

[0033]

[0034] ;

[0035] The global semantic feature f2 is obtained through max pooling. The specific feature extraction formula is as follows:

[0036] ;

[0037] By concatenating these two feature vectors f1 and f2 2, Obtain the shape code f, which contains the overall contour information of the input point cloud, specifically:

[0038] .

[0039] By employing the aforementioned technical solutions, the 3D CNN branch excels at capturing the edge structures and local details of the liver and tumors, while the Transformer branch models global topological relationships through an attention mechanism, understanding the overall morphology and spatial layout of the organ. Relying solely on local features is insufficient to grasp the global structure, while purely global information may overlook crucial details; therefore, the synergistic effect of both is paramount. During feature fusion, the 3D CNN branch, guided by the Transformer, optimizes local geometric representations to conform to the overall morphology, while the Transformer branch, aided by the detailed information supplemented by the CNN, enhances its sensitivity to boundary regions and morphological changes. This complementary interaction mechanism ensures that the shape encoding f possesses both local accuracy and global consistency, thereby providing a more complete and accurate shape prior for subsequent point cloud generation and improving reconstruction quality.

[0040] Optionally, the multi-stage point generation module includes a coarse point generator and two fine point generators.

[0041] Optionally, a coarse point generator can be used to generate a size of [size missing]. The methods for the initial point cloud P0 include:

[0042]

[0043]

[0044] A1. Using MLP and Reshape, the shape code f is decoded into a feature vector matrix to obtain the decoded point cloud P. c ;

[0045] A2. Transform the features using the cascaded CTrans2 modules;

[0046] A3. Point cloud features are converted into three-dimensional coordinates using MLP and Reshape, generating a size... Point cloud P c ;

[0047] A4. Point cloud P c The point cloud P is merged with the input contour point cloud P and sampled using the farthest point sampling algorithm to obtain a coarse point cloud P0.

[0048] By adopting the above technical solution, the coarse point generator generates a preliminary point cloud P0 of size N0×3 to capture the overall three-dimensional shape of the liver and tumor, providing a coarse but complete three-dimensional structural representation.

[0049] Optionally, the refinement point generator includes MLP, CTrans2, and Skip Transformer modules. The refinement point generator refines the point cloud P generated in the previous stage. i Achieving fine-grained point cloud structure P i＋1 The reconstruction methods include:

[0050] B1, shape encoding f and point cloud P i The point cloud features generated after processing by MLP and CTrans2 are used as input to the STrans module.

[0051]

[0052]

[0053] Point cloud P i Feature fusion with shape encoding f yields enhanced point cloud features f. p Features were further refined using MLP and CTrans2 operations to construct a higher-order representation f. s The P0 obtained by the coarse point generator and the P1 obtained by the first refined point generator are both collectively referred to as P. i ;

[0054] B2. In the STrans module, q and k are concatenated and fed into the MLP to obtain the feature neighbor v. On the other hand, the relation calculation module generates the relation matrix between points. The K nearest neighbor algorithm (KNN) is used to calculate the position code δ of the point. δ is added to the relation matrix and fed into AttenMLP to generate the attention weight distribution.

[0055]

[0056]

[0057]

[0058] δ represents the pair of point cloud P i The positional encoding is obtained using the K-Nearest Neighbors (KNN) algorithm, w A For point cloud attention weights,

[0059] B3. Multiply the generated attention weights element-wise with the feature vector v with positional encoding to obtain a weighted feature representation;

[0060] B4. Generate high-resolution point cloud features by processing weighted features through MLP:

[0061]

[0062] f output These are high-resolution point cloud features generated by the STrans module;

[0063] B5. The features output by the STrans module are fed into SharedMLP to calculate the point offset. :

[0064]

[0065] Offset and point cloud P after copying by r times i Adding them together yields the refined, fine-grained point cloud P. i＋1 ;

[0066]

[0067] Tile (P) i The r) identifier will mark the point cloud P i It was copied r times.

[0068] By adopting the above technical solution, two refinement point generator modules were used to gradually refine the data in order to reconstruct a dense 3D liver tumor point cloud structure.

[0069] In summary, this application includes at least the following beneficial effects:

[0070] 1. This application proposes a point cloud completion network (GRFormer) suitable for liver and tumor reconstruction. This network introduces a 3D grid as an intermediate representation of the point cloud and combines it with a Transformer mechanism to successfully reconstruct high-quality, complete 3D shapes with precise geometric details. This enables more accurate and efficient liver and tumor reconstruction, providing clinicians with precise 3D reconstruction tools to assist in disease assessment and surgical planning, thus improving the accuracy of disease diagnosis and aiding in the development of treatment plans.

[0071] 2. The dual-branch feature extraction architecture of this application captures local and global information respectively, thereby providing a comprehensive and accurate feature representation for subsequent point cloud generation.

[0072] 3. The multi-stage point generation module of this application, by integrating Transformer modules with different structures, ensures that the generated shape is fully restored in terms of accuracy and detail. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 This is a flowchart for reconstructing the 3D shape of the liver and tumor from CT images.

[0075] Figure 2 This is the overall architecture of GR Former.

[0076] Figure 3 It is the feature extraction module.

[0077] Figure 4 It is a coarse point generator.

[0078] Figure 5 It is a refinement point generator.

[0079] Figure 6 It is the Cross Transformer (CTrans) module.

[0080] Figure 7 It is the Skip Transformer (STrans) module. Detailed Implementation

[0081] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0082] The following is in conjunction with the appendix Figures 1 to 7 This application will be described in further detail.

[0083] This application discloses a point cloud completion tumor 3D reconstruction system based on meshing and attention mechanisms.

[0084] Reference Figures 1 to 7 A point cloud-based tumor 3D reconstruction system based on meshing and attention mechanisms includes: a data acquisition module, an image preprocessing module, and a model reconstruction module.

[0085] The data acquisition module is used to collect and acquire two-dimensional CT image data.

[0086] The image preprocessing module includes: preprocessing two-dimensional CT images, extracting the contour point cloud data of the liver and tumor based on the precisely delineated liver and tumor regions, and normalizing them to construct a high-precision three-dimensional point cloud dataset of the liver and tumor.

[0087] The model reconstruction module is used to perform three-dimensional reconstruction on preprocessed two-dimensional CT image data to generate three-dimensional medical images, including a point cloud completion network of mesh and geometrically perceptive Transformer, extracting contour information from two-dimensional segmentation mask, and predicting and reconstructing the complete surface shape of the liver and tumor from the contour information.

[0088] The point cloud completion network consists of a feature extraction module and a multi-stage point cloud generation module.

[0089] The feature extraction module adopts a dual-branch structure, including a first branch and a second branch, which are used to extract shape codes f from the input sparse and incomplete contour point cloud P.

[0090] The feature extraction module takes the contour point cloud P as input.

[0091] The first branch: The 3D CNN (3D Convolutional Network) based branch extracts local geometric features of the liver and tumors by converting the point cloud into a 3D grid. The input data is processed using a grid layer, converting it to 64... 3A 3D grid with dimensions of 32, 64, 128, and 256 is used for hierarchical feature extraction via four cascaded (four is ideal, as too many convolutional modules can lead to information loss) 3D CNNs. These feature dimensions progressively capture feature representations from local geometric structure to global semantic information. To further enhance feature representation capabilities, a two-layer fully connected network is constructed with layer sizes of 2048 and 1024. The resulting 3D grid has dimensions of 32, 64, 128, and 256 through non-linear transformations. The local geometric features f1; the input partial point cloud P is converted into a voxel mesh using the meshing module. The mesh features are extracted through four stacked 3D convolutional layers, and the local features f1 of the point cloud are extracted through a fully connected layer.

[0092] The formula for extracting f1:

[0093]

[0094]

[0095] .

[0096] The second branch: The Transformer-based branch dynamically evaluates the importance of each point by establishing long-distance dependencies, capturing global contextual information, and extracting global semantic features f2.

[0097] Based on the Transformer architecture, the Cross Transformer (CTrans) module was used. The input contour point cloud P was first processed by an MLP to generate a dimension of... The feature vector matrix X is given, where each row represents the feature vector of a point. This feature matrix X serves as the input to the CTrans module, which extracts global contextual information through cascaded CTrans1 and CTrans2 modules. Finally, max pooling is used to obtain the global semantic feature f2.

[0098] Based on different feature aggregation requirements, corresponding query values ​​are constructed, resulting in two CTrans module structure variants, specifically: , .

[0099] The CTrans1 module uses downsampled features as query input. By applying farthest-point sampling to the feature matrix X, it obtains a sample of size X. (n / d refers to the input data having n points, after sampling using the farthest point, only n / d points are retained, and each point has c coordinate features, so it is (n / d)×c) downsampled feature matrix Y, which serves as the query for the CTrans1 module; the CTrans2 module uses feature matrix X as query input and establishes the overall structural association of the point cloud through a global feature interaction mechanism.

[0100] X represents the point cloud features extracted by MLP, and Y represents the downsampled features of X obtained by using the farthest point sampling method, where d is the downsampling ratio. The CTrans1 module uses Y as the query input, and W is the weight.

[0101] , ,

[0102]

[0103] ;

[0104] The CTrans2 module uses the feature matrix X as the query input, specifically:

[0105] , ,

[0106]

[0107] ;

[0108] Q, K, and V are the inputs to the Transformer module, referring to the query, key, and value, respectively. They are obtained by transforming the input features X through different weight matrices W. Q determines the positions to focus on, K calculates the relevance to other positions, and V provides the actual feature content. The weighting of V is determined by the matching degree of Q and K, enabling the model to dynamically focus on important information.

[0109] The global semantic feature f2 is obtained through max pooling. The specific feature extraction formula is as follows:

[0110] ;

[0111] By using branches based on 3D CNN and Transformer respectively, local geometric features f1 and global semantic features f2 of the input point cloud were obtained. These two feature vectors f1 and f2 were then concatenated. 2, Obtain the shape code f, which contains the overall contour information of the input point cloud, specifically:

[0112] .

[0113] The Transformer-based branch utilizes an attention mechanism to construct long-distance dependencies, dynamically evaluating the importance weights of each location point in the input data. This mechanism precisely quantifies the contribution of different regions in the feature extraction process by calculating the correlation scores among all location points. This global interactive modeling approach enables the Transformer branch to comprehensively capture global contextual information in the input data and extract global features with significant semantic discriminative power, including key information such as the malignant characteristics of tumors and lesion boundary features.

[0114] The 3D CNN branch excels at capturing the edge structures and local details of the liver and tumors, while the Transformer branch models global topological relationships through an attention mechanism, understanding the overall morphology and spatial layout of the organ. Relying solely on local features is insufficient to grasp the global structure, while purely global information may overlook crucial details; therefore, the synergistic effect of both is critical. During feature fusion, the 3D CNN branch, guided by the Transformer, optimizes local geometric representations to conform to the overall morphology, while the Transformer branch, aided by the detailed information supplemented by the CNN, enhances sensitivity to boundary regions and morphological changes. The dual-branch feature extraction module extracts both local geometric details and global semantic features of the liver and tumors. The local feature branch extracts fine structural information, while the global feature branch models overall morphological relationships; their complementary fusion provides multi-level feature support for subsequent 3D reconstruction. This complementary interaction mechanism ensures that the shape encoding f possesses both local accuracy and global consistency, thus providing a more complete and accurate shape prior for subsequent point cloud generation and improving reconstruction quality.

[0115] Guided by shape encoding, the multi-stage point generation module uses point generators with different structures to gradually reconstruct a complete and clear 3D point cloud shape of the liver and tumor from coarse to fine.

[0116] The multi-stage point generation module includes a coarse point generator and two fine point generators.

[0117] Coarse point cloud generator: In this module, the shape code f is decoded through the MLP, Reshape and CTrans2 modules to obtain a coarse point cloud P0 that can contain global information, which will serve as the seed point cloud for subsequent reconstruction and refinement processes.

[0118] Generate a size of [size] using a coarse point generator. The methods for the initial point cloud P0 include:

[0119] A1. Using MLP and Reshape, the shape code f is decoded into a feature vector matrix to obtain the decoded point cloud P. c ;

[0120] A2. By transforming features through three cascaded CTrans2 modules, more complex and detailed feature relationships can be captured.

[0121] A3. Point cloud features are processed using MLP and Reshape to convert feature vectors into three-dimensional coordinates and generate size... Point cloud P c ;

[0122] A4. Point cloud P c The point cloud P is merged with the input contour point cloud P and sampled using the farthest point sampling algorithm to obtain a coarse point cloud P0.

[0123]

[0124]

[0125] The coarse point generator generates an initial point cloud P0 of size N0×3 to capture the overall three-dimensional shape of the liver and tumor, providing a coarse but complete three-dimensional structural representation.

[0126] The point refinement generator includes MLP, CTrans2, and Skip Transformer modules. It refines the point cloud P generated in the previous stage. i Achieving fine-grained point cloud structure P i＋1 The reconstruction methods include:

[0127] B1, shape encoding f and point cloud P i The point cloud features generated after processing by MLP and CTrans2 are used as input to the STrans module.

[0128]

[0129]

[0130] Point cloud P i Feature fusion with shape encoding f yields enhanced point cloud features f. p Features were further refined using MLP and CTrans2 operations to construct a higher-order representation f. s The P0 obtained by the coarse point generator and the P1 obtained by the first refined point generator are both collectively referred to as P. i ;

[0131] B2. In the STrans module, q and k are concatenated (concatenation refers to the operation of connecting and merging multiple feature tensors along a certain dimension (usually the channel dimension), which is a feature fusion method. If the size of q and k is n×c, concatenation along the channel dimension is concatenation along the c dimension, and the size of the feature obtained after concatenation is n×2c) and then fed into the MLP to obtain the feature neighbor v. On the other hand, the relationship calculation module generates the relationship matrix between points, uses the K nearest neighbor algorithm (KNN) to calculate the position encoding δ of the points, adds δ to the relationship matrix and feeds it into AttenMLP to generate the attention weight distribution;

[0132]

[0133]

[0134]

[0135] δ represents the pair of point cloud P i The positional encoding is obtained using the K-Nearest Neighbors (KNN) algorithm, w A For point cloud attention weights,

[0136] B3. Multiply the generated attention weights element-wise with the feature vector v with positional encoding to obtain a weighted feature representation;

[0137] B4. Generate high-resolution point cloud features by processing weighted features through MLP:

[0138]

[0139] f output These are high-resolution point cloud features generated by the STrans module;

[0140] B5. The features output by the STrans module are fed into SharedMLP to calculate the point offset. :

[0141]

[0142] Offset and point cloud P after copying by r times i Adding them together yields the refined, fine-grained point cloud P. i＋1 ;

[0143]

[0144] Tile (P) i The r) identifier will mark the point cloud P i It was copied r times.

[0145] By introducing a 3D Grid as an intermediate representation of the point cloud and combining it with the Transformer mechanism and a multi-stage point generation module, a complete and clear 3D point cloud shape of the liver and tumor is reconstructed progressively from coarse to fine granularity. Through a cascaded coarse point generator and two fine point generators, we achieve coarse-to-fine point cloud modeling of the liver and tumor. The advantages of this method are: first, it rapidly reconstructs the overall morphology of the organ using a low-resolution point cloud; then, it precisely depicts tumor details through two levels of progressive refinement. This layered and progressive structure ensures reconstruction efficiency while gradually improving the geometric accuracy of key areas, ultimately achieving a synergistic optimized representation of anatomical structures and lesion areas.

[0146] The above are merely preferred embodiments of the invention and are not intended to limit the invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A point cloud completion system for 3D tumor reconstruction based on meshing and attention mechanisms, characterized in that, include: Data acquisition module, image preprocessing module, model reconstruction module; The data acquisition module is used to collect and acquire two-dimensional CT image data; The image preprocessing module includes: preprocessing two-dimensional CT images; The model reconstruction module is used to perform three-dimensional reconstruction on preprocessed two-dimensional CT image data to generate three-dimensional medical images, including a point cloud completion network of mesh and geometrically perceptive Transformer, extracting contour information from two-dimensional segmentation mask, and predicting and reconstructing the complete surface shape of the liver and tumor from the contour information. The point cloud completion network consists of a feature extraction module and a multi-stage point cloud generation module. The feature extraction module adopts a dual-branch structure to extract shape encoding f from the input sparse and incomplete contour point cloud P. It includes a first branch: a 3D CNN-based branch that extracts local geometric features by converting the point cloud into a 3D grid. It also includes a second branch: a Transformer-based branch that dynamically evaluates the importance of each point by establishing long-distance dependencies, captures global contextual information, and extracts global semantic features. Based on the Transformer architecture, the Cross Transformer (CTrans) module is adopted. The input contour point cloud P is used to generate the original point cloud feature matrix X through MLP. The original point cloud feature matrix X is input into the CTrans module, and global context information is extracted through the cascaded CTrans1 and CTrans2 modules to extract global semantic features. Based on different feature aggregation requirements, corresponding query values ​​are constructed, resulting in two CTrans module structure variants: the CTrans1 module uses downsampled features as query input and applies farthest point sampling to the original point cloud feature matrix X; the CTrans2 module uses the original point cloud feature matrix X as query input and establishes the overall structural association of the point cloud through a global feature interaction mechanism. Guided by shape encoding, the multi-stage point generation module gradually reconstructs a complete and clear 3D point cloud shape of the liver and tumor from coarse to fine using point generators with different structures.

2. The point cloud completion tumor 3D reconstruction system based on meshing and attention mechanisms according to claim 1, characterized in that, Feature extraction module The contour point cloud P is used as input; The first branch processes the input data using a grid layer, converting it to 64. 3 A 3D grid of dimension 1 is obtained through hierarchical feature extraction using cascaded 3D CNNs, constructing a two-layer fully connected network, and obtaining the dimension 1 through non-linear transformation. The local geometric feature f1; the extraction formula for f1: ； The second branch is based on the Transformer architecture. The input contour point cloud P is first generated using an MLP to obtain a dimension of... The original point cloud feature matrix X is used as the input of the CTrans module. Global context information is extracted through the cascaded CTrans1 and CTrans2 modules, and finally global semantic features f2 are obtained through max pooling. The local geometric features f1 obtained from the 3D CNN branch and the global semantic features f2 obtained from the Transformer branch are concatenated to form a comprehensive shape code f; This represents the four 3D convolutional layers in the model, with the index i∈{1,2,3,4} used to distinguish different layers; This represents two fully connected layers, with indices i∈{1,2} representing the first and second fully connected layers, respectively.

3. The point cloud completion tumor 3D reconstruction system based on meshing and attention mechanisms according to claim 2, characterized in that, Hierarchical feature extraction is performed using four cascaded 3D CNNs with feature dimensions of 32, 64, 128 and 256 respectively. A two-layer fully connected network is constructed with layer sizes of 2048 and 1024 respectively. The input partial point cloud P is converted into a voxel mesh using the meshing module. The mesh features are extracted through four stacked 3D convolutional layers, and the local features f1 of the point cloud are extracted through a fully connected layer. R represents the number of voxels in the depth direction of the voxel mesh, which together with H and W constitutes the 3D spatial mesh, representing the length or depth dimension of the 3D space. H represents the number of voxels in the height direction of the voxel grid; W represents the number of voxels in the width direction of the voxel grid; C represents the number of feature channels. Each voxel (r, h, w) is not a single number, but a vector containing C features, including: original information, handcrafted features, and learned features. Based on the feature aggregation requirements, corresponding query values ​​are constructed, resulting in two variations of the CTrans module structure: , ; The CTrans1 module uses downsampled features as query input. It applies farthest-point sampling to the original point cloud feature matrix X to obtain a feature of size X. The downsampled feature matrix Y is used as the query for the CTrans1 module; the CTrans2 module uses the original point cloud feature matrix X as the query input and establishes the overall structural association of the point cloud through a global feature interaction mechanism. In the middle: The point cloud set input to the CTrans module contains n points, where each point is represented by a c-dimensional coordinate feature vector; X is the original point cloud feature matrix extracted by MLP, and Y is the downsampled feature matrix obtained by using the farthest point sampling method on X, where d is the downsampling ratio. The CTrans1 module uses Y as the query input, and W is the weight. , , ； W includes , and , is a set of parameter matrices that perform a linear transformation on the input features, mapping the original features to different representation spaces to generate queries, keys, and values; Project the input into the query space; Project the input onto the key space; Project the input onto the value space; The downsampled feature matrix Y is processed through... Obtained by linear transformation, This represents the problem or requirement carried by each point after downsampling; The original point cloud feature matrix X is processed through... Obtained by linear transformation; It represents the identifier or content summary of each point in the original point cloud, used for query matching; The original point cloud feature matrix X is processed through... Obtained by linear transformation, This represents the information content that is actually to be extracted from each point in the original point cloud; The CTrans2 module uses the original point cloud feature matrix X as the query input, specifically: , , ； The global semantic feature f2 is obtained through max pooling. The specific feature extraction formula is as follows: ； By concatenating these two feature vectors f1 and f2 2, Obtain the shape code f, which contains the overall contour information of the input point cloud, specifically: ； , , All data originates from the original point cloud feature matrix X. The purpose is to allow each point within the input original point cloud feature matrix X to interact with all other points, thereby capturing the contextual dependencies within the point cloud. From the downsampled feature matrix Y, and The purpose is to allow a few downsampled feature matrices Y to query and aggregate information from the original point cloud feature matrix X.

4. The point cloud completion tumor 3D reconstruction system based on meshing and attention mechanisms according to claim 2, characterized in that, The multi-stage point generation module includes a coarse point generator and two fine point generators.

5. The point cloud completion tumor 3D reconstruction system based on meshing and attention mechanisms according to claim 4, characterized in that, Generate a size of [size] using a coarse point generator. The methods for the initial point cloud P0 include: A1. Using MLP and Reshape, the shape code f is decoded into a feature vector matrix to obtain the decoded point cloud P. c ; A2. Transform the features using the cascaded CTrans2 modules; A3. Point cloud features are converted into three-dimensional coordinates using MLP and Reshape, generating a size... Point cloud P c ; A4. Point cloud P c The point cloud P is merged with the input contour point cloud P and sampled using the farthest point sampling algorithm to obtain a coarse point cloud P0.

6. The point cloud completion tumor 3D reconstruction system based on meshing and attention mechanisms according to claim 5, characterized in that, The point refinement generator includes MLP, CTrans2, and Skip Transformer modules. It refines the point cloud P generated in the previous stage. i Achieving fine-grained point cloud structure P i＋1 The reconstruction methods include: B1, shape encoding f and point cloud P i The point cloud features generated after processing by MLP and CTrans2 are used as input to the STrans module. Point cloud P i Feature fusion with shape encoding f yields enhanced point cloud features f. p Features were further refined using MLP and CTrans2 operations to construct a higher-order representation f. s The P0 obtained by the coarse point generator and the P1 obtained by the first refined point generator are both collectively referred to as P. i ; B2. In the STrans module, q and k are concatenated and fed into the MLP to obtain the feature neighbor v. On the other hand, the relation calculation module generates the relation matrix between points. The K nearest neighbor algorithm (KNN) is used to calculate the position code δ of the point. δ is added to the relation matrix and fed into AttenMLP to generate the attention weight distribution. δ represents the pair of point cloud P i The positional encoding is obtained using the K-Nearest Neighbors (KNN) algorithm, w A For point cloud attention weights, B3. Multiply the generated attention weights element-wise with the feature vector v with positional encoding to obtain a weighted feature representation; B4. Generate high-resolution point cloud features by processing weighted features through MLP: f output These are high-resolution point cloud features generated by the STrans module; B5. The features output by the STrans module are fed into SharedMLP to calculate the point offset. : Offset and point cloud P after copying by r times i Adding them together yields the refined, fine-grained point cloud P. i＋1 ; Tile (P) i The r) identifier will mark the point cloud P i It was copied r times; Similar to the CTrans module, both q and k are represented by common features. It is derived through projections of different linear transformations, i.e., q= k= In this structure, q mainly carries the self-characteristic representation of each point as the center point, which is used to actively initiate queries; while k constitutes the environmental context representation of all points as neighbors, which is used to respond to queries.

Images