Medical volume representation learning method and device based on gaussian mask autoencoder

Abstract

The application discloses a medical volume representation learning method and device based on a Gaussian mask autoencoder. The method comprises the following steps: inputting three-dimensional medical body data to be processed into a pre-trained Gaussian mask autoencoder; and outputting three-dimensional representation of the three-dimensional medical body data by using the Gaussian mask autoencoder. The pre-training of the Gaussian mask autoencoder comprises the following steps: obtaining a visible body block label from three-dimensional medical body training data; inputting the visible body block label and its position code into an encoder to extract an output category label; taking a decoder label sequence composed of the output category label of the encoder, a Gaussian query label and the visible body block label as an input of a decoder to output three-dimensional Gaussian primitive parameters; and calculating a reconstruction loss, and performing back propagation and optimization update. The medical volume representation learning framework can break through the limitation of discrete voxel reconstruction, represent an anatomical structure by using geometric primitives, and realize efficient feature extraction of the encoder and downstream reusability of the decoder.

Description

Technical Field

[0001] This invention belongs to the technical field of medical image analysis, specifically relating to a medical volume representation learning method and device based on Gaussian mask autoencoder. Background Technology

[0002] With the rapid development of medical imaging technology, three-dimensional volumetric imaging modalities such as computed tomography (CT) and magnetic resonance imaging (MRI) have become indispensable cornerstones of modern clinical diagnosis, treatment planning, and prognostic assessment. Deep learning technology has shown great potential in the automated analysis of this data, but its performance is often limited by the scarcity of large-scale expert-annotated datasets. To address this issue, self-supervised pre-training has become a mainstream solution, aiming to learn general feature representations using massive amounts of unlabeled 3D data to improve the performance of downstream tasks such as segmentation, registration, and classification.

[0003] In existing self-supervised learning paradigms, masked image modeling (MIM) has attracted much attention due to the high anatomical similarity of medical volumetric data. Existing MIM methods (such as MAE, SwinUNETR, etc.) typically employ a "mask-reconstruction" strategy: the input 3D image is divided into several image patches, some regions are randomly masked, and then the model is trained to reconstruct the original pixel or voxel intensity of the masked region based on visible contextual information (voxel-level regression).

[0004] However, after in-depth analysis of existing technologies, it was found that this traditional method based on discrete voxel regression has the following three main technical bottlenecks and shortcomings when applied to medical volumetric data: (1) There is a conflict between discrete reconstruction objectives and the continuity of anatomical structures. Existing MIM methods typically regress discrete voxel intensity values ​​of the masked region directly. While this "fill-in-the-blank" training objective forces the model to focus on local textures, it ignores the integrity and continuity of medical anatomical structures. Human organs and tissues have continuous geometric topological structures in three-dimensional space, and discrete voxel regression struggles to capture this underlying semantic continuity and geometric abstraction. Models often tend to perform local pixel interpolation rather than learn anatomical features with shape consistency, resulting in a lack of deep understanding of the global anatomical structure in the learned representation.

[0005] (2) The decoder representation lacks transferability. In traditional voxel-level MIM architectures, the decoder is designed solely to recover low-level pixel intensity information. Once pre-training is complete, the decoder is typically discarded, and its learned features are rarely used for downstream tasks. This means that a significant amount of computational resources are spent training a module that has no value in subsequent applications. Furthermore, since the decoder only has pixel-level reconstruction capabilities, it cannot be directly used as high-quality prior knowledge to accelerate generative tasks such as 3D CT reconstruction, limiting the zero-shot application potential of pre-trained models in the field of image reconstruction.

[0006] (3) Sparse anatomical distribution leads to parameter inefficiency. Unlike natural images, 3D medical images are inherently sparse in semantic and intensity distribution. Anatomical organs (i.e., foreground regions of interest) typically occupy only a small portion of the 3D volume space (e.g., abdominal organs may only occupy about 11.8% of the total volume), with the remainder being mostly background or air. Existing methods use dense voxel grids to represent and reconstruct the entire volume, resulting in significant computational redundancy and parameter waste. This redundant voxel representation fails to effectively utilize the sparsity of medical images, making it difficult to achieve optimal training efficiency and representation compactness.

[0007] Therefore, there is an urgent need for a medical volume representation learning method and device based on Gaussian mask autoencoder. Summary of the Invention

[0008] The main objective of this invention is to provide a medical volume representation learning method and related equipment. Its novel medical volume representation learning framework can overcome the limitations of discrete voxel reconstruction, utilize more efficient and continuous geometric primitives to represent anatomical structures, and simultaneously achieve efficient feature extraction of the encoder and downstream reusability of the decoder, thereby overcoming the shortcomings of the prior art.

[0009] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: One embodiment of the present invention provides a medical volume representation learning method based on Gaussian mask autoencoder, comprising: inputting three-dimensional medical volume data to be processed into a pre-trained Gaussian mask autoencoder; and using the Gaussian mask autoencoder to output a three-dimensional representation of the three-dimensional medical volume data, wherein the three-dimensional representation is used for downstream tasks; The pre-training of the Gaussian mask autoencoder includes the following steps: S1: Obtain 3D medical volume training data, and extract visible volume markers from the 3D medical volume training data through random masking; S2: Input the visible block markers and their position codes into the encoder, extract the context features and global semantic features of the visible block markers, and use them as the encoder output category markers; S3: The decoder label sequence consisting of the encoder output category label, Gaussian query label, and visible block label is used as the input of the decoder, and three-dimensional Gaussian primitive parameters corresponding one-to-one with the Gaussian query label are output through multiple prediction heads; S4: Based on the three-dimensional Gaussian primitive parameters, generate three-dimensional reconstructed volume data through differentiable volume rendering technology; calculate the reconstruction loss of the three-dimensional reconstructed volume data and its corresponding original three-dimensional volume data in the masked region, and perform backpropagation and optimization updates on the encoder, decoder and prediction head parameters based on the reconstruction loss to obtain a masked autoencoder that has completed self-supervised pre-training.

[0010] In a preferred embodiment, the downstream task includes a medical image analysis task, and the three-dimensional representation is also used for a three-dimensional Gaussian reconstruction initialization task; The medical image analysis task uses the pre-trained Gaussian mask autoencoder as a feature extractor to perform medical image segmentation, classification, or registration tasks. The 3D Gaussian reconstruction initialization task uses the set of 3D Gaussian meta-parameters obtained by forward reasoning of the target volume data or its projection by the Gaussian mask autoencoder as the initial value for the 3D Gaussian reconstruction optimization task.

[0011] In a preferred embodiment, step S1 includes: acquiring the three-dimensional medical volume data to be pre-trained, and acquiring the three-dimensional medical volume training data after preprocessing the three-dimensional medical volume data through resampling, intensity clipping and normalization; The volume blocks in the three-dimensional medical volume training data are divided into cube blocks to obtain multiple volume block labels; the volume block labels are randomly masked according to a predetermined mask ratio to obtain visible volume block labels.

[0012] In a preferred embodiment, the three-dimensional Gaussian meta-parameters include at least center position data, scale data, rotation data, and intensity data; the plurality of prediction heads include a center position head, a scale head, a rotation head, and an intensity density head; the step of outputting three-dimensional Gaussian meta-parameters corresponding one-to-one with the Gaussian query markers through the plurality of prediction heads includes: The output value of the center position head is activated using the Sigmoid function to obtain center position data constrained to a preset spatial range; The output value of the scale head is activated using the Sigmoid function to obtain scale data constrained to a preset scale range; The output value of the rotating head is normalized using L2 to obtain rotation data in the form of a unit quaternion; The output value of the strength density head is activated using the Sigmoid function to obtain strength parameters constrained to a preset strength range.

[0013] In a preferred embodiment, the weight parameters of the center position head, the scale head, the rotation head, and the intensity density head are all initialized using Xavier uniform initialization; the bias parameters of the center position head and the rotation head are initialized to first initial data; the bias parameters of the scale head are initialized to a first preset value, so that the initial scale data activated by the Sigmoid function is the second initial data; and the bias parameters of the intensity density head are initialized to a second preset value, so that the initial intensity parameters activated by the Sigmoid function are the third initial data.

[0014] In a preferred embodiment, generating 3D reconstructed volume data using microvoid rendering technology includes: The three-dimensional Gaussian element parameters are input into the differentiable volume rendering module, and the three-dimensional reconstructed volume data is obtained by accumulating the contributions of each Gaussian element on the discrete mesh.

[0015] In a preferred embodiment, step S3 further includes: setting multiple levels from coarse to fine through a hierarchical residual refinement structure, wherein the first level is used for the prediction of basic three-dimensional Gaussian meta-parameters; Subsequent levels are based on the three-dimensional Gaussian parameters of the previous level, and the three-dimensional Gaussian parameters are obtained by expanding and refining the residuals of the predicted parameters through residual blocks. The 3D Gaussian meta-parameters obtained from each level are stitched together to generate 3D reconstructed volume data.

[0016] An embodiment of the present invention also provides an electronic device, the electronic device including a memory and at least one processor, the memory storing a computer program, and the processor executing the computer program to cause the electronic device to perform the steps of any of the methods described above.

[0017] An embodiment of the present invention also provides a computer-readable storage medium storing a computer program that, when executed by at least one processor, implements the steps of the method described in any of the preceding claims.

[0018] An embodiment of the present invention also provides a computer program product, the computer program product including a computer program, which, when executed by at least one processor, implements the steps of the method described above.

[0019] Compared to existing technologies, this invention addresses the continuity problem in anatomical structure representation by replacing traditional discrete voxel intensity regression with predicted continuous 3D Gaussian primitive parameters (position, rotation, scaling, intensity). This allows the model to learn deep semantic features with geometric consistency and anatomical continuity, thus avoiding the neglect of human tissue topology by traditional methods. To significantly improve the parameter efficiency of representation learning, 3D Gaussian primitives are used to sparsify the spatial representation, adapting to the sparse distribution of foreground targets (organs / tissues) in medical volumetric data. Complex 3D volumes can be accurately represented with a very small number of parameters, eliminating computational redundancy and storage waste caused by full voxel meshes. To achieve cross-task reuse and zero-shot initialization of the decoder, the technical solution breaks the limitation that pre-trained decoders must be discarded in related technologies, constructing a physically meaningful differentiable Gaussian rendering decoder. This decoder not only assists encoder learning but can also be directly transferred to downstream 3D CT reconstruction tasks, serving as a zero-shot initializer to significantly accelerate the reconstruction convergence process and improve reconstruction quality. To improve the performance of downstream tasks in low-data-volume scenarios, the technical solution can provide a pre-training framework that can extract features with strong generalization ability, which can significantly improve the accuracy and robustness of medical image segmentation, classification and registration tasks in the case of extremely scarce labeled data. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the overall framework of MedGMA provided in this application.

[0022] Figure 2 This is a schematic diagram of the network structure of MedGMA and MedGMA* provided in this application.

[0023] Figure 3 This is a schematic diagram of the convergence curve for zero-sample initialization of 3DGS medical reconstruction provided in this application.

[0024] Figure 4 This is a schematic diagram of the pre-training process of a Gaussian mask autoencoder provided in this application. Detailed Implementation

[0025] The invention will be more fully understood through the following detailed description, which should be read in conjunction with the accompanying drawings. Detailed embodiments of the invention are disclosed herein; however, it should be understood that the disclosed embodiments are merely exemplary of the invention, which may be embodied in various forms. Therefore, the specific functional details disclosed herein should not be construed as limiting, but rather as the basis for the claims and as intended to teach those skilled in the art to employ the representative basis of the invention in different ways in any suitable detailed embodiment.

[0026] To facilitate understanding of this case, some concepts and application scenarios will be explained first.

[0027] 3D Gaussian Splatting (3DGS) is a 3D scene reconstruction and real-time rendering technology that uses a set of learnable 3D Gaussian ellipsoids to represent the scene. Through a differentiable rendering pipeline, it achieves better training speed and rendering quality.

[0028] Its three-dimensional Gaussian primitive representation is: (1) Each three-dimensional Gaussian element is located at its center. Covariance Matrix Define and jointly characterize the form and distribution of the primitive in three-dimensional space.

[0029] (2) To facilitate network prediction and constraints, the covariance matrix Σ is decomposed into: Where S=diag(s) is derived from the scale vector It is indicated that R is composed of rotational quaternions. Parameterization.

[0030] (3) Each Gaussian element also carries an intensity / density parameter I or α, representing the central intensity or contribution weight of that element. Thus, each Gaussian element can be represented by an 11-dimensional parameter vector. express.

[0031] (4) During volume rendering, the Gaussian field is evaluated at the discrete grid positions of the target volume data, and the contribution of all Gaussians to each grid point is accumulated to obtain the rendered volume data.

[0032] 3D Gaussian splashing performs well in rendering natural scenes and has begun to be applied to medical image reconstruction. However, in the application scenario of medical image reconstruction, it is mainly used as a rendering tool and has not yet been effectively embedded as a representation learning method into the mask autoencoder framework to solve the limitations brought about by the aforementioned voxel reconstruction.

[0033] To address the aforementioned problems in existing self-supervised pre-training techniques for medical images, the main objective of this invention is to provide a medical volume representation learning method and system based on a Medical Gaussian Masked Autoencoder (MedGMAE).

[0034] This invention aims to overcome the technical bottlenecks of discrete voxel reconstruction, such as discontinuous anatomical structures, large parameter redundancy, and the inability to reuse decoders, by introducing 3D Gaussian primitives as intermediate representations. The method is described first, followed by the equipment.

[0035] Example 1 See Figure 4 This embodiment provides a medical volume representation learning method based on Gaussian mask autoencoder, including: inputting the three-dimensional medical volume data to be processed into a pre-trained Gaussian mask autoencoder; and using the Gaussian mask autoencoder to output a three-dimensional representation of the three-dimensional medical volume data, wherein the three-dimensional representation is used for downstream tasks. The downstream tasks include medical image analysis tasks, and the three-dimensional representation is also used for three-dimensional Gaussian reconstruction initialization tasks. The medical image analysis task uses the pre-trained Gaussian mask autoencoder as a feature extractor to perform medical image segmentation, classification, or registration tasks. The 3D Gaussian (3DGS) reconstruction initialization task uses the set of 3D Gaussian meta-parameters obtained by forward reasoning of the target volume data or its projection by the Gaussian mask autoencoder as the initial value for the 3D Gaussian reconstruction optimization task.

[0036] Specifically, in medical image analysis tasks, for example, after pre-training, the encoder is used as a general 3D medical representation extractor for fine-tuning or linear evaluation in tasks such as segmentation, classification, and registration. The 3D Gaussian reconstruction initialization task (3DGS medical reconstruction zero-sample initialization task) involves, for example, performing forward inference on the input volume data or its projection in a 3D Gaussian representation CT reconstruction task to obtain a Gaussian point set and parameters as initial values ​​for 3DGS optimization, thereby significantly accelerating the convergence of subsequent reconstruction training.

[0037] The pre-training of the Gaussian mask autoencoder includes the following steps: S101: Obtain three-dimensional medical volume training data, and obtain visible volume block markers from the three-dimensional medical volume training data through random masking; In an exemplary embodiment, S101 includes: S201, Obtain the three-dimensional medical volume data to be pre-trained, and obtain the three-dimensional medical volume training data after preprocessing the three-dimensional medical volume data through resampling, intensity clipping and normalization; S202, the volume blocks in the three-dimensional medical volume training data are divided into cube blocks to obtain multiple volume block markers; the volume block markers are randomly masked according to a predetermined mask ratio to obtain visible volume block markers.

[0038] As an example, MedGMAE pre-training includes steps P1 and P2. Step P1 is the input acquisition and preprocessing process, and step P2 is the patchify and masking process.

[0039] In step P1, the three-dimensional medical volume data (CT / MRI, etc.) to be pre-trained is obtained, and preprocessing such as resampling, intensity clipping and normalization can be performed. In a specific implementation, the pre-trained volume data is randomly clipped into 96×96×96 blocks as network input, and a random clipping strategy based on positive and negative samples can be adopted, with multiple blocks sampled for each volume data for training.

[0040] In step P2, the input volume block is divided into cubic patches (e.g., patch size is 16×16×16 or 12×12×12), resulting in N patch tokens (e.g., when the input is 96×96×96 and the patch size is 12×12×12, N=512). The patch tokens are then randomly masked according to a predetermined masking ratio; in one specific implementation, the masking ratio is 75%, meaning that only about 25% of the visible patch tokens are sent to the encoder.

[0041] S102: Input the visible block markers and their position codes into the encoder, extract the context features and global semantic features of the visible block markers, and use them as the encoder output category markers; As an example, building upon the previous example, MedGMAE pre-training includes step P3 following step P2: Step P3: Encoder Feature Extraction. Input the visible patch token and positional encoding into a 3D vision Transformer encoder (ViT-Large) to extract the contextual features and global semantics of the visible patch token. Example configuration: embedding dimension 1536, attention heads 16, Transformer layers 12, MLP ratio 4.0, Drop Path rate 0.1.

[0042] S103: The decoder label sequence consisting of the encoder output category label, Gaussian query label, and visible block label is used as the input of the decoder, and three-dimensional Gaussian primitive parameters corresponding one-to-one with the Gaussian query label are output through multiple prediction heads; In an exemplary embodiment, the three-dimensional Gaussian meta-parameters include at least center location data, scale data, rotation data, and intensity data; the plurality of prediction heads include a center location head, a scale head, a rotation head, and an intensity density head; the step of outputting three-dimensional Gaussian meta-parameters corresponding one-to-one with the Gaussian query tags through the plurality of prediction heads includes: The output value of the center position head is activated using the Sigmoid function to obtain center position data constrained to a preset spatial range; The output value of the scale head is activated using the Sigmoid function to obtain scale data constrained to a preset scale range; The output value of the rotating head is normalized using L2 to obtain rotation data in the form of a unit quaternion; The output value of the strength density head is activated using the Sigmoid function to obtain strength parameters constrained to a preset strength range.

[0043] As an example, building on the previous example, MedGMAE pre-training includes steps P4 and P5 following step P3.

[0044] Step P4: Decoder Construction and Gaussian Query. Construct a lightweight Transformer decoder whose input token sequence consists of the encoder's output class token, a preset number of Gaussian query tokens, and visible patch tokens. Example configuration: decoder embedding dimension 528, attention heads 16, layers 8; number of Gaussian query tokens 512; and a linear layer projects the encoder features from 1536 dimensions to 528 dimensions. The decoder token sequence can be represented as:

[0045] in, Output category tags for the encoder. Mark the visible blocks. This is a preset number of Gaussian query tokens. With a 75% mask and N=512, the decoder input length is approximately 640 tokens.

[0046] Step P5: Gaussian parameter prediction. The decoder output corresponds one-to-one with a Gaussian query token, and each Gaussian query token predicts the parameters of a Gaussian primitive. To ensure parameter validity and training stability, four dedicated linear prediction heads can be set up: (a) Center position head: Outputs 3D components of μ, constrained to [0,1]^3 by Sigmoid (can be mapped to real coordinates according to block size); (b) Scale Header: Outputs the 3D components of s, constrains them to [0,1]^3 using Sigmoid and combines them with bias initialization to obtain a reasonable initial scale; (c) Rotating head: Outputs 4-dimensional components of φ and performs L2 normalization to obtain a unit quaternion; (d) Intensity density head: Outputs a 1-dimensional component of α or I, constrained to [0,1] by Sigmoid.

[0047] Furthermore, the weight parameters of the center position head, the scale head, the rotation head, and the intensity density head are all initialized using Xavier uniform initialization; the bias parameters of the center position head and the rotation head are initialized to first initial data; the bias parameters of the scale head are initialized to a first preset value, so that the initial scale data activated by the Sigmoid function is the second initial data; and the bias parameters of the intensity density head are initialized to a second preset value, so that the initial intensity parameters activated by the Sigmoid function are the third initial data.

[0048] As an example, to further improve training stability based on the previous example, a custom initialization can be used: the weights of each prediction head are initialized uniformly using Xavier; in the biases, the position and rotation head biases are 0; the scale head bias is -1.386 (approximately 0.2 after Sigmoid); and the density head bias is -0.405 (approximately 0.5 after Sigmoid).

[0049] In an exemplary embodiment, step S3 further includes: setting multiple levels from coarse to fine through a hierarchical residual refinement structure, wherein the first level is used for the prediction of basic three-dimensional Gaussian meta-parameters; Subsequent levels are based on the three-dimensional Gaussian parameters of the previous level, and the three-dimensional Gaussian parameters are obtained by expanding and refining the residuals of the predicted parameters through residual blocks. The 3D Gaussian meta-parameters obtained from each level are stitched together (and the union is taken) to generate 3D reconstructed volume data.

[0050] In this case, as an example, a hierarchical residual refinement structure MedGMAE* is further provided: Multiple levels l∈{0,1,2} are set from coarse to fine, where Level 0 contains N0 basic Gaussians; Level 1 expands to m1×N0 Gaussians; and Level 2 expands to m2×N0 Gaussians. Parameter dependencies are established between levels, and the refined residuals are predicted through residual blocks, progressively capturing finer volumetric data details in a "coarse-to-fine" manner. For example, a hierarchical reduction constraint is imposed on the scale parameter: ,in Used to control the residual amplitude, Δs l >0 is used to ensure that the scale decreases monotonically, and can be taken as 0. , For position parameters, the following can be used: Ultimately, multi-level Gaussian sets are concatenated for rendering and loss calculation, achieving stronger reconstruction capabilities.

[0051] S104: Based on the three-dimensional Gaussian primitive parameters, generate three-dimensional reconstructed volume data through differentiable volume rendering technology; calculate the reconstruction loss of the three-dimensional reconstructed volume data and its corresponding original three-dimensional volume data in the masked region, and perform backpropagation and optimization updates on the encoder, decoder and prediction head parameters based on the reconstruction loss to obtain a masked autoencoder that has completed self-supervised pre-training.

[0052] In one exemplary embodiment, generating 3D reconstructed volume data using microvoid rendering technology includes: The three-dimensional Gaussian element parameters are input into the differentiable volume rendering module, and the three-dimensional reconstructed volume data is obtained by accumulating the contributions of each Gaussian element on the discrete mesh.

[0053] As an example, based on the previous example, step P5 is followed by steps P6 and P7.

[0054] Step P6: Differentiable Volume Rendering and Reconstruction Loss. The predicted k 3D Gaussian primitive parameters are input into the differentiable volume rendering module, and the Gaussian contributions are accumulated on a discrete mesh to obtain the rendered volume data. During training, the reconstruction loss is calculated only for the original masked region. For example, mean squared error (MSE) is used to measure the difference between the rendered volume data and the original volume data at the masked voxels, thereby prompting the network to perform reasonable "filling" based on the 3D structure in missing regions.

[0055] Step p7: Parameter Update. Based on the reconstruction loss, backpropagation and optimization updates are performed on the encoder, decoder, and prediction head parameters to complete self-supervised pre-training. Finally, the self-supervised pre-trained masked autoencoder is obtained.

[0056] As an example, a scheme for self-supervised pre-training of MedGMAE for 3D medical volume data is provided, including: (1) Data preparation: Obtain a three-dimensional medical volume dataset (e.g., CT volume data). Perform necessary resampling and intensity normalization on the volume data; in a specific implementation, the CT intensity can be clipped to a preset HU interval and linearly normalized.

[0057] (2) Volume sampling: Sample 96×96×96 volumes from each volume data as input; for example, the RandCropByPosNegLabeld strategy can be used to sample 3 volumes from each volume data to take into account the foreground / background distribution.

[0058] (3) Patch (block) division and masking: Divide the input block into cubic patches of 12×12×12 or 16×16×16 to form a Patch token sequence; randomly mask according to a 75% masking ratio, and retain about 25% of the visible tokens.

[0059] (4) Encoder: ViT-Large is used as the encoder with an embedding dimension of 1536, an attention head of 16, a layer of 12, an MLP ratio of 4.0, a Dropout of 0, a Drop Path of 0.1, and only visible tokens are processed to reduce computation.

[0060] (5) Decoder: Construct a lightweight Transformer decoder with an embedding dimension of 528, an attention head of 16, and a layer of 8; introduce 512 Gaussian query tokens; and perform encoder-to-decoder feature projection through a 1536→528 linear layer.

[0061] (6) Gaussian prediction: Set four prediction heads for position, scale, rotation, density and intensity, output the 11-dimensional Gaussian parameters corresponding to each query, and apply activation constraints and bias initialization (scale bias -1.386, density bias -0.405).

[0062] (7) Differentiable volume rendering and loss: The Gaussian set is rendered as volume data, and the MSE reconstruction loss is calculated only in the masked region; the network parameters are updated by backpropagation.

[0063] (8) Training setup: For example, the AdamW optimizer is used with an initial learning rate of 1e-5 and cosine annealing learning rate scheduling; batch size of 8, training for about 100K steps, and a fixed random seed (e.g., 41) to ensure reproducibility.

[0064] As another example, a scheme for MedGMAE* for multi-level residual refinement is provided, including: Building upon the previous example, a multi-level residual refinement module is introduced, with levels l∈{0,1,2} progressing from coarse to fine. Level 0 outputs N0 basic Gaussians; Level 1 outputs m1×N0 Gaussians and predicts the residuals relative to Level 0; Level 2 outputs m2×N0 finer-grained Gaussians and predicts the residuals. A monotonically decreasing constraint is introduced for the scale residuals: ,in And it is acceptable , For positional residuals, the following can be used: The Gaussian sets from each layer are stitched together and then rendered. The reconstruction loss is calculated in the mask region to achieve a more refined reconstruction effect and faster convergence.

[0065] As yet another example, a scheme for transferring a pre-trained encoder to downstream tasks is provided, including: The pre-trained encoder obtained in Example 1 or Example 2 is used as the initial weights of the downstream network (e.g., UNETR). In the segmentation task, fine-tuning is performed using the Dice coefficient as an indicator. In the classification task, the volume data can be resampled to 1.5×1.5×3.0 mm, the HU interval can be cropped to [-1000, 1000] and normalized to [0, 1]. The input volume size is set to 192×192×96, and training is performed for 100 epochs with a batch size of 96, a learning rate of 3e-2, and a weight decay of 0.05. In the registration task, evaluation is performed using Dice or similarity metrics. Experiments show that this pre-training method can improve generalization and efficiency on low-label data for various tasks.

[0066] As yet another example, a zero-sample initialization scheme for 3DGS medical CT reconstruction is provided: In a 3D Gaussian representation CT reconstruction scene, a forward inference is first performed on the input volume data or its projection data to obtain the Gaussian point set and its parameters as the initial values ​​for 3DGS optimization; then, this initialization is used for iterative optimization in the reconstruction optimization stage.

[0067] Experiments show that, compared with traditional initialization methods, the present invention can significantly reduce training time (e.g., by about 31%-37.2% under different projection number settings) and reduce the number of iterations required to reach the quality threshold, while maintaining comparable final PSNR / SSIM reconstruction quality.

[0068]

[0069] Table 1: Comparison of fine-tuning results of different methods using different proportions of data on downstream segmentation tasks on different datasets.

[0070] Table 2: Comparison of CT reconstruction time, number of iterations to reach quality threshold, and final PSNR / SSIM under different projection numbers.

[0071] Table 3: Comparison of results of different methods on the classification task of the CT-RATE dataset

[0072] Table 4: Comparison of results of different methods on the registration task on the IXI and OASIS datasets Table 1 above compares three paradigms—Training from Scratch, General Self-supervised Methods, and Medical Self-supervised Methods—under the dimension of pretraining methods. It shows the comparison of different methods on the AMOS (Ji et al., 2022), FLARE'22 (Ma et al., 2024), BTCV (Landman et al., 2015), and SegTHOR (Lambert et al., 2020) datasets with different data proportions. Bold data represents the best results in that column, and underlined data represents the next best results in that column.

[0073] Table 2 above mainly compares the reconstruction performance and efficiency of 3DGR (the original initialization method) and MedGMAE / MedGMAE* (the model proposed in this application is initialized by estimating Gaussian points through zero-shot inference) under three projection numbers: 80, 120, and 160. Values ​​are reported as mean ± standard deviation, with the best results highlighted in bold. Under all projection numbers, MedGMAE* or MedGMAE has significantly lower latency than 3DGR. The iter (P=35) and iter (S=90%) of MedGMAE / MedGMAE* are significantly lower than those of 3DGR, indicating faster convergence. MedGMAE achieves the highest PSNR (Peak Signal-to-Noise Ratio) under 120 / 160 projections. MedGMAE / MedGMAE *While maintaining or even improving the quality of reconstruction, efficiency was significantly increased.

[0074] Table 3 above mainly compares the results of different methods on the classification task of the CT-RATE dataset, showing the performance comparison of different pre-training strategies for 3D medical image classification on the CT-RATE dataset. The experiments compared two paradigms: scratch training and fine-tuning. Under the same data conditions, MedGMAE outperforms existing self-supervised methods.

[0075] Table 4 above mainly compares the results of different methods on the registration task on the IXI and OASIS datasets, showing the performance comparison of different pre-training methods on the medical image registration task, and evaluating them on the IXI (MRI brain images) and OASIS (cross-modal brain images) datasets respectively.

[0076] It can be considered that the core idea of ​​the technical solution provided in this embodiment is that, in the framework of mask autoencoder, instead of directly regressing / reconstructing the discrete intensity of the masked voxels, a set of continuous three-dimensional Gaussian primitive parameters are predicted, and the Gaussian set is restored to volume data through differentiable volume rendering. Thus, the learning objective is transformed from "voxel-level local interpolation" to "geometric primitive-driven structural reasoning", which is more in line with the continuity and sparsity characteristics of anatomical structures.

[0077] The technical solution provided in this embodiment addresses the continuity problem in anatomical structure representation by predicting continuous 3D Gaussian primitive parameters (position, rotation, scaling, intensity) to replace traditional discrete voxel intensity regression. This enables the model to learn deep semantic features with geometric consistency and anatomical continuity, thereby avoiding the neglect of human tissue topology by traditional methods.

[0078] To significantly improve the parameter efficiency of representation learning, 3D Gaussian primitives are used to sparsely represent the space to adapt to the sparse distribution of foreground targets (organs / tissues) in medical volume data. Complex 3D volumes can be accurately represented with a very small number of parameters, thereby eliminating the computational redundancy and storage waste caused by full voxel mesh.

[0079] To achieve cross-task reuse and zero-shot initialization of the decoder, the technical solution breaks the limitation of discarding pre-trained decoders in related technologies and constructs a physically meaningful differentiable Gaussian rendering decoder. This decoder can not only assist the encoder in learning, but also be directly transferred to downstream 3D CT reconstruction tasks, serving as a zero-shot initializer to significantly accelerate the reconstruction convergence process and improve reconstruction quality.

[0080] To improve the performance of downstream tasks in low-data-volume scenarios, the technical solution can provide a pre-training framework that can extract features with strong generalization ability, which can significantly improve the accuracy and robustness of medical image segmentation, classification and registration tasks when labeled data is extremely scarce (e.g., 1% labeling ratio).

[0081] In summary, it has the following advantages and positive effects: (1) Stronger geometric consistency: Compared with voxel-level discrete intensity regression, this application reconstructs anatomical structures with continuous three-dimensional Gaussian units, shifting the learning objective from local interpolation to geometric reasoning, which is more in line with the continuity of anatomical structures such as organs / blood vessels, and is conducive to obtaining shape-consistent and transferable three-dimensional representations.

[0082] (2) More efficient parameters and computation: By utilizing the sparsity of medical body data (anatomical structures occupy only a small portion of the total space), the number of parameters can be significantly reduced by replacing dense voxel representation with Gaussian set; the paper's experiments show a reduction of approximately 99.25% in the number of parameters and demonstrate significant efficiency gains in reconstruction scenarios.

[0083] (3) Improved transfer performance in downstream tasks: In multi-dataset segmentation, classification and registration tasks, the pre-trained encoder shows better transfer ability; for example, in the registration task, compared with the best baseline trained from scratch, the method brings an improvement of about 1.1% and 3.9% on IXI and OASIS respectively, and further improves by about 1.3% on OASIS compared with the existing pre-trained method VoCo, reflecting cross-modal (no MRI modality) generalization ability.

[0084] (4) 3DGS medical reconstruction convergence is significantly accelerated: see Figure 3 In CT reconstruction experiments with different projection number settings, zero-sample initialization based on the pre-trained model of this invention can reduce training time by approximately 31.0%, 35.0%, and 37.2%; at the same time, the number of iterations required to reach the PSNR=35 and SSIM=90% thresholds is reduced by an average of approximately 39.4% and 28.1%, and statistical test results show that the improvement in training efficiency is significant (p<0.001).

[0085] (5) More stable training: By using activation constraints (Sigmoid, quaternion normalization) of the dedicated prediction head and bias initialization strategies for scale / density, a reasonable initial distribution of parameters can be obtained, reducing the risk of instability and gradient anomalies in the early stage of training; and the fine-grained reconstruction expression ability and convergence speed can be further improved through the multi-layer residual refinement structure (MedGMAE*).

[0086] Example 2 This embodiment provides an electronic device, which includes a memory and at least one processor. The memory stores a computer program, and the processor executes the computer program to enable the electronic device to perform the steps of the method as described in any one of Embodiment 1.

[0087] Example 3 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by at least one processor, implements the steps of the method described in any one of Embodiment 1.

[0088] Example 4 This embodiment provides a computer program product, which includes a computer program that, when executed by at least one processor, implements the steps of the method described in any one of Embodiment 1.

[0089] Although the invention has been described with reference to illustrative embodiments, those skilled in the art will understand that various other changes, omissions, and / or additions can be made without departing from the spirit and scope of the invention, and that elements of the described embodiments can be substituted with substantially equivalents. Furthermore, many modifications can be made without departing from the scope of the invention to adapt particular situations or materials to the teachings of the invention. Therefore, this document is not intended to limit the invention to the specific embodiments disclosed for carrying out the invention, but rather to include all embodiments falling within the scope of the appended claims.

Claims

1. A medical volume representation learning method based on Gaussian mask autoencoder, characterized in that, include: The 3D medical volume data to be processed is input into a pre-trained Gaussian mask autoencoder; The Gaussian mask autoencoder is used to output a three-dimensional representation of the three-dimensional medical body data, which is used for downstream tasks. The pre-training of the Gaussian mask autoencoder includes the following steps: S1: Obtain 3D medical volume training data, and extract visible volume markers from the 3D medical volume training data through random masking; S2: Input the visible block markers and their position codes into the encoder, extract the context features and global semantic features of the visible block markers, and use them as the encoder output category markers; S3: The decoder label sequence consisting of the encoder output category label, Gaussian query label, and visible block label is used as the input of the decoder, and three-dimensional Gaussian primitive parameters corresponding one-to-one with the Gaussian query label are output through multiple prediction heads; S4: Based on the three-dimensional Gaussian primitive parameters, generate three-dimensional reconstructed volume data through differentiable volume rendering technology; calculate the reconstruction loss of the three-dimensional reconstructed volume data and its corresponding original three-dimensional volume data in the masked region, and perform backpropagation and optimization updates on the encoder, decoder and prediction head parameters based on the reconstruction loss to obtain a masked autoencoder that has completed self-supervised pre-training.

2. The medical volume characterization learning method of claim 1, wherein, The downstream tasks include medical image analysis tasks, and the three-dimensional representation is also used for three-dimensional Gaussian reconstruction initialization tasks. The medical image analysis task is used to perform medical image segmentation, classification, or registration tasks by using the pre-trained Gaussian mask autoencoder as a feature extractor. The 3D Gaussian reconstruction initialization task uses the set of 3D Gaussian meta-parameters obtained by forward reasoning of the target volume data or its projection by the Gaussian mask autoencoder as the initial value for the 3D Gaussian reconstruction optimization task.

3. The medical volume representation learning method of claim 1, wherein Step S1 includes: acquiring the three-dimensional medical volume data to be pre-trained, and acquiring the three-dimensional medical volume training data after preprocessing the three-dimensional medical volume data through resampling, intensity clipping and normalization; The volume blocks in the three-dimensional medical volume training data are divided into cube blocks to obtain multiple volume block labels; the volume block labels are randomly masked according to a predetermined mask ratio to obtain visible volume block labels.

4. The medical volume representation learning method of claim 1, wherein The three-dimensional Gaussian meta-parameters include at least center position data, scale data, rotation data, and intensity data; the multiple prediction heads include a center position head, a scale head, a rotation head, and an intensity density head; the output of three-dimensional Gaussian meta-parameters corresponding one-to-one with the Gaussian query markers through the multiple prediction heads includes: The output value of the center position head is activated using the Sigmoid function to obtain center position data constrained to a preset spatial range; The output value of the scale head is activated using the Sigmoid function to obtain scale data constrained to a preset scale range; The output value of the rotating head is normalized using L2 to obtain rotation data in the form of a unit quaternion; The output value of the strength density head is activated using the Sigmoid function to obtain strength parameters constrained to a preset strength range.

5. The medical volume characterization learning method of claim 4, characterized in that The weight parameters of the center position head, the scale head, the rotation head, and the intensity density head are all initialized using Xavier uniform initialization; the bias parameters of the center position head and the rotation head are initialized to the first initial data; the bias parameters of the scale head are initialized to the first preset value, so that the initial scale data activated by the Sigmoid function is the second initial data. The bias parameter of the intensity density head is initialized to a second preset value so that the initial intensity parameter after activation by the Sigmoid function is the third initial data.

6. The medical volume representation learning method of claim 1, wherein The generation of 3D reconstructed volume data through micronizable volume rendering technology includes: The three-dimensional Gaussian element parameters are input into the differentiable volume rendering module, and the three-dimensional reconstructed volume data is obtained by accumulating the contributions of each Gaussian element on the discrete mesh.

7. The medical volume characterization learning method of claim 1, wherein, Step S3 also includes: setting up multiple levels from coarse to fine through a hierarchical residual refinement structure, wherein the first level is used for the prediction of basic three-dimensional Gaussian meta-parameters; Subsequent levels are based on the three-dimensional Gaussian parameters of the previous level, and the three-dimensional Gaussian parameters are obtained by expanding and refining the residuals of the predicted parameters through residual blocks. The 3D Gaussian meta-parameters obtained from each level are stitched together to generate 3D reconstructed volume data.

8. An electronic device, comprising: The electronic device includes a memory and at least one processor, the memory storing a computer program, and the processor executing the computer program to cause the electronic device to perform the steps of the method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by at least one processor, implements the steps of the method according to any one of claims 1-7.

10. A computer program product, characterised in that, The computer program product includes a computer program that, when executed by at least one processor, implements the steps of the method according to any one of claims 1-7.

Images