Multimodal medical image segmentation method based on sparse hypergraph diffusion network

Abstract

The application belongs to the technical field of image processing, and specifically discloses a multi-modal medical image segmentation method based on a sparse hypergraph diffusion network, which comprises the following steps: inputting a registered multi-modal medical image into a hierarchical encoder to extract multi-scale features and construct a context feature pyramid; taking the pyramid as a condition, driving an inverse stochastic differential equation to iteratively denoise and generate a target segmentation mask through a sparse hypergraph diffusion network, wherein each stage of the decoder performs prediction, quantization, gating, correction, dynamic identification and fine processing of difficult pixels; finally, a predictor and a corrector are used to solve an equation, the denoising result is converted into a segmentation probability map, and the final segmentation label is obtained through post-processing. The application improves the multi-modal information fusion effect and segmentation precision, is suitable for multi-modal medical image segmentation of brain tumors, liver tumors and the like, and provides reliable support for clinical diagnosis.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a multimodal medical image segmentation method based on sparse hypergraph diffusion networks. Background Technology

[0002] Medical image segmentation is a fundamental and critical task in the field of image processing technology. Its goal is to accurately identify anatomical structures or pathological regions of interest in medical images, such as brain tumors or liver tumors. The results are an important basis for clinical diagnosis, quantitative assessment of diseases, surgical planning, and the formulation of radiotherapy plans.

[0003] In recent years, deep learning technology has driven improvements in medical image segmentation performance, with architectures such as U-Net and its variants showing significant effectiveness in single-modal segmentation. However, in clinical practice, doctors rely on multimodal images such as T1, T1c, T2, and FLAIR sequences from MRI, and complementary information from CT and MRI combinations to define complex lesions. Existing technologies have significant shortcomings in handling multimodal segmentation: First, multimodal information fusion is insufficient, and early input stitching or mid-term feature aggregation strategies struggle to capture nonlinear correlations between modalities, lacking unified theoretical guidance. Second, computational resources are statically allocated, resulting in homogenized processing of all image regions, leading to wasted resources in simple regions and insufficient processing of complex regions such as lesion boundaries. Third, the ability to model high-order structures is weak; convolution operations are limited by local receptive fields, and while Transformer can capture long-distance dependencies, it still models independent pixels / image blocks, failing to effectively express complex semantic structures composed of multiple pixels.

[0004] Therefore, developing segmentation methods that can efficiently integrate multimodal information, dynamically allocate computing resources, and enhance the ability to model high-order structures has become an urgent need in the field of medical image segmentation. Summary of the Invention

[0005] To address this issue, the present invention provides a multimodal medical image segmentation method based on sparse hypergraph diffusion networks, thereby solving the aforementioned technical problems.

[0006] This invention provides a multimodal medical image segmentation method based on a sparse hypergraph diffusion network, comprising the following steps: extracting multi-scale features from the registered multimodal medical image through a hierarchical encoder and constructing a context feature pyramid.

[0007] Based on the aforementioned context feature pyramid, a sparse hypergraph diffusion network is used to drive an inverse stochastic differential equation to iteratively denoise the initial noise and generate a target segmentation mask.

[0008] Based on the target segmentation mask, a denoised segmentation feature map is obtained by solving the inverse stochastic differential equation through a predictor and a corrector. The denoised segmentation feature map is then converted into a segmentation probability map and post-processed to obtain the final segmentation label.

[0009] Preferably, the multimodal medical image is a multi-sequence magnetic resonance imaging, including T1-weighted, T1-contrast-enhanced, T2-weighted, and fluid-attenuated inversion recovery sequence.

[0010] Preferably, the inverse stochastic differential equation is a variance-preserving stochastic differential equation, whose conditional fractional function is approximated by the sparse hypergraph diffusion network.

[0011] Preferably, the Monte Carlo random ensemble is achieved by injecting independent and identically distributed Gaussian noise after the convolutional layer of the base convolutional module, or by keeping the Dropout layer active during inference and performing N forward propagations on the same input.

[0012] Preferably, the sparse hypergraph diffusion network inference process includes generating a sparse binary gated graph, specifically including: performing average pooling along the channel dimension on the cognitive uncertainty graph obtained by the Monte Carlo ensemble to obtain a single-channel uncertainty graph; selecting the k_sparse positions with the largest values ​​in the graph using the TopK function and setting their values ​​back to their original values, while setting the remaining positions to zero; and then binarizing it using an indicator function, where k_sparse is a hyperparameter controlling sparsity.

[0013] Preferably, the sparse hypergraph diffusion network inference process includes constructing a local multimodal hypergraph, wherein the node set of the local multimodal hypergraph includes query tokens from the base feature graph, local spatial tokens from the spatial neighborhood of the query tokens, and global context tokens sampled from corresponding positions in the context feature pyramid.

[0014] Preferably, the set of superedges of the local multimodal hypergraph is a predefined set of superedge primitives based on medical priors, which includes at least one of the following: an enhancement core superedge for identifying enhanced tumors, an edema region superedge for identifying peritumoral edema, a structural topology superedge for capturing local shape continuity, and a multimodal fusion superedge for performing cross-modal association.

[0015] In another aspect, this application also provides a multimodal medical image segmentation system based on a sparse hypergraph diffusion network, including: a context feature pyramid construction module, used to extract multi-scale features from the registered multimodal medical image through a hierarchical encoder and construct a context feature pyramid;

[0016] The target segmentation mask generation module is used to generate a target segmentation mask by iteratively denoising from the initial noise based on the context feature pyramid and driving an inverse stochastic differential equation using a sparse hypergraph diffusion network.

[0017] The segmentation label acquisition module is used to obtain a denoised segmentation feature map by solving the inverse stochastic differential equation through a predictor and a corrector based on the target segmentation mask, convert the denoised segmentation feature map into a segmentation probability map, and obtain the final segmentation label through post-processing.

[0018] Preferably, the multimodal medical image is a multi-sequence magnetic resonance imaging, including T1-weighted, T1-contrast-enhanced, T2-weighted, and fluid-attenuated inversion recovery sequence.

[0019] Preferably, the inverse stochastic differential equation is a variance-preserving stochastic differential equation, whose conditional fractional function is approximated by the sparse hypergraph diffusion network.

[0020] This invention provides global guidance through a contextual feature pyramid constructed by a hierarchical encoder, and combines it with a local multimodal hypergraph to capture high-order feature dependencies, thus solving the problem of insufficient multimodal fusion. It can dynamically allocate computing resources, and with the help of a cognitive uncertainty map generated by Monte Carlo random ensemble and sparse binary gating, it only refines difficult pixels, avoiding resource waste. Relying on a conditional diffusion model and predictor-corrector sampling, it ensures that the overall structure of the segmentation result is reasonable, and the hypergraph correction improves the accuracy of local details. It outperforms Unet in liver and brain tumor segmentation, providing more reliable support for clinical diagnosis. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0022] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0023] Figure 1 This is a flowchart of a multimodal medical image segmentation method based on a sparse hypergraph diffusion network, provided in an embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram of the context feature pyramid construction process provided in an embodiment of the present invention.

[0025] Figure 3This is a schematic diagram of the dynamic loop process of the decoder provided in an embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram showing the comparison of liver tumor segmentation on the Lits2017 dataset provided in this embodiment of the invention.

[0027] Figure 5 This is a schematic diagram comparing brain tumor segmentation using the Brats2019 dataset provided in an embodiment of the present invention.

[0028] Figure 6 This is a schematic diagram of a multimodal medical image segmentation system based on a sparse hypergraph diffusion network, provided as an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] like Figure 1 As shown, this embodiment of the invention discloses a multimodal medical image segmentation method 100 based on a sparse hypergraph diffusion network, which includes the following steps: S1, extracting multi-scale features from the registered multimodal medical image through a hierarchical encoder and constructing a context feature pyramid.

[0031] S2, based on the context feature pyramid, a sparse hypergraph diffusion network is used to drive the inverse stochastic differential equation to iteratively denoise the initial noise and generate the target segmentation mask.

[0032] S3. Based on the target segmentation mask, the inverse stochastic differential equation is solved by the predictor and corrector to obtain a denoised segmentation feature map. The denoised segmentation feature map is converted into a segmentation probability map and then post-processed to obtain the final segmentation label.

[0033] In some embodiments, for step S1, the input of this step is a set of registered multimodal medical images, which specifically needs to meet the following conditions: Registration requirements: Multimodal medical images need to be spatially aligned in advance through rigid or non-rigid registration algorithms to ensure that the spatial positions of the same anatomical structure or pathological region in different modal images are completely matched, so as to avoid deviations in subsequent feature extraction and fusion due to spatial offset between modalities.

[0034] Modal type: In typical application scenarios, multimodal medical images are multi-sequence magnetic resonance imaging (MRI), including but not limited to T1-weighted (T1), T1-contrast-enhanced (T1c), T2-weighted (T2), and fluid attenuation inversion recovery (FLAIR) sequences; it can also be extended to other multimodal combinations such as computed tomography (CT) and MRI, positron emission tomography (PET) and CT, depending on the clinical needs of the target segmentation task (such as brain tumor or liver tumor segmentation).

[0035] Data format: Let the input multimodal medical image set be... Where M represents the total number of modalities, such as M=4 in the BraTS brain tumor segmentation task. This represents the image data of the m-th modality, with dimension . H is the image height, W is the image width, and D is the image depth, i.e., the number of slices in the 3D volumetric data; if processing 2D slice data, then D=1. Dimensions simplified to This method is applicable to both 2D and 3D data and requires no additional adaptation.

[0036] Preferably, the Swing Transformer architecture is used as the hierarchical encoder, denoted as . Its core advantage lies in its ability to efficiently capture multi-scale spatial dependencies and cross-modal feature associations. The specific configuration is as follows: Network structure: The SwinTransformer encoder contains L downsampling stages, for example, L=4. Each stage consists of multiple SwinTransformer blocks, layer normalization, and window attention modules. The window attention module balances computational efficiency and long-distance feature capture capability through a sliding window mechanism, avoiding the high computational complexity problem of traditional Transformer global attention.

[0037] Input adaptation: The multimodal medical image set Y is stitched together along the channel dimension to form a sequence with dimension 1. or The input tensor is directly fed into the Swing Transformer encoder; the concatenation operation can preserve the original feature information of each modality, laying the foundation for subsequent multi-scale feature fusion.

[0038] In one embodiment, such as Figure 2The diagram illustrates the construction process of the context feature pyramid. The hierarchical encoder generates multi-scale features covering different resolutions through multi-stage downsampling and feature transformation, and constructs the context feature pyramid C. The specific process is as follows: S201, multi-stage feature extraction, including stage 1, stages 2-L, and feature transformation. Specifically, stage 1 uses the highest resolution: after the input tensor is processed by the Swin Transformer block, it is downsampled through a convolutional layer with a stride of 2, and the output feature map resolution is... Where H1=H / 2, W1=W / 2, D1=D / 2, and C1 is the number of feature channels in this stage, for example, C1=96.

[0039] Stages 2 through L employ progressive downsampling: each subsequent stage repeats the Transformer block processing and downsampling operation on the output feature map of the previous stage, with the resolution halved in each stage, i.e., H. l =H l-1 / 2、W l =W l-1 / 2、D l =D l-1 / 2, the number of channels doubles in stages, i.e., C l =C l-1 ×2, ultimately generating L feature maps of different scales.

[0040] Feature transformation: The Swin Transformer block at each stage transforms multimodal input features into semantically meaningful feature representations through window attention and residual connections, while preserving the unique anatomical structures or pathological region features of each modality, such as the specific features of the T1c modality for the tumor enhancement area and the specific features of the FLAIR modality for the edema area.

[0041] S202, Context Feature Pyramid Integration, summarizes the multi-scale feature maps output from L stages to form a context feature pyramid; specifically... ,in This represents the feature map of the m-th mode at the l-th scale, with dimension 1. (3D) or (2D).

[0042] Therefore, the pyramid has a global conditional guidance function and remains fixed in the subsequent reverse diffusion process. It can serve as a global knowledge base for the sparse hypergraph diffusion network. Feature maps of different scales provide high-resolution detailed information (shallow layer, l is small) and high semantic abstract information (deep layer, l is large). Multimodal features provide cross-dimensional complementary information, which together support accurate denoising and segmentation decisions.

[0043] Optionally, to improve the efficiency of subsequent training and inference of the sparse hypergraph diffusion network, the context feature pyramid C can be preprocessed and stored. Specifically, the feature maps of each scale and modality in the feature pyramid are standardized, such as by layer normalization, to eliminate feature bias caused by differences in the distribution of feature values ​​of different modalities, ensuring the effectiveness of subsequent feature fusion. During the training phase, this step is performed once for all samples in the entire training set, and the context feature pyramid C of each sample is stored to the hard disk in a serialized format. In subsequent training iterations, the C of the corresponding sample is directly loaded from the hard disk as input, avoiding repeated high-computation-cost encoder feature extraction operations and significantly improving training efficiency.

[0044] In some embodiments, for step S2, 1. For the basic configuration of the Reverse Stochastic Differential Equation (ReverseSDE), this step uses the context feature pyramid C constructed in S1 as the global condition, and achieves iterative denoising by driving the variance-preserving stochastic differential equation (VP-SDE) through a sparse hypergraph diffusion network (SHDN). For example, the core configuration is as follows: 1.1 Definition of Reverse SDE: The reverse process describes the evolution of generating a target segmentation mask from initial Gaussian noise, and its mathematical expression is: Where x: the state variable evolving with time t during the diffusion process, i.e., the noisy segmentation map, with a time interval of t. , where T is the initial noise time, for example, T=1.0.

[0045] : A predefined noise scheduling function is used, employing a linear scheduling strategy, from like =0.1 linearly increases to like =20, controlling the noise reduction rate; : Conditional fractional function, representing the noisy segmentation map at time t given a multimodal image Y. The logarithmic gradient of the probability density points in the direction of increasing data density, which is the direction of denoising.

[0046] dw: Inverse time Wiener process, also known as Brownian motion increment, simulates random perturbations in the denoising process.

[0047] 1.2 Initial noise settings; initial state x T That is, sampling is performed from the standard normal distribution N(0,I) at t=T, where I is the identity matrix and x T The dimension is consistent with the target segmentation mask, such as in 3D brain tumor segmentation. K is the number of segmentation categories.

[0048] 2. Sparse Hypergraph Diffusion Network (SHDN) structure and denoising process, specifically including: SHDN is a variant of U-Net, and its core function is to approximate the conditional fractional function. Its denoising process uses an encoder-decoder framework. The encoder extracts features from the noisy segmentation map, and the decoder achieves denoising through dynamic prediction, quantization, gating, and correction loops, as follows: 2.1 Network Input and Temporal Embedding; Input Components: SHDN receives three inputs, including the noisy segmentation map. Dimensions Scalar time t, context feature pyramid C.

[0049] Temporal embedding: Time t is converted into a high-dimensional vector through a sinusoidal position coding layer, and then output as a temporal embedding vector through a small multilayer perceptron (MLP). By incorporating features from each stage of the encoder through addition or FiLM layers, the network can perceive the noise level at the current denoising moment.

[0050] 2.2 Encoder Feature Extraction: Standard U-Net encoder structures, such as ConvNeXt blocks or ResNet blocks, are used for feature extraction. Perform multi-stage downsampling: each stage includes convolution operations, activation functions, normalization, and incorporates temporal embeddings. .

[0051] Output L multi-scale feature maps , where l is the scale level, and L is consistent with the scale number of the context feature pyramid, providing skip connection features for the decoder.

[0052] 3. Decoder Dynamic Prediction, Quantization, Gating, and Correction Loop: Specifically, the decoder iterates stage by stage from the deepest layer to the shallowest layer. In each stage l, where l ranges from L to 1, the following loop is executed, with the input being upsampled features. Its characteristics come from the output of stage l+1 and skip connection features. It comes from encoder stage l, such as Figure 3 The diagram shows the dynamic loop process of the decoder, specifically including: S301, basic feature map generation and prediction, through the basic convolution module. Achieve preliminary feature prediction: Module composition: The basic convolutional module consists of two 3×3×3 or 3×3 residual convolutional blocks, including batch normalization and ReLU activation function.

[0053] Feature fusion: and Concatenate along the channel dimension (concat), input .

[0054] Output: Generate basic feature maps , which serves as the initial denoising feature for the current stage, where i is the subsequent Monte Carlo sampling index.

[0055] S302, Cognitive Uncertainty Map Quantization, quantifies the uncertainty of basic feature predictions using a Monte Carlo stochastic ensemble model. Two implementation methods are available: Method 1: Gaussian noise injection: In the basic convolutional module... After each convolutional layer, independent and identically distributed (i.i.d.) Gaussian noise is injected, such as with a mean of 0 and a variance of 0.1, for the same input ( + Perform N forward propagations (e.g., N=4) to obtain N perturbation basic feature maps. .

[0056] Method 2: Dropout Layer Activation: This method ensures that the Dropout layers in the basic convolutional modules are activated during both training and inference, unlike traditional Dropout which is only activated during training. It performs N forward propagations on the same input to obtain N random dropout-enhanced basic feature maps. .

[0057] For uncertain calculations, for example, calculating the pixel-wise mean of N basic feature maps to obtain stable basic features. .

[0058] Cognitive uncertainty map: Calculate the pixel-wise variance of N basic feature maps to generate an uncertainty map. ,in, Dimensions and Consistency: The higher the value, the more uncertain the model's segmentation prediction for that pixel is, i.e., the higher the probability of it being a "difficult pixel".

[0059] S303, Sparse Binary Gated Graph Generation Based on Cognitive Uncertainty Graph Sparse gating is generated through TopK selection and binarization to identify the set of difficult pixels. Specifically, this includes channel average pooling: Perform average pooling along the channel dimension to compress the multi-channel uncertainty graph into a single-channel uncertainty graph. Dimensions This eliminates channel redundancy and focuses on spatial location uncertainty.

[0060] TopK selection: Call the TopK function, input... The hyperparameter k_sparse controls sparsity; for example, setting it to 5% of the total number of voxels in the current layer will only retain a portion of the output. A temporary graph with the k_sparsest median positions set to the maximum and the rest set to zero.

[0061] Binarization: via indicator function Convert the temporary graph into a sparse binary gated graph. If the value at a certain position in the temporary graph is > 0, Mark as a difficult pixel; otherwise , marked as simple pixels.

[0062] Determining the set of difficult pixels: Set of difficult pixels Subsequent fine-tuning is performed only on the pixels in that set.

[0063] S304, Local Multimodal Hypergraph Construction and Feature Refinement, targeting each difficult pixel. By constructing a dynamic hypergraph and calculating feature correction terms through attention propagation, high-order multimodal information fusion is achieved.

[0064] Specifically, firstly, a local multimodal hypergraph is constructed. , including node sets That is, a set of feature tokens, in which nodes consist of three types of tokens, covering local-global and multimodal information.

[0065] Specifically, query token: retrieve Feature vector of pixel p , representing the basic predicted features of the current pixel.

[0066] Local spatial token: Retrieves the tokens of all pixels within a 3×3×3 or 3×3 spatial neighborhood of pixel p. The feature vectors consist of either 27 or 9 elements, denoted as... To capture the continuity of local structures.

[0067] Global context token: From the l-scale of the context feature pyramid C, sample one modal feature vector for each spatial location corresponding to pixel p, such as T1, T1c, T2, and FLAIR modalities. This token is denoted as... Introducing multimodal global information

[0068] Node set integration: Each node feature is processed by a small MLP ( ) is mapped to a latent space of a uniform dimension, where, It is an MLP that maps the original features of nodes to the latent space.

[0069] Also includes superedge sets This refers to hyperedge primitives based on medical priors. Hyperedges are used to model higher-order associations between nodes. For example, at least one of the following hyperedge primitives is predefined: Enhanced core hyperedge: By utilizing the specificity of the T1c mode for tumor enhancement regions, the correction of enhanced tumor regions can be strengthened.

[0070] Edema area extends beyond the edge: By utilizing the sensitivity of the FLAIR / T2 mode to the edema region, the segmentation of the edema region is optimized.

[0071] Structural topology hyperedges: It captures local spatial topological relationships and avoids the breakage of segmentation boundaries.

[0072] Multimodal fusion hyperedge: It integrates global information from all modalities to resolve the ambiguity problem of single modality.

[0073] Next, the feature correction term is calculated through hypergraph attention propagation. The propagation process consists of two steps: node-to-hyperedge aggregation and hyperedge-to-node propagation. The final output is the feature correction term for pixel p. .

[0074] Specifically, node-to-hyperedge aggregation: for each hyperedge Through aggregation functions (Max-Pooling or Mean-Pooling) integrates the latent space features of all nodes within the hyperedge to generate the hyperedge representation. : ,in Let e ​​be the node features contained in the hyperedge. Map node features to the latent space.

[0075] Propagation from superedge to node: Using the query token as the target, the attention weights between it and all superedges e containing it are calculated using an attention mechanism. The modified term is obtained by weighted summation of the hyperedge representation. .

[0076] Specifically, this includes attention weight calculation, which involves generating query vectors for query tokens using an MLP. Key vector represented by hyperedge Weights are calculated using dot product similarity: Where sim(⋅,⋅) is the dot product similarity function. For all Included superedges.

[0077] Correction item generation: via Generate the Value vector represented by the hyperedge, and then sum them by weights to obtain... : Subsequently, the sparse correction map is fused with the gate, specifically including the sparse correction map construction: collecting all hard pixels. Correction items Construct a sparse correction graph , Only the difficult pixel position is The remaining positions are 0; Gated fusion: stabilizes the basic features With sparse correction graph Using binary gating graphs Pixel-by-pixel multiplication and fusion yield the final output features of the decoder stage l. : ,in For element-wise multiplication, correction terms are only added to difficult pixel positions, while the basic features of simple pixel positions remain unchanged, thus achieving efficient allocation of computing resources.

[0078] Finally, the decoder stage iterates and outputs the fractional function, with each decoder stage l outputting... Then, it is upsampled to l-1 stage resolution via transposed convolution, and used as the l-1 stage resolution. .

[0079] After iterating to the shallowest layer l=1, the final output features are mapped to the same level using a 1×1 convolutional layer. Dimensionally Consistent Fractional Function Estimation That is, conditional fractional function Approximate value.

[0080] Preferably, the score matching loss function is minimized. To train the core objective, ensure that SHDN accurately approximates the conditional fractional function:

[0081] in The parameter is SHDN network, This represents all learnable parameters of the network. This represents the score matching loss function. Indicates time (Uniformly distributed U(0,T), true segmentation mask) (Data distribution) ),noise ) Expectation; , Signal scaling factor and noise scaling factor of forward SDE It is a time-dependent weighting function used to balance the loss contribution under different noise levels; The L2 norm squared, i.e., the sum of squares of the elements of a vector, measures the difference between the SHDN output and the ideal fractional function. The error.

[0082] In one embodiment, preferably, the total training loss function of the sparse hypergraph diffusion network is a weighted sum of multiple loss terms. The core loss ensures the approximate accuracy of the fractional function, while the auxiliary loss provides semantic supervision and sparsity constraints, ultimately achieving dual-objective optimization of segmentation accuracy and computational efficiency. Its mathematical expression is: ,in, All learnable parameters of SHDN include those of the encoder, decoder, and hypergraph attention module; : Core score matching loss, used to train the SHDN exact approximation conditional score function; : Auxiliary segmentation loss provides direct semantic supervision, accelerates network convergence, and improves the category accuracy of segmentation masks; Sparsity regularization loss encourages the sparsity of sparse binary gated graphs and avoids the waste of computational resources caused by overactivation of hypergraph correction modules. , : Loss weight hyperparameters, which control the contribution ratio of auxiliary segmentation loss and sparsity regularization loss respectively (typical values: =1.0, =0.05, which can be determined by validation set optimization based on the specific segmentation task (such as brain tumors or liver tumors).

[0083] Specifically, for auxiliary segmentation loss The auxiliary segmentation loss provides explicit semantic supervision to the network by deriving the noise-free segmentation map prediction value from the score function and calculating the loss with the ground truth label. This overcomes the limitation of the score matching loss, which only focuses on noise prediction. The specific steps are as follows: First, the noise-free segmentation map prediction value is derived. Specifically, the score function is estimated based on the SHDN output. By combining the mathematical properties of the forward diffusion process, the predicted value of the noise-free segmentation map can be directly deduced: the noisy segmentation map at time t during the forward diffusion process. Compared with the initial noise-free segmentation map The relationship is: ,in, For signal scaling factor, This is the noise scaling factor. It is Gaussian noise.

[0084] According to fraction matching theory, an ideal fractional function satisfies , and SHDN output Approximate conditional fractional function Combining Bayes' theorem, it can be derived that .

[0085] Will Substituting into the forward diffusion formula, the inverse solution yields the noise-free segmentation prediction value: ,in, Dimension and Real Segmentation Mask Consistent, each channel corresponds to a predicted feature for one category.

[0086] The combined loss of Dice loss and cross-entropy loss (DiceCELoss) is used to calculate With real labels The difference between the two loss functions can be addressed by combining the overlap of segmented regions with the class probability distribution. This combination loss function is suitable for common class imbalance problems in medical image segmentation, such as the fact that the pixel ratio of tumor regions is much lower than that of the background.

[0087] Real segmentation mask Convert to one-hot encoding format: If the segmentation task contains K categories, such as K=4 in brain tumor segmentation, corresponding to background, necrotic core, edema area, and enhanced tumor area, then This is a K-channel binary tensor, where in the k-th channel (k=1,2,...,K), the target category pixel position has a value of 1, and the other positions have a value of 0; the purpose of preprocessing is to make the true label and the target category pixel value equal. The channel dimensions and numerical distribution, i.e. the 0-1 probability range, are consistent to ensure the effectiveness of loss calculation.

[0088] Calculate the Dice loss for all K categories and take the average to obtain the final Dice loss contribution.

[0089] Specifically, cross-entropy loss measures Category probability distribution and true label The difference; the auxiliary segmentation loss is the weighted sum of the Dice loss and the cross-entropy loss, where the default weights are all 1.0, which can be adjusted according to the task.

[0090] Specifically, for the sparsity regularization loss, L1 regularization is applied to the cognitive uncertainty graph, encouraging the sparse binary gating graph to activate only a small number of "hard pixels," i.e., regions where the model prediction is uncertain. This avoids the surge in computational cost caused by the global activation of the hypergraph correction module. Specifically, L1 regularization terms are used to calculate... L1 regularization has the property of "sparse induction," meaning it tends to make most elements equal to 0, which can effectively reduce uncertainty graphs. Values ​​in the low to medium uncertainty region, thereby reducing The number of active pixels.

[0091] During training, minimize It will make In the TopK operation, the uncertainty value of "simple pixels" (regions predicted by the model) approaches 0, while the uncertainty value of "difficult pixels" is relatively preserved. A small number of pixels with higher median values ​​will be The hypergraph correction module is activated, performing fine-grained processing only on these difficult pixels, thus concentrating computational resources on key regions. Typical sparsity control is as follows: The number of activated pixels does not exceed 5% of the total number of pixels in the current stage.

[0092] In some embodiments, for step S3, before performing predictor-corrector sampling, the basic parameter configuration and initial state settings of the sampling process need to be completed.

[0093] Specifically, this includes: 1.1 Discretization of the time interval, specifically, discretizing the continuous time interval [T, 0] of the back-diffusion (where T is the initial noise time, typically T = 1.0) into... The discrete time series {t0, t1, ..., t} is formed by several equally spaced sub-intervals. Nsteps}, where: t0=0 (final noise-free moment), t Nsteps =T (initial noise time).

[0094] Time step Where i is the discrete time step index, starting from N steps Up to 1, A negative value indicates that time progresses from T to 0; N steps As a hyperparameter, it is necessary to balance sampling efficiency and segmentation accuracy. Typical values ​​are 200 (3D medical image segmentation) or 100 (2D medical image segmentation), N steps The larger the value, the higher the sampling accuracy, but the longer the inference time.

[0095] 1.2 Initial Noise Sample Generation. Specifically, initial noisy samples are obtained by sampling from the standard normal distribution N(0,I). That is, the state at time t=T, where: The dimension of I is consistent with the target segmentation mask; I is the identity matrix, which ensures that the dimensions of each pixel in the initial noise are independent, which is consistent with the probability assumption of back diffusion.

[0096] 1.3 Loading the Context Feature Pyramid. Specifically, the context feature pyramid C, which was pre-computed and stored in step S1, is loaded as a fixed condition input for the sparse hypergraph diffusion network during the sampling process, ensuring the guiding role of multimodal information in the denoising direction.

[0097] According to an embodiment of the present invention, the predictor-corrector solves the inverse stochastic differential equation, specifically, for each discrete time step i, it starts from N steps Up to step 1, the predictor rapidly advances and the corrector error corrects alternately, gradually reducing the initial noise. Denoising is achieved by using a mask that closely approximates the true segmentation mask. That is, the target segmentation mask.

[0098] Specifically, the predictor is based on the Euler method of probabilistic flow ODE. The core function of the predictor is to rapidly update the sample state along the denoising direction, using the Euler method to solve the probabilistic flow ordinary differential equation (ProbabilityFlowODE) corresponding to ReverseSDE, thus achieving a one-step rapid advancement of the sample state.

[0099] Specifically, the probabilistic flow ODE is derived by neglecting the random perturbation term (dw) in the ReverseSDE, and its expression is: ,in, The rate of change of the sample state over time, i.e., the direction of noise reduction; The noise scheduling function is consistent with that used during the training phase. The fractional function estimate output by SHDN provides gradient guidance for the direction of denoising.

[0100] Next, the Euler method is used to numerically solve the probability flow ODE through state update, and the samples at time step i are... Updated to the predicted sample at time step i-1 The formula is: ,in, : The intermediate sample state output by the predictor; The discrete time step is negative here; the Euler method has low computational complexity, which can significantly reduce the sampling time of each time step and ensure overall inference efficiency.

[0101] The corrector, based on Langevin dynamics error correction, plays a crucial role in correcting the numerical error of the predictor. This is achieved by executing J-step Langevin dynamics iterations (e.g., J=2), which adjust the predicted samples... To restore the true data distribution and improve denoising accuracy: for predicted samples The state update formula for the j-th step (j ranges from 0 to J-1) of the Langevin iteration is: ,in, The initial input of the corrector is the output of the predictor; where : The corrected sample after the (j+1)th iteration; where The Langevined step length of time step i is a small positive number, such as... =0.001, which needs to decrease as time t decreases. The closer t is to 0, the less sample noise there is. Step size The smaller the value, the less likely it is to over-correct and disrupt the existing segmentation structure; New noise sampled from the standard normal distribution N(0,I) injects a suitable amount of randomness into the iterative process, helping the samples to escape local optima. SHDN is for the current calibration sample and time. The output score function guides the samples to adjust towards higher data density.

[0102] Subsequently, the corrector output is determined. Specifically, after completing the J-step Langevin iteration, the final iteration result is taken. The sample state at time step i-1 ,Right now: By employing multi-step Langevin correction, the first-order numerical error of the predictor's Euler method is offset, ensuring that the sample state always closely matches the true evolution trajectory of ReverseSDE, thus providing an accurate initial state for denoising in subsequent time steps.

[0103] This is followed by multi-timestep iterative denoising. Specifically, the predictor advancement and corrector correction processes are repeated until all N are completed. steps Through iterations at discrete time steps, a noise-free sample at time t=0 is finally obtained. That is, x0, which is a multi-channel segmentation feature map, with each channel corresponding to a segmentation feature of a category.

[0104] Finally, segmentation probability map transformation and post-processing are performed. Specifically, the final denoised result x0 is converted into segmentation labels that can be used for clinical analysis, which requires probability normalization and post-processing operations.

[0105] Specifically, this includes, firstly, the generation of the segmentation probability map. A softmax function is performed on the K channels of x0, converting the K class features of each pixel into a class probability distribution, resulting in the segmentation probability map P: ,in, The probability value of the k-th channel of the segmentation probability map P at pixel i,j,k, with a value range of [0,1]. The final denoising result is the feature value of the k-th channel at pixels i,j,k. This represents the confidence level that the pixel belongs to the k-th category. The higher the probability, the stronger the segmentation confidence.

[0106] Next, initial segmentation labels are generated. Specifically, an argmax operation is performed along the channel dimension of the segmentation probability map P, selecting the class with the highest probability for each pixel as the final class for that pixel, thus obtaining the initial segmentation label map. : , The initial segmentation label map contains the category labels (integers, ranging from 1 to K, corresponding to different anatomical structures or pathological regions) at pixels i, j, and k; for example, in a brain tumor segmentation task. 1 represents the background, 2 represents the necrotic core, 3 represents the edema area, and 4 represents the enhanced tumor area.

[0107] Next, post-processing optimization involves eliminating the initial segmentation label map. To improve the clinical applicability of segmentation results and identify isolated noise regions (such as small missegmented regions), the following post-processing operations are performed: Connectivity component analysis is conducted using algorithms such as 8-neighbor connectivity analysis or 26-neighbor connectivity analysis. Connectivity detection is performed on the regions corresponding to each category: the volume or area of ​​each connected region is calculated; a volume threshold V is set. th(Typically, it is 1% of the average volume of the target region, such as V in brain tumor segmentation) th =10 voxels), remove all volumes smaller than V. th Isolated connected regions.

[0108] Therefore, the segmentation label image after post-processing optimization is the final segmentation result, which can be directly used for clinical diagnosis, disease quantitative assessment or surgical planning. Its format is compatible with medical image annotation tools such as 3DSlicer, and supports doctors' visual verification and adjustment.

[0109] According to embodiments of the present invention, the cognitive uncertainty of the model regarding the current prediction is estimated online through a Monte Carlo random ensemble in the model's decoding path. This uncertainty map is used as a dynamically generated gating signal to sparsely activate a computational module dedicated to handling complex regions. This design allows the model to concentrate more computational resources on pixel regions that it itself considers "difficult," such as blurred tissue boundaries, while using standard, efficient computational paths for "simple" regions. Its potential technical effect is to improve the segmentation accuracy of key regions without significantly increasing, and may even reduce, the average computational load.

[0110] For the identified difficult-to-segment regions, this invention constructs a local multimodal hypergraph. The nodes of the hypergraph consist of features from multiple modalities and scales, while the hyperedges are designed to represent predefined, clinically relevant combinations of high-order features. By performing attention propagation on the hypergraph, the model can learn many-to-many feature dependencies beyond the pixel-pair level. This approach provides a structured framework for deep fusion of multimodal information, helping the model to more accurately understand and segment lesions with complex internal components.

[0111] This invention places the segmentation task within the framework of a conditional diffusion model. The generation of the target segmentation mask is modeled as a reverse stochastic process that starts with Gaussian noise and progressively denoises under the guidance of a multimodal image. This process itself imposes an implicit prior on the continuity and smoothness of the segmentation result. When this framework is combined with the aforementioned sparse hypergraph correction module, the aim is to leverage the generative capabilities of the diffusion model to ensure the overall structural rationality of the segmentation result, while simultaneously ensuring the accuracy of local details through the hypergraph module.

[0112] like Figure 4 The diagram showing the comparison of liver tumor segmentation on the Lits2017 dataset, and... Figure 5 The diagram shown illustrates the comparison of brain tumor segmentation on the Brats2019 dataset. Based on this invention, the results of liver tumor segmentation on the Lits2017 dataset and brain tumor segmentation on the Brats2019 dataset are superior to those of the traditional Unet segmentation.

[0113] Specifically, such as Figure 4 As shown, the tumor region segmentation of the present invention is more complete: Unet is prone to "missed segmentation" due to blurred edges of liver tumors and similar gray levels to surrounding blood vessels / bile ducts, such as small-volume tumor lesions and tumor infiltration areas. The present invention can accurately capture the complete tumor region and reduce missed detection by fusing the density features of the tumor in CT images with global contextual information through multimodal supermap fusion.

[0114] The invention provides more precise boundaries: Unet segmentation of liver tumor boundaries is prone to "sawtooth noise" or "boundary offset", such as mistakenly classifying normal liver tissue into the tumor area, or the tumor boundary not conforming to the actual anatomical location. The invention uses the global structural constraints of the diffusion model and sparse hypergraph to correct the "difficult regions" of the boundary, resulting in smoother boundaries and smaller deviations from the actual annotation.

[0115] This invention has less background interference: Unet is prone to misclassifying the adipose tissue and intestines around the liver as tumor-related areas. This invention focuses only on the tumor and its boundary areas through uncertainty quantification, and combines medical priors of the hypergraph, such as the spatial relationship between liver tumors and blood vessels, to effectively eliminate background interference and reduce false positive segmentation.

[0116] like Figure 5 As shown, specifically, the present invention distinguishes multiple sub-regions more accurately. Brain tumors include nested sub-regions such as whole tumor (WT), tumor core (TC), and enhanced tumor (ET). Unet is prone to "region confusion", such as misclassifying edema areas as tumor cores or missing enhanced tumor areas. The present invention can accurately distinguish each sub-region by using modality-specific hyperedges such as T1c modality to identify ET and FLAIR modality to identify edema areas, and the degree of overlap between the segmented regions is higher.

[0117] This invention offers more sensitive detection of small lesions: Unet is prone to missing small tumor cores or enhanced lesions with insignificant grayscale differences. This invention, through a dynamic hypergraph correction module, invests additional computing resources in these "difficult small lesions" and, combined with multimodal global features, can effectively identify small lesions and reduce the risk of missed diagnosis.

[0118] The overall structure of this invention is more in line with anatomical logic: Unet segmentation results are prone to "fragmentation" due to interference from local features, such as breaks in the tumor region or conflicts with the brain sulcus structure. The conditional diffusion model of this invention applies a continuity prior to the segmentation results, and combined with the modeling of the brain tumor topology by the hypergraph, the segmentation results are more in line with the brain anatomy and have stronger overall consistency.

[0119] Figure 6 A multimodal medical image segmentation system 600 based on a sparse hypergraph diffusion network is illustrated. The system implementation is similar to... Figure 1Corresponding to the method embodiment shown, it specifically includes: a context feature pyramid construction module 601, which is used to extract multi-scale features based on the registered multimodal medical image through a hierarchical encoder and construct a context feature pyramid.

[0120] The target segmentation mask generation module 602 is used to generate a target segmentation mask by iteratively denoising from the initial noise based on the context feature pyramid as a condition and by using a sparse hypergraph diffusion network to drive an inverse stochastic differential equation.

[0121] The segmentation label acquisition module 603 is used to obtain a denoised segmentation feature map by solving the inverse stochastic differential equation through a predictor-corrector based on the target segmentation mask, convert the denoised segmentation feature map into a segmentation probability map, and obtain the final segmentation label through post-processing.

[0122] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multimodal medical image segmentation method based on sparse hypergraph diffusion networks, characterized in that, Includes the following steps: Based on the registered multimodal medical images, multi-scale features are extracted through a hierarchical encoder to construct a context feature pyramid; Based on the context feature pyramid, a sparse hypergraph diffusion network is used to drive an inverse stochastic differential equation to iteratively denoise and generate a target segmentation mask from initial noise. The inference process of the sparse hypergraph diffusion network includes: generating a basic feature map through basic convolutional modules; calculating a cognitive uncertainty map through Monte Carlo stochastic ensemble; generating a sparse binary gating map based on the cognitive uncertainty map; constructing a local multimodal hypergraph based on the sparse binary gating map and calculating feature correction terms through hypergraph attention propagation; fusing the correction terms with the basic feature map to output a denoised feature map as the target segmentation mask. Based on the target segmentation mask, a denoised segmentation feature map is obtained by solving the inverse stochastic differential equation through a predictor and a corrector. The denoised segmentation feature map is then converted into a segmentation probability map and post-processed to obtain the final segmentation label.

2. The multimodal medical image segmentation method based on sparse hypergraph diffusion networks according to claim 1, characterized in that, The multimodal medical images are multi-sequence magnetic resonance imaging, including T1-weighted, T1-contrast-enhanced, T2-weighted, and fluid-attenuated inversion recovery sequences.

3. The multimodal medical image segmentation method based on sparse hypergraph diffusion networks according to claim 1, characterized in that, The inverse stochastic differential equation is a variance-preserving stochastic differential equation, and its conditional fractional function is approximated by the sparse hypergraph diffusion network.

4. The multimodal medical image segmentation method based on sparse hypergraph diffusion networks according to claim 1, characterized in that, include: The inference process of the sparse hypergraph diffusion network includes Monte Carlo random ensemble, which is achieved by injecting independent and identically distributed Gaussian noise after the convolutional layer of the basic convolutional module, or by keeping the Dropout layer active during inference and performing forward propagation on the same input.

5. The multimodal medical image segmentation method based on a sparse hypergraph diffusion network according to claim 4, characterized in that, The inference process of the sparse hypergraph diffusion network includes generating a sparse binary gated graph. Specifically, it includes performing average pooling along the channel dimension on the cognitive uncertainty graph obtained by the Monte Carlo ensemble to obtain a single-channel uncertainty graph, selecting the k_sparse positions with the largest values ​​in the graph using the TopK function and setting their values ​​back to their original values, while setting the remaining positions to zero, and then binarizing it using an indicator function, where k_sparse is a hyperparameter that controls sparsity.

6. The multimodal medical image segmentation method based on a sparse hypergraph diffusion network according to claim 5, characterized in that, The sparse hypergraph diffusion network inference process includes constructing a local multimodal hypergraph, wherein the node set of the local multimodal hypergraph includes query tokens from the base feature graph, local spatial tokens from the spatial neighborhood of the query tokens, and global context tokens sampled from corresponding positions in the context feature pyramid.

7. The multimodal medical image segmentation method based on a sparse hypergraph diffusion network according to claim 6, characterized in that, The set of superedges in the local multimodal hypergraph is a predefined set of superedge primitives based on medical priors. The set of superedge primitives includes at least one of the following: enhancement core superedges for identifying enhanced tumors, edema region superedges for identifying peritumoral edema, structural topology superedges for capturing local shape continuity, and multimodal fusion superedges for performing cross-modal associations.

8. A multimodal medical image segmentation system based on a sparse hypergraph diffusion network, characterized in that, include: The context feature pyramid construction module is used to extract multi-scale features from the registered multimodal medical images through a hierarchical encoder and construct a context feature pyramid. The target segmentation mask generation module is used to generate a target segmentation mask by iteratively denoising from initial noise based on the context feature pyramid and driven by an inverse stochastic differential equation using a sparse hypergraph diffusion network. The inference process of the sparse hypergraph diffusion network includes: generating a basic feature map through a basic convolutional module; calculating a cognitive uncertainty map through Monte Carlo stochastic ensemble; generating a sparse binary gating map based on the cognitive uncertainty map; constructing a local multimodal hypergraph based on the sparse binary gating map and calculating a feature correction term through hypergraph attention propagation; and fusing the correction term with the basic feature map to output a denoised feature map as the target segmentation mask. The segmentation label acquisition module is used to obtain a denoised segmentation feature map by solving the inverse stochastic differential equation through a predictor and a corrector based on the target segmentation mask, convert the denoised segmentation feature map into a segmentation probability map, and obtain the final segmentation label through post-processing.

9. A multimodal medical image segmentation system based on a sparse hypergraph diffusion network according to claim 8, characterized in that, The multimodal medical images are multi-sequence magnetic resonance imaging, including T1-weighted, T1-contrast-enhanced, T2-weighted, and fluid-attenuated inversion recovery sequences.

10. A multimodal medical image segmentation system based on a sparse hypergraph diffusion network according to claim 8, characterized in that, The inverse stochastic differential equation is a variance-preserving stochastic differential equation, and its conditional fractional function is approximated by the sparse hypergraph diffusion network.

Images