Multi-source domain kidney magnetic resonance image segmentation method based on double-branch knowledge aggregation

Abstract

The application provides a multi-source domain kidney magnetic resonance image segmentation method based on double-branch knowledge aggregation, and relates to the field of medical imaging.The image segmentation method comprises constructing a double-branch knowledge aggregation image segmentation model; and training the double-branch knowledge aggregation model using multiple sequence kidney magnetic resonance images with segmentation labels, so that the model has the ability to segment multi-source domain kidney magnetic resonance images.The double-branch knowledge aggregation model comprises a shared encoder module, a singular value knowledge aggregation module, and a first knowledge aggregation branch formed by an independent decoder D1, a picture contrast knowledge aggregation module, an independent decoder D2, and a domain alignment module.The model architecture uses double branches, integrates two different knowledge aggregation implementation methods, and the two implementation methods complement each other, so that the whole model achieves more accurate image segmentation results.

Description

Technical Field

[0001] This invention relates to the field of medical imaging, and more particularly to a method for segmenting multi-source domain nuclear magnetic resonance images. Background Technology

[0002] The kidneys are one of the most vital organs in the human body and also one of the most commonly diseased organs. Magnetic resonance imaging (MRI) is a common non-invasive method for evaluating kidney diseases. Renal MRI can not only be used for the qualitative diagnosis of renal space-occupying lesions, becoming an important tool for the stratified management of renal tumors, but it can also quantitatively evaluate the degree of acute and chronic kidney injury, serving as an important tool for exploring the pathogenesis of acute and chronic kidney injury. For example, kidney volume is one of the markers of renal function impairment; in acute kidney injury, kidney volume is significantly enlarged, while in the middle and late stages of chronic kidney injury, it is relatively reduced in size. In previous clinical practice, kidney volume was used as a substitute for the length of the superior and inferior diameters of the kidney, but this was mainly used in renal ultrasound measurements and could not be directly measured in renal MRI. Measuring kidney volume in renal MRI is more advantageous, but it requires manually outlining the kidney contour, which is very time-consuming and labor-intensive.

[0003] Chinese patent application number ZL201811622330.9, entitled "Three-Dimensional Medical Image Processing Device and Method," discloses a method for segmenting three-dimensional medical images based on neural networks. This method uses a convolutional neural network model trained with a large amount of labeled data to segment a single sequence of renal MRI images and calculate the kidney volume. This method achieves automated segmentation of single-sequence renal MRI images and has been applied clinically.

[0004] However, renal MRI generally employs multiplanar and multiparametric imaging and multiparametric sequences, such as different imaging sequences from Siemens MRI equipment, such as T1 FL2D, T1 VIBE COR, T1 VIBE TRA, and T2 PCA, which provide different tissue physiological and pathological information. However, the imaging mechanisms, signal intensities, and noise distributions of different sequences vary greatly, and the feature representations at the pixel level may be completely different. It is difficult for neural networks to learn and integrate such heterogeneous features across sequences at the same time, which can easily lead to feature confusion.

[0005] Therefore, there is a need to develop a more adaptable comprehensive algorithm that is suitable for the needs of automated kidney segmentation in multi-parameter and multi-planar imaging. Summary of the Invention

[0006] To meet the needs of automated segmentation in multi-planar and multi-parameter renal magnetic resonance imaging and improve the adaptability of automated magnetic resonance imaging segmentation, this invention proposes a multi-source domain renal magnetic resonance image segmentation method based on dual-branch knowledge aggregation.

[0007] First aspect: A multi-source domain renal MRI image segmentation method based on dual-branch knowledge aggregation, characterized in that the method includes the following steps: A dual-branch knowledge aggregation image segmentation model is constructed, comprising: a first knowledge aggregation branch consisting of a shared encoder module, a singular value knowledge aggregation module, and an independent decoder D1; a second knowledge aggregation branch consisting of a graph comparison knowledge aggregation module and an independent decoder D2; and a domain alignment module. The dual-branch knowledge aggregation image segmentation model is trained using multiple sequences of kidney MRI images with segmentation labels. N parameter sequences of kidney MRI images are collected, each sequence containing M original images and corresponding segmentation labels. The original image of one sequence is selected as the target domain, and the original images and segmentation labels of the other N-1 sequences are selected as the source domain. The images are input into the dual-branch knowledge aggregation image segmentation model for one round of training. The source and target domain samples are changed for the next round of training. A total of N rounds of training are conducted, where N and M are both natural numbers greater than 1. Verify the reliability of the trained dual-branch knowledge aggregation image segmentation model; The kidney MRI image to be segmented is input into the trained dual-branch knowledge aggregation image segmentation model to obtain the segmentation result.

[0008] Furthermore, the singular value knowledge aggregation module in the dual-branch knowledge aggregation image segmentation model includes: a local feature map convolutional block. basis matrix Fully connected layer Feature recovery convolutional blocks and reconstructed feature discriminator ; The input to the singular value knowledge aggregation module is the original domain feature map output by the shared coding module. The output is a domain feature map with spatial semantic enhancement. ; The singular value knowledge aggregation module performs the following functions: S121: The domain feature map output from the shared coding module Classified by spatial location a small piece ; S122: Each small block Convolutional blocks using local feature mapping Mapping to feature space In this context, it becomes a d-dimensional local feature vector. ; S123: The data will come from N domains, each domain... All local feature vectors are concatenated, with the concatenation order being that local feature vectors from the same spatial location in different domains are grouped together to form a feature matrix. ; S124: Will Reconstructed The reconstructed feature matrix By integrating the local spatial features of each domain, the formula is expressed as follows: , In the above formula As a feature space The basis matrix is ​​a globally learnable matrix. Let be the coefficient matrix, representing In the basis matrix The projection on the surface, through the fully connected layer To fit: ; S125: Reconstruct the feature matrix Split into n is the domain index, and j is the spatial location index; S126: Combining products from the same domain Arranged and pieced together according to spatial location ; S127: Recovering Convolutional Blocks from Features Will Restore to .

[0009] Furthermore, the graph contrastive knowledge aggregation module in the dual-branch knowledge aggregation image segmentation model includes an average pooling submodule, a graph convolutional network G, and an upsampling submodule; The graph comparison knowledge aggregation module receives its input from the original feature map from the shared encoding module. The output is a domain feature map that incorporates topological knowledge. ; The graph comparison knowledge aggregation module performs the following functions: S141: The domain feature map output from the shared coding module Classified by spatial location Small feature blocks ; S142: Take the average of each small feature block as the prototype vector of that local region. ; S143: Will As nodes, a graph feature matrix is ​​used. To represent a set of nodes, S144: Construct an adjacency matrix To express the relationship between nodes. Each element in the array represents the similarity between nodes using cosine similarity. S145: Will and Input a graph convolutional network G and perform data augmentation on the graph. ; S146: Through the upsampling submodule, Upsampling restored to , This represents the domain feature graph after incorporating topological knowledge.

[0010] Furthermore, the domain alignment module includes a gradient inversion layer (GRL) and N-1 discriminators. N-1 represents the number of source domains; the input to the domain alignment module comes from the original feature map output by the shared encoder module. The domain feature map with spatial semantic enhancement output by the singular value knowledge aggregation module The domain feature map output by the graph comparison knowledge aggregation module incorporates topological knowledge. The output of the domain alignment module is the domain alignment difference of the first knowledge aggregation branch. Domain alignment differences of the second knowledge aggregation branch .

[0011] Furthermore, the dual-branch knowledge aggregation model is trained using multiple sequences of kidney MRI images with segmentation labels, and the total loss function is:

[0012] In the above formula The loss function for the first knowledge aggregation branch. The loss function for the second knowledge aggregation branch. For the domain alignment difference of the first knowledge aggregation branch, For the domain alignment differences of the second knowledge aggregation branch, The source domain number, .

[0013] Furthermore, the loss function of the first knowledge aggregation branch for: , in It is a balance parameter. Let the segmentation loss function be the first knowledge aggregation branch. Reconstruct the loss function for the singular value module. Basis matrix For full-rank and stable constraint functions, for The constraint function for a low-rank matrix. for The constraint function for a block diagonal matrix; The basis matrix For full-rank and stable constraint functions as follows: , in For the basis matrix The constraint function must be full rank, obtained by applying the basis matrix. Perform singular value decomposition (SVD) to obtain the basis matrix. All singular values ​​are listed and sorted in descending order of value. Constrain smaller singular values ​​to be far from 0 to ensure the basis matrix The singular values ​​are all non-negative, indirectly constraining Full rank of a matrix: , , Basis matrix The constraint function must be stable, through the basis matrix. The stability of a matrix is ​​calculated using its condition number. The smaller the condition number, the more stable the matrix is. The condition number is calculated as the ratio of the largest singular value to the smallest singular value, as shown in the following formula: ; The constraint functions for low-rank matrices as follows: Will Divided into a small piece ,right Perform Singular Value Decomposition (SVD) to obtain all singular values ​​{ , ,constraint Minimize the sum of singular values ​​to guarantee that each small block It is a low-rank matrix. ; The Constraint functions for block diagonal matrices as follows: , ; in It is a diagonal mask. It is the value of the m-th element when that element is within the diagonal region of the block. The value is 0 otherwise the function. constraint The remaining elements in the set are close to 0.

[0014] Furthermore, the loss function of the second knowledge aggregation branch... as follows:

[0015] in It is a balance parameter. Let the segmentation loss function be the second knowledge aggregation branch. The InfoNCE loss function is used for information-noise contrast estimation. The information noise contrast estimation InfoNCE loss function Specifically as follows: Randomly discard some nodes to obtain the first view: ; Randomly discard some edges to obtain a second view: ; The first view of the first Sample As a query sample, the corresponding sample in the second view The remaining samples in the second view are positive samples, and the remaining samples in the third view are negative samples. Therefore, the fourth... Information-noise contrast estimation of InfoNCE loss for each sample:

[0016] In the above formula Let represent the temperature parameter, sim represent the similarity function, and B be the total number of samples; Swap the views of the query samples to the second view. sample As a query sample, the corresponding sample of the first view As positive samples, the remaining samples in the first view are negative samples, resulting in a symmetric loss: ; Calculate the InfoNCE loss for all samples based on the information-noise contrast, and then take the average: .

[0017] Furthermore, the random discarding of some nodes is specifically as follows: Set ratio ,exist Random sampling Line it and assign its value to 0, and get The set of sampled row numbers is represented as ,Will Chinese correspondence If all elements in the row and column are assigned the value 0, then... .

[0018] Furthermore, the random discarding of some edges is specifically as follows: Set ratio ,exist Random sampling in the upper triangular matrix 1 element, and set their values ​​to 0, in In the lower triangular matrix, the corresponding symmetric elements are assigned the value 0, resulting in... , It remains unchanged.

[0019] The second aspect: An electronic device includes a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the image segmentation method as described in the first aspect.

[0020] The beneficial effects of this invention are as follows: (1) The concept of multi-source domains is used to define renal MRI images of different sequences. A knowledge aggregation strategy is adopted to integrate, refine, and optimize the feature information from renal MRI images of multiple sources and different sequences, forming a more robust and effective unified knowledge representation. The method proposed in this invention can utilize complementary information between different domains to allow the network to learn common features and discriminative knowledge from each domain. This enables the model to correctly process inputs from different domains, identify common features, and avoid being confused by different feature representations from different domains. Thus, it possesses the ability to correctly segment renal MRI images of different sequences. Figure 2-5 In the study, it is evident that the knowledge aggregation-based multi-source domain NMR image segmentation method outperforms other multi-source domain image segmentation algorithms in segmenting NMR images of different sequences. (2) A dual-branch model architecture was used, which integrates two different knowledge aggregation implementation methods. The two implementation methods complement each other and are fused at the loss function level, so that the whole model can achieve more comprehensive knowledge fusion. The ablation experiment of dual-branch model was carried out. The results show that the segmentation result of dual-branch is better than the segmentation result of any single branch. (3) The singular value knowledge aggregation module is used to map the feature vectors of the same local position of different sequence NMR images into a sub-feature space. A learnable basis matrix is ​​used to represent the global feature space. Then, the combination of the coefficient matrix and the basis matrix is ​​used to map all local feature vectors. By constraining the singular values ​​of the corresponding coefficient matrix, the correlation between the same local positions in each domain is improved and the correlation between different local positions is reduced, so that the model can better learn the characteristics of each local structure in the image and enhance the ability to distinguish the region of interest. (4) The graph contrastive knowledge aggregation module is used to extract the topological relationships of each spatial local feature and to model different sequences of MRI images as graph structures. By randomly deleting some edges or nodes, the original graph is enhanced to generate different views. These views are consistent in structure but different in some details, thus providing more learning signals for the model. During the training process, the model uses contrastive loss to constrain the network to learn more discriminative node or graph representations, thereby improving the model's understanding of graph structures. Attached Figure Description

[0021] Figure 1 This is a flowchart of the multi-source domain renal MRI image segmentation method based on dual-branch knowledge aggregation of the present invention; Figure 2 This is a schematic diagram of the overall framework of the dual-branch knowledge aggregation image segmentation model of the present invention; Figure 3 This invention compares the image segmentation method of this invention with other image segmentation algorithms for kidney segmentation results in rz image sequences; Figure 4 This invention compares the image segmentation method of this invention with other image segmentation algorithms for kidney segmentation results in rt image sequences; Figure 5 This invention compares the image segmentation method of this invention with other image segmentation algorithms for kidney segmentation results in rh image sequences; Figure 6 This invention compares the image segmentation method of this invention with other image segmentation algorithms for kidney segmentation results in rb image sequences; Wherein (a) is the input image, (b) is the input image and its kidney segmentation label, (c) is the input image and the kidney segmentation result after using the image segmentation method of the present invention, (d) is the input image and the kidney segmentation result of the DoCR algorithm, (e) is the input image and the kidney segmentation result of the MCLS algorithm, (f) is the input image and the kidney segmentation result of the StandardGAN algorithm, (g) is the input image and the kidney segmentation result of the M2CD algorithm, (h) is the input image and the kidney segmentation result of the M3DA algorithm, and (i) is the input image and the kidney segmentation result of the DRT algorithm. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the invention is not limited to the specific embodiments disclosed below.

[0024] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0025] While this invention makes various references to certain modules in the model according to embodiments of the invention, any number of different modules can be used and run on computing devices and / or processors. The modules are merely illustrative, and different aspects of the method may use different modules.

[0026] It should be understood that when a unit or module is described as "connected" or "coupled" to other units, modules, or blocks, it can refer to a direct connection or coupling, or communication with other units, modules, or blocks, or the presence of intermediate units, modules, or blocks, unless the context explicitly indicates otherwise. The term "and / or" as used herein can include one or more related columns. Any and all combinations of the project.

[0027] This invention uses flowcharts to illustrate the steps performed by the method according to embodiments of the invention. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more steps may be removed from them.

[0028] Some descriptions of this invention are provided in conjunction with renal magnetic resonance imaging (MRI) images. It should be understood that this is for illustrative purposes and not intended to limit the scope of the invention. The methods of this invention can be used to process images or image data from other organs using other imaging modalities, including, for example, digital radiography (DR) algorithms, computed tomography (CT) algorithms, medical ultrasound imaging algorithms, multimodal algorithms, or the like, or any combination thereof. Exemplary multimodal algorithms may include positron emission tomography-computed tomography (PET-CT) algorithms, positron emission tomography-magnetic resonance imaging (PET-MRI) algorithms, etc.

[0029] Figure 1 This is a flowchart of the multi-source domain renal MRI image segmentation method based on dual-branch knowledge aggregation, as described in this invention. Includes the following steps: S1: Construct a dual-branch knowledge aggregation image segmentation model, wherein the dual-branch knowledge aggregation image segmentation model includes: The first knowledge aggregation branch consists of a shared encoder module, a singular value knowledge aggregation module, and an independent decoder D1; the second knowledge aggregation branch consists of a graph comparison knowledge aggregation module and an independent decoder D2; and a domain alignment module. S2: Train the dual-branch knowledge aggregation model using multiple sequences of kidney MRI images with segmentation labels. Collect N parameter sequences of kidney MRI images. Each parameter sequence contains M original images and corresponding segmentation labels. Select the original image of one sequence as the target domain, and the original images and segmentation labels of the other N-1 sequences as the source domain. Input the images into the dual-branch knowledge aggregation image segmentation model for one round of training. Change the sample selection of the source domain and target domain and perform the next round of training. Train for a total of N rounds. S3: Verify the reliability of the trained dual-branch knowledge aggregation image segmentation model; S4: Input the kidney MRI image to be segmented into the trained dual-branch knowledge aggregation image segmentation model to obtain the segmentation result.

[0030] The following will explain each step in detail.

[0031] S1: Figure 2 This is a schematic diagram of the overall framework of a dual-branch knowledge aggregation image segmentation model constructed through some embodiments. As shown in the figure, the dual-branch knowledge aggregation image segmentation model of the present invention includes the following modules: S11: Shared encoder module, The shared encoder module is equipped with a multi-source domain nuclear magnetic resonance image input interface. The output of the shared encoder module is sent to the singular value knowledge aggregation module, the graph comparison knowledge aggregation module, and the domain alignment module, respectively.

[0032] The shared encoder module is used to process the input multi-source domain nuclear magnetic resonance images. Mapped to a unified feature space, it becomes the original domain feature. picture; S12: Singular Value Knowledge Aggregation Module The input to the singular value knowledge aggregation module comes from the shared encoder module, and the output is sent to the domain alignment module and the independent decoder D1. The singular value knowledge aggregation module and the independent decoder D1 form the first knowledge aggregation branch.

[0033] The singular value knowledge aggregation module is used to aggregate local spatial features from various domain feature maps and reconstruct domain feature maps with enhanced spatial semantics. ; S13: Independent decoder D1, The input to the independent decoder D1 comes from the singular value knowledge aggregation module, and its output is the model output.

[0034] Independent decoder D1 is used to enhance the spatial semantics of the domain feature map. With the original domain feature map After averaging, the result is decoded into the first knowledge aggregation branch domain segmentation graph. ; S14: Graph Comparison Knowledge Aggregation Module The input to the graph comparison knowledge aggregation module comes from the shared encoder module, and the output is sent to the domain alignment module and the independent decoder D2. The graph comparison knowledge aggregation module and the independent decoder D2 form the second knowledge aggregation branch. The graph comparison knowledge aggregation module uses graph convolutional networks to aggregate and enhance the spatial topological knowledge of each domain feature map, and reconstructs the spatially topologically enhanced domain feature maps. ; S15: Independent decoder D2, The input to the independent decoder D2 comes from the graph alignment knowledge aggregation module, and its output is the model output. The independent decoder D2 is used to enhance the spatial topology of the domain feature map. With the original domain feature map After averaging, the result is decoded into the second knowledge aggregation branch domain segmentation graph. .

[0035] The model's final output is a segmentation map corresponding to the input MRI image. .

[0036] The above has completed the model for multi-source domain MRI image segmentation. However, in order to complete the overall training of the model and achieve the fusion of the two branches, a shared domain alignment module needs to be added to the back end of the singular value knowledge aggregation module and the graph comparison knowledge aggregation module. This module will be used during the overall training of the model to ensure that the output feature maps of the image feature maps from each source domain can be as close as possible after knowledge aggregation.

[0037] S16 is the domain alignment module, which is used to measure the differences between multi-domain fused feature maps during model training.

[0038] The following sections provide further detailed explanations of the singular value knowledge aggregation module, the graph comparison knowledge aggregation module, and the domain alignment module.

[0039] The singular value aggregation module includes: local feature map convolutional blocks. basis matrix Fully connected layer Feature recovery convolutional blocks and reconstructed feature discriminator .

[0040] The singular value knowledge aggregation module performs the following functions: S121: The domain feature map output from the shared coding module E Classified by spatial location a small piece The formula is expressed as: , In the above formula, n is the domain index and j is the spatial location index; in some embodiments Set to 4; S122: Each small block Convolutional blocks using local feature mapping Mapping to feature space In this context, it becomes a d-dimensional local feature vector. ; ; S123: The data will come from N domains, each domain... All local feature vectors are concatenated, with the concatenation order being that local feature vectors from the same spatial location in different domains are grouped together to form a feature matrix. The order in which the feature matrices are assembled is very important. Arranging feature vectors that belong to different domains but are located in the same local space together is the logical basis for constructing constraint functions during later training. , ; S124: Construct a globally learnable matrix As a feature space The basis matrix, the characteristic space any characteristic matrix in It can be used Linear representation: , In the above formula Represents the coefficient matrix. Each row vector express row vectors in In the basis matrix Projection on; Through the fully connected layer To fit : ， Through the fully connected layer Later rebuilt The reconstructed feature matrix By integrating the local spatial features of each domain, the formula is expressed as follows: ; S125: The reconstructed feature matrix Split into n is the domain index, and j is the spatial location index; S126: Combining products from the same domain Arranged and pieced together according to spatial location ; S127: Recovering Convolutional Blocks from Features Will Restore to

[0041] , Domain feature map representing spatial semantic enhancement; Discriminator Used to measure the singular value module's affinity for the feature matrix during the model training phase. The reconstruction losses.

[0042] The graph comparison knowledge aggregation module includes an average pooling submodule, a graph convolutional network G, and an upsampling submodule; The graph comparison knowledge aggregation module performs the following functions: S141. The domain feature map output from the shared coding module E. Classified by spatial location A small feature block, represented as ; S142: The average of each small feature block in the spatial dimension is taken as the prototype vector of that local region, represented as follows: In this embodiment, Set to 4; This step can be achieved through average pooling of the original features, with the pooling sliding window size set to... ,Right now:

[0043] S143: Prototype vector for each local region It is used as a node to construct the graph model. , The set of nodes represents the convergence of prototype vectors from all domains: , Using a graph feature matrix To represent it as: , edge set express The relationships between all nodes in the process; S144: Construct an adjacency matrix Come to To model the topological relationships between nodes, graph comparison knowledge aggregation uses cosine similarity to represent the similarity between nodes, and thus the adjacency matrix values ​​of the two nodes. , in , They represent The first in and The prototype vector of the first row; S145: Will and Input a graph convolutional network G and perform data augmentation on the graph. ; S146: Through the upsampling submodule, Upsampling restored to , This represents the domain feature graph after incorporating topological knowledge.

[0044] The domain alignment module includes a gradient inversion layer (GRL) and N-1 discriminators. Here, N-1 represents the number of input source domains during training, and the input comes from the original feature maps output by the shared encoder module. The domain feature map with spatial semantic enhancement output by the singular value knowledge aggregation module The domain feature map output by the graph comparison knowledge aggregation module incorporates topological knowledge. The output of the domain alignment module is the domain alignment difference of the first knowledge aggregation branch. Domain alignment differences of the second knowledge aggregation branch The details are as follows: S161: Domain alignment differences in the first knowledge aggregation branch Domain alignment differences in the second knowledge aggregation branch The formula is as follows: ], In the above formula, This is the source domain fusion feature map for the first knowledge aggregation branch. This is the target domain fusion feature map for the first knowledge aggregation branch. The domain number is the source domain number. B is the target domain index, and B is the total number of samples. S162: ], In the above formula, This is the source domain fusion feature map for the second knowledge aggregation branch. For the second knowledge aggregation branch, the target domain fusion feature map is... The domain number is the source domain number. B is the target domain index, and B is the total number of samples.

[0045] S2: After the model is built, it needs to be trained. The image segmentation model based on dual-branch knowledge aggregation is trained end-to-end.

[0046] First, training data is collected, including N parameter sequences of renal MRI images. Each parameter sequence image is labeled with a kidney segmentation label. The original image of one sequence is selected as the target domain, and the original images and segmentation labels of the other N-1 sequences are used as the source domain. These are input into the model for one round of training. The source and target domain samples are changed for the next round of training, for a total of N rounds. In some embodiments, the dataset consists of four parameter sequences of renal MRI images: T1 fl2d (simply denoted as rz), T1 vibe cor (simply denoted as rt), T1 vibe tra (simply denoted as rh), and T2PCA (simply denoted as rb). Each of the four parameter sequence images comes from 100 patients, and the four parameter sequence images may come from different patients. Each parameter sequence image is labeled with a kidney segmentation label. During the training phase, three of the parameter sequence images and their kidney segmentation labels are used as the source domain, and the other parameter sequence image is used as the target domain without a kidney segmentation label.

[0047] Next, the loss function is constructed. The two knowledge aggregation branches are trained simultaneously, and the total loss function is:

[0048] In the above formula The loss function for the first knowledge aggregation branch. The loss function for the second knowledge aggregation branch. For the domain alignment difference of the first knowledge aggregation branch, For the domain alignment differences of the second knowledge aggregation branch, The source domain number, .

[0049] S21: The core of the first knowledge aggregation branch is the singular value knowledge aggregation module, and the core of training the singular value knowledge aggregation module lies in how to train the basis matrix. and fully connected layer Among them, the fully connected layer It is used to generate the fitting coefficient matrix. constraint coefficient matrix To achieve training of fully connected layers The purpose of training is to ensure that the reconstructed feature matrix retains the common features of each domain while eliminating the distinctive features belonging to each domain, minimizing the difference from the original feature matrix, and ultimately achieving the correct segmentation result.

[0050] To achieve the aforementioned training objectives, the concept of singular values ​​is introduced. Performing Singular Value Decomposition (SVD) on a matrix yields its singular values. These singular values ​​can be considered similar to "information weights" of a matrix; larger singular values ​​correspond to more critical information, while smaller ones are negligible. Therefore, they are frequently used in scenarios such as data dimensionality reduction and image compression. The singular value aggregation module performs SVD on the basis matrix... sum coefficient matrix Singular Value Decomposition (SVD) is performed, and constraints on the singular values ​​are introduced when constructing the loss function to influence the basis matrix. sum coefficient matrix Moving closer to the training goals.

[0051] To achieve the training objectives, the following aspects need to be considered: S211, Segmentation Loss Function of the First Knowledge Aggregation Branch as follows:

[0052] In the above formula, This is a segmentation map of the source domain NMR image output by the independent decoder D1. The corresponding source domain NMR image segmentation label is N-1, where N-1 is the number of source domains; S212, Singular Value Reconstruction Loss Function as follows: ; By reconstructing the discriminator The feature matrices before and after the reconstruction of the singular value module are identified, and the difference between the two should be minimized as much as possible; S213, Basis Matrix For full-rank and stable constraint functions as follows: , in For the basis matrix The constraint function must be full rank, obtained by applying the basis matrix. Perform singular value decomposition (SVD) to obtain the basis matrix. All singular values ​​are listed and sorted in descending order of value. Constrain smaller singular values ​​to be far from 0 to ensure the basis matrix The singular values ​​are all non-negative, indirectly constraining Full rank of a matrix: , ; Basis matrix The constraint function must be stable, through the basis matrix. The stability of a matrix is ​​calculated using its condition number. The smaller the condition number, the closer the matrix is ​​to stability. The condition number is calculated as the ratio of the largest singular value to the smallest singular value, i.e.: ; S214 constraint functions for low-rank matrices as follows: Will Divided into a small piece ,right Perform Singular Value Decomposition (SVD) to obtain all singular values ​​{ , ,constraint Minimize the sum of singular values ​​to guarantee that each small block It is a low-rank matrix. ; S215 Constraint functions for block diagonal matrices as follows: , ; in It is a diagonal mask. It is the value of the m-th element when that element is within the diagonal region of the block. The value is 0 otherwise the function. constraint The remaining elements in the set are close to 0.

[0053] In summary, the loss function of the first knowledge aggregation branch... for: , in It is a balance parameter. Let the segmentation loss function be the first knowledge aggregation branch. Reconstruct the loss function for the singular value module. Basis matrix For full-rank and stable constraint functions, for The constraint function for a low-rank matrix. for The constraint function for a block diagonal matrix.

[0054] S22. The core of the second knowledge aggregation branch is the graph comparison knowledge aggregation module, and the core of training the graph comparison knowledge aggregation module is training a graph convolutional network. The training goal is to enable the graph feature matrix after the graph convolutional network to learn the topological knowledge between local features, and at the same time achieve the correct segmentation result.

[0055] To achieve the above training objectives, the following aspects need to be considered: S221, Segmentation loss function of the second knowledge aggregation branch as follows:

[0056] In the above formula, This is a segmentation map of the source domain NMR image output by the independent decoder D2. The corresponding source domain NMR image segmentation label is N-1, where N-1 is the number of source domains; S222. Use two methods, random node loss and random edge loss, to generate a new view: According to a certain proportion Randomly discard some nodes to obtain a new graph. Specifically, as follows: Random sampling Line it and assign its value to 0, and get The set of sampled row numbers is represented as ,Will Chinese correspondence If all elements in the row and column are assigned the value 0, then... In some embodiments, Set to 0.2; Inputting the new view with the discarded nodes into the graph convolutional network G, we get: ; According to a certain proportion Randomly discard some edges to obtain a new graph. Specifically, as follows: Random sampling in the upper triangular matrix Each element is set to 0, because The symmetry also needs to be The corresponding symmetric elements in the lower triangular matrix are assigned the value 0, and the final result is... , Then it remains unchanged; in some embodiments, Set to 0.2; Inputting the new view with the discarded edges into the graph convolutional network G, we get: ; The first view of the first Sample As a query sample, the corresponding sample in the second view The remaining samples in the second view are positive samples, and the remaining samples in the third view are negative samples. Therefore, the fourth... Information-noise contrast estimation of InfoNCE loss for each sample:

[0057] In the above formula Let represent the temperature parameter, sim represent the similarity function, and B be the total number of samples; in some embodiments, ... Set to 0.05; The similarity function uses a Gaussian kernel function, and is calculated as follows:

[0058] In the above formula The standard deviation parameter is used to adjust the sensitivity to similarity errors; in some embodiments, It was set to 0.005; Accordingly, leveraging the symmetry of the two views, the views of the query samples are swapped to obtain the corresponding samples from the second view. This is the query sample, the first view. sample As positive samples, the remaining samples in the first view are negative samples, resulting in a symmetric loss:

[0059] Calculate the InfoNCE loss for all samples based on the information-noise contrast, and then take the average: ; In summary, the loss function of the second knowledge aggregation branch... as follows:

[0060] in It is a balance parameter. Let the segmentation loss function be the second knowledge aggregation branch. The InfoNCE loss function is used for information noise contrast estimation.

[0061] S3: The model building and training are now complete. To verify the reliability of the model, a comparative experiment will be conducted.

[0062] After the model is trained, a kidney MRI image of any parameter sequence is input into the trained model, and the model outputs a kidney segmentation prediction. The consistency between the model's output kidney segmentation prediction and the kidney segmentation label is evaluated using the Dice similarity coefficient (simply denoted as DSC) and the intersection-over-union ratio (simply denoted as IoU).

[0063] S31. To verify the superiority of the dual-branch model over the single-branch model, ablation experiments were conducted. These ablation experiments involved training only one branch of graph contrastive knowledge aggregation, training only one branch of singular value knowledge aggregation, and training with both branches. Four sequences of kidney MRI images with segmentation labels were collected to form the MRMP dataset. Four sets of experiments were conducted on each dataset, with each set verifying the model using rz, rt, rh, and rb as the target domains. The ablation experiment results on the dataset are shown in Tables 1 and 2. The DSC results show that the model performs the worst when only the graph contrastive knowledge aggregation method is used. In contrast, when only the singular value knowledge aggregation method is used, the average DSC and IoU of the model in the four domains are improved by 0.29% and 0.59%, respectively. This shows that the singular value knowledge aggregation method can not only achieve knowledge aggregation, but also effectively maintain the similarity of the same local features in each domain. The average DSC and IoU of the dual-branch model segmentation results are further improved by 1.81% and 1.88%, respectively. The ablation experiment fully demonstrates the advantages of the dual-branch model in multi-source domain MRI image segmentation.

[0064] Table 1. DSC (%) results of the dual-branch knowledge aggregation model in ablation experiments on the MRMP dataset.

[0065] Table 2. IoU (%) results of the dual-branch knowledge aggregation model in ablation experiments on the MRMP dataset.

[0066] S32. To verify the advantages of the multi-source domain kidney MRI image segmentation method based on dual-branch knowledge aggregation of the present invention compared with other methods, a comparative experiment was conducted.

[0067] Several state-of-the-art multi-source domain adaptation algorithms were selected and their performance compared on the MRMP dataset. These methods include DRT (Dynamic Transfer), DoCR (Domain-specific Convolution Reconstruction), StandardGAN (Standard Generative Adversarial Network), M3SDA (Moment Matching for Multi-Source Domain Adaptation), MCLS (Multi-source domain adaptation with Collaborative Learning for Semantic Segmentation), and M2CD (Multimodal Change Detection with Mixture of Experts and Self-Distillation).

[0068] In addition, two sets of reference experiments were set up, denoted as Supervised and w / o. Supervised represents the results of strongly supervised experiments, i.e., training the U-Net network directly using data and labels from the target domain, and then testing it on the target domain. w / o represents the results of training the U-Net network directly using data and labels from the source domain without domain adaptation training, and then testing it on the target domain.

[0069] See the comparison chart of segmentation results. Figure 3 , Figure 4 , Figure 5 and Figure 6 The experimental data are shown in Tables 3 and 4.

[0070] Table 3. Performance comparison of the dual-branch knowledge aggregation model and the comparative method on the MRMP dataset using DSC (%).

[0071] Table 4. Performance comparison of the dual-branch knowledge aggregation model with other comparative methods on the MRMP dataset (IoU %)

[0072] The above results demonstrate that the trained dual-branch knowledge aggregation image segmentation model has high reliability in segmenting multi-sequence kidney MRI images, and has significant advantages compared to other multi-source domain image segmentation algorithms.

[0073] S4: Finally, the trained model is used. The kidney MRI image to be segmented is input into the trained dual-branch knowledge aggregation image segmentation model to obtain the segmentation result. The kidney MRI image to be segmented is a sequence from the multi-sequence MRI images used by the training model.

[0074] Based on the above method, this invention also proposes an electronic device for multi-source domain renal MRI image segmentation, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the image segmentation method as described above.

[0075] The basic concepts have been described above. It is clear that the above disclosure is merely illustrative and does not constitute a limitation of the invention. Although not explicitly stated herein, various modifications, improvements, and corrections may be made to the invention by those skilled in the art. Such modifications, improvements, and corrections are suggested in this invention and therefore remain within the spirit and scope of the exemplary embodiments of the invention.

Claims

1. A multi-source domain renal MRI image segmentation method based on dual-branch knowledge aggregation, characterized in that, The method includes the following steps: A dual-branch knowledge aggregation image segmentation model is constructed, comprising: a first knowledge aggregation branch consisting of a shared encoder module, a singular value knowledge aggregation module, and an independent decoder D1; a second knowledge aggregation branch consisting of a graph comparison knowledge aggregation module and an independent decoder D2; and a domain alignment module. The dual-branch knowledge aggregation image segmentation model is trained using multiple sequences of kidney MRI images with segmentation labels. N parameter sequences of kidney MRI images are collected, each sequence containing M original images and corresponding segmentation labels. The original image of one sequence is selected as the target domain, and the original images and segmentation labels of the other N-1 sequences are selected as the source domain. The images are input into the dual-branch knowledge aggregation image segmentation model for one round of training. The source and target domain samples are changed for the next round of training. A total of N rounds of training are conducted, where N and M are both natural numbers greater than 1. Verify the reliability of the trained dual-branch knowledge aggregation image segmentation model; The kidney MRI image to be segmented is input into the trained dual-branch knowledge aggregation image segmentation model to obtain the segmentation result.

2. The image segmentation method according to claim 1, characterized in that, The singular value knowledge aggregation module in the dual-branch knowledge aggregation image segmentation model includes: a local feature mapping convolutional block. basis matrix Fully connected layer Feature recovery convolutional blocks and reconstructed feature discriminator ; The input to the singular value knowledge aggregation module is the original domain feature map output by the shared coding module. The output is a domain feature map with spatial semantic enhancement. ; The singular value knowledge aggregation module performs the following functions: S121: The domain feature map output from the shared coding module Classified by spatial location a small piece ; S122: Each small block Convolutional blocks using local feature mapping Mapping to feature space In this context, it becomes a d-dimensional local feature vector. ; S123: The data will come from N domains, each domain... All local feature vectors are concatenated, with the concatenation order being that local feature vectors from the same spatial location in different domains are grouped together to form a feature matrix. ; S124: Will Reconstructed The reconstructed feature matrix By integrating the local spatial features of each domain, the formula is expressed as follows: ， In the above formula As a feature space The basis matrix is ​​a globally learnable matrix. Let be the coefficient matrix, representing In the basis matrix The projection on the surface, through the fully connected layer To fit: ; S125: Reconstruct the feature matrix Split into n is the domain index, and j is the spatial location index; S126: Combining products from the same domain Arranged and pieced together according to spatial location ; S127: Recovering Convolutional Blocks from Features Will Restore to .

3. The image segmentation method according to claim 1, characterized in that, The graph contrastive knowledge aggregation module in the dual-branch knowledge aggregation image segmentation model includes an average pooling submodule, a graph convolutional network G, and an upsampling submodule. The graph comparison knowledge aggregation module receives its input from the original feature map from the shared encoding module. The output is a domain feature map that incorporates topological knowledge. ; The graph comparison knowledge aggregation module performs the following functions: S141: The domain feature map output from the shared coding module Classified by spatial location Small feature blocks ; S142: Take the average of each small feature block as the prototype vector of that local region. ; S143: Will As nodes, a graph feature matrix is ​​used. To represent a set of nodes, S144: Construct an adjacency matrix To express the relationship between nodes. Each element in the array represents the similarity between nodes using cosine similarity. S145: Will and Input a graph convolutional network G and perform data augmentation on the graph. ； S146: Through the upsampling submodule, Upsampling restored to , This represents the domain feature graph after incorporating topological knowledge.

4. The image segmentation method according to claim 1, characterized in that, The domain alignment module includes a gradient inversion layer (GRL) and N-1 discriminators. N-1 represents the number of source domains; the input to the domain alignment module comes from the original feature map output by the shared encoder module. The domain feature map with spatial semantic enhancement output by the singular value knowledge aggregation module The domain feature map output by the graph comparison knowledge aggregation module incorporates topological knowledge. The output of the domain alignment module is the domain alignment difference of the first knowledge aggregation branch. Domain alignment differences of the second knowledge aggregation branch .

5. The image segmentation method according to claims 1-4, characterized in that, The dual-branch knowledge aggregation model is trained using multiple sequences of kidney MRI images with segmentation labels, and the total loss function is: In the above formula The loss function for the first knowledge aggregation branch. The loss function for the second knowledge aggregation branch. For the domain alignment difference of the first knowledge aggregation branch, For the domain alignment differences of the second knowledge aggregation branch, The source domain number, .

6. The image segmentation method according to claim 5, characterized in that, Loss function of the first knowledge aggregation branch for: , in It is a balance parameter. Let the segmentation loss function be the first knowledge aggregation branch. Reconstruct the loss function for the singular value module. Basis matrix For full-rank and stable constraint functions, for The constraint function for a low-rank matrix. for The constraint function for a block diagonal matrix; The basis matrix For full-rank and stable constraint functions as follows: ， in For the basis matrix The constraint function must be full rank, obtained by applying the basis matrix. Perform singular value decomposition (SVD) to obtain the basis matrix. All singular values ​​are listed and sorted in descending order of value. Constrain smaller singular values ​​to be far from 0 to ensure the basis matrix The singular values ​​are all non-negative, indirectly constraining Full rank of a matrix: ， ， Basis matrix The constraint function must be stable, through the basis matrix. The stability of a matrix is ​​calculated using its condition number. The smaller the condition number, the more stable the matrix is. The condition number is calculated as the ratio of the largest singular value to the smallest singular value, as shown in the following formula: ； The constraint functions for low-rank matrices as follows: Will Divided into a small piece ,right Perform Singular Value Decomposition (SVD) to obtain all singular values. , ,constraint Minimize the sum of singular values ​​to guarantee that each small block It is a low-rank matrix. ； The Constraint functions for a block diagonal matrix as follows: ， ; in It is a diagonal mask. It is the value of the m-th element when that element is within the diagonal region of the block. The value is 0 otherwise the function. constraint The remaining elements in the set are close to 0.

7. The image segmentation method according to claim 5, characterized in that, Loss function of the second knowledge aggregation branch as follows: in It is a balance parameter. Let the segmentation loss function be the second knowledge aggregation branch. The InfoNCE loss function is used for information-noise contrast estimation. The information noise contrast estimation InfoNCE loss function Specifically as follows: Randomly discard some nodes to obtain the first view: ； Randomly discard some edges to obtain a second view: ； The first view of the first Sample As a query sample, the corresponding sample in the second view The remaining samples in the second view are positive samples, and the remaining samples in the third view are negative samples. Therefore, the fourth... Information-noise contrast estimation of InfoNCE loss for each sample: In the above formula Let represent the temperature parameter, sim represent the similarity function, and B be the total number of samples; Swap the views of the query samples to the second view. sample As a query sample, the corresponding sample of the first view As positive samples, the remaining samples in the first view are negative samples, resulting in a symmetric loss: ； Calculate the InfoNCE loss for all samples based on the information-noise contrast, and then take the average: 。 8. The image segmentation method according to claim 7, characterized in that, The random discarding of some nodes is as follows: Set ratio ,exist Random sampling Line it and assign its value to 0, and get The set of sampled row numbers is represented as ,Will Chinese correspondence If all elements in the row and column are assigned the value 0, then... .

9. The image segmentation method according to claim 7, characterized in that, The random discarding of certain edges is as follows: Set ratio ,exist Random sampling in the upper triangular matrix 1 element, and set their values ​​to 0, in In the lower triangular matrix, the corresponding symmetric elements are assigned the value 0, resulting in... , It remains unchanged.

10. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the image segmentation method according to any one of claims 1 to 9.

Images