Small sample lumbar vertebra segmentation method based on space prior information and prototype iteration updating
By using spatial prior information and iterative update methods under small sample data, the problem of low lumbar segmentation accuracy is solved, and the rapid and accurate lumbar marking effect is achieved.
Patent Information
- Application Number
- CN202411617676.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-13
- Publication Date
- 2025-03-11
AI Technical Summary
The prior art is difficult to effectively perform lumbar segmentation under small sample data, especially when there is interclass similarity in lumbar spine images, and there is a lack of methods to effectively utilize spatial position information, resulting in low labeling accuracy.
A small sample lumbar segmentation method based on spatial prior information and iterative update of prototypes is proposed. Through the small sample feature extraction module, spatial prior information extraction module, spatial fusion module and iterative update module, the clinical doctor's learning mechanism is simulated and the prototype information is gradually updated to improve segmentation accuracy.
This method can improve the accuracy of lumbar spine segmentation under small sample data, effectively utilize spatial position information, solve the problem of low labeling accuracy, and achieve fast, accurate and high-throughput lumbar spine labeling.
Smart Images
Figure CN119600283A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of lumbar vertebra segmentation, and more specifically, to a small sample lumbar vertebra segmentation method based on spatial prior information and prototype iterative updating. Background Art
[0002] Vertebral multi-class segmentation based on spinal images. It plays an important role in the orthopedic diagnosis and treatment process, including the evaluation, diagnosis, surgical planning and image guidance of various spinal diseases. Spine segmentation from magnetic resonance (MR) images has a great impact on the diagnosis and treatment of various spinal diseases, which relies heavily on instance segmentation. In order to reduce the workload of doctors performing these manual annotations of lumbar vertebrae, automatic segmentation using deep learning has great prospects. Unfortunately, existing segmentation frameworks rely on supervised training and a large amount of densely labeled data, which often does not achieve good results when the amount of data is small.
[0003] Compared with natural images, lumbar images have their own characteristics due to the existence of inter-class similarity, that is, the shape and appearance similarity in adjacent spinal structures, which leads to the fact that their spatial location information is a factor to be considered. The above problems may lead to changes in the spatial distribution between the query image and the support set, and current technologies, such as prototype networks, do not explicitly take into account this spatial location information. In clinical practice, experienced clinicians can borrow information from query images, such as CT or MRI scans of patients, to fine-tune or calibrate his prior knowledge to determine the spatial location information of the target tissue. Inspired by the above observations, we propose a small sample lumbar segmentation network VerFSS-Net (Vertebra Few-shot Segmentation Network) whose principle is to imitate the learning mechanism of expert clinicians to gradually update their prototype information. Summary of the invention
[0004] In view of the above problems existing in the prior art, the present invention provides a small sample lumbar vertebra segmentation method based on spatial prior information and prototype iterative update, which can improve the accuracy of lumbar vertebra segmentation.
[0005] As a first aspect of the present invention, a small sample lumbar vertebra segmentation method based on spatial prior information and prototype iterative update is provided, and the small sample lumbar vertebra segmentation method based on spatial prior information and prototype iterative update comprises the following steps:
[0006] Step S1: acquiring a lumbar vertebra MR image sample set including a plurality of sample lumbar vertebra MR images;
[0007] Step S2: constructing a small sample lumbar vertebrae annotation model, and using the lumbar vertebrae MR image sample set to train the small sample lumbar vertebrae annotation model to obtain a trained small sample lumbar vertebrae annotation model;
[0008] Step S3: inputting the lumbar vertebra MR image to be labeled into the trained small sample lumbar vertebra labeling model for labeling, so as to obtain a lumbar vertebra labeling prediction result of the lumbar vertebra MR image to be labeled.
[0009] Furthermore, the step of acquiring a lumbar vertebra MR image sample set including a plurality of sample lumbar vertebra MR images further includes:
[0010] Dividing the lumbar MR image sample set into a training sample set, a verification sample set and a test sample set;
[0011] A training support set and a training query set are constructed according to the training sample set, a verification support set and a verification query set are constructed according to the verification sample set, and a test support set and a test query set are constructed according to the test sample set; wherein, adjacent slices of each sample lumbar vertebra MR image in the training sample set are respectively used as the training support set and the training query set, all slices of a sample lumbar vertebra MR image in the verification sample set are used as the verification query set, and the middle slice of another sample lumbar vertebra MR image in the verification sample set is used as the verification support set; all slices of a sample lumbar vertebra MR image in the test sample set are used as the test query set, and the middle slice of another sample lumbar vertebra MR image in the test sample set is used as the test support set;
[0012] Among them, the slices in the training support set, the slices in the verification support set and the slices in the test support set all have vertebrae labels, and the vertebrae labels are obtained by annotating the vertebrae regions of the slices in the training support set, the vertebrae regions of the slices in the verification support set and the vertebrae regions of the slices in the test support set using ITK-SNAP.
[0013] Furthermore, the method of training the small sample lumbar vertebrae labeling model using the lumbar vertebrae MR image sample set to obtain the trained small sample lumbar vertebrae labeling model further includes:
[0014] Using the training sample set to train the small sample lumbar vertebrae labeling model, and during the training process, using the verification sample set to verify the small sample lumbar vertebrae labeling model;
[0015] After the small sample lumbar vertebrae labeling model is trained, the test sample set is used to test and evaluate the performance of the trained small sample lumbar vertebrae labeling model.
[0016] Furthermore, the training of the small sample lumbar vertebrae labeling model using the training sample set further includes:
[0017] The small sample lumbar vertebrae annotation model includes a small sample feature extraction module, a spatial prior information extraction module, a spatial fusion module and an iterative update module;
[0018] Inputting the training support set and the training query set into the small sample feature extraction module to obtain the vertebral features F of the slices in the training query set;
[0019] The training support set and the training query set are input into the spatial prior information extraction module to obtain the spatial prior information of the slices in the training query set.
[0020] The vertebral features F and their spatial prior information in the slices of the training query set are Input into the spatial fusion module for attention mechanism fusion to obtain preliminary prediction results of vertebral annotation of slices in the training query set;
[0021] Inputting preliminary prediction results of vertebral annotation of slices in the training query set into the iterative update module to output final prediction results of vertebral annotation of slices in the training query set;
[0022] The final prediction results of the vertebrae annotations of the slices in the training query set are spliced together to obtain the complete lumbar vertebrae annotation prediction results of the slices in the training query set.
[0023] Furthermore, the step of inputting the training support set and the training query set into the small sample feature extraction module to obtain the vertebral features F of the slices in the training query set further includes:
[0024] The small sample feature extraction module is a small sample shared feature encoder, which includes three submodules, each of which includes a 3×3 convolution layer, a normalization layer, and an activation layer. The training support set and the training query set are merged and then sequentially passed through the three submodules to extract the associated features of the training support set and the training query set. The associated features are then subjected to channel dimension reduction through a 1×1 convolution, and finally decomposed to obtain the feature F of the training query set. q and the feature F of the training support set s ;
[0025] According to the feature F of the training support set s and the vertebral labels m of the training support set s The prototype feature P of the training support set is calculated, wherein the calculation formula of the prototype feature P of the training support set is as follows:
[0026]
[0027] In the above formula, s represents the training support set, F s represents the features of the training support set, m s represents the vertebral labels of the training support set, ⊙ represents the Hadamard product used for masked average pooling, and P represents the calculated prototype features of the training support set;
[0028] Calculate the feature F of the training query set q The cosine similarity between the prototype feature P of the training support set and the inverse of it, and then the sigmoid activation function is used to generate the foreground probability map of the training query set The calculation formula is as follows:
[0029]
[0030]
[0031] Among them, (h, w) represents each pixel in the sample lumbar MR image, q represents the training query set, σ is the sigmoid activation function, T is the segmentation threshold, and α is the scaling factor.
[0032] Furthermore, the training support set and the training query set are input into the spatial prior information extraction module to obtain the spatial prior information of the slices in the training query set. Also included:
[0033] The spatial prior information extraction module includes 4 downsampling modules and 4 upsampling modules, each of which includes 2D convolution and ReLU activation function, and each of which includes 1*1 convolution, upsampling, 2D convolution and ReLu activation function; the training support set and the training query set are merged and then sequentially passed through 4 downsampling modules and 4 upsampling modules, and the vertebral label m of the training support set is obtained. s Perform spatial transformation to obtain spatial prior information of the training query set
[0034] Furthermore, the vertebral features F and their spatial prior information of the slices in the training query set are The input is input into the spatial fusion module for attention mechanism fusion to obtain the preliminary prediction result of vertebral annotation of the slice in the training query set, and further includes:
[0035] The spatial prior information of the training query set is transformed along the channel dimension and the foreground probability map of the training query set Then use another linear transformation to obtain the actual content information value, and finally get the fused decoding feature F dec , where the decoded feature F dec The initial prediction result of vertebral annotation as a slice in the training query set
[0036]
[0037]
[0038]
[0039] Where Att ij represents the attention information of the training query set features, v ij Represents the weight matrix W v The actual content information value calculated, i, j represents each pixel on the sample lumbar MR image, and softmax represents the normalization function. represents the query feature of the training query set, represents the input element currently being processed, T represents the matrix transpose, q represents the training query set, k ij Represents the key feature of the training query set, used with Compare to determine the correlation, xy represents the length and width of the corresponding feature, ij represents the length and width of the corresponding feature.
[0040] Furthermore, the step of inputting the preliminary prediction results of the vertebrae annotation of the slices in the training query set into the iterative update module to output the final prediction results of the vertebrae annotation of the slices in the training query set further includes:
[0041] Preliminary prediction results of vertebral annotations for slices in the training query set The calculation formula is as follows:
[0042]
[0043] Among them, i,j represents each pixel point on the sample lumbar MR image, Represents the background features of each pixel on the sample lumbar MR image, Represents the foreground features of each pixel on the sample lumbar MR image;
[0044] Then the features of the training query set after iteration are The calculation formula is as follows:
[0045]
[0046] Among them, F q(i,j) represents the features of the initial training query set, P n Represents the prototype features after n iterations;
[0047] Then the prototype feature P after iteration n+1 times n+1 The calculation formula is as follows:
[0048]
[0049] in, Represents the features of the training query set after the iteration With the features F of the initial training query set q The cross entropy loss between them, N is the number of iterations of model loss back propagation, v is the parameter in the model training process, Is the differential symbol.
[0050] Furthermore, during the training process of the small sample lumbar vertebrae labeling model, the calculation formula of the training loss L of the small sample lumbar vertebrae labeling model is:
[0051] L=L seg +L reg
[0052] Among them, L seg is the main loss of the small sample lumbar vertebrae annotation model, L reg Auxiliary loss for the small sample lumbar vertebra annotation model;
[0053] The main loss L of the small sample lumbar vertebrae annotation model seg The calculation formula is:
[0054]
[0055] The auxiliary loss L of the small sample lumbar vertebrae annotation model reg The calculation formula is:
[0056]
[0057] Among them, H and W represent the length and width of the sample lumbar MR image, q represents the training query set, s represents the training support set, and m q represents the vertebrae labels of the training query set, Represents the vertebral annotation prediction results of the training query set, m s represents the vertebrae labels of the training support set, represents the vertebral annotation prediction results of the training support set, and (h,w) represents each pixel in the sample lumbar MR image.
[0058] As a second aspect of the present invention, a small sample lumbar vertebra segmentation system based on spatial prior information and prototype iterative update is provided, and the small sample lumbar vertebra segmentation system based on spatial prior information and prototype iterative update comprises:
[0059] An acquisition module, used for acquiring a lumbar vertebra MR image sample set including a plurality of sample lumbar vertebra MR images;
[0060] A training module, used for constructing a small sample lumbar vertebrae annotation model, and training the small sample lumbar vertebrae annotation model using the lumbar vertebrae MR image sample set to obtain a trained small sample lumbar vertebrae annotation model;
[0061] The labeling module is used to input the lumbar vertebra MR image to be labeled into the trained small sample lumbar vertebra labeling model for labeling, so as to obtain the lumbar vertebra labeling prediction result of the lumbar vertebra MR image to be labeled.
[0062] A small sample lumbar segmentation method based on spatial prior information and prototype iterative update provided by the present invention has the following beneficial effects: lumbar MR images can be batch processed and accurately annotated in the original image, and lumbar images can be efficiently annotated; the present invention utilizes small sample feature extraction, prior information extraction, prototype iterative update and vertebral fusion to effectively mine local and global prior information in lumbar images and solve the problem of low annotation accuracy; the present invention utilizes the prototype iteration mechanism to fully learn the collaborative interaction relationship between multiple tasks, further improve the robustness and generalization ability of the small sample lumbar annotation model, and can achieve fast, accurate and high-throughput lumbar annotation. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation of the present invention.
[0064] Figure 1 This is a comparison chart between labeling using a traditional 3D segmentation network model and labeling using the small sample lumbar vertebra labeling model of the present invention.
[0065] Figure 2 This is a structural diagram of the small sample lumbar vertebrae annotation model provided by the present invention.
[0066] Figure 3 This is a structural diagram of the small sample feature extraction module provided by the present invention.
[0067] Figure 4 This is a structural diagram of the spatial prior information extraction module provided by the present invention.
[0068] Figure 5 This is a structural diagram of the iterative update module provided by the present invention.
[0069] Figure 6 A schematic diagram of the division of the training sample set, verification sample set and test sample set provided by the present invention.
[0070] Figure 7-Figure 8 This is a comparison chart of the lumbar vertebrae annotation results of the small sample lumbar vertebrae annotation model of the present invention and the lumbar vertebrae annotation results of other 3D segmentation network models.
[0071] Fig. 9 This is a flow chart of the small sample lumbar vertebra segmentation method based on spatial prior information and prototype iterative update provided by the present invention.
[0072] Fig.10 Schematic diagram of each vertebra in a sample lumbar MR image provided by the present invention. DETAILED DESCRIPTION
[0073] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the specific implementation method, structure, characteristics and effects of a small sample lumbar segmentation method based on spatial prior information and prototype iterative update proposed by the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.
[0074] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the present invention described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0075] In this embodiment, a small sample lumbar segmentation method based on spatial prior information and prototype iterative update is provided. Fig. 9 As shown, the small sample lumbar vertebra segmentation method based on spatial prior information and prototype iterative update includes the following steps:
[0076] Step S1: acquiring a lumbar vertebra MR image sample set including a plurality of sample lumbar vertebra MR images;
[0077] Preferably, if Figure 6As shown, the acquiring of a lumbar vertebra MR image sample set including a plurality of sample lumbar vertebra MR images further includes:
[0078] Dividing the lumbar MR image sample set into a training sample set, a verification sample set and a test sample set;
[0079] A training support set and a training query set are constructed according to the training sample set, a verification support set and a verification query set are constructed according to the verification sample set, and a test support set and a test query set are constructed according to the test sample set; wherein, adjacent slices of each sample lumbar vertebra MR image in the training sample set are respectively used as the training support set and the training query set, all slices of a sample lumbar vertebra MR image in the verification sample set are used as the verification query set, and the middle slice of another sample lumbar vertebra MR image in the verification sample set is used as the verification support set; all slices of a sample lumbar vertebra MR image in the test sample set are used as the test query set, and the middle slice of another sample lumbar vertebra MR image in the test sample set is used as the test support set;
[0080] Among them, the slices in the training support set, the slices in the verification support set and the slices in the test support set all have vertebrae labels, and the vertebrae labels are obtained by annotating the vertebrae regions of the slices in the training support set, the vertebrae regions of the slices in the verification support set and the vertebrae regions of the slices in the test support set using ITK-SNAP.
[0081] In the embodiment of the present invention, Figure 6 As shown in the figure, since the research is conducted in the field of small samples, only 10 sample lumbar MR images are selected as the training sample set, 8 sample lumbar MR images are selected as the verification query set in the verification stage, 4 sample lumbar MR images are selected as the verification support set in the verification stage, 21 sample lumbar MR images are selected as the test query set in the test stage, and 6 sample lumbar MR images are selected as the test support set in the test stage. Since the small sample is trained using a task-based method, it needs to be divided into a support set and a query set. In the training stage, the adjacent slices of a sample lumbar MR image are used as the training support set and the training query set respectively; in the verification and testing stages, all slices of a lumbar MR image are used as the query set, and the middle slice of another lumbar MR image is used as the support set; the slices in the query set are traversed from the sample lumbar MR image in turn, and multiple model annotation results of the slices are obtained, and then the multiple model annotation results of the slices are spliced.
[0082] It should be noted that the lumbar MRI image is a 3D image, and a slice is a 2D plane structure in the 3D image. It can be understood that a 3D image is composed of several 2D slices. For example, the size of a lumbar MR image is 18*128*256, which can be understood as 18 pictures of 128*256 size, and the number of its slices is 18.
[0083] It should be noted that ITK-SNAP is used to annotate the vertebrae region that supports centralized slicing. The pixel values of all L5 vertebrae in the slice region of interest are set to 1, the pixel values of all L4 vertebrae in the slice region of interest are set to 2, the pixel values of all L3 vertebrae in the slice region of interest are set to 3, the pixel values of all L2 vertebrae in the slice region of interest are set to 4, the pixel values of all L1 vertebrae in the slice region of interest are set to 5, and all pixel values outside the slice region of interest are set to 0. Fig.10 As shown, the lumbar spine MR image includes L1 vertebra, L2 vertebra, L3 vertebra, L4 vertebra, and L5 vertebra from top to bottom.
[0084] like Figure 1 As shown, Figure 1 A is a traditional 3D segmentation network model. The traditional 3D segmentation network requires a lot of computing power. Figure 1 In A, the input is a lumbar MRI image, and after passing through the traditional 3D segmentation network model, the automatic labeling result of the lumbar vertebra is obtained. This result is obtained in one go, but its accuracy is low. Figure 1 B is the small sample lumbar vertebra annotation model of the present invention. The present invention uses the idea of small sample learning to randomly select adjacent slices as support sets and query sets in each epoch of the training phase. Figure 1 In B, the input is a slice in the query set. In addition, slices in the support set are input in sequence from L1, L2, L3, L4, and L5 to assist in training the small sample lumbar vertebrae labeling model. If the auxiliary training vertebrae label is L1, the small sample lumbar vertebrae labeling model outputs the automatic labeling result of L1. If the auxiliary training vertebrae label is L2, the small sample lumbar vertebrae labeling model outputs the automatic labeling result of L2, and so on... Finally, these automatic labeling results of the vertebrae in the query set slices are spliced together to obtain the complete vertebrae labeling results of the query set slices.
[0085] Step S2: constructing a small sample lumbar vertebrae annotation model, and using the lumbar vertebrae MR image sample set to train the small sample lumbar vertebrae annotation model to obtain a trained small sample lumbar vertebrae annotation model;
[0086] Preferably, the step of training the small sample lumbar vertebrae labeling model using the lumbar vertebrae MR image sample set to obtain the trained small sample lumbar vertebrae labeling model further includes:
[0087] Using the training sample set to train the small sample lumbar vertebrae labeling model, and during the training process, using the verification sample set to verify the small sample lumbar vertebrae labeling model;
[0088] After the small sample lumbar vertebrae labeling model is trained, the test sample set is used to test and evaluate the performance of the trained small sample lumbar vertebrae labeling model.
[0089] Specifically, Figure 2 As shown, the training of the small sample lumbar vertebrae labeling model using the training sample set also includes:
[0090] The small sample lumbar vertebrae annotation model includes a small sample feature extraction module, a spatial prior information extraction module, a spatial fusion module and an iterative update module; wherein, Figure 2 The left part of shows that in the training process, adjacent slices i and j are taken from a lumbar MRI image as the training support set and the training query set, respectively, where the training support set has vertebrae labels. In order to ensure the weight sharing of feature extraction, it is necessary to perform feature concatenation on the training support set and the training query set.
[0091] Inputting the training support set and the training query set into the small sample feature extraction module to obtain the vertebral features F of the slices in the training query set;
[0092] Furthermore, if Figure 3 As shown, the inputting of the training support set and the training query set into the small sample feature extraction module to obtain the vertebral features F of the slices in the training query set also includes:
[0093] The small sample feature extraction module is mainly a small sample shared feature encoder, which includes three submodules, each of which includes a 3×3 convolution layer, a normalization layer and an activation layer. The number of layers processed is 3, 4, and 23 respectively. The training support set and the training query set are merged and then passed through the three submodules in sequence to extract the associated features of the training support set and the training query set. The number of channels finally obtained is 1024. The associated features are then subjected to channel dimension reduction through 1×1 convolution, and finally decomposed to obtain the feature F of the training query set. q and the feature F of the training support set s ;
[0094] Specifically, Figure 3As shown in Figure 1, in order to enable the neural network to learn the association information between the training support set and the training query set, the training support set slices and the training query set slices are merged. In this chapter, the number of slices after merging is recorded as K+1, where 1 represents the number of slices of the training query set and K represents the number of slices of the training support set. Then, the shared feature extractor f is used to extract the training support set slices. θ Extract features from the merged slices, which ensures that the weights of the training support set and the training query set are shared. ResNet-101 with pre-trained weights is used as the backbone network for feature extraction. It consists of a 7×7 convolution layer with a step size of 2, a maximum pooling layer, and three residual blocks with multiple sub-block modules. After the first maximum pooling, the output feature of the first residual block in the network is 4 times smaller than the input resolution, and its number of channels becomes 64. Then the feature map passes through three residual network extraction modules, which include 3×3 convolution, normalization, activation and other operations. The number of layers processed is 3, 4, and 23 respectively, and the final number of channels is 1024. Then 1×1 convolution is performed for channel dimension reduction, and finally decomposed to obtain the feature F of the training support set. s and the features F of the training query set q .
[0095] In the embodiment of the present invention, Figure 3 As shown, (K+1)*1*256*128, (K+1) 1 represents the number of query set slices, and K represents the number of support set slices. In 1*256*128, 1 represents the number of channels, 256 represents the length, and 128 represents the width. After convolution, normalization, activation, and maximum pooling, the 1*256*128 tensor becomes 64*64*32. Then, after 3 convolutions, normalization, and activation, 64*64*32 becomes 256*64*32. After 4 convolutions, normalization, and activation, 256*64*32 becomes 512*64*32. After 23 convolutions, normalization, and activation, 512*64*32 becomes 1024*64*32. Because there are (K+1) slices, the resulting tensor size is (K+1)*1024*64*32. After that, after 1*1 channel dimensionality reduction, the features of the support set are obtained: K*512*32*16 and the features of the query set are obtained: 1*512*32*16.
[0096] The prototype feature P prediction is performed through non-parametric metric learning. Among them, the feature F of the training support set is pooled by global average pooling. s and the vertebral labels m of the training support set s The prototype feature P of the training support set is calculated, wherein the calculation formula of the prototype feature P of the training support set is as follows:
[0097]
[0098] In the above formula, s represents the training support set, F s represents the features of the training support set, m s represents the vertebral labels of the training support set, ⊙ represents the Hadamard product used for masked average pooling, and P represents the calculated prototype features of the training support set;
[0099] The feature F of the training support set s The size of is adjusted to the label size (H, W). In order to perform threshold processing on the predicted result, the present invention calculates the feature F of the training query set. q The cosine similarity between the prototype feature P of the training support set and the inverse of it, and then the sigmoid activation function is used to generate the foreground probability map of the training query set The calculation formula is as follows:
[0100]
[0101]
[0102] Among them, (h, w) represents each pixel in the sample lumbar MR image, q represents the training query set, σ is the sigmoid activation function, T is the segmentation threshold, which is set to 12, and α is the scaling factor, which is set to 20.
[0103] The training support set and the training query set are input into the spatial prior information extraction module to obtain the spatial prior information of the slices in the training query set.
[0104] Furthermore, if Figure 4 As shown, the training support set and the training query set are input into the spatial prior information extraction module to obtain the spatial prior information of the slices in the training query set. Also included:
[0105] The spatial prior information extraction module includes 4 downsampling modules and 4 upsampling modules, each of which includes 2D convolution and ReLU activation function, and each of which includes 1*1 convolution, upsampling, 2D convolution and ReLu activation function; the training support set and the training query set are merged and then sequentially passed through 4 downsampling modules and 4 upsampling modules, and the vertebral label m of the training support set is obtained. s Perform spatial transformation to obtain spatial prior information of the training query set in,
[0106] Specifically, the spatial prior information extraction module aims to provide rough spatial location information to the backbone network, using a network similar to 2DUnet. sp (ω) is used to implement it, and the maximum number of feature channels is set to 32. The training support set and the training query set are concatenated to serve as the multi-channel input of this module. After operations such as convolution activation, the highest-level features are obtained. In the decoding process, a skip connection is used to obtain preliminary position information. Since the query label and the support label contain the same semantic target, the support label after spatial transformation is more consistent with the query label. Through the label M of the support set s The spatial prior information S of the target can be obtained prior .
[0107] In the embodiment of the present invention, Figure 4 As shown in the figure, the input of the prior information extraction module is 2*256*128. Among them, 2 is composed of a query set slice and a support set slice, 256 represents the length, and 128 represents the width. Then it enters the encoding stage. After 2D convolution and ReLU activation function, it becomes 16*128*64, and after 2D convolution and ReLU activation function, it becomes 32*64*32. After 2D convolution and ReLU activation function, it becomes 32*32*16. After 2D convolution and ReLU activation function, it becomes 32*16*8. After 2D convolution and ReLU activation function, it becomes 32*16*8. Then it enters the decoding stage; after skip connection, 1*1 convolution, upsampling, 2D convolution, ReLu activation function, it becomes 32*32*16; after skip connection, 1*1 convolution, upsampling, 2D convolution, ReLu activation function, it becomes 32*64*32; after skip connection, 1*1 convolution, upsampling, 2D convolution, ReLu activation function, it becomes 16*128*64, and then it is combined with the vertebral label m of the training support set s Perform spatial transformation to obtain 1*256*128 spatial prior information.
[0108] The vertebral features F and their spatial prior information in the slices of the training query set are Input into the spatial fusion module for attention mechanism fusion to obtain preliminary prediction results of vertebral annotation of slices in the training query set;
[0109] Furthermore, the vertebral features F and their spatial prior information of the slices in the training query set are The input is input into the spatial fusion module for attention mechanism fusion to obtain the preliminary prediction result of vertebral annotation of the slice in the training query set, and further includes:
[0110] The spatial prior information of the training query set is transformed along the channel dimension and the foreground probability map of the training query set Then use another linear transformation to obtain the actual content information value, and finally get the fused decoding feature F dec , where the decoded feature F dec The initial prediction result of vertebral annotation as a slice in the training query set
[0111]
[0112]
[0113]
[0114] Where Att ij represents the attention information of the training query set features, v ij Represents the weight matrix W v The actual content information value calculated, i, j represents each pixel on the sample lumbar MR image, and softmax represents the normalization function. represents the query feature of the training query set, represents the input element currently being processed, T represents the matrix transpose, q represents the training query set, k ij Represents the key feature of the training query set, used with Compare to determine the correlation, xy represents the length and width of the corresponding feature, ij represents the length and width of the corresponding feature.
[0115] In this embodiment of the present invention, for a given feature F of a training query set q First, the query feature q of the training query set is generated by linear transformation xy and the key feature k of the training query set ij , where q xy =W q F q , k ij =W k F q In this way, we can q To calculate the attention information Att ij Among them, W q Refers to the weight matrix of the query features of the training query set in the network, W k Refers to the weight matrix of the key features of the training query set in the network, q xy Refers to the query features of the training query set after network training, and xy refers to the length and width of the feature. k ijIt refers to the result of the key feature of the training query set after network training, and ij refers to the length and width of the feature.
[0116] It should be noted that value refers to the actual content information, which provides the final output data when calculating attention. In the attention mechanism, the part corresponding to value is the object that the model weights according to the calculated attention weight. The calculated attention weight determines the contribution of each value to the final output.
[0117] Inputting preliminary prediction results of vertebral annotation of slices in the training query set into the iterative update module to output final prediction results of vertebral annotation of slices in the training query set;
[0118] Furthermore, if Figure 5 As shown, the step of inputting the preliminary prediction results of the vertebral annotation of the slices in the training query set into the iterative update module to output the final prediction results of the vertebral annotation of the slices in the training query set also includes:
[0119] Preliminary prediction results of vertebral annotations for slices in the training query set The calculation formula is as follows:
[0120]
[0121] Among them, i,j represents each pixel point on the sample lumbar MR image, Represents the background features of each pixel on the sample lumbar MR image, Represents the foreground features of each pixel on the sample lumbar MR image;
[0122] Then the features of the training query set after iteration are The calculation formula is as follows:
[0123]
[0124] Among them, F q (i,j) represents the features of the initial training query set, P n Represents the prototype features after n iterations;
[0125] Then the prototype feature P after iteration n+1 times n+1 The calculation formula is as follows:
[0126]
[0127] in, Represents the features of the training query set after the iteration With the features F of the initial training query set qThe cross entropy loss between them, N is the number of iterations of model loss back propagation, v is the parameter in the model training process, Is the differential symbol.
[0128] Specifically, Figure 5 As shown in Figure 1, multiple iterations are required in the iterative update module. In each iteration, Q represents the features of the training query set, S represents the features of the training support set, and the features S of the training support set are used to obtain the prototype features P of the training support set through global average pooling (MAP). Then the cosine similarity between the prototype features P and the features Q of the training query set is calculated to obtain the foreground probability map of the training query set. The spatial prior information of the training query set And the foreground probability map of the training query set After fusion, the vertebral annotation prediction results of the training query set are obtained The vertebrae annotation prediction results of the training query set After feature extraction and 1*1 convolution with the feature Q of the training query set, the updated training query set feature Q' is obtained. Then, Q and Q' are cross-entropy loss calculated, and the prototype feature P of the training support set is updated through this cross-entropy loss to obtain the vertebrae annotation prediction result. Converge to the optimal solution; after the iterative process is completed, the updated prototype feature P is used to obtain the final prediction result of the vertebral annotation of the training query set.
[0129] It should be noted that the iterative update module follows the QIL strategy and borrows information from the features of the training query set to update the prototype feature P during the training phase. The iterative update module gradually updates the prototype feature P during the iteration process so that the vertebral annotation prediction results of the training query set are Converge to the optimal solution.
[0130] The final prediction results of the vertebrae annotations of the slices in the training query set are spliced together to obtain the complete lumbar vertebrae annotation prediction results of the slices in the training query set.
[0131] Preferably, during the training of the small sample lumbar vertebrae labeling model, the calculation formula of the training loss L of the small sample lumbar vertebrae labeling model is:
[0132] L=L seg +L reg
[0133] Among them, L seg is the main loss of the small sample lumbar vertebrae annotation model, L reg Auxiliary loss for the small sample lumbar vertebra annotation model;
[0134] The main loss L of the small sample lumbar vertebrae annotation modelseg The calculation formula is:
[0135]
[0136] The auxiliary loss L of the small sample lumbar vertebrae annotation model reg The calculation formula is:
[0137]
[0138] Among them, H and W represent the length and width of the sample lumbar MR image, q represents the training query set, s represents the training support set, and m q represents the vertebrae labels of the training query set, Represents the vertebral annotation prediction results of the training query set, m s represents the vertebrae labels of the training support set, represents the vertebral annotation prediction results of the training support set, and (h,w) represents each pixel in the sample lumbar MR image.
[0139] Step S3: inputting the lumbar vertebra MR image to be labeled into the trained small sample lumbar vertebra labeling model for labeling, so as to obtain a lumbar vertebra labeling prediction result of the lumbar vertebra MR image to be labeled.
[0140] As another embodiment of the present invention, a small sample lumbar vertebra segmentation system based on spatial prior information and prototype iterative update is provided, and the small sample lumbar vertebra segmentation system based on spatial prior information and prototype iterative update comprises:
[0141] An acquisition module, used for acquiring a lumbar vertebra MR image sample set including a plurality of sample lumbar vertebra MR images;
[0142] A training module, used for constructing a small sample lumbar vertebrae annotation model, and training the small sample lumbar vertebrae annotation model using the lumbar vertebrae MR image sample set to obtain a trained small sample lumbar vertebrae annotation model;
[0143] The labeling module is used to input the lumbar vertebra MR image to be labeled into the trained small sample lumbar vertebra labeling model for labeling, so as to obtain the lumbar vertebra labeling prediction result of the lumbar vertebra MR image to be labeled.
[0144] In an embodiment of the present invention, the small sample lumbar vertebrae annotation model includes a small sample feature extraction module, a spatial prior information extraction module, a spatial fusion module and an iterative update module; wherein the small sample feature extraction module includes a residual network encoder and a channel dimension reduction encoder, the residual network encoder is used to extract the associated features of the support set and the query set images, and the channel dimension reduction encoder is used to split and reduce the channel dimension of the extracted associated features. The spatial prior information extraction module is used to mine the global prior dependency between lumbar vertebrae images, so as to obtain the global features of the lumbar vertebrae images; the spatial fusion module is used to fuse the downsampled features of the vertebrae and the spatial prior information of the vertebrae, and the decoding module is used to convert the fused features into the final output of the network, that is, the result of automatic annotation of the lumbar vertebrae.
[0145] In an embodiment of the present invention, during the training process of the small sample lumbar vertebra annotation model, the training support set and the training query set are fused into the small sample feature extraction module. During the feature extraction process, the model parameters of the training support set and the training query set are shared, and the prototype features are obtained through the training support set to match the similarity with the training query set. The training support set and the training query set are spatially aligned through a 2D spatial decoding network to obtain the spatial prior information of the lumbar vertebra and fuse the prior information with the decoding features. In the test phase, the prediction results and the training query set features are used to calculate the loss, the prototype features are repeatedly updated, and the updated prototype features are used for final annotation.
[0146] It should be noted that Python is used to extract lumbar MR image information and label image information to obtain a two-dimensional or three-dimensional lumbar image, and the slices are generated into a two-dimensional matrix or a three-dimensional matrix; the pytorch framework is used to predict and output it.
[0147] In order to verify and illustrate the technical effect of the present method, this embodiment selects a traditional technical solution labeling method and a manual labeling method to compare with the method of the present invention to verify the real effect of the present method.
[0148] The qualitative analysis results of the comparative experiment are as follows Figure 7 As shown, Figure 7 The first column is the support set label, the second and third columns are the slices and labels to be annotated, and the following are the comparison results. In contrast, the L5 and L4 vertebrae are more completely annotated, and there is no lumbar dislocation. In L1, L2, and L3, incorrect lumbar vertebrae are detected. This is due to the similar shapes of vertebrae in the lumbar structure. VerFSS-Net can avoid this situation. In contrast, PANet, ADNet, and Q-Net do not add the position information of the vertebrae, so they detect two consecutive vertebrae with only one label as the support set.
[0149] Qualitative analysis of 3D network comparison experiments Figure 8 As shown in the figure, it is not difficult to find in the experiment that the labeling effect of L5 and L4 vertebrae is better, and there are fewer errors. However, in the 3D segmentation network, there are still prediction errors or even unpredictable situations. In particular, there are many errors in the L1 vertebra area. Most networks will label the T12 vertebra as the L1 vertebra, which will result in the continuous labeling of two adjacent vertebrae. This is due to the positional relationship of the L1 vertebra. In contrast, this algorithm can avoid the occurrence of prediction misalignment to the greatest extent.
[0150] The proposed VerFSS-Net is compared with some original learning methods, including PANet, ADNet, and Q-Net. Therefore, the annotation results of VerFSS-Net are still compared with the annotation results of other 3D segmentation networks, including 3D Unet, 3D ResUnet, 3D DeepLabv3+, and GCSN.
[0151] Table 1 Comparative experiments based on DSC indicators, unit % (for small sample networks, bold indicates the best results)
[0152]
[0153] Table 2 Comparative experiments based on DSC indicators, unit % (for 3D segmentation network, bold indicates the best results)
[0154]
[0155] It can be seen from the above comparison data that the method of the present invention is superior to other solutions in accuracy, labeling precision and efficiency. It can achieve rapid removal in a shorter time than traditional solutions. Traditional labeling methods are only applicable to specific images and cannot be promoted on a large scale. The manual cutting method and the method of the present invention have a wide range of applications, but the traditional solution is inferior to the method of the present invention in other data. The comparison results show that the method of the present invention can quickly, efficiently, accurately and automatically label lumbar images.
[0156] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.
[0157] The small sample lumbar segmentation method based on spatial prior information and prototype iterative update provided in the embodiment of the present invention can batch process lumbar MR images and accurately and efficiently annotate lumbar images, thereby solving the problems of high subjectivity, slow speed and low precision of manual annotation.
[0158] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of the technical solution of the present invention.
Claims
1. A small sample lumbar vertebra segmentation method based on spatial prior information and prototype iterative update, characterized in that: The small sample lumbar vertebra segmentation method based on spatial prior information and prototype iterative update comprises the following steps: Step S1: acquiring a lumbar vertebra MR image sample set including a plurality of sample lumbar vertebra MR images; Step S2: constructing a small sample lumbar vertebrae annotation model, and using the lumbar vertebrae MR image sample set to train the small sample lumbar vertebrae annotation model to obtain a trained small sample lumbar vertebrae annotation model; Step S3: inputting the lumbar vertebra MR image to be labeled into the trained small sample lumbar vertebra labeling model for labeling, so as to obtain a lumbar vertebra labeling prediction result of the lumbar vertebra MR image to be labeled.
2. The small sample lumbar vertebra segmentation method based on spatial prior information and prototype iterative update according to claim 1, characterized in that: The step of acquiring a lumbar vertebra MR image sample set including a plurality of sample lumbar vertebra MR images further includes: Dividing the lumbar MR image sample set into a training sample set, a verification sample set and a test sample set; A training support set and a training query set are constructed according to the training sample set, a verification support set and a verification query set are constructed according to the verification sample set, and a test support set and a test query set are constructed according to the test sample set; wherein, adjacent slices of each sample lumbar vertebra MR image in the training sample set are respectively used as the training support set and the training query set, all slices of a sample lumbar vertebra MR image in the verification sample set are used as the verification query set, and the middle slice of another sample lumbar vertebra MR image in the verification sample set is used as the verification support set; all slices of a sample lumbar vertebra MR image in the test sample set are used as the test query set, and the middle slice of another sample lumbar vertebra MR image in the test sample set is used as the test support set; Among them, the slices in the training support set, the slices in the verification support set and the slices in the test support set all have vertebrae labels, and the vertebrae labels are obtained by annotating the vertebrae regions of the slices in the training support set, the vertebrae regions of the slices in the verification support set and the vertebrae regions of the slices in the test support set using ITK-SNAP.
3. The small sample lumbar vertebra segmentation method based on spatial prior information and prototype iterative update according to claim 2, characterized in that: The method of training the small sample lumbar vertebrae labeling model by using the lumbar vertebrae MR image sample set to obtain the trained small sample lumbar vertebrae labeling model further includes: Using the training sample set to train the small sample lumbar vertebrae labeling model, and during the training process, using the verification sample set to verify the small sample lumbar vertebrae labeling model; After the small sample lumbar vertebrae labeling model is trained, the test sample set is used to test and evaluate the performance of the trained small sample lumbar vertebrae labeling model.
4. The small sample lumbar vertebra segmentation method based on spatial prior information and prototype iterative update according to claim 3, characterized in that: The step of using the training sample set to train the small sample lumbar vertebrae labeling model further includes: The small sample lumbar vertebrae annotation model includes a small sample feature extraction module, a spatial prior information extraction module, a spatial fusion module and an iterative update module; Inputting the training support set and the training query set into the small sample feature extraction module to obtain the vertebral features F of the slices in the training query set; The training support set and the training query set are input into the spatial prior information extraction module to obtain the spatial prior information of the slices in the training query set. The vertebral features F and their spatial prior information in the slices of the training query set are Input into the spatial fusion module for attention mechanism fusion to obtain preliminary prediction results of vertebral annotation of slices in the training query set; Inputting preliminary prediction results of vertebral annotation of slices in the training query set into the iterative update module to output final prediction results of vertebral annotation of slices in the training query set; The final prediction results of the vertebrae annotations of the slices in the training query set are spliced together to obtain the complete lumbar vertebrae annotation prediction results of the slices in the training query set.
5. The small sample lumbar vertebra segmentation method based on spatial prior information and prototype iterative update according to claim 4, characterized in that: The step of inputting the training support set and the training query set into the small sample feature extraction module to obtain the vertebral features F of the slices in the training query set further includes: The small sample feature extraction module is a small sample shared feature encoder, which includes three submodules, each of which includes a 3×3 convolution layer, a normalization layer, and an activation layer. The training support set and the training query set are merged and then sequentially passed through the three submodules to extract the associated features of the training support set and the training query set. The associated features are then subjected to channel dimension reduction through a 1×1 convolution, and finally decomposed to obtain the feature F of the training query set. q and the feature F of the training support set s ; According to the feature F of the training support set s and the vertebral labels m of the training support set s The prototype feature P of the training support set is calculated, wherein the calculation formula of the prototype feature P of the training support set is as follows: In the above formula, s represents the training support set, F s represents the features of the training support set, m s represents the vertebral labels of the training support set, ⊙ represents the Hadamard product used for masked average pooling, and P represents the calculated prototype features of the training support set; Calculate the feature F of the training query set q The cosine similarity between the prototype feature P of the training support set and the inverse of it, and then the sigmoid activation function is used to generate the foreground probability map of the training query set The calculation formula is as follows: Among them, (h, w) represents each pixel in the sample lumbar MR image, q represents the training query set, σ is the sigmoid activation function, T is the segmentation threshold, and α is the scaling factor.
6. The small sample lumbar vertebra segmentation method based on spatial prior information and prototype iterative update according to claim 5, characterized in that: The training support set and the training query set are input into the spatial prior information extraction module to obtain the spatial prior information of the slices in the training query set. Also included: The spatial prior information extraction module includes 4 downsampling modules and 4 upsampling modules, each of which includes 2D convolution and ReLU activation function, and each of which includes 1*1 convolution, upsampling, 2D convolution and ReLu activation function; the training support set and the training query set are merged and then sequentially passed through 4 downsampling modules and 4 upsampling modules, and the vertebral label m of the training support set is obtained. s Perform spatial transformation to obtain spatial prior information of the training query set 7. The small sample lumbar vertebra segmentation method based on spatial prior information and prototype iterative update according to claim 6, characterized in that: The vertebral features F and their spatial prior information of the slices in the training query set are The input is input into the spatial fusion module for attention mechanism fusion to obtain the preliminary prediction result of vertebral annotation of the slice in the training query set, and further includes: The spatial prior information of the training query set is transformed along the channel dimension and the foreground probability map of the training query set Then use another linear transformation to obtain the actual content information value, and finally get the fused decoding feature F dec , where the decoded feature F dec The initial prediction result of vertebral annotation as a slice in the training query set Where Att ij represents the attention information of the training query set features, v ij Represents the weight matrix W v The actual content information value calculated, i, j represents each pixel on the sample lumbar MR image, and softmax represents the normalization function. represents the query feature of the training query set, represents the input element currently being processed, T represents the matrix transpose, q represents the training query set, k ij Represents the key feature of the training query set, used with Compare to determine the correlation, xy represents the length and width of the corresponding feature, ij represents the length and width of the corresponding feature.
8. The small sample lumbar vertebra segmentation method based on spatial prior information and prototype iterative update according to claim 7, characterized in that: The step of inputting the preliminary prediction results of the vertebrae annotation of the slices in the training query set into the iterative update module to output the final prediction results of the vertebrae annotation of the slices in the training query set further includes: Preliminary prediction results of vertebral annotations for slices in the training query set The calculation formula is as follows: Among them, i,j represents each pixel point on the sample lumbar MR image, Represents the background features of each pixel on the sample lumbar MR image, Represents the foreground features of each pixel on the sample lumbar spine MR image; Then the features of the training query set after iteration are The calculation formula is as follows: Among them, F q (i, j) represents the features of the initial training query set, P n Represents the prototype features after n iterations; Then the prototype feature P after iteration n+1 times n+1 The calculation formula is as follows: in, Represents the features of the training query set after the iteration With the features F of the initial training query set q The cross entropy loss between them, N is the number of iterations of model loss back propagation, v is the parameter in the model training process, Is the differential symbol.
9. The small sample lumbar vertebra segmentation method based on spatial prior information and prototype iterative update according to claim 3, characterized in that: During the training process of the small sample lumbar vertebrae labeling model, the calculation formula of the training loss L of the small sample lumbar vertebrae labeling model is: L=L seg +L reg Among them, L seg is the main loss of the small sample lumbar vertebrae annotation model, L reg Auxiliary loss for the small sample lumbar vertebra annotation model; The main loss L of the small sample lumbar vertebrae annotation model seg The calculation formula is: The auxiliary loss L of the small sample lumbar vertebrae annotation model reg The calculation formula is: Among them, H and W represent the length and width of the sample lumbar MR image, q represents the training query set, s represents the training support set, and m q represents the vertebrae labels of the training query set, Represents the vertebral annotation prediction results of the training query set, m s represents the vertebrae labels of the training support set, represents the vertebral annotation prediction results of the training support set, and (h,w) represents each pixel in the sample lumbar MR image.
10. A small sample lumbar vertebra segmentation system based on spatial prior information and prototype iterative update, used to implement the small sample lumbar vertebra segmentation method based on spatial prior information and prototype iterative update according to any one of claims 1 to 9, characterized in that: The small sample lumbar segmentation system based on spatial prior information and prototype iterative update includes: An acquisition module, used for acquiring a lumbar vertebra MR image sample set including a plurality of sample lumbar vertebra MR images; A training module, used for constructing a small sample lumbar vertebrae annotation model, and training the small sample lumbar vertebrae annotation model using the lumbar vertebrae MR image sample set to obtain a trained small sample lumbar vertebrae annotation model; The labeling module is used to input the lumbar vertebra MR image to be labeled into the trained small sample lumbar vertebra labeling model for labeling, so as to obtain the lumbar vertebra labeling prediction result of the lumbar vertebra MR image to be labeled.