Zero-shot brain lesion segmentation method, system, device and medium based on prompt learning

Abstract

The present invention discloses a zero-sample brain lesion segmentation method, system, device and medium based on prompt learning, including: using the available FLAIR mode of healthy magnetic resonance imaging without lesions, and then completing two different prompting schemes according to the high and low signals of tumors compared with abnormal tissues by doctors, and obtaining a validation set and a training set respectively. According to setting different hyperparameters, training models under different hyperparameters are obtained. The obtained model is subjected to hyperparameter integration to obtain the final model for testing, and the test is completed to obtain the final segmentation result. The advantages of the present invention are: it has higher accuracy and recall rate in brain lesion segmentation tasks, improves the efficiency of lesion segmentation, and reduces the intensity of demand for manpower and financial resources.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a zero-sample brain lesion segmentation method, system, device and medium based on prompt learning. Background Art

[0002] Brain lesion segmentation is an image segmentation technology based on medical imaging data. It aims to automatically identify lesions (such as tumors, strokes, and hemorrhages) in brain images and separate them from normal brain tissue. This technology is crucial for the accurate diagnosis and treatment of brain diseases.

[0003] Brain lesion segmentation is typically achieved using technologies such as computer graphics, machine learning, and deep learning. Deep learning typically employs convolutional neural networks (CNNs) to train models for automatic segmentation. This training requires the use of large amounts of medical imaging data to improve model accuracy and generalization capabilities.

[0004] Brain lesion segmentation technology can help doctors more quickly identify lesions and more accurately measure their size, shape, and location. It can also compare before and after treatment to assess treatment effectiveness. This technology has been widely used in clinical medicine and has become a key tool in the diagnosis and treatment of brain diseases.

[0005] Most of the methods for brain lesion segmentation rely on supervised learning, which requires a large amount of manual lesion annotation as data for model training. [1] . It takes a lot of time to label data, so semi-supervised [2] The self-supervised method was proposed one after another, which can make full use of a large amount of unlabeled data as an auxiliary training model, but it still cannot get rid of the dependence on labeled data. [4] It has also begun to flourish, such as self-encoders, such as AutoEncoder (AE) [4] and Variational AutoEncoder(VAE) [4] or fAnoGan [5]However, these methods are all based on models for autoregressive tasks. During the training phase, these methods learn only to reconstruct healthy brain images, allowing the model to master the feature space of healthy images. During the testing phase, when images of patients with brain lesions are input, the model still learns to reconstruct the patient's healthy brain map based on the features of the healthy image. It calculates the difference between the input brain lesion image and the reconstructed "healthy image" from the model, and converts the difference image into a brain lesion annotation using a threshold. These methods can completely eliminate the reliance on labeled data, but require human intervention with prior knowledge, such as threshold selection, and have complex model post-processing. This post-processing aims to remove noise from small local connected areas and make the segmentation results smoother, but it also inevitably causes segmentation omissions in some small lesions. For other brain lesion segmentation tasks, especially those with diffusely distributed small lesions, it is very difficult and performs poorly.

[0006] As can be seen from the above, the existing technology has the following defects:

[0007] 1. Traditional unsupervised brain lesion segmentation methods often require a priori thresholds to segment lesions, which may lead to inaccurate segmentation results.

[0008] 2. Traditional unsupervised brain lesion segmentation methods usually require complex post-processing, such as morphological operations, which may reduce the performance of the algorithm.

[0009] 3. It does not have good generalization ability and needs to be predicted on different datasets, which requires fine-tuning or retraining.

[0010] References

[0011] [1]Isensee F, Jaeger PF, Kohl SAA, et al.nnU-Net: a self-configuringmethod for deep learning-based biomedical image segmentation[J].Naturemethods, 2021,18(2):203-211;

[0012] [2]Cui W, Liu Y, Li Y, et al. Semi-supervised brain lesion segmentation with an adapted mean teacher model [C] / / International Conference on Information Processing in Medical Imaging. Springer, Cham, 2019:554-565;

[0013] [3]Zhang X, Xie W, Huang C, et al. Self-supervised tumor segmentation through layer decomposition[J]. arXiv preprint arXiv:2109.03230,2021;

[0014] [4]Baur C, Wiestler B, Albarqouni S, et al.Deep autoencoding models for unsupervised anomaly segmentation in brain MR images[C] / / International MICCAIbrainlesion workshop. Springer, Cham, 2018:161-169;

[0015] [5]Schlegl T, P, Waldstein SM, et al. f-AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks [J]. Medical image analysis, 2019, 54: 30-44. Summary of the Invention

[0016] The present invention addresses the shortcomings of existing technologies and provides a zero-shot brain lesion segmentation method, system, device, and medium based on prompt learning. This method can complete brain lesion segmentation tasks using only unlabeled data and can be directly generalized to different datasets without fine-tuning or retraining.

[0017] In order to achieve the above object of the invention, the technical solution adopted by the present invention is as follows:

[0018] A zero-shot brain lesion segmentation method based on cue learning, comprising the following steps:

[0019] S1: In the prompting stage, the available FLAIR modality of healthy MRI images without lesions is used, and then two different prompting schemes are completed based on the doctor's high and low signals compared with tumors and abnormal tissues, and a validation set and a training set are obtained respectively.

[0020] S2: In the learning phase, different hyperparameter settings are used to obtain trained models with different hyperparameters. Specifically, zero-shot learning is prone to overfitting to the distribution of the training data, and a large learning rate (lr) can help to address this overfitting problem in model selection. However, the selection and formulation of hyperparameters may not generalize well to different datasets. Therefore, we integrate the models trained with different hyperparameters. This means setting different learning rates and training epochs. For example, lr∈{0.1, 0.01, 0.001}, and training epochs∈{10, 20, 50, 100}, where the validation set is used to select the model for each hyperparameter combination.

[0021] S3: In the testing phase, the obtained model is hyperparameter-integrated to obtain the final test model, the test is completed, and the final segmentation result is obtained.

[0022] Furthermore, S1 is as follows:

[0023] Given a set of FLAIR modality MRI images without tumor lesions where X i represents the i-th image, and N is the total number of images. For each image X i For example, to generate a synthetic image X with high signal or low signal areas and the corresponding label Y is expressed as follows:

[0024] X=T(X i )⊙A+X i ⊙(1-A)

[0025]

[0026] Where T(·) represents a grayscale transformation of an image, A is a weighted image ranging from 0 to 1, ⊙ represents pixel-wise multiplication, and a is a threshold that determines the relationship between A and the label Y.

[0027] To simulate the grayscale information, blur it to reduce the description of the texture. T(·) is expressed as follows:

[0028] T(X i )=λBlur(X i )

[0029] Among them, λ is used to control the amplitude distribution range of high and low signals. When 0<λ<1, the signal is low, and when λ>1, the signal is high.

[0030] Furthermore, in S1, the shape and location distribution of the tumor are determined by first generating a polyhedron with a frequency of f. The edges of the icosahedron are subdivided f times, and each vertex is projected onto a parametric sphere of unit radius centered at the origin. The resulting tumor mask M needs to be distributed throughout the brain.

[0031] Furthermore, S2 is as follows:

[0032] The validation set modifies the prompt details as follows:

[0033] T(X i )=λX i

[0034] A=M

[0035] a=0

[0036] Furthermore, in S3, based on the nnU-Net framework, the learning rate was increased tenfold to obtain models with different hyperparameter settings. These models were then integrated to obtain the final model for testing on different test sets.

[0037] The present invention also discloses a zero-sample brain lesion segmentation system based on prompt learning, which can be used to implement the above-mentioned zero-sample brain lesion segmentation method based on prompt learning. Specifically, it includes: a prompt module, a learning module and a testing module;

[0038] The prompt module uses the FLAIR modality of healthy MRI images without lesions, and then completes two different prompt schemes based on the doctor's high and low signals compared with tumors and abnormal tissues, obtaining a validation set and a training set respectively.

[0039] The learning module obtains training models under different hyperparameters by setting different hyperparameters, and the validation set is used for model selection.

[0040] The testing module integrates the hyperparameters of the obtained model to obtain the final test model, completes the test, and obtains the final segmentation result.

[0041] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned zero-sample brain lesion segmentation method based on prompt learning is implemented.

[0042] The present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned zero-sample brain lesion segmentation method based on prompt learning.

[0043] Compared with the prior art, the advantages of the present invention are:

[0044] 1. Improved segmentation accuracy: Compared with traditional unsupervised medical image segmentation methods, this method has higher precision and recall in brain lesion segmentation tasks. The DICE index is improved by 45.3% and 59.8% on the BRATs2021 dataset and BTFLAIR dataset, respectively, compared with the best-performing control group.

[0045] 2. Improved efficiency: This method does not require manually labeled data and can be trained using only images of healthy individuals without the need for brain lesion images, thereby greatly improving the efficiency of lesion segmentation.

[0046] 3. Reduce labor intensity: This method does not require doctors to manually label brain lesion images, thus saving professional doctors from spending about half a day labeling each image and reducing the demand for manpower and financial resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of a zero-sample brain lesion segmentation method according to an embodiment of the present invention;

[0048] Figure 2 This is a qualitative example segmentation result diagram of zero-sample tumor segmentation in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0050] like Figure 1 As shown in the figure, a zero-shot brain lesion segmentation method based on cue learning is presented. Taking the most typical brain lesion, tumor, as an example, the zero-shot brain lesion segmentation architecture is divided into the following stages.

[0051] The first is the prompting stage, which uses the available FLAIR modality of healthy MRI images without lesions. Then, based on the doctor's prior knowledge of the tumor (high and low signals compared with abnormal tissue), two different prompting schemes are completed to obtain a validation set and a training set.

[0052] The second stage is the learning stage. By setting different hyperparameters, training models under different hyperparameters are obtained. The validation set can be used for model selection, thereby avoiding overfitting on the generated data.

[0053] The obtained model is hyperparameter integrated to obtain the final test model, further completing the testing phase.

[0054] (1) Prompt stage:

[0055] Given a set of FLAIR modality MRI images without tumor lesions where X i represents the i-th image, and N is the total number of images. For each image X i In terms of, generating a synthetic image X with high signal or low signal areas and the corresponding label Y can be expressed as follows:

[0056] X=T(X i )⊙A+X i ⊙(1-A)

[0057]

[0058] Where T(·) represents a grayscale transformation of an image, and A is a weighted image ranging from 0 to 1.

[0059] ⊙ represents pixel-wise multiplication. a is a threshold that determines the relationship between A and the label Y.

[0060] The key to this is how to design the transformation T(·) and the weighted image A. This example assumes that generating an image that more closely resembles the tumor's appearance is beneficial for the final tumor segmentation task. However, since no annotation information is available, this example will not delve into the details of how to generate the tumor, as this is quite challenging.

[0061] Tumors always have large clusters of abnormally high signals or necrotic low signals. Therefore, this embodiment simulates this grayscale information and uses blurring to reduce the description of texture. T(·) can be expressed as follows:

[0062] T(X i )=λBlur(X i )

[0063] Among them, λ is used to control the amplitude distribution range of high and low signals. When 0<λ<1, the signal is low, and when λ>1, the signal is high.

[0064] Furthermore, the shape and location of the tumor are also crucial. To this end, this example first generates a polyhedron with a frequency of f. This is done by subdividing the edges of an icosahedron f times and projecting each vertex onto a parametric sphere of unit radius centered at the origin. The resulting tumor mask must be distributed throughout the brain.

[0065] (2) Training phase:

[0066] The design is the same as the hint stage, but the texture details are designed differently in order to solve the problem of over-learning of the model on the generated data. In the hint design, the generation of the tumor is very simple compared to the real tumor. Specifically, the model will learn the useful parts of the generated data in the early stages of learning, and the training loss curve and the real online test set loss curve will decrease. However, in the later stages, the model will overfit, and the training set loss will continue to decrease, but the test set loss will increase. However, in the zero-shot learning setting, there is no real labeled data to constrain the model selection. Therefore, a method needs to be designed to prevent the overfitting problem of the model, so the validation set task design is proposed. Some details of the hint have been modified as follows:

[0067] T(X i )=λX i

[0068] A=M

[0069] a=0

[0070] (3) Testing phase:

[0071] In fact, increasing the learning rate can benefit model learning when the data distributions of the training and test sets are very different. Therefore, based on the nnU-Net framework, this embodiment increases the learning rate tenfold to obtain models under different hyperparameter settings. These models are then integrated to obtain the final model, which is then tested on different test sets (model integration reference: Tebaldi C, Knutti R. The use of the multi-model ensemble in probabilistic climate projections [J]. Philosophical transactions of the royal society A: mathematical, physical and engineering sciences, 2007, 365 (1857): 2053-2075).

[0072] In terms of zero-sample tumor segmentation, the same model can currently be used to complete tests on different test sets. Each dataset is multi-center and multi-scanner data, and can all be tested on the trained model. It shows good segmentation performance on a total of more than 1,300 samples, and does not require any labeled data or data from the same domain for training.

[0073] The qualitative segmentation results are as follows Figure 2As shown in the figure, 6 examples were randomly selected from the public dataset BraTS2021 (1251 test data) and the private dataset BTFLAIR (67 test data), and the zero-shot segmentation method based on prompt learning (PL-BTS) of this embodiment was compared with the existing unsupervised methods (reconstruction model: autoencoder (AE), variational autoencoder (VAE), vector quantized variational autoencoder (VQ-VAE), adversarial generation network: f-AnoGAN, Transformer-based method: dense convolutional transformer autoencoder (DCTAE), basic vision transformer autoencoder (BTAE), and hierarchical transformer autoencoder with skip connections (HTAES)), as well as 3D implicit fields (3DIF) and constrained unsupervised anomaly segmentation (CUAS) for control results. Attached with real annotations for reference, we can see that in randomly selected qualitative analysis subjects, the PL-BTS method of this embodiment significantly outperforms nine other unsupervised control methods. Even in areas where the periventricular hyperintense areas are adjacent to hyperintense tumors, the method of this embodiment is able to effectively distinguish these non-tumor hyperintense areas, effectively improving the model's segmentation performance. It also demonstrates stronger generalization capabilities on a test set of over 1,300 cases from multiple centers and scanners across two datasets.

[0074] The quantitative experimental results are shown in Table 1. The method of this embodiment has significant improvements in dice coefficient, precision, and recall compared to the control group method (compared with the best-performing control group, dice is improved by 45.3% and 59.8% on the brats2021 dataset and BTFLAIR dataset, respectively).

[0075] Table 1 Quantitative segmentation results of zero-sample tumor segmentation

[0076]

[0077] In another embodiment of the present invention, a zero-shot brain lesion segmentation system based on prompt learning is provided. The system can be used to implement the above-mentioned zero-shot brain lesion segmentation method based on prompt learning, and specifically includes: a prompt module, a learning module, and a testing module;

[0078] The prompt module uses the FLAIR modality of healthy MRI images without lesions, and then completes two different prompt schemes based on the doctor's high and low signals compared with tumors and abnormal tissues, obtaining a validation set and a training set respectively.

[0079] The learning module obtains training models under different hyperparameters by setting different hyperparameters, and the validation set is used for model selection.

[0080] The testing module integrates the hyperparameters of the obtained model to obtain the final test model, completes the test, and obtains the final segmentation result.

[0081] In another embodiment of the present invention, a terminal device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of a zero-sample brain lesion segmentation method based on prompt learning, comprising the following steps:

[0082] S1: In the prompting stage, the available FLAIR modality of healthy MRI images without lesions is used, and then two different prompting schemes are completed based on the doctor's high and low signals compared with tumors and abnormal tissues, and a validation set and a training set are obtained respectively.

[0083] S2: Learning phase: By setting different hyperparameters, trained models with different hyperparameters are obtained. Specifically, the models trained with different hyperparameters are integrated. This means setting different learning rates and training iterations to combine models. The validation set is used to select the model for each hyperparameter combination.

[0084] S3: In the testing phase, the obtained model is hyperparameter-integrated to obtain the final test model, the test is completed, and the final segmentation result is obtained.

[0085] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and, of course, the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory.

[0086] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the above-mentioned embodiment related to a zero-shot brain lesion segmentation method based on prompt learning; the processor may load and execute the following steps:

[0087] S1: In the prompting stage, the available FLAIR modality of healthy MRI images without lesions is used, and then two different prompting schemes are completed based on the doctor's high and low signals compared with tumors and abnormal tissues, and a validation set and a training set are obtained respectively.

[0088] S2: Learning phase: By setting different hyperparameters, trained models with different hyperparameters are obtained. Specifically, the models trained with different hyperparameters are integrated. This means setting different learning rates and training iterations to combine models. The validation set is used to select the model for each hyperparameter combination.

[0089] S3: In the testing phase, the obtained model is hyperparameter-integrated to obtain the final test model, the test is completed, and the final segmentation result is obtained.

[0090] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0092] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0094] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the implementation methods of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A zero-shot brain lesion segmentation method based on cue learning, characterized in that The following steps are involved: S1: Prompt stage, using the available FLAIR modality of healthy MRI images without lesions, and then completing two different prompting schemes based on the doctor's high and low signals compared with tumors and abnormal tissues, respectively, to obtain a validation set and a training set; S1 is as follows: Given a set of FLAIR modality MRI images without tumor lesions where X i Represents the i-th image, N is the total number of images; for each image X i For example, to generate a synthetic image X with high signal or low signal areas and the corresponding label Y is expressed as follows: X=T(X i )⊙A+X i ⊙(1-A) Where T(·) represents a grayscale transformation of an image, A is a weighted image ranging from 0 to 1; ⊙ represents pixel-level multiplication; a is a threshold that determines the relationship between A and label Y; To simulate the information of grayscale, blur it to reduce the description of texture; T(·) is expressed as follows: T(X i )=λBlur(X i ) Among them, λ is used to control the amplitude distribution range of high and low signals; when 0<λ<1, the signal is low, and when λ>1, the signal is high; To determine the shape and location of the tumor, we first generate a polyhedron with a frequency of f. We then subdivide the edges of an icosahedron f times and project each vertex onto a parametric sphere of unit radius centered at the origin. The resulting tumor mask M needs to be distributed throughout the brain. S2: Learning phase: By setting different hyperparameters, training models under different hyperparameters are obtained. Specifically, the models trained under different hyperparameters are integrated; that is, different learning rates and training algebraic combinations are set. The validation set is used to select the model under each hyperparameter combination. The validation set modifies the prompt details as follows: T(X i )=λX i A=M a＝0 S3: In the testing phase, the obtained model is hyperparameter-integrated to obtain the final test model, the test is completed, and the final segmentation result is obtained.

2. The zero-shot brain lesion segmentation method based on cue learning according to claim 1, characterized in that: In S3, based on the nnU-Net framework, the learning rate is increased tenfold to obtain models under different hyperparameter settings; and these models are integrated to obtain the final model for testing on different test sets.

3. A zero-shot brain lesion segmentation system based on cue learning, characterized by: The system can be used to implement the zero-sample brain lesion segmentation method based on prompt learning as described in claim 1 or 2; Zero-shot brain lesion segmentation system, including: prompt module, learning module and testing module; The prompt module uses the FLAIR modality of healthy MRI images without lesions and then completes two different prompt schemes based on the doctor's judgment of the high and low signals of tumors compared with abnormal tissues, obtaining a validation set and a training set respectively; The learning module obtains training models under different hyperparameters by setting different hyperparameters, and the validation set is used for model selection; The testing module integrates the hyperparameters of the obtained model to obtain the final test model, completes the test, and obtains the final segmentation result.

4. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the zero-sample brain lesion segmentation method based on prompt learning according to claim 1 or 2 is implemented.

5. A computer-readable storage medium, characterized in that: A computer program is stored on a computer-readable storage medium, and when the program is executed by a processor, the method for zero-sample brain lesion segmentation based on prompt learning according to claim 1 or 2 is implemented.

Images