Medical image segmentation method, device and equipment based on deep learning full life cycle, medium and program product
By combining the fusion of image and text features, the problems of anatomical structure changes and scanning center differences in medical image segmentation throughout the life cycle are solved, and segmentation accuracy and efficiency are improved, providing strong support for medical image analysis.
Patent Information
- Application Number
- CN202411014379.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-15
AI Technical Summary
The existing full-life cycle medical image segmentation method based on deep learning is difficult to adapt to the anatomical changes of different age stages and scanning centers, and ignores text information, resulting in limited segmentation performance.
By collecting medical image data from different life cycle stages and sources, a large-scale data set with suboptimal annotation is formed for pre-training, obtain text information related to image morphology, and retrain the segmentation model with text information as knowledge prompts. Combine image and text features for fusion.
It significantly improves the accuracy and efficiency of medical image segmentation throughout the life cycle, adapts to image data at different ages and scanning centers, and improves the practicality and reliability of the model in actual medical scenarios.
Smart Images

Figure CN118967713A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a medical image segmentation method, apparatus, equipment, medium and program product based on the full life cycle of deep learning. Background Art
[0002] The whole life cycle refers to the entire time span from birth to the end of life. Medical image segmentation of the whole life cycle is of great significance for revealing the anatomical structure and pathological changes of the human body in different life cycles, promoting basic research in life sciences, optimizing early diagnosis and intervention of diseases, and developing personalized medical solutions. Segmenting tissues or organs throughout the life cycle based on a certain modality of medical images can more finely analyze the volume changes and functional characteristics of tissues or organs, thereby providing more accurate data support for the study of the mechanism of tissue or organ development and pathological research.
[0003] However, manual labeling is time-consuming and laborious, so developing automatic and accurate full-life cycle medical image segmentation methods has become a task of great clinical significance.
[0004] Methods based on deep learning have been widely used in medical image segmentation tasks and have shown excellent performance. For example, U-Net based on convolutional neural networks, UNETR based on vision transformer, and Swin UNETR based on Swin Transformer. However, full-life cycle medical image segmentation based on deep learning has the following challenges:
[0005] 1) Changes in anatomical structure at different ages, such as rapid development of anatomical structure in newborns and infants, relative stability of anatomical structure in adults, and degeneration of organs and tissues in old age;
[0006] 2) Different scanning centers and image acquisition protocols will also lead to certain anatomical deviations.
[0007] In addition, existing methods only use image information and ignore the importance of text information, which leads to certain limitations in segmentation performance. Therefore, it is particularly important to combine image and text information to develop new segmentation methods to improve the accuracy and reliability of medical image segmentation throughout the life cycle. Summary of the invention
[0008] In view of the shortcomings of the prior art mentioned above, the purpose of the present application is to provide a technical solution to the challenges of how to overcome age changes, scanning protocol differences and insufficient information utilization to improve the accuracy and robustness of life cycle segmentation.
[0009] To achieve the above-mentioned objectives and other related objectives, the first aspect of the present application provides a medical image segmentation method based on deep learning throughout the entire life cycle, including: collecting medical image data at different life cycle stages and from different sources to form a large-scale data set with suboptimal annotations and preprocessing it; pre-training a segmentation model based on the large-scale data set with suboptimal annotations; obtaining text information related to the image morphology, and retraining the pre-trained segmentation model using the text information as a knowledge prompt, so that the segmentation model can perform full life cycle segmentation of medical images.
[0010] In some embodiments of the first aspect of the present application, the medical image segmentation method includes pre-training a segmentation model using single-modality data.
[0011] In some embodiments of the first aspect of the present application, the segmentation model adopts an encoder-decoder architecture; the encoder is used to gradually reduce the spatial dimension of the image and increase the feature dimension to extract image features; the decoder is used to gradually restore the spatial dimension of the image and reduce the feature dimension to perform an inverse conversion from features to segmentation maps.
[0012] In some embodiments of the first aspect of the present application, the text information related to the image morphology is obtained, and the pre-trained segmentation model is retrained using the text information as a knowledge prompt, which includes: generating a corresponding descriptive sentence based on the text information related to the image morphology; the text information related to the image morphology includes scanning device information, scanning parameter information, and subject age and gender information; the descriptive sentence is input into a trained text encoder to extract text features; the extracted text features capture background information related to the image; the extracted text features are fused with the image features and input into the pre-trained segmentation model for retraining.
[0013] In some embodiments of the first aspect of the present application, the method of converting text information into a descriptive sentence includes: designing a template for a descriptive sentence, wherein the template includes placeholders for filling in information; filling the collected text information related to the image morphology into the placeholders in the template to generate a descriptive sentence; and optimizing the descriptive sentence using natural language processing technology.
[0014] In some embodiments of the first aspect of the present application, the method of fusing the extracted text features with image features includes: transforming the dimensions of the text features to match the image features through dimensionality transformation; and splicing the matched text features and image features in the sequence length dimension to form input features of the segmentation model.
[0015] In some embodiments of the first aspect of the present application, the method of retraining the segmentation model includes: initializing the model using the weights of the pre-trained model; freezing the encoder parameters in the segmentation model and only updating the learnable parameters and decoder parameters.
[0016] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides a medical image segmentation device based on deep learning full life cycle, including: a data acquisition module, used to collect medical image data at different life cycle stages and from different sources, to form a large-scale data set with suboptimal annotations and perform preprocessing; a first-stage training module, used to pre-train the segmentation model based on a large-scale data set with suboptimal annotations; a second-stage training module, used to obtain text information related to image morphology, and to retrain the pre-trained segmentation model using the text information as knowledge prompts, so that the segmentation model can perform full life cycle segmentation of medical images.
[0017] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the medical image segmentation method based on the full life cycle of deep learning.
[0018] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer implements the medical image segmentation method based on the full life cycle of deep learning.
[0019] To achieve the above-mentioned purpose and other related purposes, the fifth aspect of the present application provides a computer device, including a memory, a processor and a computer program stored in the memory; the processor executes the computer program to implement the medical image segmentation method based on the full life cycle of deep learning.
[0020] As described above, the medical image segmentation method, device, equipment, medium and program product based on deep learning full life cycle of the present application have the following beneficial effects:
[0021] (1) Improving segmentation accuracy: In this way, the model not only utilizes the features of the image itself, but also combines the knowledge contained in the text information, making feature extraction more targeted and effective.
[0022] (2) Full life cycle coverage: This method significantly improves the accuracy and efficiency of medical image segmentation throughout the life cycle and can adapt to image data from different age groups and different collection centers.
[0023] (3) Clinical application support: It provides strong support for medical image analysis and clinical applications, and improves the practicality and reliability of the model in actual medical scenarios.
[0024] In summary, this scheme combines image features with text features, enabling the model to more comprehensively understand and analyze image content when processing medical images, thereby improving the accuracy and efficiency of segmentation tasks. This method is particularly suitable for processing medical image data from diverse sources and with complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 Shown is a flowchart of a medical image segmentation method based on deep learning throughout the life cycle in one embodiment of the present application.
[0026] Figure 2 Shown is a flowchart of model retraining in one embodiment of the present application.
[0027] Figure 3 Shown is a flowchart of a medical image segmentation method based on deep learning throughout the life cycle in one embodiment of the present application.
[0028] Figure 4 Shown is a structural schematic diagram of a medical image segmentation device based on the full life cycle of deep learning in one embodiment of the present application.
[0029] Figure 5 Shown is a schematic diagram of the structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION
[0030] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.
[0031] The medical image segmentation method, device, equipment, medium and program product based on the full life cycle of deep learning provided in the embodiments of the present application have a wide range of applications in the medical field. For example, it can track the development of human organs, provide important support for the diagnosis and treatment of diseases, and long-term monitoring of health conditions. Not only that, the surgical area can be accurately determined by image segmentation technology to help doctors specify more accurate surgical plans and provide real-time navigation during surgery; or it can be used to determine the exact location and size of the tumor to optimize the distribution of affine doses and reduce damage to surrounding tissues; image segmentation technology can also help researchers observe the effects of drugs on diseased tissues to evaluate the efficacy of drugs. In addition, image segmentation technology has been well and deeply applied in epidemiological research, rehabilitation medicine, biomedical research, telemedicine, health monitoring and management, etc.
[0032] Before further describing the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are applicable to the following interpretations:
[0033] (1) Data annotation: In the field of machine learning and artificial intelligence, data annotation refers to the process of classifying, labeling, or categorizing raw data so that the training algorithm can recognize and learn patterns in the data. For example, in an image recognition task, data annotation may include labeling specific objects in the image.
[0034] (2) Medical images: Medical images refer to images used for medical diagnosis, treatment planning, disease monitoring, etc., usually generated by medical imaging equipment such as X-ray machines, CT scanners, MRI machines, etc. Medical images include but are not limited to X-ray images, CT images, MRI images, ultrasound images, etc.
[0035] (3) Text encoder: In natural language processing, a text encoder is a model or algorithm that converts text data into numerical form, allowing machine learning models to process text data. Common text encoders include word embedding and sequence-to-sequence models based on deep learning.
[0036] (4) Loss function: In machine learning, the loss function is a function used to measure the difference between the model's predicted value and the actual value. The smaller the value of the loss function, the more accurate the model's prediction.
[0037] (5) Encoder and decoder: Encoder and decoder are two key components in sequence processing tasks. The encoder is responsible for converting the input sequence into a fixed-length internal representation, while the decoder converts this internal representation back into an output sequence. This structure is very common in sequence-to-sequence models, such as machine translation and text summarization tasks.
[0038] (6) Feature fusion: Feature fusion is a technique used to combine features from different sources or obtained through different processing methods to improve the performance of the model. In deep learning, feature fusion can occur in networks at different levels, enhancing the representation ability of the model by merging feature maps from different layers.
[0039] Figure 1 A schematic flow chart of a medical image segmentation method based on the full life cycle of deep learning in an embodiment of the present invention is shown. The specific process mainly includes the following steps:
[0040] Step S11, collect medical image data from different life cycle stages and different sources, form a large-scale dataset with suboptimal annotations and perform preprocessing.
[0041] It should be understood that the medical image data collected in the embodiments of the present application need to cover all age groups from newborns to the elderly to ensure that the model can learn the anatomical structure characteristics of different life cycle stages. Generally speaking, the more widely representative the collected data is, including different ages, races, body shapes, etc., the better the generalization ability of the model. These data should have different sources, and can come from different acquisition centers, scanning centers, and use different image acquisition protocols, etc., so as to simulate various problems that may be encountered in practical applications. The more medical image data collected, the richer the model can learn and the more robust it can be.
[0042] After the medical image data is collected, preprocessing should be performed, such as intensity normalization, contrast adjustment and other preprocessing. Intensity normalization refers to adjusting the intensity value of the image to a uniform range, usually 0 to 255 (for 8-bit images) or 0 to 1, which helps to eliminate the intensity value differences caused by different scanning devices or different scanning parameters, and ensures that the image data is compared and analyzed on the same scale. Contrast adjustment refers to improving the visual effect of the image and enhancing the distinction between different structures in the image. By increasing the contrast, the subtle structures in the image can be made clearer, which helps to highlight the area of interest, such as distinguishing different tissue types in medical images. Contrast adjustment can be achieved through methods such as histogram equalization (adjusting the histogram of the image to make it more evenly distributed) or contrast-limited adaptive histogram equalization.
[0043] Step S12: pre-training a segmentation model based on the large-scale dataset with suboptimal annotations.
[0044] This stage is the first stage of model training. In this stage, the segmentation model needs to be pre-trained using large-scale suboptimally labeled data. The input information of the model is the medical image, and the output information is the segmentation result. The goal of this stage is to pre-train a model in the medical image segmentation task so that it can recognize and segment the regions of interest in the image (such as organs, tumors, etc.).
[0045] Preferably, the segmentation model is pre-trained using unimodal data. Unimodal data (e.g., using only CT scan images) is easier to acquire and process than multimodal data (e.g., combining different types of scan images such as CT images and MRI images), especially on large-scale data sets. Unimodal data can reduce the complexity of model design, making the model easier to train and optimize. This is especially important in the pre-training stage because it allows the model to learn common features with fewer computing resources. Using unimodal data can speed up the pre-training process because the model only needs to focus on learning one type of image features, thereby improving training efficiency. In addition, in the field of medical images, due to the particularity of the domain and scenario, there are usually a large number of unimodal data sets that can be used to train the model to learn rich feature representations.
[0046] The dataset with suboptimal annotation mentioned above refers to an annotation dataset that contains useful information but may have certain errors or incompleteness. In the field of medical image segmentation, the factors causing suboptimal annotation of datasets may be annotation errors, inconsistent annotations, incomplete annotations, insufficient granularity of annotations, or technical limitations. Among them, annotation errors are usually due to human factors, resulting in the annotation of some images not being consistent with the actual situation; inconsistent annotations are usually due to the fact that different annotators may have different recognition of the structure and boundary determination in the image, resulting in inconsistent annotation results; incomplete annotations are due to the fact that the annotations of some images may not cover all the structures or details that need to be segmented; insufficient granularity of annotations means that the annotations are too rough or not accurate enough to the level of precision required for medical image segmentation; technical limitations mean that some subtle anatomical structures may not be accurately captured and labeled.
[0047] It should be understood that despite the above problems with datasets with suboptimal annotations, they still contain valuable information and can be used for pre-training of models. By training on these data, the model can learn basic features and patterns, laying the foundation for further training on higher-quality small-scale fine-labeled data. This approach can improve the robustness of the model to data of different annotation qualities and maximize the performance of the model under limited high-quality data conditions. In addition, pre-training reduces the demand for computing resources because the model converges faster on small-scale data and reduces the number of iterations.
[0048] For example, the segmentation model adopts an encoder-decoder architecture. The encoder gradually reduces the spatial dimension of the image and increases the feature dimension to extract image features; the decoder gradually restores the spatial dimension of the image and reduces the feature dimension to achieve the reverse conversion from feature to segmentation map. A suitable loss function is selected to measure the difference between the predicted segmentation map and the true segmentation map.
[0049] The training process includes: During the forward propagation process, the medical image is input into the model, the features are extracted through the encoder, and then the preliminary segmentation results are generated through the decoder. The loss value between the predicted segmentation map and the true segmentation map is calculated using the defined loss function. During the back propagation process, according to the loss value, the gradient of the loss relative to the model parameters is calculated through the back propagation algorithm. The parameters of the model are updated using gradient descent or its variants (such as the Adam optimizer) to reduce the loss. Repeat the above forward propagation, loss calculation, back propagation, and parameter update steps until the performance of the model on the validation set can no longer be significantly improved or the predetermined number of iterations is reached.
[0050] Preferably, considering that in medical image analysis, the dataset is unbalanced due to differences in disease incidence, data collection difficulty, annotation cost, data sharing restrictions, image quality and availability, etc., it is preferred to use the Dice loss function to deal with the problem of data imbalance. Among them, the disease incidence refers to the low incidence of certain diseases or pathological conditions in the overall population, resulting in less relevant image data. The difficulty of data collection refers to the fact that some pathological conditions may be difficult to capture by conventional means, or require special equipment and conditions for scanning, which limits the acquisition of relevant data. The annotation cost refers to the fact that annotation of medical images usually requires professional medical knowledge and experience. For some complex pathological conditions, the annotation cost is high, resulting in limited acquisition of annotated data. Data sharing restrictions refer to the fact that due to privacy and ethical issues, some medical image data may not be shared among researchers, limiting the scale and diversity of the dataset. Image quality and availability refer to the fact that high-quality medical images usually need to be obtained under strictly controlled conditions, and these conditions may be difficult to achieve in actual clinical practice, resulting in a limited number of high-quality images available. All these may lead to the problem of unbalanced datasets.
[0051] The Dice loss function effectively solves the data imbalance problem in medical image segmentation by measuring the overlap between the predicted segmentation and the true segmentation. It does not rely on the frequency of the category, but focuses on the accuracy of the segmentation, thereby reducing the excessive influence of the majority category on the loss calculation, enhancing the detection ability of the minority category, especially small objects, so that the model can learn the features of all categories more fairly in the imbalanced data set, and improve the overall segmentation performance.
[0052] Preferably, considering that in medical image analysis, due to differences in the obviousness of pathological features, image quality, severity of lesions, type and location of lesions, rarity of diseases, etc., leading to class imbalance, it is preferred to use Focal loss function to deal with the problem of class imbalance. Among them, the obviousness of pathological features means that some lesions or abnormalities have obvious visual features, making them easy to identify and classify, while other lesions may be more subtle and difficult to distinguish in the image. Image quality refers to the quality factors such as image clarity, contrast and resolution that affect the difficulty of classifying samples. Samples in high-quality images are usually easier to classify. The severity of the lesion means that mild or early lesions may not be obvious in the image and are therefore difficult to classify, while severe lesions are easier to identify due to their significant features. The type and location of the lesion means that some types of lesions may be easier to identify because of their unique morphology or specific locations in the image, while other types of lesions may be difficult to classify because of their unclear location or atypical morphology. The rarity of the disease means that image samples of rare diseases often become difficult to classify samples due to the lack of sufficient training data, because the model does not have enough opportunities to learn its features.
[0053] The Focal loss function solves the technical problem of class imbalance by increasing the weight of samples that are difficult to classify and reducing the weight of samples that are easy to classify. It pays special attention to samples that are difficult to classify correctly. In this way, the model can focus more on learning and improving the recognition ability of minority classes or difficult to identify samples, thereby improving the overall classification accuracy and generalization ability of the model. This method is particularly suitable for medical image analysis, where certain pathological conditions may be rare and difficult to identify.
[0054] Step S13: Acquire text information related to the image morphology, and use the text information as knowledge prompts to retrain the pre-trained segmentation model, so that the segmentation model can perform full life cycle segmentation on medical images.
[0055] It should be understood that in order to enable the segmentation model to adaptively learn medical images from different age groups and different acquisition centers, the embodiment of the present invention proposes to use text information that is highly correlated with the image morphology (such as scanning equipment, scanning parameters, subject age and gender, etc.) as prompts to guide the learning of the model.
[0056] Figure 2 The sub-steps performed to implement the technical content of the above step S13 are shown:
[0057] Step S131: Generate a corresponding descriptive sentence based on text information related to the image morphology; the text information related to the image morphology includes scanning device information, scanning parameter information, and subject age and gender information.
[0058] It should be understood that in medical image analysis, "image morphology" generally refers to the shape, size, and appearance of the anatomical structure displayed in the image. Different scanning devices (such as CT, MRI, X-ray, etc.) and scanning parameters (such as resolution, contrast, scanning layer thickness, etc.) will affect the quality and characteristics of the image; by generating descriptive sentences from these text information and inputting them into the segmentation model, it can help the segmentation model understand how the image was acquired, so as to better interpret the features in the image. The age and gender of the subjects are important factors affecting the anatomical structure and physiological characteristics of the human body. Individuals of different ages and genders may have significant differences in anatomical structure. For example, the bones of children and adults are different in morphology and maturity; by generating descriptive sentences from these text information and inputting them into the segmentation model, it can help the model learn these differences and take them into account when analyzing images.
[0059] Exemplarily, the method of converting text information into a descriptive sentence includes: designing a template for a descriptive sentence, wherein the template includes placeholders for filling in information; filling the collected text information related to the image morphology into the placeholders in the template to generate a descriptive sentence; and optimizing the descriptive sentence using natural language processing technology.
[0060] A template for a descriptive sentence, such as "The scan was performed using [device type], the parameters were set to [parameter list], and the subject was [age] years old [sex]". If the collected text information is a CT scan with a resolution of 1.0×1.0×1.0, and the subject is a 45-year-old female, then the descriptive sentence obtained after filling in the template can be "The scan was performed using CT, the resolution was set to 1.0×1.0×1.0, and the subject was a 45-year-old female". Subsequently, natural language processing (NLP) techniques are used to optimize the fluency and readability of the sentence, including grammar checking, synonym replacement, sentence reconstruction, etc.
[0061] Step S132: input the descriptive sentence into a trained text encoder to extract text features; the extracted text features capture background information related to the image.
[0062] Exemplarily, these descriptive sentences are input into a trained text encoder (Biomed CLIP) to extract corresponding text features. These text features capture important background information related to the image and provide additional context for the segmentation model. The trained text encoder is a deep learning model that has been trained on a large amount of text data so that it can understand and process natural language. The feature extraction process of the text encoder is usually to process these sentences through its internal neural network structure (such as a transformer or a recurrent neural network). The text encoder can capture the semantic information and contextual relationships in the sentence and convert the natural language into a fixed-length vector form, namely, text features.
[0063] Step S133: The extracted text features are fused with the image features and then input into the pre-trained segmentation model for retraining.
[0064] In order to embed the extracted text features into the segmentation model and fuse them with the image features, an embodiment of the present invention introduces text features into the deep encoder of the segmentation model, that is, the text features are spliced with the image features in the sequence length dimension after dimensional transformation as the input features of the segmentation model.
[0065] Exemplarily, the method of fusing text features with image features includes: transforming the dimension of the text features to match the image features through dimensional transformation; splicing the matched text features and image features in the sequence length dimension to form the input features of the segmentation model. Among them, since text features and image features usually have different dimensions, they need to be properly transformed to ensure that the two can be effectively combined. For example, text features are fixed-length vectors extracted from a text editor, while image features can be multi-dimensional arrays (such as feature maps extracted from a convolutional neural network). Therefore, the dimension of the text features can be increased by copying or extending operations to make them consistent with the dimension of the image features.
[0066] "Retraining" in the embodiments of the present invention is a process of fine-tuning the model through knowledge prompts. In the fine-tuning stage, the weights of the model obtained by pre-training are used to initialize the model, freeze the encoder parameters, and only update the learnable parameters and decoder parameters, thereby significantly improving the convergence speed of the model.
[0067] Specifically, before the fine-tuning phase begins, the current model is initialized using the model weights pre-trained on a large dataset. The encoder is the part of the model responsible for feature extraction. During the fine-tuning phase, the encoder's parameters are frozen, that is, their values are not updated during the training process. The purpose of this is to retain the useful features that the encoder has learned. Learnable parameters usually refer to those parameters in the model that can be updated during the training process, such as the parameters in the decoder. The decoder is another part of the model that is responsible for converting the output of the encoder into the final prediction result, such as a segmentation map. Since the encoder parameters are frozen, the model only needs to update the decoder and a few learnable parameters during the fine-tuning phase, which means that the model's training process can converge faster because the number of parameters that need to be learned is reduced. By only updating the decoder and learnable parameters, the model can adapt to the characteristics of a specific dataset more quickly without having to learn all the features from scratch, thus saving computing resources and significantly improving the convergence speed of the model.
[0068] It should be understood that in image segmentation tasks, convolutional neural networks are usually used as encoders to extract image features. These networks gradually extract abstract features of the image through multiple layers of convolutional layers and pooling layers. "Deep encoders" refer to multiple levels of convolution and pooling operations in the network, and each layer extracts different levels of features of the image, from shallow edge and texture information to deep and more abstract semantic information.
[0069] In summary, by introducing text features as prompts, the segmentation model of medical images can not only utilize the features of the image itself, but also combine the knowledge contained in the text information, so that it is more targeted and effective when extracting useful features. Furthermore, this application can significantly improve the accuracy and efficiency of medical image segmentation throughout the life cycle, providing strong support for medical image analysis and clinical applications.
[0070] Figure 3 A specific embodiment of the present invention is shown, which is intended to perform image segmentation on a brain magnetic resonance image of a 50-year-old male, and the implementation process is as follows:
[0071] 1. Data preprocessing.
[0072] 1.1) Collect medical image data from different life cycle stages and different collection centers, including medical image data of newborns, children, adolescents, adults and the elderly.
[0073] 1.2) Preprocess the collected medical image data, such as intensity normalization and other operations.
[0074] 2. Model training.
[0075] 2.1) The first stage: pre-training the segmentation model. In this stage, the segmentation model is pre-trained using large-scale data with suboptimal annotations. The input information of the model is the medical image, and the output information is the segmentation result. The segmentation loss function (such as Dice loss and Focal loss, etc.) is used to supervise the learning of the network. Through this stage, the model can preliminarily learn the basic features and segmentation rules of medical images.
[0076] 2.2) The second stage: fine-tuning the pre-trained model. Fine-tune the pre-trained model on a relatively small scale of manually annotated data. In order to improve the segmentation performance of the model at different age stages and under dynamic changes in morphology, and to speed up the training speed, the present invention introduces knowledge-driven learnable parameters in the input space. The specific steps are as follows:
[0077] Knowledge hint generation: Using biological attributes that are highly correlated with the segmented target morphology (such as age, gender, health status, etc.), embed this text information into the template to generate a descriptive sentence. For example: "This is a brain magnetic resonance image obtained from a 50-year-old male in good health." The generated sentence is input into a pre-trained text encoder (such as BiomedCLIP) to extract text features.
[0078] Learnable parameter initialization: In order to effectively embed text information into the model, multiple learnable parameters are predefined and initialized using the extracted text features. These parameters are called “knowledge hints”.
[0079] Fine-tune the model with knowledge hints: Introduce knowledge hints as input information at the deep level of the model's encoder. In order to align image features with text features, the text features are transformed in dimension and then concatenated with the image features in the sequence length dimension as the input of the model. The output of the model is the segmentation label, which is constrained by the segmentation loss function (such as Dice loss and Focal loss). In the fine-tuning stage, the model is initialized with the weights of the pre-trained model, the encoder parameters are frozen, and only the learnable parameters and decoder parameters are updated, which significantly improves the convergence speed of the model.
[0080] It should be noted that, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit the difference. Words such as "exemplary" or "for example" represent examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0081] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, ab, ac, bc or abc, where a, b, c can be single or multiple.
[0082] Figure 4 4 is a schematic diagram of the structure of a medical image segmentation device based on the full life cycle of deep learning provided in an embodiment of the present application. The medical image segmentation device 400 includes a data acquisition module 401, a first-stage training module 402, and a second-stage training module 403.
[0083] The data acquisition module 401 is used to collect medical image data of different life cycle stages and different sources, form a large-scale suboptimally annotated data set and perform preprocessing. The first-stage training module 402 is used to pre-train the segmentation model based on a large-scale suboptimally annotated data set. The second-stage training module 403 is used to obtain text information related to image morphology, and use the text information as knowledge prompts to retrain the pre-trained segmentation model, so that the segmentation model can perform full life cycle segmentation on medical images.
[0084] It should be understood that the specific process of each module executing the above corresponding steps has been described in detail in the above method embodiment, and for the sake of brevity, it will not be repeated here.
[0085] It should also be understood that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.
[0086] Figure 5 is a schematic block diagram of a computer device provided in an embodiment of the present application. Figure 5As shown in FIG. 5 , the computer device includes: at least one processor 501, a memory 502, at least one network interface 503, and a user interface 505. The various components in the device are coupled together via a bus system 504. It is understood that the bus system 504 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 504 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 504 is not described in detail. Figure 5 In the specification, various buses are labeled as bus systems.
[0087] The user interface 505 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.
[0088] It is understood that the memory 502 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.
[0089] The memory 502 in the embodiment of the present invention is used to store various types of data to support the operation of the electronic terminal 500. Examples of these data include: any executable program used to operate on the electronic terminal 500, such as an operating system 5021 and an application 5022; the operating system 5021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 5022 can include various applications, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. The XX method provided by the embodiment of the present invention can be included in the application 5022.
[0090] The method disclosed in the above embodiment of the present invention can be applied to the processor 501, or implemented by the processor 501. The processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 501 or the instruction in the form of software. The above processor 501 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 501 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 501 may be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0091] In an exemplary embodiment, the electronic terminal 500 may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.
[0092] According to the method provided in the embodiments of the present application, the present application also provides a computer program product, which includes: computer program code, when the computer program code runs on a computer, the computer executes the medical image segmentation method based on the full life cycle of deep learning of the embodiment.
[0093] According to the method provided in the embodiment of the present application, the present application also provides a computer-readable storage medium, which stores a program code. When the program code runs on a computer, the computer executes the medical image segmentation method based on the full life cycle of deep learning of the embodiment.
[0094] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program and / or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may, for example, communicate through local and / or remote processes according to signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).
[0095] Those of ordinary skill in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0096] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0097] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0098] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0099] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0100] In the above embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When loading and executing computer program instructions (programs) on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integrations. Available media may be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., high-density digital video discs (DVDs), or semiconductor media (e.g., solid state disks (SSDs), etc.).
[0101] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.
[0102] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0103] In summary, the present application provides a medical image segmentation method, device, equipment, medium and program product based on the full life cycle of deep learning. The present application combines image features with text features, so that the model can more comprehensively understand and analyze the image content when processing medical images, thereby improving the accuracy and efficiency of the segmentation task. This method is particularly suitable for processing medical image data from diverse sources and with complex backgrounds. Specifically, the present application can improve the segmentation accuracy: in this way, the model not only utilizes the features of the image itself, but also combines the knowledge contained in the text information, making the feature extraction more targeted and effective; full life cycle coverage: this method significantly improves the accuracy and efficiency of medical image segmentation throughout the life cycle, and can adapt to image data of different age stages and different collection centers; clinical application support: provides strong support for medical image analysis and clinical applications, and improves the practicality and reliability of the model in actual medical scenarios. Therefore, the present application effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.
[0104] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.
Claims
1. A medical image segmentation method based on deep learning full life cycle, characterized in that: include: Collect medical image data from different life cycle stages and different sources to form large-scale datasets with suboptimal annotations and perform preprocessing; Pre-training a segmentation model based on the large-scale suboptimally annotated dataset; Text information related to the image morphology is obtained, and the pre-trained segmentation model is retrained using the text information as a knowledge prompt, so that the segmentation model can perform full life cycle segmentation on the medical image.
2. The medical image segmentation method based on deep learning full life cycle according to claim 1 is characterized in that: The medical image segmentation method includes pre-training a segmentation model using single-modality data.
3. The medical image segmentation method based on deep learning full life cycle according to claim 1 is characterized in that: The segmentation model adopts an encoder-decoder architecture; the encoder is used to gradually reduce the spatial dimension of the image and increase the feature dimension to extract image features; The decoder is used to progressively restore the spatial dimensions of the image and reduce the feature dimensions to perform the inverse transformation from features to segmentation maps.
4. The medical image segmentation method based on deep learning full life cycle according to claim 1, characterized in that: The step of acquiring text information related to the image morphology and retraining the pre-trained segmentation model using the text information as a knowledge prompt includes: Generate a corresponding descriptive sentence based on text information related to the image morphology; the text information related to the image morphology includes scanning device information, scanning parameter information, and subject age and gender information; Inputting the descriptive sentence into a trained text encoder to extract text features; the extracted text features capture background information related to the image; The extracted text features are fused with the image features and then input into the pre-trained segmentation model for retraining.
5. The medical image segmentation method based on deep learning full life cycle according to claim 4 is characterized in that: The method of converting text information into a descriptive sentence includes: designing a template for a descriptive sentence, wherein the template includes a placeholder for filling in information; filling the collected text information related to the image morphology into the placeholder in the template to generate a descriptive sentence; and optimizing the descriptive sentence using natural language processing technology.
6. The medical image segmentation method based on deep learning full life cycle according to claim 4 is characterized in that: The method of fusing the extracted text features with the image features includes: transforming the dimensions of the text features to match the image features through dimensional transformation; and splicing the matched text features and image features in the sequence length dimension to form input features of the segmentation model.
7. The medical image segmentation method based on deep learning full life cycle according to claim 1, characterized in that: The method of retraining the segmentation model includes: using the weights of the model obtained by pre-training to initialize the model; freezing the encoder parameters in the segmentation model, and only updating the learnable parameters and the decoder parameters.
8. A medical image segmentation device based on deep learning full life cycle, characterized in that: include: The data acquisition module is used to collect medical image data from different life cycle stages and different sources, form a large-scale dataset with suboptimal annotations, and perform preprocessing; The first stage training module is used to pre-train the segmentation model based on a large-scale dataset with suboptimal annotations; The second stage training module is used to obtain text information related to image morphology and use the text information as knowledge prompts to retrain the pre-trained segmentation model so that the segmentation model can perform full life cycle segmentation of medical images.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the medical image segmentation method based on deep learning full life cycle as described in any one of claims 1 to 7 is implemented.
10. A computer program product, characterized in that The computer program product includes a computer program code, and when the computer program code runs on a computer, the computer implements the medical image segmentation method based on deep learning full life cycle as described in any one of claims 1 to 7.
11. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the medical image segmentation method based on deep learning full life cycle as described in any one of claims 1 to 7.
Citation Information
Cited By
Image processing system for organ delineation
TWI918337B
