Cardiac fat segmentation method based on language cues

Abstract

The present application relates to the technical field of medical data analysis, and discloses a heart fat segmentation method based on language prompts, step S1: acquiring a multi-modal image, and extracting heart fat image features based on a visual model; step S2: performing text description on the label, extracting features of the segmentation intention of the text description by using an existing large language model, and converting the features into a task prompt vector that can be understood by a subsequent module; and step S3: constructing a decoding network, in which the input heart fat image Embedding sequence is comprehensively decoded according to the requirements of the task prompt vector to obtain a segmentation result, mainly solving the problems of insufficient heart fat data, shape uncertainty, small volume, and close brightness of the surrounding tissue, and having the advantages of reducing cost, improving performance, high flexibility, processing incomplete data, and reusing pre-training.

Description

Technical Field

[0001] This invention relates to the field of medical data analysis technology, and more specifically, to a method for cardiac fat segmentation based on verbal cues. Background Technology

[0002] Cardiac fat comprises epicardial adipose tissue (EAT) and pericardial adipose tissue (PAT). The pericardium is a protective layer surrounding the heart, consisting of an inner serous pericardium and an outer fibrous pericardium. The inner serous pericardium can be further divided into visceral and parietal layers. EAT is the adipose tissue surrounding the visceral layer of the pericardium, while PAT is the adipose tissue surrounding the parietal layer. The size and shape of cardiac fat are closely related to various cardiovascular diseases. Since cardiovascular disease accounts for one-third of all deaths annually, clinical research on cardiovascular diseases has significant social value. EAT is closer to the heart than PAT, and there is evidence that it plays a direct role in the development of coronary atherosclerosis and cardiomyopathy. Its thickness has been shown to be associated with metabolic syndrome and coronary artery disease, as well as the progression of coronary artery calcification. Its volume and density are associated with major adverse cardiac events in asymptomatic subjects. EAT quantity is also considered a good marker of cardiovascular disease. Furthermore, EAT plays a role in visceral obesity, cardiac morphology, liver enzymes, insulin resistance, and fasting blood glucose, making it an accurate treatment target. For PAT, its volume is correlated with atrial fibrillation. Evidence suggests that computer-aided methods for quantifying EAT and PAT, especially on large datasets, can be used to predict major adverse cardiovascular events. This has significant clinical implications for the identification of cardiac fat and the diagnosis and treatment of related diseases. However, cardiac fat segmentation faces many challenges. These include limited data, the complex anatomy of the heart, significant variations in fat shape and uneven distribution, the difficulty in distinguishing between epicardial and pericardial fat tissues due to their similar brightness intensities, the fact that the pericardium is often too thin to be fully visible in CT or MRI images, and the difficulty in defining the boundaries of different parts of the heart. Therefore, many problems remain to be solved for the automatic and accurate segmentation of cardiac fat, and manual or semi-automatic segmentation is often used in clinical practice.

[0003] Traditional cardiac image segmentation typically requires extensive domain knowledge and manual intervention. Furthermore, different methods are employed depending on the specific image being segmented, sometimes requiring a combination of methods. These methods can be broadly categorized into three main types: pixel-based clustering, graph theory-based methods, and deep semantic segmentation. Pixel-based clustering methods generally include thresholding (based on single numerical values ​​such as image brightness), clustering (classifying images into predefined categories based on extracted feature similarity, such as k-means, spectral clustering, Meanshift, and SLIC), histograms (based on statistical methods), and edge detection (utilizing domain prior knowledge). Graph theory-based segmentation is represented by algorithms such as NormalizedCut, GraphCut, and GrabCut. Deep semantic segmentation encompasses various algorithms developed alongside deep learning in recent years, from early algorithms like FCN, DeepLab, and PSPNet to currently popular algorithms like UNet, SegNet, and ResUNet. With the emergence of Transformers and large-scale models, Transformers are increasingly used for complex segmentation; cardiac fat segmentation is a subfield of image segmentation. Cardiac anatomy is complex, and cardiac fat segmentation tasks are similar for myocardial segmentation and whole-cardiac substructure segmentation. Most cardiac fat segmentation is based on CT images; segmentation from MRI is more difficult than from CT because the pericardium is often blurred due to partial volume effects. The vast majority of existing EAT and PAT segmentation methods focus on quantifying EAT from CT images.

[0004] Defects and shortcomings of existing technology:

[0005] Traditional methods suitable for cardiac fat segmentation include thresholding segmentation, region filling algorithms, adaptive contour segmentation, and atlas-based methods (which use pre-annotated atlases and registration techniques to register the atlas with the image to be segmented, thus obtaining the segmentation result). Deep learning-based registration can achieve relatively fast segmentation. However, due to the complexity of fat, the data distribution of atlases may differ significantly from that of real patients, leading to this method being gradually replaced by other segmentation approaches. Another method uses the pericardium to assist in cardiac fat segmentation. The pericardium has a smooth, thin, almost elliptical contour; once identified, the extent of cardiac fat can be easily defined. The challenge of this method is that the pericardium is very thin, typically less than 2 mm, making it difficult to identify in images. In cardiac fat segmentation, frameworks like UNet are most commonly used, and data augmentation is employed to artificially increase the amount of data. It is worth noting that the severe imbalance between the proportion of heart fat and background, i.e., the number of heart fat pixels is much smaller than the number of background pixels, can cause problems during CNN training. Some recent works have used focus loss to improve this problem, but these problems are still not satisfactory in the case of heart fat.

[0006] Furthermore, since cardiac MRI scans are multi-slice scans with large inter-slice spacing and significant variations in image structure between slices (anisotropy), 2.5DU-Net is currently widely used for segmentation tasks involving anisotropic data to reduce additional computational costs. However, for this task, 2.5DU-Net's performance is not ideal. This is due not only to the large inter-slice differences but also to insufficient annotation, which prevents the model from effectively learning the inter-slice correlation information. Because the shape of cardiac fat is uncertain, as is the shape of the pericardial effusion adjacent to it, and because these tissues exhibit very similar brightness on MRI images, it is difficult to effectively distinguish these structures without prior constraints, such as anatomical, geometric, or topological constraints.

[0007] Secondly, medical data in this field is extremely scarce; there are currently no publicly available datasets for MRI pericardial fat segmentation. Data sets for other cardiac outcomes are also limited. This makes it difficult for existing techniques to achieve satisfactory results on such small datasets. If different datasets are used, besides differences in dataset modality and format, even the necessary labels may be unavailable; for example, no publicly available datasets for cardiac fat can currently be found.

[0008] Finally, existing models lack support for training on various mixed data, such as missing labels and immutable label categories. Traditional classification / segmentation requires a fixed number of categories beforehand, but with our ever-expanding data, the number of categories is constantly increasing. Furthermore, some information between categories is difficult to express using traditional methods, such as "myocardial ring" and "scars on the myocardial ring," the latter being clearly a subset of the former; or "ventricle / left ventricle / right ventricle," the former obviously encompassing the latter two labels. Additionally, traditional models require training data to contain complete labels (i.e., no missing labels), which is impossible across different datasets; often, each dataset only has partial labels, leading to label inconsistencies between datasets. To address the above issues, we propose using a large language model to encode each task. Instead of using a large language model to encode each label, we convert each label into a text description and then use the existing large language model to extract features of the segmentation intent (text prompt) described in the text. This text prompt is then converted into a task prompt vector that subsequent modules can understand. Combined with the feature extraction from the previously pre-trained model, we obtain image features (image embedding) and text features (prompt), which are then fed into the segmentation network to obtain the segmentation result, thus solving the above problems. Because the large language model can capture the relevant features between the input text, it has a stronger expressive ability for the prompts we want to segment. In addition, in the segmentation network, each prompt corresponds to only one result, which indirectly solves the problem of incomplete labels for some data. Experiments show that this approach is more effective. In specific training, each segmentation task is described in text form. Each dataset has more or less different tasks, and these tasks may overlap. Text descriptions can well express this information and segmentation intent. For example, tasks such as "Please segment the left ventricle"; "Please segment the left ventricle and its scar"; "Please segment the scar inside the left ventricle"; and "Please segment the left and right ventricles" are provided. In addition, we designed multiple model training tasks to address the problem that one-dimensional models often only utilize the labeled data, maximizing the use of information from the metadata dataset.

[0009] In view of this, the present invention provides a method for cardiac fat segmentation based on language prompts. Summary of the Invention

[0010] To overcome the problems in the prior art, this invention proposes a language-cue-based cardiac fat segmentation method, which combines a large model and a large number of cardiac datasets to complete the segmentation of epicardial fat.

[0011] To address the aforementioned problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a method for cardiac fat segmentation based on language prompts, comprising the following steps:

[0012] Step S1: Acquire multimodal images and extract cardiac fat image features based on a visual model;

[0013] Step S2: Describe the annotations in text, extract the segmentation intent features of the text description using the existing large language model, and convert them into task prompt vectors that can be understood by subsequent modules;

[0014] Step S3: Construct a decoding network. In the decoding network, the input cardiac fat image embedding sequence is comprehensively decoded according to the requirements of the task prompt vector to obtain the segmentation result.

[0015] As a preferred embodiment of the present invention, the application logic of the visual model is as follows:

[0016] Step S11: Process the multimodal image and the modality type information corresponding to the multimodal image to generate an embedding sequence;

[0017] Step S12: Use the Transformer architecture to extract features from the Embedding sequence to obtain cardiac fat image features, wherein the cardiac fat image features are the Embedding sequence.

[0018] As a preferred embodiment of the present invention, the logic for obtaining the Embedding sequence is as follows:

[0019] Converting the image into an embedding sequence: The multimodal image is segmented into modal patches of fixed size, and each modal patch is mapped to an image embedding sequence;

[0020] Convert modal types to their corresponding modal embeddings: Modal types are encoded using integers, and the corresponding embeddings are used as modal embeddings.

[0021] Assemble Embedding sequences with different information: Concatenate the image embedding sequence and the modal embedding sequence to generate an embedding sequence.

[0022] As a preferred embodiment of the present invention, the process of mapping an embedding to a corresponding feature using a self-attention mechanism is as follows. First, a linear mapping module is used to map each embedding to three sets of vectors: Q, K, and V. Here, Q is equivalent to the eye used for searching, K is equivalent to the representation of the current embedding, and V is the corresponding feature.

[0023] As a preferred embodiment of the present invention, the application logic of the language model is as follows:

[0024] Each label is converted into a corresponding text description, and the text features of the segmentation intent of the text description are extracted using a language model.

[0025] After obtaining the text features, we need to convert them into task prompt vectors, which output the text features as fixed lengths.

[0026] As a preferred embodiment of the present invention, the decoding processing logic includes the following steps:

[0027] Step S31: Based on the cardiac fat image features extracted in Step S1 and the task cue vector extracted in Step S2, label the embedding sequence corresponding to the cardiac fat image features as sequence A, and label the embedding sequence corresponding to the task cue vector as sequence B.

[0028] Step S32: Simultaneously feed sequence A and sequence B into the bidirectional attention network to obtain the perception results of sequence A and sequence B respectively;

[0029] Step S33: The A sequence perception result is processed through multiple deconvolution layers to generate multiple candidate images. The B sequence perception result is processed through the B-to-A cross-attention Transformer block. The multilayer perceptron obtains the weights of these candidate images. Finally, the multiple images are weighted according to these weights to obtain the segmentation result.

[0030] Secondly, the present invention provides a model training process for a language-cue-based cardiac fat segmentation method, characterized by comprising the following steps:

[0031] Step X1: Data collection and processing;

[0032] Step X2: Merge multiple datasets to generate a training dataset;

[0033] Step X3: Extract the training task from the merged dataset during training;

[0034] Step X4: Encode the visual model of the input image and modality information to obtain the image embedding sequence.

[0035] Step X5: Calculate the contrast similarity. Since model training is performed simultaneously on a batch of images, assume each batch has bs training data points;

[0036] Step X6: Calculate the reconstruction similarity, where:

[0037] Step X61: Subtask 1: Calculate the model's ability to reconstruct images;

[0038] Step X62: Subtask 2: Calculate the model's modal transfer capability;

[0039] Step X63: Subtask 3: Calculate the model's ability to segment images.

[0040] Thirdly, the present invention provides an electronic device, characterized in that it comprises:

[0041] At least one processor; and

[0042] A memory that is communicatively connected to the at least one processor;

[0043] The memory stores instructions that can be executed by the at least one processor to cause the electronic device to perform the language-cue-based cardiac fat segmentation method.

[0044] Fourthly, the present invention provides a storage medium storing a computer program thereon, characterized in that, when the computer program is executed by a processor, it implements the aforementioned language-guided cardiac fat segmentation method.

[0045] The technical effects and advantages of the language-cued cardiac fat segmentation method of this invention are as follows: This invention utilizes a large amount of existing data from other medical fields, combined with a small amount of cardiac fat data, and employs the learning approach of large models and a large amount of data for training. It mainly solves the problems of insufficient cardiac fat data, its shape uncertainty, small size, and similar brightness to surrounding tissues. It has advantages such as reduced cost, improved performance, high flexibility, processing of incomplete data, and reuse of pre-training. Attached Figure Description

[0046] Figure 1 The flowchart and model processing logic of the cardiac fat segmentation method based on language prompts of this invention are shown below;

[0047] Figure 2 This is a schematic diagram of the attention mechanism related to the visual model processing logic of the present invention;

[0048] Figure 3 This is a logical diagram showing the relationship between the various parts of the training dataset for the language-cue-based cardiac fat segmentation method of this invention.

[0049] Figure 4 This is a flowchart illustrating the training logic and training task of the cardiac fat segmentation method based on language prompts of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] Example 1

[0052] Please see Figure 1-2 As shown in this embodiment, the cardiac fat segmentation method based on language prompts includes the following steps:

[0053] Step S1: Acquire multimodal images and extract cardiac fat image features based on a visual model;

[0054] It should be noted that the multimodal images include MRI images, CT images, and ultrasound images. MRI images are further subdivided into T1, T2, and other modalities. Different types of datasets have different image modalities. For example, images used for cardiac myocardial ring and ventricular segmentation are mainly obtained from MRI images, while images used for cardiac fat segmentation are mainly from CT image datasets. To fully utilize these valuable datasets, we need models that can process images of different modalities simultaneously. Images of different modalities have different spatial resolutions, pixel intensities, and anatomical structural information representations, each with its own characteristics. Therefore, a multimodal visual model is needed to adapt to the input of multimodal images, understand the content of the input images, and unify the encoding to prepare for subsequent segmentation.

[0055] The application logic of the visual model is as follows:

[0056] Step S11: Process the multimodal image and the modality type information corresponding to the multimodal image to generate an embedding sequence;

[0057] It should be noted that although images already contain modal information, images of different modalities may have some similarities, but the information they contain may be completely different. Therefore, specifying the modal type allows subsequent steps to encode specific modalities more specifically, which can reduce the difficulty of image recognition for the system and improve the ability of the visual model to quickly recognize the feature information carried by multimodal images. Since subsequent processing requires the use of embedding sequences, the goal of this step is to convert the image and its corresponding modal type into embedding sequences for use in subsequent steps.

[0058] The logic for obtaining the Embedding sequence is as follows:

[0059] The image is converted into an embedding sequence: the multimodal image is segmented into modal patches of fixed size, and each modal patch is mapped to an image embedding sequence; this mapping process is performed using a multilayer perceptron.

[0060] Convert modality types to corresponding modality embeddings: Modality types are encoded using integers, such as 1 for CT and 2 for MRIT1; each integer corresponds to an embedding vector, that is, different modality types correspond to different embedding vectors, and there are as many different embedding vectors as there are modalities; when it is necessary to convert a modality type to a modality embedding, the embedding corresponding to the modality is found as the modality embedding.

[0061] Assemble embedding sequences with different information. Concatenate the image embedding sequence and the modal embedding sequence to generate an embedding sequence. Then, add a positional code to each embedding in this sequence, finally obtaining an embedding sequence that can be processed by the Transformer.

[0062] Step S12: Use the Transformer architecture to extract features from the Embedding sequence and convert the features into an Embedding sequence for easier subsequent computation;

[0063] It should be noted that a "self-attention" mechanism is used here to realize the transformation from embedding to features. The Transformer block uses multiple attention mechanisms to obtain a set of corresponding features, and then uses a linear mapping module to restore this set of feature sequences to the embedding sequence. Multiple Transformer blocks are chained together to form the Transformer encoder, and the embedding sequence output by the last block is the encoded result.

[0064] The process of mapping an embedding to its corresponding feature using a self-attention mechanism is as follows: First, a linear mapping module is used to map each embedding to three vectors: Q, K, and V. Q acts as the eye for searching, K represents the current embedding, and V represents the corresponding feature. Then, Q is used to calculate the distance between it and the K of all embeddings to determine the relationship (closeness). If they are close, the current embedding is considered to have a closer relationship with its corresponding feature (V), and the corresponding feature weight is given a higher weight. We can then calculate the weights based on the distances and sum these features (V) to obtain the transformed feature encoding of the current embedding. The same operation is performed on each embedding in the series to obtain the transformed feature sequence. Multi-head attention uses the self-attention mechanism multiple times, each time using a different method (weight), essentially providing multiple opportunities for feature detection, each time detecting a different feature sequence. Therefore, as... Figure 2 As shown, the result of multi-head attention is to transform the embedding sequence into a set of feature sequences.

[0065] In addition, since the input includes the embedding corresponding to the modality type, the modality type is considered at every step in this feature extraction process.

[0066] Step S2: Describe the annotations in text, extract the segmentation intent features of the text description using the existing large language model, and convert them into task prompt vectors that can be understood by subsequent modules.

[0067] The application logic of the language model is as follows:

[0068] Each label is converted into a corresponding text description, and a language model is used to extract the text features of the segmentation intent of the text description. Text description is chosen to replace the traditional segmentation representation in order to better integrate various different datasets for training.

[0069] After obtaining the text features, we need to convert them into task prompt vectors. For this conversion, we use a small network such as an RNN or MLP to learn how to extract task prompt vectors from the text features. The input is the text features, and the output is a fixed-length prompt vector.

[0070] It's important to note that in this step, it's unnecessary to train a separate text feature extraction network. Currently, training large language models is very costly, and many pre-trained large language models are available in academia, such as LLaMA and T5. These models can be used directly or fine-tuned to extract text features. Furthermore, since our goal is not to use a specific large language model but to extract text features, a pre-trained text encoder model, such as BERT, can also be used for feature extraction.

[0071] Step S3: Construct a decoding network. In the decoding network, the input cardiac fat image embedding sequence is comprehensively decoded according to the requirements of the task prompt vector, and the final output is the segmentation target data.

[0072] It's important to note that the decoding network consists of a bidirectional attention network and a fusion output network. The preceding steps encode the image and task cue vector separately, generating separate embedding sequences without considering their relationship. The bidirectional attention network addresses this by considering the relationship between these two embedding sequences for further decoding. This utilizes "cross-attention," which, unlike "self-attention," takes two embedding sequences as input. Assuming these sequences are A and B, the cross-attention process from A to B is as follows: a linear mapping is used to map A to Q. A Map B to K B V B Then put Q A K B V B The output features are calculated in the same way as Q, K, and V in a self-attention mechanism. Multi-head cross-attention applies cross-attention multiple times with different weights to obtain a corresponding set of features. It can be seen that if A and B are the same sequence, then it is equivalent to self-attention. The block that maps features back to embeddings using a multi-head cross-attention mechanism and linear mapping is called a cross-attention Transformer block, as shown in the attached figure. Figure 2 The logic of "A's attention to B (cross-attention)".

[0073] The decoding processing logic includes the following steps:

[0074] Step S31: Based on the cardiac fat image features extracted in Step S1 and the task cue vector extracted in Step S2, label the embedding sequence corresponding to the cardiac fat image features as sequence A, and label the embedding sequence corresponding to the task cue vector as sequence B.

[0075] Step S32: Simultaneously feed sequence A and sequence B into the bidirectional attention network to obtain the perception results of sequence A and sequence B respectively;

[0076] Specifically, as an example, the input sequences A and B are sequentially passed through multiple bidirectional attention blocks to obtain new sequences A and B.

[0077] The logic of the bidirectional attention block is as follows: Inputs are A and B, outputs are A* and B*, and A* and B* are used as inputs to the next layer network to achieve concatenation of multiple blocks. For input B, it first passes through a self-attention Transformer block, then through a cross-attention Transformer block (B to A), and finally through a multilayer perceptron to obtain the new B*. For input A, it passes through a cross-attention Transformer block (A to B*) to obtain the new A*.

[0078] Step S33: The A sequence perception result is processed through multiple deconvolution layers to generate multiple candidate images. The B sequence perception result is processed through the B-to-A cross-attention Transformer block. The multilayer perceptron obtains the weights of these candidate images, and finally, the images are weighted according to these weights to obtain the segmentation result.

[0079] Example 2

[0080] Please see Figure 3-4 As shown, the model training process of the cardiac fat segmentation method based on language prompts described in this embodiment includes the following steps:

[0081] The goal of this training is to maximize the use and mining of various information in the existing dataset, and to meet the needs of the continuously growing dataset and increasing annotations, so that the model can have better versatility.

[0082] Step X1: Dataset Collection and Processing. Our datasets come in multiple formats, such as MRI k-space, MATLAB, NII, and JPG. We first extract the images and corresponding metadata from these datasets. The metadata includes information such as image modality, size, and resolution, all of which is converted to JSON format. Based on the metadata, the image layer resolution is uniformly adjusted to 1 mm / pixel using quadratic linear interpolation, while maintaining inter-layer resolution. Images are normalized to 0-1 for different modalities, discarding the highest 1% noise in the histogram and matching the histogram to the global average histogram for that modality, with a background value of 0. Then, the dataset is globally registered to the average template using the Ants registration tool. Finally, the layer images are cropped to 192x192 pixels. Note that different datasets may have significantly different scan orientations; currently, registration is performed according to the scan orientation along five directions: three planes of the body, the long axis of the heart, and the short axis of the heart.

[0083] We defined a unified annotation numbering system, with different annotations corresponding to different numbers. For annotation information, we need to uniformly map the annotation information to the numbers we defined. Then, the annotations need to correspond to an image resolution of 1 mm / pixel. Resolution adjustment uses nearest neighbor elements, and after performing the same transformation as the corresponding image registration, the size is cropped to 192x192.

[0084] After image preprocessing, the images need to be compressed to meet the requirements of the training merged dataset. This process first splits the image and its corresponding annotations (if any, the same applies below) into 2D sequences and converts them to float16 type. The corresponding annotations are then split into one-hot encoded values ​​(even if split according to annotation category, each category has one mask; pixels belonging to that category have a value of 1, and pixels not belonging to that category have a value of 0). Thus, one image will correspond to multiple masks. Because the mask and image are the same size, the mask and image are also saved to the image file, and the position indices of the image and annotations are recorded, as shown in the attached image file. Figure 3 The "Image" in the text.

[0085] The next step is to construct the training task for a single dataset. For each training task, four pieces of information are needed during training: image, modality type, training task type, and model output. Since the model input and output are both in images, they can be represented by integers; the modality type and training task type can also be represented by different integers. Therefore, each training task can be represented as (image modality ID, image index, task type ID, model output image index), as shown in the appendix. Figure 3The term "pair" refers to the fact that when we build the dataset, we describe all the tasks that can be built based on the current image and save them in the pair file.

[0086] Step X2: Merge multiple datasets to generate a training dataset.

[0087] First, we prepare textual descriptions for the training tasks. To improve the model's generalization ability, we prepare multiple similar textual descriptions for each training task. We then use a language model to extract features from these descriptions beforehand. Finally, we save these textual features together into a text-embedding file, as shown in the attached file. Figure 3 The "Task Description" section contains textual features of different descriptions for each task. These features are stored sequentially, and an index is used to store the starting address and number of each task, as shown in the appendix. Figure 3 The "task information" in the middle.

[0088] Next, multiple datasets are merged. Images from multiple datasets are merged into one, as shown in the attached image. Figure 3 The "Image" in the text refers to the image index. Because the image index is renamed during image merging, for task pairs, the corresponding indexes need to be modified before merging, as shown in the attached diagram. Figure 3 The word "pair" in the sentence.

[0089] Step X3: Extract the training task from the merged dataset during training.

[0090] First, before each training session is divided into multiple epochs, the training tasks are randomly shuffled, and task information is extracted in order of batch size. Extracting information from all tasks completes one epoch of training.

[0091] Next, the input and output images are retrieved from the image database based on the index. Random data augmentation is then performed on both the images and annotations simultaneously. Data augmentation methods include: random rotation and flipping, random adjustment of brightness and contrast, and random cropping to 144x144.

[0092] Based on the task ID, randomly select one text feature from the text-embedding (because each task has multiple descriptive features) to correspond to the task's text feature.

[0093] Finally, we extracted the data needed for training: images (144x144), modality types, task description features, and the model's expected output (144x144).

[0094] Step X4: Encode the visual model for the input image and modality information. First, apply S11 of Example 1 to obtain the image embedding sequence.

[0095] Randomly delete 3 / 4 of the Embedding sequence. Then apply step S12 of Example 1.

[0096] Perform the above process twice, resulting in two different sequences A1 and A2, as shown in the appendix. Figure 4 The "multimodal image encoder (during training)" in the text.

[0097] Step X5: Calculate contrastive similarity. Since model training is performed simultaneously on a batch of images, assume each batch contains bs training data points. We take the first embedding from A1 and A2 of each training data point, resulting in two embedding sequences of length bs. Then, we calculate the similarity between each element of these two embedding sequences. Because embeddings at the same position come from the same input image, their features should be similar. Embeddings at different positions come from different images, therefore their features should not be similar. Based on this principle, we can define a corresponding loss function, namely the contrastive similarity loss function.

[0098] Step X6: Calculate the reconstruction similarity, as shown in the attached figure. Figure 4 The "training task" in the text.

[0099] Step X61: Subtask 1: Calculate the model's ability to reconstruct the image. This subtask training process is similar to a guessing game. The input image has most of its contents removed. If the model can predict the original, complete image content, it can be considered that the model truly understands the image information and performs feature encoding. Since the output sequences A1 and A2 of X4 are only 1 / 4 of the original, for subsequent calculations, we use a special embedding to fill in the originally deleted positions in A1 and A2, obtaining a complete sequence. This special embedding simply tells the model that the feature at this position has been deleted and needs to be predicted by the model. Finally, applying S2 and S3 of Example 1, we obtain the two model prediction outputs R1 and R2. Since the model input is the same image, regardless of which part of the image is deleted, the model output should be consistent with the target image. Based on this principle, we can define a corresponding loss function, namely, the reconstruction loss.

[0100] Step X62: Subtask 2: Calculate the model's modality transfer capability. This process is the same as X61, except that it outputs an image of a different modality.

[0101] Step X63: Subtask 3: Calculate the model's ability to segment images. This process is the same as X61, except that the output is a label. We use soft-decision for learning each label (i.e., the output value is no longer limited to 0 and 1, but is between 0 and 1), so that segmentation can be treated as a reconstruction task, unifying the training process.

[0102] This invention proposes a model-based approach, trained on a large number of other publicly available datasets, that allows the model to indirectly learn various medical domain knowledge from cardiac MRI images. This significantly improves performance on our task and addresses the shortcomings of existing technologies. Large models combined with large training data have proven effective for many tasks, but not every task has access to such a large amount of data. Therefore, a fundamental approach is to utilize other medical datasets, even those with different research focuses. For example, data from other tasks (such as left atrial segmentation datasets) can be used to assist the current task. While the data from each individual research topic may be limited, the cumulative effect of these smaller datasets is substantial. Therefore, the entire solution requires a suitable training method to balance these diverse datasets and achieve the training objectives, not only for the current task of pericardial fat segmentation but also for other tasks.

[0103] Therefore, we divide the technical solution into two parts: one corresponding to the model, and the other corresponding to the data and training. The model part comprises three components: a large visual model and a large language model as data inputs, and a decoding network as the output. The dataset training part is mainly used to work with the model to process data and achieve the fusion training of various datasets.

[0104] Example 3

[0105] In an exemplary embodiment, an electronic device is also provided, comprising:

[0106] At least one processor; and

[0107] A memory that is communicatively connected to the at least one processor;

[0108] The memory stores instructions that can be executed by the at least one processor to cause the electronic device to perform a voice-guided cardiac fat segmentation method.

[0109] For example, computer-readable storage media can be read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage devices.

[0110] In an exemplary embodiment, a storage medium is also provided on which a computer program is stored, which, when executed by a processor, implements a language-cue-based cardiac fat segmentation method.

[0111] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0112] It should be understood that determining B based on A does not mean determining B solely based on A; it also means determining B based on A and / or other information.

[0113] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0114] The above description is only an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0115] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only one method, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0116] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0117] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for cardiac fat segmentation based on language prompts, characterized in that, Includes the following steps: Step S1: Acquire multimodal images and extract cardiac fat image features based on a visual model; The application logic of the visual model is as follows: Step S11: Process the multimodal image and the modality type information corresponding to the multimodal image to generate an embedding sequence; the logic for obtaining the embedding sequence is as follows: Converting the image into an embedding sequence: The multimodal image is segmented into modal patches of fixed size, and each modal patch is mapped to an image embedding sequence; Convert modal types to corresponding modal embeddings: Modal types are encoded using integers, and the corresponding embeddings are used as a sequence of modal embeddings; Assemble Embedding sequences with different information: Concatenate image embedding sequences and modal embedding sequences to generate an embedding sequence; Step S12: Use the Transformer architecture to extract features from the Embedding sequence to obtain cardiac fat image features, wherein the cardiac fat image features are image feature Embedding sequences; Step S2: Describe the annotations in text, extract the segmentation intent features of the text description using the existing large language model, and convert them into task prompt vectors that can be understood by subsequent modules; Step S3: Construct a decoding network. In the decoding network, the input image feature embedding sequence is comprehensively decoded according to the requirements of the task prompt vector to obtain the segmentation result. The decoding processing logic includes the following steps: Step S31: Based on the cardiac fat image features extracted in Step S1 and the task cue vector extracted in Step S2, label the image feature embedding sequence corresponding to the cardiac fat image features as sequence A, and label the embedding sequence corresponding to the task cue vector as sequence B. Step S32: Simultaneously feed sequence A and sequence B into the bidirectional attention network to obtain the perception results of sequence A and sequence B respectively; Step S33: The A sequence perception result is processed through multiple deconvolution layers to generate multiple candidate images. The B sequence perception result is processed through the B-to-A cross-attention Transformer block. The multilayer perceptron obtains the weights of these candidate images. Finally, the multiple images are weighted according to these weights to obtain the segmentation result.

2. The cardiac fat segmentation method based on language prompts according to claim 1, characterized in that, The Transformer architecture includes multiple Transformer blocks arranged in series. Each Transformer block maps the Embeddings in the Embedding sequence to corresponding features through a self-attention mechanism. The process is as follows: First, a linear mapping module is used to map each embedding into three sets of vectors: Q, K, and V. Q is equivalent to the eye used for finding the embedding, K is equivalent to the representation of the current embedding, and V is the corresponding feature.

3. The cardiac fat segmentation method based on language prompts according to claim 2, characterized in that, The distance between vector Q and the vector K of all embeddings is calculated. The weights are calculated based on the distances and these features V are weighted together to obtain the feature encoding after the current embedding is transformed.

4. The cardiac fat segmentation method based on language prompts according to claim 1, characterized in that, The application logic of the large language model is as follows: Each label is converted into a corresponding text description, and the text features of the segmentation intent of the text description are extracted using a large language model. After obtaining the text features, the text features are converted into task prompt vectors, and the task prompt vectors output the text features in a fixed length.

5. The cardiac fat segmentation method based on language prompts according to claim 1, wherein the model training process of the cardiac fat segmentation method based on language prompts is characterized in that, Includes the following steps: Step X1: Data collection and processing; Step X2: Merge multiple datasets to generate a training dataset; Step X3: Extract the training task from the merged dataset during training; Step X4: Encode the input image and modal information using a visual model to obtain the image embedding sequence; Step X5: Calculate the comparison similarity; Step X6: Calculate the reconstruction similarity; Step X6 includes: Step X61: Subtask 1: Calculate the model's ability to reconstruct images; Step X62: Subtask 2: Calculate the model's modal transfer capability; Step X63: Subtask 3: Calculate the model's ability to segment images.

6. An electronic device, comprising: include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause the electronic device to perform the language-cue-based cardiac fat segmentation method according to any one of claims 1-4.

7. A storage medium having stored thereon a computer program, characterized in that When executed by a processor, the computer program implements the language-cue-based cardiac fat segmentation method as described in any one of claims 1-4.

Images