Interactive Medical Image Segmentation Method and System Based on Multimodal Information Integration

Abstract

This invention relates to medical image semantic segmentation methods, and discloses an interactive medical image segmentation method and system based on multimodal information integration. Its multimodal information integration-based semi-automatic image segmentation method can fuse textual information and image features, combined with user-interactive human-computer interaction, to achieve semi-automatic 3D semantic segmentation. This invention simplifies the annotation process through technical means, achieving semi-automatic segmentation and annotation of medical 3D image semantics and interactivity, thereby significantly reducing the cost of pixel-level precise annotation of 3D image data.

Description

Technical Field

[0001] This invention relates to a semantic segmentation method for medical images, and more particularly to an interactive segmentation method and system for medical images based on multimodal information integration. Background Technology

[0002] In the field of precision diagnosis using medical AI, developing image semantic segmentation models requires a large amount of finely labeled data. However, acquiring pixel-level finely labeled images is extremely time-consuming and labor-intensive, resulting in very high costs. This significantly hinders the technological development of precision medicine. Most existing technologies reduce the amount of annotation work by decreasing the granularity of the annotation.

[0003] Existing technology 1: Patent CN111462163A gradually enhances the model's global sensitivity to the target region through erasure training. However, the lack of a large amount of finely labeled data during model training can easily lead to imbalance, causing the target region that the model is not sensitive to to remain insensitive, and may even damage the accuracy of the model in segmenting well-performing regions.

[0004] Existing technology 2: Patent CN113269778B uses thyroid ultrasound images containing localized bounding boxes as weakly supervised information, and obtains training labels using probabilistic gradient annotation. It iterative training continuously updates the parameters and training labels of the deep learning weakly supervised segmentation network. However, this method is difficult to apply to CT or MRI images with larger fields of view and more complex conditions, and the extremely poor training annotation effect in the first stage makes it difficult for the model training process to converge.

[0005] Existing technology 3: Patent CN117830332A uses a dynamic hybrid augmentation mechanism and uncertainty-guided pixel-level contrastive learning to narrow or even eliminate the gap between sparse annotation-based methods and dense annotation-based methods, thereby improving the accuracy of weakly supervised segmentation.

[0006] The above methods all aim to compensate for insufficient quantity or quality of finely labeled data to meet the data requirements for model training through technical means. However, reducing the quantity or quality of finely labeled data will inevitably lead to a certain degree of reduction in model performance. Summary of the Invention

[0007] This invention addresses the problem in existing technologies that cannot change the reliance on large-scale, precisely labeled data for model training, and provides an interactive medical image segmentation method and system based on multimodal information integration.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0009] An interactive medical image segmentation method based on multimodal information integration includes the following methods:

[0010] Text information extraction transforms a short text into a one-dimensional embedded semantic feature F of length p. text ;

[0011] Image feature extraction utilizes the first half of the ViT3D network as the image feature encoder. The 3D image is cropped into local blocks and stitched into a block sequence, which is then input into the image feature encoder to convert the 3D image into a one-dimensional image feature F of length q. image ;

[0012] A human-computer interaction module is constructed that takes the relative position coordinates (x, y, z, i) of a user click as input, where x, y, and z are image voxel coordinates, and i is the point category, where i = 1 represents a foreground point and i = 0 represents a background point; these coordinates are then converted into a one-dimensional interaction feature F of length l. inter ;

[0013] Construct a dynamic memory knowledge matrix M, using text features F text As the key, based on the relative position of the segmentation label to which the category belongs in the sample, from the image features F of the sample. image Local features belonging to this category are extracted and resampled to h dimensions through a fully connected layer to obtain F. local As values, all samples in the training data form a key-value pair memory knowledge matrix M;

[0014] The binary segmentation result is obtained by consisting of multiple deconvolutional layers and one transformer layer, taking the final features as input and outputting the binary segmentation result.

[0015] Preferably, text information extraction is achieved by constructing a text information extraction network, which transforms short text corpora into one-dimensional embedded semantic features F of length p. text The text information extraction network consists of a stack of four transformer layers.

[0016] As a preferred embodiment, the text information is segmented and converted into token vectors using a medical imaging dictionary before extraction.

[0017] As a preferred method, the acquisition of the medical image dictionary includes: treating a word consisting of consecutive English words, numbers, and hyphens as a single English word; for Chinese words, manually annotating N examples of medical image report texts and training a hidden Markov chain model.

[0018] Hidden Markov chain model is used to perform word segmentation prediction on a large number of reports, and words not included in the segmentation results are added to the dictionary. Each entry in the dictionary consists of a unique numeric ID and a word. This process continues until a total of M reports have been segmented and the dictionary has not been updated.

[0019] As a preferred approach, the image feature extraction module is pre-trained using a multi-task method; a complete ViT3D model is constructed and two sub-task modules are spliced ​​together, namely a semantic segmentation task module and a similarity matching module;

[0020] Each sub-task module is composed of multiple layers of transformer structures assembled sequentially. The loss function for the semantic segmentation task module is the Dice loss function L. dice The similarity matching module uses triplet matching loss L. match The final loss was borne by L dice and L match Calculate the weighted average.

[0021] Preferably, the similarity matching module receives three feature vectors (x1, x2, x3) from three image data processed by ViT3D, where x1 and x2 represent images of the same examination type, and x1 and x3 represent images of different examination types; the loss function L... match Using triplet matching loss, L match =max(d(x1,x2)-d(x1,x3)+ε,0), where d is the L2 distance.

[0022] Preferably, the knowledge matrix M is dynamically updated during model training. The update process includes:

[0023] After the t-th iteration, the knowledge matrix M is remembered. t The key of category c is Its value is At iteration t+1, for the same text input, the model obtains and Interpolation with the feature calculation at step t can yield and in For multi-head attention operators;

[0024] After step t+1 Where norm represents normalization, thus yielding M t+1 This enables the updating of the knowledge matrix.

[0025] As a preferred approach, during model derivation, the F calculated from textual information... text The cosine similarity is calculated sequentially with the keys in the knowledge matrix M, and the value corresponding to the key with the highest similarity is taken as the knowledge feature F. klg and with image features F image Calculate multi-head attention to obtain the fused knowledge feature F. att And then with F image and F interThe final features are formed after being spliced ​​together.

[0026] To address the aforementioned technical problems, this invention also provides a medical image interactive segmentation system based on multimodal information integration, comprising:

[0027] The text information extraction module transforms short text snippets into one-dimensional embedded semantic features F of length p. text ;

[0028] The image feature extraction module uses the first half of the ViT3D network as the image feature encoder. The 3D image is cropped into local blocks and stitched into a block sequence, which is then input into the image feature encoder to convert the 3D image into a one-dimensional image feature F of length q. image ;

[0029] A human-computer interaction module is constructed that takes the relative position coordinates (x, y, z, i) of a user click as input, where x, y, and z are image voxel coordinates, and i is the point category, where i = 1 represents a foreground point and i = 0 represents a background point; these coordinates are then converted into a one-dimensional interaction feature F of length l. inter ;

[0030] Construct a dynamic memory knowledge matrix module M, using text features F text As the key, based on the relative position of the segmentation label to which the category belongs in the sample, from the image features F of the sample. image Local features belonging to this category are extracted and resampled to h dimensions through a fully connected layer to obtain F. local As values, all samples in the training data form a key-value pair memory knowledge matrix M;

[0031] The binary segmentation result output module consists of multiple deconvolutional layers and one transformer layer. It takes the final features as input and outputs the binary segmentation result.

[0032] This invention, by adopting the above technical solutions, has significant technical effects:

[0033] The multimodal information-integrated image semi-automatic segmentation method proposed in this invention can integrate text information and image features, combined with user-click human-computer interaction, to achieve semi-automatic three-dimensional semantic segmentation.

[0034] This invention simplifies the annotation process through technical means, enabling semi-automatic segmentation and annotation of medical 3D images for semantic and interactive purposes, thereby significantly reducing the cost of pixel-level precision-annotated 3D image data. Attached Figure Description

[0035] Figure 1 This is the algorithm structure diagram of the present invention.

[0036] Figure 2This is a schematic diagram of entities and relationships in the entity target of the diagnostic results of this invention.

[0037] Figure 3 This is a pre-training task diagram for the image feature extraction module of this invention. Detailed Implementation

[0038] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0039] Example 1

[0040] Pre-trained image feature extraction module. First, 50,000 and 20,000 CT scan images of various examination sites and diseases were collected respectively, including artificially smeared lesion areas and disease categories, as the original training set and test set.

[0041] Step 1. Iterate through the images of the same category in the original training set and test set, and combine them into pairs. Combine each pair with an image of a different category to form a triplet sample pair, which serves as the training set and test set for the similarity matching module.

[0042] Step 2. Construct the complete ViT3D model and assemble two sub-task modules: a semantic segmentation module and a similarity matching module. Each sub-task module consists of a multi-layer transformer structure assembled sequentially. The loss function for the semantic segmentation module is the Dice loss function L. dice The similarity matching module uses triplet matching loss L. match The final loss was borne by L dice and L match Calculate the weighted average.

[0043] Step 3. The triplet samples are used to train the similarity matching module and the semantic segmentation task module by manually smearing the lesion area of ​​each sample in the training set until the model converges.

[0044] Step 4. Extract the encoder part from the trained ViT3D model and use it as the pre-trained image feature extraction module.

[0045] Example 2

[0046] Based on Example 1, this example demonstrates how users perform interactive semantic segmentation annotation using a semi-automatic medical image segmentation system.

[0047] The image data to be labeled is input into the image feature extraction module, which converts it into a one-dimensional image feature F with a length of 2048. image .

[0048] The user enters the category to be labeled in the text box: comminuted rib fracture. This category text is then transformed into a one-dimensional embedded semantic feature F of length 32 by the text information extraction module. text .

[0049] F text The cosine similarity is calculated sequentially with the keys in the knowledge matrix M, and the value corresponding to the key with the highest similarity is taken as the knowledge feature F. klg , with image features F image Calculate multi-head attention to obtain the fused knowledge feature F. att .

[0050] The user clicks on the target point (238, 92, 103) in the image using the mouse, marking it as the foreground point, i.e., i = 1. This is then input into the human-computer interaction module and converted into a one-dimensional interactive feature F of length 32. inter .

[0051] Integrate knowledge features F att With F image and F inter The concatenated data forms the final feature, which is then input into the decoding module to output the binary segmentation result.

[0052] The segmentation result is unsatisfactory, with false positive segmentation regions existing. An interactive click is made at coordinates (341, 145, 101) within a false positive segmentation region, and this point is marked as the background point, i.e., i = 0. This coordinate point (341, 145, 101, 0) is concatenated with the coordinates of the previously clicked point and input into the human-computer interaction module to obtain a new interaction feature F′ of length 32. inter .

[0053] Integrate knowledge features F att With F image and F′ inter The concatenated data forms the final feature, which is then input into the decoding module to output a new binary segmentation result.

[0054] If the segmentation result is unsatisfactory, repeat until the segmentation result meets expectations, thus completing the semi-automatic segmentation and labeling.

[0055] Example 3

[0056] Based on the above embodiments, this embodiment is a medical image interactive segmentation system based on multimodal information integration, which includes:

[0057] The text information extraction module transforms short text snippets into one-dimensional embedded semantic features F of length p. text ;

[0058] The image feature extraction module uses the first half of the ViT3D network as the image feature encoder. The 3D image is cropped into local blocks and stitched into a block sequence, which is then input into the image feature encoder to convert the 3D image into a one-dimensional image feature F of length q. image ;

[0059] A human-computer interaction module is constructed that takes the relative position coordinates (x, y, z, i) of a user click as input, where x, y, and z are image voxel coordinates, and i is the point category, where i = 1 represents a foreground point and i = 0 represents a background point; these coordinates are then converted into a one-dimensional interaction feature F of length l. inter ;

[0060] Construct a dynamic memory knowledge matrix module M, using text features F text As the key, based on the relative position of the segmentation label to which the category belongs in the sample, from the image features F of the sample. image Local features belonging to this category are extracted and resampled to h dimensions through a fully connected layer to obtain F. local As values, all samples of the training data form a key-value pair memory knowledge matrix M; the binary segmentation result output module consists of multiple deconvolutional layers and one transformer layer, taking the final features as input and outputting the binary segmentation result.

Claims

1. An interactive segmentation method for medical images based on multimodal information integration, the method comprising: Text information extraction transforms a short text into a one-dimensional embedded semantic feature F of length p. text ; Image feature extraction utilizes the first half of the ViT3D network as the image feature encoder. The 3D image is cropped into local blocks and stitched into a block sequence, which is then input into the image feature encoder to convert the 3D image into a one-dimensional image feature F of length q. image ; A human-computer interaction module is constructed that takes the relative position coordinates (x, y, z, i) of a user click as input, where x, y, and z are image voxel coordinates, and i is the point category, where i = 1 represents a foreground point and i = 0 represents a background point; these coordinates are then converted into a one-dimensional interaction feature F of length l. inter ; Construct a dynamic memory knowledge matrix M, using text features F text As the key, based on the relative position of the segmentation label to which the category belongs in the sample, from the image features F of the sample. image Local features belonging to this category are extracted and resampled to h dimensions through a fully connected layer to obtain F. local As a value, all samples of the training data form the memory knowledge matrix M; The binary segmentation result is obtained by a combination of multiple deconvolutional layers and a single transformer layer. The final features are used as input, and the output is the binary segmentation result. During model inference, the F-values ​​obtained from the text information are used to calculate the binary segmentation result. text The cosine similarity is calculated sequentially with the keys in the knowledge matrix M, and the value corresponding to the key with the highest similarity is taken as the knowledge feature F. klg and with image features F image Calculate multi-head attention to obtain the fused knowledge feature F. att And then with F image and F inter The final features are formed after being spliced ​​together.

2. The interactive medical image segmentation method based on multimodal information integration according to claim 1, characterized in that, Text information extraction is achieved by constructing a text information extraction network, which transforms short text corpora into one-dimensional embedded semantic features F of length p. text The text information extraction network consists of a stack of four transformer layers.

3. The interactive medical image segmentation method based on multimodal information integration according to claim 1, characterized in that, Before extracting text information, it is segmented into words by a medical imaging dictionary and converted into token vectors.

4. The interactive medical image segmentation method based on multimodal information integration according to claim 3, characterized in that, The methods for obtaining the medical image dictionary include: treating consecutive English words, numbers, and hyphens as a single English word; for Chinese words, manually annotating N examples of medical image report texts and training a hidden Markov chain model. Hidden Markov chain model is used to perform word segmentation prediction on a large number of reports, and words not included in the segmentation results are added to the dictionary. Each entry in the dictionary consists of a unique numeric ID and a word. This process continues until a total of M reports have been segmented and the dictionary has not been updated.

5. The interactive medical image segmentation method based on multimodal information integration according to claim 1, characterized in that, The image feature extraction module is pre-trained using a multi-task approach; a complete ViT3D model is constructed and two sub-task modules are spliced ​​together: a semantic segmentation module and a similarity matching module. Each sub-task module is composed of multiple layers of transformer structures assembled sequentially. The loss function for the semantic segmentation task module is the Dice loss function L. dice The similarity matching module uses triplet matching loss L. match The final loss was borne by L dice and L match Calculate the weighted average.

6. The interactive medical image segmentation method based on multimodal information integration according to claim 5, characterized in that, The similarity matching module receives three feature vectors (x1, x2, x3) from three image data processed by ViT3D, where x1 and x2 represent images of the same examination type, and x1 and x3 represent images of different examination types; the loss function L... match Using triplet matching loss, L match =max(d(x1,x2)-d(x1,x3)+ε,0), where d is the L2 distance.

7. The interactive medical image segmentation method based on multimodal information integration according to claim 1, characterized in that, The knowledge matrix m is dynamically updated during model training. The update process includes: After the t-th iteration, the key of category c in the memory knowledge matrix m is Its value is At iteration t+1, for the same text input, the model obtains and Interpolation with the feature calculation at step t can yield and in For multi-head attention operators; After step t+1 Where norm represents normalization, thus yielding M t+1 This enables the updating of the knowledge matrix.

8. A medical image interactive segmentation system based on multimodal information integration, characterized in that, include: The text information extraction module transforms short text snippets into one-dimensional embedded semantic features F of length p. text ; The image feature extraction module uses the first half of the ViT3D network as the image feature encoder. The 3D image is cropped into local blocks and stitched into a block sequence, which is then input into the image feature encoder to convert the 3D image into a one-dimensional image feature F of length q. image ; A human-computer interaction module is constructed that takes the relative position coordinates (x, y, z, i) of a user click as input, where x, y, and z are image voxel coordinates, and i is the point category, where i = 1 represents a foreground point and i = 0 represents a background point; these coordinates are then converted into a one-dimensional interaction feature F of length l. inter ; Construct a dynamic memory knowledge matrix module M, using text features F text As the key, based on the relative position of the segmentation label to which the category belongs in the sample, from the image features F of the sample. image Local features belonging to this category are extracted and resampled to h dimensions through a fully connected layer to obtain F. local As a value, all samples of the training data form the memory knowledge matrix M; The binary segmentation output module consists of multiple deconvolutional layers and one transformer layer. It takes the final features as input and outputs the binary segmentation result. For obtaining the binary segmentation result, during model inference, the F-values ​​calculated from the text information are used. text The cosine similarity is calculated sequentially with the keys in the knowledge matrix M, and the value corresponding to the key with the highest similarity is taken as the knowledge feature F. klg and with image features F image Calculate multi-head attention to obtain the fused knowledge feature F. att And then with F image and F inter The final features are formed after being spliced ​​together.

Images