Medical image interactive segmentation method and system based on multi-modal information penetration

Abstract

The invention relates to a medical image semantic segmentation method, and discloses a medical image interactive segmentation method and system based on multi-modal information penetration, and the multi-modal information penetration image semi-automatic segmentation method can fuse text information and image features and realize semi-automatic three-dimensional semantic segmentation in combination with a human-computer interaction mode of user clicking. According to the method, the labeling operation process is simplified through the technical means, semi-automatic segmentation labeling of medical three-dimensional image semantics and interaction is achieved, and then the cost of pixel-level fine labeling of three-dimensional image data is reduced on a large scale.

Description

Technical Field

[0001] The present invention relates to a medical image semantic segmentation method, and in particular to a medical image interactive segmentation method and system based on multimodal information communication. Background Art

[0002] In the field of medical artificial intelligence precision diagnosis, the development of image semantic segmentation models requires a large amount of precisely labeled data, and pixel-level precisely labeled images require a lot of time and manpower to obtain, which is extremely costly. This has greatly hindered the development of technology in the field of precision medicine. Most existing technologies reduce the workload of labeling by reducing the granularity of labeling.

[0003] Prior art 1: Patent CN111462163A gradually enhances the model's global sensitivity to the target area through erasing training. However, the model training process lacks a large amount of precisely labeled data support and is prone to imbalance, which makes the target area to which the model is insensitive remain insensitive, and may even destroy the accuracy of the model's better segmented areas.

[0004] Prior art 2: Patent CN113269778B uses thyroid ultrasound images containing positioning bounding boxes as weak supervision information, uses probabilistic gradient annotation to obtain training annotations, and continuously updates deep learning weak supervision segmentation network parameters and training labels in an iterative training manner. However, this method is difficult to apply to CT or MRI images with a larger field of view and more complex situations, and the model training process will be difficult to converge due to the extremely poor training annotation effect in the first stage.

[0005] Prior art 3: Patent CN117830332A uses a dynamic hybrid augmentation mechanism and uncertainty-guided pixel-level contrastive learning to narrow or even eliminate the gap between sparse annotation-based methods and dense annotation-based methods, thereby improving the accuracy of weakly supervised segmentation.

[0006] The idea behind the above methods is to use technical means to make up for the fact that the quantity or quality of precise annotations is insufficient to meet the data requirements for model training, but reducing the quantity or quality of precise annotations will inevitably lead to a certain degree of reduction in model effect. Summary of the invention

[0007] In view of the problem in the prior art that model training needs to rely on large-scale accurately labeled data and cannot be changed, the present invention provides an interactive medical image segmentation method and system based on multimodal information penetration.

[0008] In order to solve the above technical problems, the present invention is solved by the following technical solutions:

[0009] An interactive medical image segmentation method based on multimodal information integration includes:

[0010] Extract text information and convert short text into a one-dimensional embedded semantic feature F of length p. text ;

[0011] To extract image features, the first half of the ViT3D network is used as the image feature encoder. The 3D image is cut into blocks and then input into the image feature encoder to convert the 3D image into a one-dimensional image feature F with a length of q. image ;

[0012] Construct a human-computer interaction module with the relative position coordinates (x, y, z, i) of the user's click as input, where x, y and z are the image voxel coordinates, i is the point category, when i = 1 represents the foreground point, i = 0 represents the background point; convert the above coordinate points into a one-dimensional interaction feature F with a length of l inter ;

[0013] Construct a dynamic memory knowledge matrix M based on text features F text As the key, according to the relative position of the segmentation annotation of the category in the sample, the image feature F of the sample is obtained. image Extract the local features belonging to the category and resample them to h dimensions through a fully connected layer to obtain F local As values, all samples of the training data form a key-value pair memory knowledge matrix M;

[0014] The acquisition of binary segmentation results consists of multiple layers of deconvolution layers and one layer of transformer, which takes the final features as input and outputs the binary segmentation results.

[0015] As a preferred method, the text information is extracted by constructing a text information extraction network to convert the short text corpus into a one-dimensional embedded semantic feature F with a length of p. text ; The text information extraction network consists of a 4-layer stacked transformer structure.

[0016] Preferably, the text information is characterized in that it is segmented and converted into token vectors using a medical imaging dictionary before extraction.

[0017] As a preferred method, the method for obtaining the medical imaging dictionary includes: treating words consisting of consecutive English words, numbers, and hyphens as one English word; for Chinese words, manually annotating N medical imaging report texts and training a hidden Markov chain model;

[0018] A hidden Markov chain model is used to predict the word segmentation of a large number of reports, and the words not included in the word segmentation results are added to the dictionary. Each entry in the dictionary consists of a non-repeating numeric id and a word. This process continues until the dictionary is not updated after M reports are segmented.

[0019] As a preferred method, the image feature extraction module is pre-trained in a multi-task manner; a complete ViT3D model is constructed, and two sub-task modules are spliced, the two sub-task modules are a semantic segmentation task module and a similarity matching module;

[0020] Each subtask module is composed of multiple layers of transformer structures spliced ​​sequentially. The loss function of the semantic segmentation task module is the Dice loss function L dice , the similarity matching module uses triple matching loss L match , the final loss is L dice and L match Calculate the weighted average.

[0021] Preferably, the similarity matching module receives three feature vectors (x1, x2, x3) of three image data after ViT3D, wherein x1 and x2 are images of the same inspection type, and x1 and x3 are images of different inspection types; the loss function L match Using triplet matching loss, L match =max(d(x1,x2)-d(x1,x3)+ε,0), where d is the L2 distance.

[0022] Preferably, the memory knowledge matrix M is dynamically updated during the model training process, and the updating process includes:

[0023] After the tth iteration, the memory knowledge matrix M t The key of class c is Its value is At the t+1th iteration, for the same text input, the model obtains and Interpolation with the feature calculation in step t can be obtained and in is a multi-head attention operator;

[0024] After step t+1 Among them, norm represents normalization, and then we get M t+1 , to realize the update of memory knowledge matrix.

[0025] As a preference, when modeling, the F calculated from the text information text Calculate the cosine similarity with the keys in the memory knowledge matrix M in turn, and take the value corresponding to the key with the highest similarity as the knowledge feature F klg , and the image feature F image Calculate multi-head attention and obtain the fusion knowledge feature F att , and then with F image and F interAfter splicing, they form the final feature.

[0026] In order to solve the above technical problems, the present invention also provides a medical image interactive segmentation system based on multimodal information communication, which includes:

[0027] The text information extraction module converts the short text into a one-dimensional embedded semantic feature F of length p. text ;

[0028] The image feature extraction module uses the first half of the ViT3D network as the image feature encoder. The 3D image is cut into blocks and then input into the image feature encoder to convert the 3D image into a one-dimensional image feature F with a length of q. image ;

[0029] Construct a human-computer interaction module with the relative position coordinates (x, y, z, i) of the user's click as input, where x, y and z are the image voxel coordinates, i is the point category, when i = 1 represents the foreground point, i = 0 represents the background point; convert the above coordinate points into a one-dimensional interaction feature F with a length of l inter ;

[0030] Construct a dynamic memory knowledge matrix M module based on text features F text As the key, according to the relative position of the segmentation annotation of the category in the sample, the image feature F of the sample is obtained. image Extract the local features belonging to the category and resample them to h dimensions through a fully connected layer to obtain F local As values, all samples of the training data form a key-value pair memory knowledge matrix M;

[0031] The binary segmentation result output module consists of multiple layers of deconvolution layers and one layer of transformer. It takes the final features as input and outputs the binary segmentation result.

[0032] The present invention has significant technical effects due to the adoption of the above technical solution:

[0033] The multimodal information-penetrating image semi-automatic segmentation method proposed in the present invention can integrate text information and image features, and combine with the human-computer interaction mode of user clicks to achieve semi-automatic three-dimensional semantic segmentation.

[0034] The present invention simplifies the annotation operation process through technical means, realizes the semi-automatic segmentation and annotation of medical three-dimensional image semantics and interaction, and thus greatly reduces the cost of pixel-level precise annotation of three-dimensional image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is the algorithm structure diagram of the present invention

[0036] Figure 2It is a schematic diagram of entities and relationships in the diagnosis result entity target of the present invention.

[0037] Figure 3 It is a pre-training task diagram of the image feature extraction module of the present invention. DETAILED DESCRIPTION

[0038] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0039] Example 1

[0040] Pre-trained image feature extraction module. First, 50,000 and 20,000 CT scan images of various examination sites and diseases, including manually painted lesion areas and disease categories, were collected as the original training set and test set, respectively.

[0041] Step 1. Traverse the images of the same category in the original training set and test set, and combine them with an image of a different category to form a triplet sample pair as the training set and test set of the similarity matching module.

[0042] Step 2. Build a complete ViT3D model and connect two subtask modules: semantic segmentation task module and similarity matching module. Each subtask module consists of multiple layers of transformer structures connected in sequence. The loss function of the semantic segmentation task module is the Dice loss function L dice , the similarity matching module uses triple matching loss L match , the final loss is L dice and L match Calculate the weighted average.

[0043] Step 3. The triplet sample training set and the manually smeared lesion area annotations of each sample in it are used to train the similarity matching module and the semantic segmentation task module until the model converges.

[0044] Step 4. Extract the encoder part of the trained ViT3D model as the pre-trained image feature extraction module.

[0045] Example 2

[0046] Based on Example 1, in this example, the user uses a medical image-based semi-automatic segmentation system to perform interactive semantic segmentation annotation.

[0047] The image data to be annotated is input into the image feature extraction module and converted into a one-dimensional image feature F with a length of 2048. image .

[0048] The user enters the category to be labeled in the text box: rib comminuted fracture. The category text is converted into a one-dimensional embedded semantic feature F with a length of 32 by the text information extraction module. text .

[0049] F text Calculate the cosine similarity with the keys in the memory knowledge matrix M in turn, and take the value corresponding to the key with the highest similarity as the knowledge feature F klg , and image feature F image Calculate multi-head attention and obtain the fusion knowledge feature F att .

[0050] The user clicks the target point (238, 92, 103) in the image through mouse interaction and marks it as the foreground point, that is, i = 1. It is input into the human-computer interaction module and converted into a one-dimensional interaction feature F with a length of 32. inter .

[0051] The fusion knowledge feature F att With F image and F inter After splicing, the final features are formed and then input into the decoding module to output the binary segmentation result.

[0052] The segmentation result is not satisfactory. There is a false positive segmentation area. Click interactively at the coordinate position (341,145,101) in the false positive segmentation area and mark it as a background point, that is, i = 0. The coordinate point (341,145,101,0) is concatenated with the coordinate point of the previous click and input into the human-computer interaction module to obtain a new interactive feature F′ with a length of 32. inter .

[0053] The fusion knowledge feature F att With F image and F′ inter After splicing, the final features are formed and then input into the decoding module to output the new binary segmentation result.

[0054] If the segmentation result is not satisfactory, repeat until the segmentation result meets the expectations and complete the semi-automatic segmentation and annotation.

[0055] Example 3

[0056] Based on the above embodiments, this embodiment is a medical image interactive segmentation system based on multimodal information communication, which includes:

[0057] The text information extraction module converts the short text into a one-dimensional embedded semantic feature F of length p. text ;

[0058] The image feature extraction module uses the first half of the ViT3D network as the image feature encoder. The 3D image is cut into blocks and then input into the image feature encoder to convert the 3D image into a one-dimensional image feature F with a length of q. image ;

[0059] Construct a human-computer interaction module with the relative position coordinates (x, y, z, i) of the user's click as input, where x, y and z are the image voxel coordinates, i is the point category, when i = 1 represents the foreground point, i = 0 represents the background point; convert the above coordinate points into a one-dimensional interaction feature F with a length of l inter ;

[0060] Construct a dynamic memory knowledge matrix M module based on text features F text As the key, according to the relative position of the segmentation annotation of the category in the sample, the image feature F of the sample is obtained. image Extract the local features belonging to the category and resample them to h dimensions through a fully connected layer to obtain F local As values, all samples of the training data constitute the key-value pair memory knowledge matrix M; the binary segmentation result output module, composed of multiple layers of deconvolution layers and 1 layer of transformer, takes the final features as input and outputs the binary segmentation result.

Claims

1. An interactive medical image segmentation method based on multimodal information integration, the method comprising: Extract text information and convert short text into a one-dimensional embedded semantic feature F of length p. text ; To extract image features, the first half of the ViT3D network is used as the image feature encoder. The 3D image is cut into blocks and then input into the image feature encoder to convert the 3D image into a one-dimensional image feature F with a length of q. image ; Construct a human-computer interaction module with the relative position coordinates (x, y, z, i) of the user's click as input, where x, y and z are the image voxel coordinates, i is the point category, when i = 1 represents the foreground point, i = 0 represents the background point; convert the above coordinate points into a one-dimensional interaction feature F with a length of l inter ; Construct a dynamic memory knowledge matrix M based on text features F text As the key, according to the relative position of the segmentation mark of the category in the sample, the image feature F of the sample is obtained. image Extract the local features belonging to the category and resample them to h dimensions through a fully connected layer to obtain F local As values, all samples of the training data form a key-value pair memory knowledge matrix M; The acquisition of binary segmentation results consists of multiple layers of deconvolution layers and one layer of transformer, which takes the final features as input and outputs the binary segmentation results.

2. The method for interactive segmentation of medical images based on multimodal information communication according to claim 1, characterized in that: The extraction of text information is done by building a text information extraction network to convert short text corpus into a one-dimensional embedded semantic feature F with a length of p. text ; The text information extraction network consists of a 4-layer stacked transformer structure.

3. The medical image interactive segmentation method based on multimodal information communication according to claim 1, characterized in that: Before extracting text information, it is segmented by medical imaging dictionary and converted into token vectors.

4. The medical image interactive segmentation method based on multimodal information penetration according to claim 3 is characterized in that: The method for obtaining the medical imaging dictionary includes: treating words consisting of consecutive English words, numbers, and hyphens as one English word; for Chinese words, manually annotating N medical imaging report texts and training a hidden Markov chain model; A hidden Markov chain model is used to predict the word segmentation of a large number of reports, and the words not included in the word segmentation results are added to the dictionary. Each entry in the dictionary consists of a non-repeating numeric id and a word. This process continues until the dictionary is not updated after M reports are segmented.

5. The medical image interactive segmentation method based on multimodal information communication according to claim 1, characterized in that: The image feature extraction module is pre-trained in a multi-task manner; a complete ViT3D model is constructed and two sub-task modules are spliced, namely the semantic segmentation task module and the similarity matching module; Each subtask module is composed of multiple layers of transformer structures spliced ​​sequentially. The loss function of the semantic segmentation task module is the Dice loss function L dice , the similarity matching module uses triple matching loss L match , the final loss is L dice and L match Calculate the weighted average.

6. The medical image interactive segmentation method based on multimodal information penetration according to claim 1, characterized in that: The similarity matching module receives three feature vectors (x1, x2, x3) of three image data after ViT3D, where x1 and x2 are images of the same inspection type, and x1 and x3 are images of different inspection types; the loss function L match Using triplet matching loss, L match =max(d(x1,x2)-d(x1,x3)+ε,0), where d is the L2 distance.

7. The medical image interactive segmentation method based on multimodal information communication according to claim 1, characterized in that: The memory knowledge matrix M is dynamically updated during the model training process. The update process includes: After the tth iteration, the memory knowledge matrix M t The key of class c is Its value is At the t+1th iteration, for the same text input, the model obtains and Interpolation with the feature calculation in step t can be obtained and in is a multi-head attention operator; After step t+1 Among them, norm represents normalization, and then we get M t+1 , to realize the update of memory knowledge matrix.

8. The medical image interactive segmentation method based on multimodal information communication according to claim 1, characterized in that: For the acquisition of binary segmentation results, when the model is deduced, the F text Calculate the cosine similarity with the keys in the memory knowledge matrix M in turn, and take the value corresponding to the key with the highest similarity as the knowledge feature F klg , and the image feature F image Calculate multi-head attention and obtain the fusion knowledge feature F att , and then with F image and F inter After splicing, they form the final feature.

9. An interactive medical image segmentation system based on multimodal information communication, characterized in that: include: The text information extraction module converts the short text into a one-dimensional embedded semantic feature F of length p. text ; The image feature extraction module uses the first half of the ViT3D network as the image feature encoder. The 3D image is cut into blocks and then input into the image feature encoder to convert the 3D image into a one-dimensional image feature F with a length of q. image ; Construct a human-computer interaction module with the relative position coordinates (x, y, z, i) of the user's click as input, where x, y and z are the image voxel coordinates, i is the point category, when i = 1 represents the foreground point, i = 0 represents the background point; convert the above coordinate points into a one-dimensional interaction feature F with a length of l inter ; Construct a dynamic memory knowledge matrix M module based on text features F text As the key, according to the relative position of the segmentation mark of the category in the sample, the image feature F of the sample is obtained. image Extract the local features belonging to the category and resample them to h dimensions through a fully connected layer to obtain F local As values, all samples of the training data constitute the key-value pair memory knowledge matrix M; the binary segmentation result output module, composed of multiple deconvolution layers and 1 transformer layer, takes the final features as input and outputs the binary segmentation result.

Images