A Matcher-based medical image sequence segmentation method

Abstract

This invention discloses a Matcher-based medical image sequence segmentation method. First, preprocessing and feature extraction are performed on images in both the target and reference medical image sequences, and a correspondence matrix is ​​calculated. A bidirectional matching strategy is used to match the reference mask with the correspondence matrix, resulting in a matching layer. This matching layer is then filtered, and k-means++ clustering is applied to generate a cue sampling set. This cue sampling set is then processed and used as input to a pre-trained SAM model to obtain a proposal list. Instance-level matching is performed between the reference mask and the predicted mask to obtain a result list. Finally, the result list is sorted to obtain the final segmentation result, which is then output. This invention employs bidirectional matching, cue sampling, and instance-level matching strategies to generate high-quality masks, achieving clearer and more accurate segmentation results.

Description

Technical Field

[0001] This invention belongs to the field of deep learning image semantic segmentation, and relates to the segmentation of medical image sequences, specifically a Matcher-based medical image sequence segmentation method. Background Technology

[0002] Matcher is a powerful image segmentation tool that integrates various general feature extraction models, such as DINOv2, CLIP, and MAE, as well as the class-independent segmentation model SAM, enabling accurate segmentation of any object. Matcher's strength lies in its self-supervised learning mechanism; it only requires a reference image and a target image for segmentation, without extensive pre-processing or manual intervention.

[0003] In the field of medical imaging, the scarcity of labeled medical images makes it difficult for fully supervised deep learning methods to achieve ideal results in medical image segmentation or classification tasks. Furthermore, acquiring data labels requires professional expertise, and data annotation is expensive and difficult to obtain sufficient labeled datasets for medical image retrieval. Therefore, applying Matcher to medical image segmentation can address the problem of the scarcity of labeled medical images. However, Matcher currently lacks the ability to process medical image sequences.

[0004] Based on the above reasons, this invention proposes a medical image sequence segmentation method based on Matcher. It uses a multi-layer matching method to filter out layers with strong correlation and extract the corresponding matching layer information. Through the comparison and fusion of multi-layer information, the segmentation performance of Matcher on medical image sequences is finally improved. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a Matcher-based medical image sequence segmentation method to solve the problem of scarce medical images with labeled data.

[0006] A Matcher-based medical image sequence segmentation method includes:

[0007] Step 1: Prepare the target medical image sequence, the reference medical image sequence, and the corresponding mask (i.e., the reference mask) for the reference medical image sequence;

[0008] The target medical image sequence and the reference medical image sequence each contain n images;

[0009] Step 2: Preprocess the images in the target medical image sequence and the reference medical image sequence. Extract features from the preprocessed images using the feature extraction model dinov2_vitg14 and calculate the corresponding relationship matrix.

[0010] Step 3: Match the reference mask with the corresponding relation matrix using a bidirectional matching strategy to obtain n. 2 One matching layer;

[0011] Step 4: Filter the matching layer to reduce the false positive rate;

[0012] Step 5: Use k-means++ clustering to process the selected matching layer, cluster the matching points in the matching layer into k clusters, and generate the corresponding prompt sampling set through local, instance and global sampling;

[0013] Step 6: Merge the local, instance, and global point sets in the prompt sampling set to obtain the merged point set. right Perform enumeration and obtain the set of points. The transformation and concatenation process yields a tensor, which is then used as input to a pre-trained SAM model to obtain the prediction mask and IoU, along with the point set. Add them together to the proposal list. After the traversal is complete, a complete list of proposals is obtained.

[0014] Step 7: Perform instance-level matching between the reference mask and the prediction mask to obtain a result list consisting of the prediction mask, score, and point coordinates of the proposal;

[0015] Choose a high-quality mask to improve matching accuracy.

[0016] Step 8: Sort the results list in descending order of scores, select the top k highest quality masks, merge multiple masks into the final segmentation result and output it.

[0017] Furthermore, the preprocessing described in step 2 involves cropping the input image to 896*896 pixels according to the dinov2_vitg14 input requirements, and finally converting its data type to Tensor type and storing it in the same tensor, which will then be used as input to the feature extraction model.

[0018] Furthermore, the specific method for step 3 is as follows:

[0019] First, the reference mask is flattened into a one-dimensional tensor, and the indices of non-zero elements are obtained. Then, using the correspondence matrix, rows corresponding one-to-one with the indices are selected, and the index of the maximum value in each row is calculated to obtain the forward matching coordinates, i.e., the forward set. The columns in the correspondence matrix with the indices of the forward matching coordinates are selected, and the index of the maximum value in each column is calculated to obtain the reverse matching coordinates. If the corresponding reverse matching coordinate is not on the reference mask, it means that the coordinate does not meet the condition. Coordinates that meet the condition are retained as matching points, and the layer containing the matching point is designated as the matching layer. Finally, n is obtained.2 A matching layer.

[0020] Furthermore, the specific method for step 4 is as follows:

[0021] To achieve multi-layer matching, the matching layer with the most inner points is selected. It is then determined whether the number of matching points exceeds a threshold (5 in this invention). The matching points of this layer are input into the pre-trained SAM model to obtain the prediction mask and IoU. After post-processing, matching labels are obtained, while points in unmatched regions are treated as negative samples. Simultaneously, corresponding operations are performed on the layers above and below to obtain the prediction mask and IoU. The correlation is calculated, and the comparison formula is as follows:

[0022]

[0023] Where P is an empirical value and thickness is the matching layer thickness. If the Q of the upper or lower layer is greater than zero, the prediction mask and IoU of the upper or lower layer are retained, and the corresponding matching labels and negative samples are obtained through post-processing. Otherwise, the results of the upper or lower layer are not retained. The thinner the layer thickness, the more it depends on the information of the upper and lower layers, thereby reducing the false positive rate of matching.

[0024] The beneficial effects of this invention are as follows:

[0025] Matcher employs self-supervised learning and applies it to medical image segmentation, addressing the problem of scarce labeled medical images. Furthermore, compared to other self-supervised methods for medical image segmentation, Matcher utilizes strategies such as bidirectional matching, cue sampling, and instance-level matching to generate high-quality masks, achieving clearer and more accurate segmentation results.

[0026] Currently, Matcher can only segment single images and its segmentation effect on medical image sequences is not good. Therefore, this invention adopts a multi-layer matching method to filter the layers with strong correlation and extract the corresponding matching layer information. Through the comparison and fusion of multi-layer information, Matcher can finally achieve accurate segmentation of medical image sequences. Attached Figure Description

[0027] Figure 1 This is a flowchart of the method according to an embodiment of the present invention.

[0028] Figure 2 This is a schematic diagram of the sequence processing method in an embodiment of the present invention.

[0029] Figure 3 This represents the segmentation prediction result for the Matcher.

[0030] The accompanying drawings show only some of the relevant structures of the present invention, not all of them. Detailed Implementation

[0031] The implementation steps of the invention are described in detail with reference to the invention description and accompanying drawings.

[0032] like Figure 1 As shown, a medical image sequence segmentation method based on Matcher includes the following steps:

[0033] Step 1: Prepare the target medical image sequence, the reference medical image sequence, and the corresponding mask for the reference medical image sequence;

[0034] Collect target medical image sequences and reference medical image sequences from the same disease to ensure case consistency and obtain the mask of the reference medical image sequence; each of the target medical image sequence and the reference medical image sequence contains n images;

[0035] Step 2: Feature extraction of the backbone network and calculation of the correspondence matrix;

[0036] First, according to the input requirements of dinov2_vitg14, the input image size is cropped to 896*896. Finally, its data type is converted to Tensor type and stored in the same tensor, which will be used as the input of the feature extraction model dinov2_vitg14 later.

[0037] The feature extraction model obtains the reference and target image features, which can be represented as a three-dimensional array F. r ∈R n×4096×1536 and F t ∈R n×4096×1536 The following operations are performed on n target image features using a single reference image feature: First, the reference image feature remains unchanged, and the target image features are transposed. Then, matrix multiplication is performed on the reference image feature and the transposed target image feature to obtain the inner product. The inner product is divided by the norms of the reference image feature and the target image feature to obtain the correspondence matrix sim∈R. 4096×4096 ;

[0038]

[0039] Where z is r Reference image features, z t For the target image features, calculate the cosine similarity between the reference image features and the target image features to obtain the corresponding relationship matrix sim.

[0040] The above operations result in n correspondence matrices sim. Then, F... r ∈R n×4096×1536 Perform the above operation on each reference image feature through iteration, and finally obtain n. 2 There are correspondence matrices sim.

[0041] Step 3: Implement a two-way matching strategy;

[0042] The reference mask is forward matched with n correspondence matrices generated from the same reference image. Specifically, the reference mask R is... 64×64 Heisei R 4096 First, obtain the indices of non-zero elements in the vector: `maskpoints`. Then, select all rows in the corresponding relation matrix `sim` whose indices are `maskpoints`, and take the maximum value in the first dimension. Return the indices of the maximum value: `forwardpoins`. This completes the forward matching. Next, select all columns in the corresponding relation matrix `sim` whose indices are `forwardpoins`, and take the maximum value in the second dimension. Return the indices of the maximum value: `reversepoins`. Compare `reversepoins` with `maskpoints`. If the corresponding reverse matching coordinate is not on the reference mask, it means that the coordinate does not meet the condition. Keep the coordinates that meet the condition as matching points, and the layer where the matching point is located as the matching layer. Finally, n is obtained. 2 A matching layer.

[0043] Step 4: Select a matching layer to reduce the false positive rate;

[0044] To achieve multi-layer matching, the matching layer with the most inner points is first selected. It is then determined whether the number of matching points exceeds a threshold (5 in this invention). The matching points of this layer are input into the pre-trained SAM model to obtain the prediction mask and IoU. After post-processing, matching labels are obtained, while points in unmatched regions are treated as negative samples. Simultaneously, corresponding operations are performed on the layers above and below to obtain the prediction mask and IoU. The correlation is calculated, and the comparison formula is as follows:

[0045]

[0046] Where P is an empirical value, which is 0.1 in this embodiment. Thickness is the matching layer thickness. If the Q of the upper or lower layer is greater than zero, the prediction mask and IoU of the upper or lower layer are retained, and the corresponding matching label and negative sample are obtained through post-processing. Otherwise, the results of the upper or lower layer are not retained. The thinner the layer thickness, the more it depends on the information of the upper and lower layers, thereby reducing the false positive rate of matching.

[0047] Step 5: Use k-means++ clustering to process the selected matching layer, cluster the matching points into k clusters, and generate the corresponding prompt sampling set through local, instance and global sampling;

[0048] The selected matching layers and the retained upper and lower matching layers are processed separately. The matching labels are used as the point set for instance hints, and their lengths are used as the label set for instance hints. Their box coordinates are calculated and used as the box set for instance hints. Next, k-means++ clustering is performed on the matching labels to obtain the cluster centers, which are used as the point set for global hints. Their lengths are used as the label set for global hints, and the box coordinates of the cluster centers are calculated and used as the box set for global hints. For each cluster label, all points in the matching labels that belong to that cluster label are extracted and used as the point set for partial hints. Their lengths are used as the label set for partial hints, and the box coordinates of the partial point set are calculated and used as the box set for partial hints.

[0049] Step 6: Merge the local, instance, and global point sets in the prompt sampling set to obtain the merged point set. right Perform enumeration and obtain the set of points. The transformation and concatenation process yields a tensor, which is then used as input to a pre-trained SAM model to obtain the prediction mask and IoU, along with the point set. Add them together to the proposal list. After the traversal is complete, a complete list of proposals is obtained.

[0050] The sets of matching layers obtained from the filtering and the retained upper and lower matching layers are processed separately. The point sets of instance hints, global hints, and partial hints are merged into a dictionary, called the merged point set. The tag set is merged into a dictionary, and the box set is merged into a dictionary, by enumerating and traversing the dictionary. The set of points obtained through traversal Convert it to a PyTorch tensor, then create a shape of... A point set label tensor with all 1s of length is created. The negative samples obtained in step 4 are also converted into PyTorch tensors, creating a negative sample tensor with a shape equal to the length of the current negative sample tensor and all zeros. Finally, the processed point set and label set are concatenated together, and the point set label tensor and negative sample tensor are concatenated together. This concatenation, along with the corresponding box set, is used as input to the pre-trained SAM model to obtain the prediction mask and IoU, and then compared with the point set... Add them together to the proposal list. After the traversal is complete, the final list of proposals is obtained;

[0051] Step 7: Perform instance-level matching between the reference mask and the predicted mask;

[0052] By traversing the proposal list in step 6, each proposal is obtained, the prediction mask in the proposal is obtained, and the Sigmoid function is used to binarize its pixels into 1s and 0s.

[0053] Obtain the index coordinates of elements in the prediction mask that are greater than 0.5, denoted as m. p Set the points in the proposal Perform coordinate mapping, and then use this point set to select corresponding elements from the prediction mask. And calculate the total number of points within it, then calculate the purity, which is the proportion of effective pixels to the total predicted mask. Then, based on... The total number of midpoints is calculated, and the coverage is determined. The coordinates of the reference mask with values ​​greater than zero are extracted and their EMD distances are calculated with the predicted mask in the proposal. Purity and coverage are also calculated. Higher purity promotes the selection of partial-level masks, while higher coverage promotes the selection of instance-level masks. A score is calculated through a weighted combination, and the predicted mask, score, and point coordinates of the proposal are stored in the results list.

[0054]

[0055]

[0056] score=α·(1-emd)+β·purity·coverage λ

[0057] Where Num(·) is the total number of points, and α, β and λ are adjustment coefficients between different metrics, which are taken as 0.6, 0.4 and 1 respectively in this invention.

[0058] Finally, the result lists of the matching layer and the result lists of the upper and lower layers are merged and sorted in descending order of score.

[0059] Step 8: Sort the results list in descending order of score, select the top k highest quality masks (k is 5 in this invention), merge multiple masks into the final segmentation result and output it.

[0060] Example 1: Lung lesion region segmentation. First, the 3D lung dataset was sliced ​​into 2D images, resulting in 1457 diseased 2D images. The corresponding reference masks were processed and binarized, converting them into 1457 reference mask images. Then, preprocessing operations such as cropping and data type conversion were performed on the lung dataset. The results were used as input to the model to obtain the final segmentation prediction. For comparison, the lung dataset was input into Matcher to obtain Matcher's segmentation prediction, as shown below. Figure 3As shown.

[0061] The above description, in conjunction with specific / preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. Those skilled in the art can make various substitutions or modifications to these described embodiments without departing from the inventive concept, and all such substitutions or modifications should be considered within the scope of protection of the present invention.

[0062] The parts of this invention not described in detail are well-known to those skilled in the art.

Claims

1. A medical image sequence segmentation method based on Matcher, characterized in that, include: Step 1: Prepare the target medical image sequence, the reference medical image sequence, and the corresponding mask of the reference medical image sequence. The target medical image sequence and the reference medical image sequence each contain n images; Step 2: Preprocess the images in the target medical image sequence and the reference medical image sequence. Extract features from the preprocessed images using the feature extraction model dinov2_vitg14 and calculate the corresponding relationship matrix. Step 3: Match the reference mask with the corresponding relation matrix using a bidirectional matching strategy to obtain n. 2 One matching layer; Step 4: Filter the matching layer to reduce the false positive rate; To achieve multi-layer matching, the matching layer with the most inner points is selected. It is then determined whether the number of matching points exceeds a threshold. The matching points of this layer are input into the pre-trained SAM model to obtain the prediction mask and IoU. After post-processing, matching labels are obtained, while points in unmatched regions are treated as negative samples. Simultaneously, corresponding operations are performed on the layers above and below to obtain the prediction mask and IoU. The correlation is calculated, and the comparison formula is as follows: Where P is an empirical value and thickness is the matching layer thickness. If Q of the upper or lower layer is greater than zero, the prediction mask and IoU of the upper or lower layer are retained, and the corresponding matching label and negative sample are obtained through post-processing. Otherwise, the results of the upper or lower layer are not retained. The thinner the layer thickness, the more it depends on the information of the upper and lower layers, thereby reducing the false positive rate of matching. Step 5: Use k-means++ clustering to process the selected matching layer, cluster the matching points in the matching layer into k clusters, and generate the corresponding prompt sampling set through local, instance and global sampling; Step 6: Merge the local, instance, and global point sets in the prompt sampling set to obtain the merged point set. right Perform enumeration and obtain the set of points. The transformation and concatenation process yields a tensor, which is then used as input to a pre-trained SAM model to obtain the prediction mask and IoU, along with the point set. Add them together to the proposal list. After the traversal is complete, a complete list of proposals is obtained. Step 7: Perform instance-level matching between the reference mask and the prediction mask to obtain a result list consisting of the prediction mask, score, and point coordinates of the proposal; Choose a high-quality mask to improve the accuracy of the match; Step 8: Sort the results list in descending order of scores, select the top k highest quality masks, merge multiple masks into the final segmentation result and output it.

2. The medical image sequence segmentation method based on Matcher according to claim 1, characterized in that, The preprocessing described in step 2 involves cropping the input image to 896*896 pixels according to the dinov2_vitg14 input requirements, and finally converting its data type to Tensor type and storing it in the same tensor, which will then be used as input to the feature extraction model.

3. A medical image sequence segmentation method based on Matcher according to claim 1 or 2, characterized in that, The specific method for step 3 is as follows: First, the reference mask is flattened into a one-dimensional tensor, and the indices of non-zero elements are obtained. Then, using the correspondence matrix, rows corresponding one-to-one with the indices are selected, and the index of the maximum value in each row is calculated to obtain the forward matching coordinates, i.e., the forward set. Next, columns in the correspondence matrix with forward matching coordinates are selected, and the index of the maximum value in each column is calculated to obtain the reverse matching coordinates. If the corresponding reverse matching coordinate is not on the reference mask, it means that the coordinate does not meet the condition. Coordinates that meet the condition are retained as matching points, and the layer containing the matching point is designated as the matching layer. Finally, n is obtained. 2 A matching layer.

Images