Tumor labeling method based on Unet improved SAM model

Abstract

The invention relates to a tumor labeling method based on a Unet improved SAM model. The tumor labeling method comprises the following steps: acquiring a tumor image to be labeled; giving a corresponding marking prompt for the region of interest of the tumor image; providing the marking prompt and the tumor image to a Unet-based improved SAM model, wherein the Unet-based improved SAM model forms a marking mask of the region of interest according to the marking prompt; wherein the improved SAM model based on the Unet comprises an original image encoder, an original prompt encoder and an original mask decoder of the SAM model; the cascaded up-sampling decoding module corresponds to each stage of the original image encoder, the multiple newly added edge enhancement attention is arranged between the up-sampling decoding module and the original image encoder, and the edge enhancement attention is matched with the cascaded up-sampling decoding module to bring edge enhancement information. Therefore, the quality of the mark mask edge is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular, to a tumor annotation method implemented based on an improved SAM model of Unet. Background Art

[0002] The Segement Anything Model (SAM) allows users to generate mask annotations for specific regions of interest through interaction. The SAM model includes: an image encoder, a prompt encoder, and a mask decoder. The image encoder processes the image using the Vision Transformer model ViT and gradually downsamples it to generate a feature map with a size of 1 / 16 of the original image. The prompt encoder accepts sparse and dense prompts, including point-like, box-like, or mask prompts, and converts the prompts into prompt embeddings. Then, the mask decoder uses the cross-attention mechanism to integrate the prompt embeddings from the feature map and the prompt encoder, and generates different masks for the image according to different prompts. In the medical field, the boundaries between some regions of interest (tumor lesions) and the background (organs and body tissues where the tumor lesions are located) are complex, and most of them are weak boundaries, complex edges, and variable shapes that are difficult to distinguish clearly. Applying the Segement Anything Model directly to medical image annotation has poor effects. Summary of the Invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present invention provides a tumor annotation method implemented based on an improved SAM model of Unet.

[0004] In a first aspect, the present invention provides a tumor annotation method implemented based on an improved SAM model of Unet, including: Obtaining a tumor image to be annotated; Giving corresponding annotation prompts for the region of interest of the tumor image, and the annotation prompts adopt any one of point-like, box-like, or mask prompts; Providing the annotation prompts and the tumor image to an improved SAM model based on Unet, and the improved SAM model based on Unet forms an annotation mask for the region of interest according to the annotation prompts; wherein, the improved SAM model based on Unet includes: the original image encoder, the original prompt encoder, and the original mask decoder of the SAM model; and additionally, cascaded upsampling decoding modules corresponding to each stage of the original image encoder are newly added, and a plurality of edge enhancement attentions are newly set between the upsampling decoding modules and the original image encoder.

[0005] Furthermore, in the improved SAM model based on Unet, the original image encoder is improved according to the Unet architecture, including: Configure upsampling decoding modules corresponding to each encoding stage of the original image encoder. The upsampling decoding modules are cascaded, and the cascaded upsampling decoding modules form four decoding stages: The upsampling decoding module in the first decoding stage decodes the tumor image encoding feature map output by the last decoding stage of the original image encoder; in the second to fourth decoding stages, for any decoding stage, the upsampling decoding module upsamples the pointwise addition result of the tumor image decoding feature map output by the upsampling decoding module in its previous decoding stage and the edge-enhanced tumor image feature map in its corresponding decoding stage to obtain a multi-scale edge-enhanced tumor image decoding feature map. After the multi-scale edge-enhanced tumor image decoding feature maps are fused step by step through the upsampling decoding modules, high-edge-quality data is provided for the generation of the final annotation mask.

[0006] Furthermore, the edge-enhanced tumor image feature maps used in the decoding stage come from the edge-enhanced attention; the three edge-enhanced attentions respectively receive the tumor image encoding feature maps output by the first to third encoding stages of the original image encoder, respectively receive the edge feature maps of the corresponding scales of the tumor image, and respectively receive the outputs of the corresponding previous upsampling decoding modules, and generate edge-enhanced tumor image feature maps by combining the three through attention.

[0007] Furthermore, for the tumor image encoding feature map output by the encoding stage, the edge-enhanced attention obtains a channel-reduced tumor image encoding feature map through convolution layer processing; For the edge feature maps in the Laplacian pyramid of the tumor image, the edge-enhanced attention converts them into edge feature attentions through convolution and non-linear activation functions; the edge feature attentions are multiplied pointwise with the corresponding channel-reduced tumor image encoding feature maps; For the tumor image decoding feature map output by the previous upsampling decoding module, the tumor image decoding feature map attention obtained through 1×1 convolution layer and activation function processing, the edge-enhanced attention converts the tumor image decoding feature map attention into a tumor image decoding feature map inverse attention, and the tumor image decoding feature map inverse attention is multiplied pointwise with the corresponding channel-reduced tumor image encoding feature map; the edge-enhanced attention processes the tumor image decoding feature map attention using Gaussian filtering, downsampling, and upsampling operations, and the tumor image decoding feature map attention subtracts the processing results of Gaussian filtering, downsampling, and upsampling to obtain a tumor image decoding feature map edge attention, and the tumor image decoding feature map edge attention is multiplied pointwise with the corresponding channel-reduced tumor image encoding feature map; The result obtained by the tumor image encoding feature map with reduced channels based on three kinds of attention is concatenated and then processed by residual convolution and gated attention to obtain a comprehensive edge-enhanced attention feature map; after the tumor image encoding feature map with reduced channels and the comprehensive edge-enhanced attention feature map are multiplied point by point, an edge-enhanced tumor image feature map is generated through a convolutional attention module.

[0008] Furthermore, the calculation process of the Laplacian pyramid is as follows: The tumor image is iteratively subjected to Gaussian filtering and downsampling to obtain a multi-layer Gaussian pyramid from large to small. For any two adjacent layers of the Gaussian pyramid, one layer of the Laplacian pyramid is obtained by subtracting the upsampling result of the lower layer from the upper layer.

[0009] Furthermore, the convolutional attention module includes channel attention and spatial attention.

[0010] Furthermore, the original mask decoder of SAM uses a dual-path attention mechanism to integrate the tumor image encoding feature map of size 1 / 16 and the prompt embedding of the prompt encoder. The decoding result of the original tumor image encoding feature map is obtained through upsampling decoding of SAM. The decoding result and the edge-enhanced tumor image decoding feature map are fused by point-by-point addition. The fused result of point-by-point addition is multiplied by the output generated by the multi-layer perceptron of the original mask decoder based on the dual-path attention output to obtain the annotation mask of the region of interest.

[0011] Furthermore, the SAM model improved based on Unet is trained. The parameters of the original SAM model are frozen during training, and the loss function for training adopts a combination of cross-entropy loss and dice loss between the true annotation mask and the predicted annotation mask.

[0012] In a second aspect, the present invention provides a tumor annotation device implemented based on a SAM model improved based on Unet, including: at least one processing unit, the processing unit is connected to a storage unit through a bus unit, the storage unit stores a computer program, and the processing unit realizes the tumor annotation method implemented based on the SAM model improved based on Unet by running the computer program stored in the storage unit.

[0013] In a third aspect, the present invention provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed, it realizes the tumor annotation method implemented based on the SAM model improved based on Unet.

[0014] The above technical solutions provided by the embodiments of the present invention have the following advantages compared with the prior art: In the newly added decoder of the present invention, in the second to fourth decoding stages, in any decoding stage, the upsampling decoding module upsamples the point-by-point addition result of the decoded feature map of the tumor image output by the upsampling decoding module of its previous decoding stage and the edge-enhanced tumor image feature map of its corresponding decoding stage to obtain a multi-scale edge-enhanced decoded feature map of the tumor image. After the multi-scale edge-enhanced decoded feature map of the tumor image is gradually fused by the upsampling decoding module, it provides data with high edge quality for the generation of the final annotation mask, which can improve the quality of the annotation.

[0015] For the edge feature map in the Laplacian pyramid of the tumor image, the edge enhancement attention converts it into edge feature attention through convolution and a non-linear activation function. The edge feature attention is multiplied point by point with the corresponding channel-reduced encoded feature map of the tumor image to enhance the edge information in the encoded feature map of the tumor image using the edge information in the edge feature map. For the decoded feature map of the tumor image output by the previous upsampling decoding module, the decoded feature map attention obtained through a 1×1 convolutional layer and an activation function is processed. The edge enhancement attention converts the decoded feature map attention of the tumor image into the anti-attention of the decoded feature map of the tumor image, and the anti-attention of the decoded feature map of the tumor image is multiplied point by point with the corresponding channel-reduced encoded feature map of the tumor image. The semantic confidence in the decoded feature map of the tumor image in the previous decoding stage is relatively high, but the resolution is low and the edges are relatively rough. The anti-attention deletes the currently predicted attention region from the decoded feature map of the tumor image in each layer to obtain the non-attention region, and the non-attention region is provided to the next layer. The edge enhancement feature is used to further perform attention analysis on the non-attention region, thereby guiding the entire upsampling decoding module to sequentially discover and supplement the details that should belong to the attention region from the non-attention region. The edge enhancement attention processes the decoded feature map attention of the tumor image using Gaussian filtering, downsampling, and upsampling operations. The decoded feature map attention of the tumor image minus the processing result of Gaussian filtering, downsampling, and upsampling gives the edge attention of the decoded feature map of the tumor image, and the edge attention of the decoded feature map of the tumor image is multiplied point by point with the corresponding channel-reduced encoded feature map of the tumor image to enhance the common edge information between the decoded feature map of the tumor image and the encoded feature map of the tumor image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings herein are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present invention and, together with the specification, are used to explain the principles of the present invention.

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0018] Figure 1 Flowchart of a tumor annotation method implemented based on an improved SAM model of Unet provided by an embodiment of the present invention; Figure 2 Architecture diagram of the improved SAM model of Unet provided by an embodiment of the present invention; Figure 3 Schematic diagram of the Laplacian pyramid provided by an embodiment of the present invention; Figure 4 Schematic diagram of edge attention provided by an embodiment of the present invention; Figure 5 Schematic diagram of a tumor annotation device implemented based on an improved SAM model of Unet provided by an embodiment of the present invention. Detailed implementation manners

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0020] It should be noted that in this article, the term "including", "comprising", or any other variation thereof is intended to cover a non-exclusive inclusion, such that a process, method, article, or device including a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or device. Without further limitation, an element defined by the phrase "including a..." does not exclude the presence of additional identical elements in the process, method, article, or device including the element.

[0021] Embodiment 1 Refer to Figure 1 As shown, the tumor annotation method implemented based on the improved SAM model of Unet provided by this application includes: S100, obtain the tumor image to be annotated. The tumor image can be a tumor image obtained by CT imaging, a tumor image obtained by magnetic resonance imaging, a tumor image obtained by PET imaging, or a tumor image obtained by ultrasonic imaging. This application supports the annotation of tumor images in multiple modalities.

[0022] S200, give corresponding annotation prompts for the region of interest of the tumor image, and the annotation prompt can be any one of dot, box, or mask prompts.

[0023] S300, provide the annotation prompt and the tumor image to the SAM model improved based on Unet, and the SAM model improved based on Unet forms an annotation mask of the region of interest according to the annotation prompt.

[0024] To improve the effect of the SAM model in the field of medical images, this application provides the SAM model improved based on Unet. As Figure 2 shown, the SAM model improved based on Unet includes: The original image encoder, original prompt encoder, and original mask decoder of the SAM model; and additional cascaded upsampling decoding modules corresponding to each stage of the original image encoder, and multiple edge enhancement attentions added between the upsampling decoding module and the original image encoder.

[0025] In the SAM model improved based on Unet, the original image encoder of the SAM model processes the tumor image using the ViT model , the ViT model includes four encoding stages, and each encoding stage uses visual attention to gradually downsample and extract multi-scale tumor image encoding feature maps of sizes 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of the tumor image , the visual attention of each encoding stage of the ViT model includes: multi-level window attention, followed by global attention after multi-level window attention, and the global attention of each stage outputs the tumor image encoding feature map externally.

[0026] In the SAM model improved based on Unet, the original prompt encoder of the SAM model receives the annotation prompt provided in step S200 and converts the annotation prompt into a prompt embedding.

[0027] In the SAM model improved based on Unet, the original image encoder is improved according to the Unet architecture, including: Configure upsampling decoding modules corresponding to each encoding stage of the original image encoder. The upsampling decoding modules are cascaded, and the cascaded upsampling decoding modules form four decoding stages. The upsampling decoding module can be implemented by a convolutional network or by an attention module including self-attention and cross-attention. The upsampling decoding module of the first decoding stage decodes the tumor image encoding feature map output by the last decoding stage of the original image encoder ; in the second to fourth decoding stages, for any decoding stage, the upsampling decoding module decodes the tumor image decoding feature map output by the upsampling decoding module of its previous decoding stage The result of point - by - point addition of the edge - enhanced tumor image feature map in its corresponding decoding stage is upsampled to obtain a multi - scale edge - enhanced tumor image decoding feature map. After the multi - scale edge - enhanced tumor image decoding feature maps are gradually fused by the upsampling decoding module, it provides data with high edge quality for the generation of the final annotation mask.

[0028] Among them, the edge - enhanced tumor image feature map used in the decoding stage comes from the edge - enhanced attention. The three edge - enhanced attentions respectively receive the tumor image encoding feature maps output from the first to the third encoding stages of the original image encoder , where k is the encoding stage, and they respectively receive the edge feature maps of the corresponding scales of the tumor image , and respectively receive the output from the corresponding previous upsampling decoding module, and combine the three to generate an edge - enhanced tumor image feature map through edge enhancement.

[0029] Specifically, as Figure 4 shown, for the tumor image encoding feature map output in the encoding stage , the edge - enhanced attention is processed by a convolutional layer with a convolution kernel of 3×3 to reduce the number of channels of the tumor image encoding feature map, obtaining a channel - reduced tumor image encoding feature map , where is the convolution of the tumor image encoding feature map .

[0030] Calculate the Laplacian pyramid of the tumor image. The multi - level feature maps in the Laplacian pyramid form a multi - scale edge feature map , and the layer where the edge feature map is located corresponds to the encoding stage. In the specific implementation process, as Figure 4 shown, the tumor image is iteratively subjected to Gaussian filtering and downsampling to obtain a multi - layer Gaussian pyramid from large to small. For any two adjacent layers of the Gaussian pyramid, one layer of the Laplacian pyramid is obtained by subtracting the upsampled result of the lower layer from the upper layer. As Figure 4, the L1 layer of the Gaussian pyramid is obtained by performing Gaussian filtering and downsampling on the tumor image. Through downsampling, the L1 layer of the Gaussian pyramid is 1 / 2 the size of the original tumor image, corresponding to the size of the encoded feature map of the tumor image in the first encoding stage of the ViT model; the L2 layer of the Gaussian pyramid is obtained by performing Gaussian filtering and downsampling on the L1 layer of the Gaussian pyramid. Through downsampling, the L2 layer of the Gaussian pyramid is 1 / 4 the size of the original tumor image, corresponding to the size of the encoded feature map of the tumor image in the second encoding stage of the ViT model; the L3 layer of the Gaussian pyramid is obtained by performing Gaussian filtering and downsampling on the L2 layer of the Gaussian pyramid. Through downsampling, the L3 layer of the Gaussian pyramid is 1 / 8 the size of the original tumor image, corresponding to the size of the encoded feature map of the tumor image in the third encoding stage of the ViT model; the L4 layer of the Gaussian pyramid is obtained by performing Gaussian filtering and downsampling on the L3 layer of the Gaussian pyramid. Through downsampling, the L4 layer of the Gaussian pyramid is 1 / 16 the size of the original tumor image, corresponding to the size of the encoded feature map of the tumor image in the fourth encoding stage of the ViT model; the L1 layer of the Laplacian pyramid is obtained by subtracting the upsampled result of the L2 layer of the Gaussian pyramid from the tumor image, with a size of 1 / 2 the size of the original tumor image. The L2 layer of the Laplacian pyramid is obtained by subtracting the upsampled result of the L3 layer of the Gaussian pyramid from the L2 layer of the Gaussian pyramid, with a size of 1 / 4 the size of the original tumor image. The L3 layer of the Laplacian pyramid is obtained by subtracting the upsampled result of the L4 layer of the Gaussian pyramid from the L3 layer of the Gaussian pyramid, with a size of 1 / 8 the size of the original tumor image. The Laplacian pyramid captures second-order changes in the tumor image at multiple scales and is used to extract high-frequency features of the edges and contours of the tumor image.

[0031] For the edge feature map in the Laplacian pyramid of the tumor image , the edge enhancement attention converts it into edge feature attention through convolution and a non-linear activation function: , where is the non-linear activation function, is the convolution performed on the edge feature map . The edge feature attention is multiplied element-wise with the corresponding channel-reduced encoded feature map of the tumor image to enhance the edge information in the encoded feature map of the tumor image using the edge information in the edge feature map.

[0032] For the decoded feature map of the tumor image output by the upper-level upsampling decoding module , the attention of the decoded feature map of the tumor image obtained by processing through a 1×1 convolutional layer and an activation function , the edge enhancement attention converts the attention of the decoded feature map of the tumor image into the anti-attention of the decoded feature map of the tumor image: , the anti-attention of the decoded feature map of the tumor image The tumor image encoding feature map with corresponding channel dimensionality reduction Perform element-wise multiplication. In the tumor image decoding feature map of the previous decoding stage, the semantic confidence is relatively high, but the resolution is low and the edges are relatively rough. The non-attended regions are obtained by removing the currently predicted attention regions from the tumor image decoding feature map of each layer through inverse attention, and the non-attended regions are provided to the next layer. The edge enhancement features are used to further perform attention analysis on the non-attended regions, thereby guiding the entire upsampling decoding module to sequentially discover and supplement the details that should belong to the attention regions from the non-attended regions.

[0033] The edge enhancement attention processes the tumor image decoding feature map attention using Gaussian filtering, downsampling, and upsampling operations , and the tumor image decoding feature map attention minus the processing results of Gaussian filtering, downsampling, and upsampling gives the tumor image decoding feature map edge attention: ; The tumor image decoding feature map edge attention is element-wise multiplied with the corresponding channel dimensionality-reduced tumor image encoding feature map to enhance the edge information shared by the tumor image decoding feature map and the tumor image encoding feature map.

[0034] The results obtained by the channel dimensionality-reduced tumor image encoding feature map based on three kinds of attention are concatenated and then processed through residual convolution and gated attention to obtain a comprehensive edge enhancement attention feature map; after the channel dimensionality-reduced tumor image encoding feature map is element-wise multiplied with the comprehensive edge enhancement attention feature map, an edge-enhanced tumor image feature map is generated through a convolutional attention module. The convolutional attention module includes channel attention and spatial attention.

[0035] S400, form an annotation contour or annotation layer according to the annotation mask, and add the annotation contour or annotation layer to the corresponding region of interest of the tumor image to achieve automatic annotation.

[0036] The original mask decoder of SAM uses a two-way attention mechanism to integrate the tumor image encoding feature map of size 1 / 16 and the prompt embedding of the prompt encoder. The decoding result of the original tumor image encoding feature map is obtained through upsampling decoding of SAM. The decoding result and the edge-enhanced tumor image decoding feature map are fused through element-wise addition, and the fused result of element-wise addition is multiplied by the output generated by the multi-layer perceptron of the original mask decoder based on the two-way attention output to obtain the annotation mask of the region of interest. The annotation mask of the region of interest is formed according to the annotation prompt and the tumor image features.

[0037] Add a new decoder branch of the UNet architecture to the original mask decoder. The new decoder branch gradually fuses the multi-scale edge-enhanced tumor image feature maps into the tumor image decoding feature maps layer by layer. Then, the final tumor image decoding feature maps are combined with the decoding results of the original tumor image encoding feature maps of the original decoder point by point. The tumor image decoding feature maps bring enhanced edge information for better annotation mask prediction.

[0038] This application solves the problem of annotating weak-boundary regions of interest in tumor images through edge enhancement attention. It uses the Laplacian pyramid to extract edge information and transmits and enhances the edge information between the original image encoder of the SAM model and the newly added upsampling decoding module to form a new decoder. The new decoder adopts a multi-scale fusion strategy of the Unet architecture to achieve the fusion of multi-scale edge enhancement information, enabling the model to focus on the edge details of the regions of interest, improving the annotation accuracy, and achieving accurate annotation of tumors with complex edges and variable shapes.

[0039] After introducing the newly added cascaded upsampling decoding modules corresponding to each stage of the original image encoder in SAM and adding multiple edge enhancement attentions between the upsampling decoding module and the original image encoder, it is necessary to train the SAM model improved based on Unet. The parameters of the original SAM model are frozen during training, and the loss function for training adopts a combination of cross-entropy loss and dice loss between the true annotation mask and the predicted annotation mask.

[0040] Example 2 Refer to Figure 5 As shown, the embodiment of the present invention provides a tumor annotation device implemented based on the SAM model improved by Unet, including: at least one processing unit, which is connected to the storage unit through the bus unit. The storage unit, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the software programs, computer-executable programs, and modules corresponding to the tumor annotation method implemented based on the SAM model improved by the present invention. The processing unit realizes the above-mentioned tumor annotation method implemented based on the SAM model improved by Unet by running the software programs, computer-executable programs, and modules stored in the storage unit, including: Obtain the tumor image to be annotated; Give corresponding annotation prompts for the region of interest of the tumor image, and the annotation prompt adopts any one of dot-shaped, box-shaped, or mask prompts; Provide the annotation prompt and the tumor image to the SAM model improved based on Unet. The SAM model improved based on Unet forms an annotation mask for the region of interest according to the annotation prompt. Among them, the SAM model improved based on Unet includes: the original image encoder, the original prompt encoder, and the original mask decoder of the SAM model; and additionally, cascaded upsampling decoding modules corresponding to each stage of the original image encoder are added, and multiple edge enhancement attentions are added between the upsampling decoding modules and the original image encoder.

[0041] Certainly, the computer program stored in the storage unit of the tumor annotation device implemented by the SAM model improved based on Unet provided in the embodiments of the present invention is not limited to the method operations described above, and can also execute the related operations in the tumor annotation method implemented by the SAM model improved based on Unet provided in any embodiment of the present invention.

[0042] Embodiment 3 The embodiments of the present invention provide a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed, it implements the tumor annotation method implemented by the SAM model improved based on Unet, including: Obtain the tumor image to be annotated; Give corresponding annotation prompts for the region of interest of the tumor image, and the annotation prompt can adopt any one of dot, box or mask prompts; Provide the annotation prompt and the tumor image to the SAM model improved based on Unet. The SAM model improved based on Unet forms an annotation mask for the region of interest according to the annotation prompt. Among them, the SAM model improved based on Unet includes: the original image encoder, the original prompt encoder, and the original mask decoder of the SAM model; and additionally, cascaded upsampling decoding modules corresponding to each stage of the original image encoder are added, and multiple edge enhancement attentions are added between the upsampling decoding modules and the original image encoder.

[0043] The computer program stored in the computer-readable storage medium provided in the embodiments of the present invention is not limited to the method operations described above, and can also execute the related operations in the tumor annotation method implemented by the SAM model improved based on Unet provided in any embodiment of the present invention.

[0044] In the embodiments provided by the present invention, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces, and the indirect coupling or communication connection of structures or units can be in electrical, mechanical or other forms.

[0045] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0046] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0047] The above are only the specific implementation manners of the present invention, enabling those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A tumor annotation method implemented based on an improved SAM model of Unet, characterized in that, Including: Obtain tumor images to be annotated; Give corresponding annotation prompts for the region of interest of the tumor image, and the annotation prompt adopts any one of dot, box or mask prompts; Provide the annotation prompt and the tumor image to the SAM model improved based on Unet. The SAM model improved based on Unet forms an annotation mask for the region of interest according to the annotation prompt. Among them, the SAM model improved based on Unet includes: the original image encoder, the original prompt encoder and the original mask decoder of the SAM model; and additionally added cascaded upsampling decoding modules corresponding to each stage of the original image encoder, and a plurality of edge enhancement attentions arranged between the upsampling decoding module and the original image encoder.

2. The tumor annotation method implemented based on the improved SAM model of Unet according to claim 1, characterized in that, In the SAM model improved based on Unet, the original image encoder is improved according to the Unet architecture, including: Configure upsampling decoding modules corresponding to each encoding stage of the original image encoder. The upsampling decoding modules are cascaded, and the cascaded upsampling decoding modules form four decoding stages: the upsampling decoding module of the first decoding stage decodes the tumor image encoding feature map output by the last decoding stage of the original image encoder; in the second to fourth decoding stages, for any decoding stage, the upsampling decoding module upsamples the point-by-point addition result of the tumor image decoding feature map output by the upsampling decoding module of its previous decoding stage and the edge-enhanced tumor image feature map of its corresponding decoding stage to obtain a multi-scale edge-enhanced tumor image decoding feature map. After the multi-scale edge-enhanced tumor image decoding feature maps are gradually fused through the upsampling decoding module, high-edge-quality data is provided for the generation of the final annotation mask.

3. The tumor annotation method implemented based on the improved SAM model of Unet according to claim 2, characterized in that, The edge-enhanced tumor image feature map used in the decoding stage comes from the edge enhancement attention; the three edge enhancement attentions respectively receive the tumor image encoding feature maps output by the first to third encoding stages of the original image encoder, respectively receive the edge feature maps of the corresponding scales of the tumor image, and respectively receive the outputs of the corresponding upper-level upsampling decoding modules, and combine the three through attention to generate an edge-enhanced tumor image feature map.

4. The tumor annotation method implemented based on the improved SAM model of Unet according to claim 3, wherein, For the tumor image encoding feature map output by the encoding stage, the edge enhancement attention processes it through a convolutional layer to obtain a channel-reduced tumor image encoding feature map; For the edge feature map in the Laplacian pyramid of the tumor image, the edge enhancement attention converts it into edge feature attention through convolution and a non-linear activation function; The edge feature attention is multiplied point by point with the corresponding channel-reduced tumor image encoding feature map; For the tumor image decoding feature map output by the upper-level upsampling decoding module, the tumor image decoding feature map attention is obtained through a 1×1 convolutional layer and an activation function. The edge enhancement attention converts the tumor image decoding feature map attention into the tumor image decoding feature map anti-attention, and the tumor image decoding feature map anti-attention is multiplied point by point with the correspondingly channel-reduced tumor image encoding feature map; the edge enhancement attention processes the tumor image decoding feature map attention using Gaussian filtering, downsampling, and upsampling operations. The tumor image decoding feature map attention minus the processing results of Gaussian filtering, downsampling, and upsampling gives the tumor image decoding feature map edge attention, and the tumor image decoding feature map edge attention is multiplied point by point with the correspondingly channel-reduced tumor image encoding feature map. The results obtained from the channel-reduced tumor image encoding feature map based on the three kinds of attention are concatenated and then processed through residual convolution and gated attention to obtain the comprehensive edge enhancement attention feature map; after the channel-reduced tumor image encoding feature map and the comprehensive edge enhancement attention feature map are multiplied point by point, an edge-enhanced tumor image feature map is generated through a convolutional attention module.

5. The tumor annotation method implemented based on the improved SAM model of Unet according to claim 4, characterized in that, The calculation process of the Laplacian pyramid is as follows: The tumor image is iteratively subjected to Gaussian filtering and downsampling to obtain multiple layers of Gaussian pyramids from large to small. For any two adjacent layers of the Gaussian pyramid, one layer of the Laplacian pyramid is obtained by subtracting the upsampled result of the lower layer from the upper layer.

6. The tumor annotation method implemented based on the improved SAM model of Unet according to claim 4, wherein, The convolutional attention module includes channel attention and spatial attention.

7. The tumor annotation method implemented based on the improved SAM model of Unet according to claim 1, wherein, The original mask decoder of SAM uses a dual-path attention mechanism to integrate the tumor image encoding feature map of size 1 / 16 and the prompt embedding of the prompt encoder. Through upsampling decoding of SAM, the decoding result of the original tumor image encoding feature map is obtained. The decoding result and the edge-enhanced tumor image decoding feature map are fused through point-by-point addition. The fused result of point-by-point addition is multiplied by the output generated by the multi-layer perceptron of the original mask decoder based on the dual-path attention output to obtain the annotation mask of the region of interest.

8. The tumor annotation method implemented based on the improved SAM model of Unet according to claim 1, characterized in that, The SAM model improved based on Unet is trained. The parameters of the original SAM model are frozen during training, and the loss function for training adopts a combination of cross-entropy loss and dice loss between the true annotation mask and the predicted annotation mask.

9. A tumor annotation device implemented based on an improved SAM model of Unet, comprising: At least one processing unit, the processing unit is connected to the storage unit through a bus unit, characterized in that the storage unit stores a computer program, and the processing unit realizes the tumor annotation method implemented by the SAM model improved based on Unet as described in any one of claims 1-8 by running the computer program stored in the storage unit.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed, it realizes the tumor annotation method implemented by the SAM model improved based on Unet as described in any one of claims 1-8.

Images