Three-dimensional medical image segmentation method and device, terminal equipment and computer program product

Abstract

The invention is suitable for the technical field of computers, and provides a three-dimensional medical image segmentation method and device, terminal equipment and a computer program product, and the method comprises the steps: obtaining an initialized first input image in a no-prompt mode; obtaining a first feature of the first input image based on an improved image encoder; obtaining a first multi-scale fusion feature of the first input image based on a multi-scale adapter and the first feature; and based on the mask decoder and the first multi-scale fusion feature, obtaining a first segmentation mask of the first input image and outputting the first segmentation mask. According to the embodiment of the invention, the accuracy of three-dimensional medical image segmentation can be effectively improved.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and in particular, relates to a three-dimensional medical image segmentation method, apparatus, terminal equipment and computer program product. Background Art

[0002] Three-dimensional medical image segmentation is of great significance in the medical field, including disease diagnosis, treatment and medical research. At present, when the original SAM (Segment Anything Model) model is applied to the field of three-dimensional medical image segmentation technology, especially when image segmentation is performed in silent scenes, the accuracy is low and accurate segmentation results cannot be obtained, which is not conducive to disease diagnosis and treatment. Summary of the invention

[0003] In view of this, embodiments of the present application provide a three-dimensional medical image segmentation method, apparatus, terminal device, and computer program product to improve the accuracy of three-dimensional medical image segmentation.

[0004] A first aspect of an embodiment of the present application provides a three-dimensional medical image segmentation method, the three-dimensional medical image segmentation method comprising:

[0005] In silent mode, obtain the initialized first input image;

[0006] Based on the improved image encoder, obtaining a first feature of the first input image;

[0007] Based on the multi-scale adapter and the first feature, obtaining a first multi-scale fusion feature of the first input image;

[0008] Based on the mask decoder and the first multi-scale fusion feature, a first segmentation mask of the first input image is obtained and output.

[0009] In one embodiment, the three-dimensional medical image segmentation method further includes:

[0010] In the prompt mode, obtaining an initialized second input image and initial prompt data;

[0011] Obtaining a prompt embedding feature of the initial prompt data based on a semantic-aware prompt encoder;

[0012] Based on the improved image encoder, obtaining a second feature of the second input image;

[0013] Based on the multi-scale adapter and the second feature, obtaining a second multi-scale fusion feature of the second input image;

[0014] Based on the mask decoder, the second multi-scale fusion feature and the hint embedding feature, a second segmentation mask of the second input image is obtained and output.

[0015] In one embodiment, the improved image encoder includes a first adapter inserted before a multi-head attention layer of each transformer block in the image encoder, and a second adapter inserted after the multi-head attention layer;

[0016] The first adapter includes a fully automatic branch and a semi-automatic branch;

[0017] The fully automatic branch includes an implicit gating module, a downsampling layer, a 3D convolution layer, and an upsampling layer;

[0018] The semi-automatic branch includes an explicit gating module, a downsampling layer, a 3D convolution layer, and an upsampling layer;

[0019] The second adapter includes a downsampling layer, a 3D convolution layer, and an upsampling layer.

[0020] In one embodiment, obtaining the first feature of the first input image includes:

[0021] The first input image is input into the improved image encoder, and in the improved image encoder, feature extraction is performed on the first input image through the fully automatic branch of the first adapter, the multi-head attention layer and the second adapter to obtain a first feature.

[0022] In one embodiment, obtaining the second feature of the second input image includes:

[0023] The second input image is input into the improved image encoder, and in the improved image encoder, feature extraction is performed on the second input image through the semi-automatic branch of the first adapter, the multi-head attention layer and the second adapter to obtain a second feature.

[0024] In one embodiment, the acquiring a first multi-scale fusion feature of the first input image based on the multi-scale adapter and the first feature includes:

[0025] Inputting the first feature into the multi-scale adapter to obtain a first multi-scale feature;

[0026] The first multi-scale features are fused to obtain the first multi-scale fused features.

[0027] In one embodiment, the acquiring a second multiscale fusion feature of the second input image based on the multiscale adapter and the second feature includes:

[0028] Inputting the second feature into the multi-scale adapter to obtain a second multi-scale feature;

[0029] The second multi-scale features are fused to obtain the second multi-scale fused features.

[0030] A second aspect of an embodiment of the present application provides a three-dimensional medical image segmentation device, the three-dimensional medical image segmentation device comprising:

[0031] A data processing module, used for obtaining an initialized first input image in a silent mode;

[0032] A feature acquisition module, configured to acquire a first feature of the first input image based on an improved image encoder;

[0033] a feature fusion module, configured to obtain a first multiscale fusion feature of the first input image based on a multiscale adapter and the first feature;

[0034] A mask decoding module is used to obtain and output a first segmentation mask of the first input image based on a mask decoder and the first multi-scale fusion feature.

[0035] A third aspect of an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the three-dimensional medical image segmentation method as described in the first aspect of the embodiment of the present application are implemented.

[0036] A fourth aspect of the embodiments of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the three-dimensional medical image segmentation method as described in the first aspect of the embodiments of the present application are implemented.

[0037] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the three-dimensional medical image segmentation method as described in the first aspect of the embodiment of the present application are implemented.

[0038] The first aspect of the embodiment of the present application provides a three-dimensional medical image segmentation method, which obtains an initialized first input image in silent mode; obtains a first feature of the first input image based on an improved image encoder; obtains a first multi-scale fusion feature of the first input image based on a multi-scale adapter and the first feature; obtains and outputs a first segmentation mask of the first input image based on a mask decoder and the first multi-scale fusion feature. The embodiment of the present application can effectively improve the accuracy of three-dimensional medical image segmentation.

[0039] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 is a schematic diagram of the MSPT architecture provided in the embodiment of the present application;

[0042] Figure 2 This is a first flow chart of the three-dimensional medical image segmentation method provided in the embodiment of the present application;

[0043] Figure 3 is a schematic diagram of the structure of the adapter provided in an embodiment of the present application;

[0044] Figure 4 This is a second flow chart of the three-dimensional medical image segmentation method provided in an embodiment of the present application;

[0045] Figure 5 It is a semantic-aware cue encoder provided in an embodiment of the present application;

[0046] Figure 6 It is a schematic diagram of the data processing flow of the multi-scale adapter and the mask decoder provided in the embodiment of the present application;

[0047] Figure 7 It is the input prompt data type provided in the embodiment of the present application;

[0048] Figure 8 is a schematic diagram of the structure of a three-dimensional medical image segmentation device provided in an embodiment of the present application;

[0049] Fig. 9 It is a schematic diagram of the structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0051] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0052] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0053] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. "Multiple" means "two" or "more than two".

[0054] Three-dimensional medical image segmentation is of great significance in the medical field, including disease diagnosis, treatment and medical research. At present, when the original SAM model is applied to the field of three-dimensional medical image segmentation technology, especially when image segmentation is performed in silent scenes, the accuracy is low and accurate segmentation results cannot be obtained, which is not conducive to the diagnosis and treatment of diseases.

[0055] The embodiment of the present application provides a 3D medical image segmentation method, which obtains an initialized first input image in silent mode; obtains a first feature of the first input image based on an improved image encoder; obtains a first multiscale fusion feature of the first input image based on a multiscale adapter and the first feature; obtains and outputs a first segmentation mask of the first input image based on a mask decoder and the first multiscale fusion feature. The embodiment of the present application can effectively improve the accuracy of 3D medical image segmentation.

[0056] Embodiment 1

[0057] SAM is a promptable segmentation model. The original SAM model includes an image encoder, a prompt encoder, and a mask decoder. It can generate segmentation masks based on prompts such as points, boxes, and texts provided by the user. Currently, when the original SAM model is applied to three-dimensional medical image segmentation, especially when image segmentation is performed in unprompted scenarios, the accuracy is low and it has certain limitations.

[0058] The present application has made corresponding improvements on the basis of the original SAM model. The improved image encoder mentioned below refers to inserting an adapter before and after the multi-head attention layer of each transformer block in the image encoder of the original SAM model, which can be applicable to both unprompted and prompted scenarios. It solves the limitation of the current SAM model that the accuracy of image segmentation is low in unprompted scenarios, and also improves the accuracy of image segmentation in prompted scenarios. The improved image encoder still uses the non-hierarchical ViT (Vision Transformer) structure, and introduces a multi-scale adapter before the mask decoder to extract hierarchical features of different levels in ViT, which can better capture information of different scales and improve the adaptability of the model.

[0059] It can be understood that the image encoder of the original SAM model contains multiple transformer blocks. The present application inserts two adapters into each of the multiple transformer blocks in the image encoder of the SAM model, wherein one adapter is inserted before the multi-head attention layer of the transformer block, and the other adapter is inserted after the multi-head attention layer of the transformer block.

[0060] The MSPT (Medical Self-Prompting Fine-Tuning Framework) architecture based on the SAM model proposed in this application is as follows Figure 1 As shown, it mainly includes:

[0061] Improved Image Encoder, used to extract features of input images;

[0062] Multi-scale Adapter, used to extract features of different scales;

[0063] Semantic-aware Prompt Encoder, which processes the initial prompt data and generates task-related prompt embedding information, thus giving the model semantic prompting capabilities;

[0064] Mask Decoder, which is used to generate the final segmentation mask to achieve 3D medical image segmentation.

[0065] like Figure 2 As shown, the three-dimensional medical image segmentation method provided in the embodiment of the present application includes the following steps S1 to S4:

[0066] Step S1: In silent mode, obtain an initialized first input image and proceed to step S2.

[0067] In the application, the silent mode means that the given input data only includes various three-dimensional medical images, including but not limited to computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), single-photon emission computed tomography (SPECT), digital subtraction angiography (DSA), micro-computed tomography (Micro-CT) and other images.

[0068] In the application, given an input image Before the input image is input to the improved image encoder, initialization processing is required. Reshape That is, the first input image is used to improve the efficiency of subsequent processing, where B, N, H and W represent the batch size, the number of adjacent slices, the height of the slice and the width of the slice, respectively.

[0069] In the application, when initializing, the following may also be included but not limited to:

[0070] The resampling process resamples the input image to a uniform target spacing to ensure that the input images have the same physical resolution and eliminate the resolution differences caused by different devices or scanning parameters;

[0071] The sliding slicing process divides the input image into multiple two-dimensional slices along the depth direction. These two-dimensional slices can be reshaped as needed to fit the model input shape;

[0072] The normalization process standardizes the image intensity, making the model more robust to images with different intensity distributions. The normalized image can be directly used for reshaping;

[0073] The data enhancement process uses a variety of data enhancement techniques, such as random rotation, flipping, erasing, cropping, scaling, translation, posterization, contrast adjustment, brightness modification, and sharpening.

[0074] Step S2: Based on the improved image encoder, obtain the first feature of the first input image, and proceed to step S3.

[0075] In the application, the initialized first input image is input to each transformer block in the improved image encoder, and then Figure 1 The first feature is obtained by processing the first adapter (self-prompt adapter or Self-PromptAdapter), residual connection and normalization layer (Add&Layer Norm), multi-head attention layer (Multi-headAttention), LoRA layer (LoRA Layers), second adapter (Adapter), normalization layer (Layer Norm), feed-forward network (Feed-forward), etc. It can be understood that in actual application, when obtaining the first feature of the first input image, only part of the above-mentioned layer structure can be used, which is not limited here.

[0076] It should be noted that Figure 1 The parameters of the first adapter, the second adapter, the LoRA layer, the multi-scale adapter, and the mask decoder are not frozen, while the parameters of the residual connection and normalization layer, the multi-head attention layer, the normalization layer, and the feedforward network layer are frozen and do not participate in training fine-tuning.

[0077] In one embodiment, step S2 specifically includes:

[0078] The first input image is input into the improved image encoder, and in the improved image encoder, feature extraction is performed on the first input image through the fully automatic branch of the first adapter, the multi-head attention layer and the second adapter to obtain a first feature.

[0079] In application, the structure of the improved image encoder is shown in Figure 1Specifically, after each transformer block in the image encoder inputs the first input image, the first adapter (self-prompt adapter or Self-PromptAdapter) combines the task-specific knowledge perception prompts to obtain an enhanced feature map to enhance the feature selection and self-prompt capabilities of the model; in the residual connection and normalization layer (Add&Layer Norm), the enhanced feature map is residually connected, and the connected feature map is normalized to stabilize the extraction process and improve the model performance; the multi-head attention layer (Multi-head Attention) uses the multi-head attention mechanism to process the feature map to generate an attention-weighted feature map to obtain the dependency between features; the LoRA layer is introduced to reduce the number of parameters that need to be updated and improve the fine-tuning efficiency; in the second adapter (Adapter), the feature map is further processed to generate an enhanced feature map; the enhanced feature map is normalized in the normalization layer (LayerNorm) to stabilize the extraction process; finally, the normalized feature map is processed through the feed-forward network (Feed-forward) to generate the final feature, i.e., the first feature.

[0080] In one embodiment, the improved image encoder includes a first adapter inserted before a multi-head attention layer of each transformer block in the image encoder, and a second adapter inserted after the multi-head attention layer;

[0081] The first adapter includes a fully automatic branch and a semi-automatic branch;

[0082] The fully automatic branch includes an implicit gating module, a downsampling layer, a 3D convolution layer, and an upsampling layer;

[0083] The semi-automatic branch includes an explicit gating module, a downsampling layer, a 3D convolution layer, and an upsampling layer;

[0084] The second adapter includes a downsampling layer, a 3D convolution layer, and an upsampling layer.

[0085] In the application, the first adapter can be specifically represented as:

[0086]

[0087] Where x represents the input image, represents the implicit / explicit gating module in the first adapter (SP-Adapter), W down represents downsampling, W up Denotes upsampling, Conv denotes a 3D convolution operation, and σ denotes an activation function.

[0088] It should be noted that the above The task-specific knowledge-aware prompt p can be parameterized, and the knowledge-aware prompt p corresponding to the fully automatic branch and the semi-automatic branch is different, that is, the knowledge-aware prompt p corresponding to the silent mode and the prompt mode is different.

[0089] In application, the fully automatic branching (also called implicit branching) of the first adapter is as follows Figure 3 As shown in b.

[0090] Initialize N (where N is equal to the number of transformer blocks or the number of first adapters or the number of second adapters) learnable knowledge-aware prompts p in the fully automatic branch, Among them, k represents the tag length, r corresponds to the dimension of the downsampled feature, so as to focus the model's attention by amplifying the features at specific locations, so that the first adapter can focus more on the medical area that needs to be segmented; combined with the knowledge-aware hint p, the implicit gating module can also obtain implicit semantic enhancement features based on the downsampled image features, which can be specifically expressed as:

[0091] z=Norm(x)·W down ;

[0092]

[0093] Among them, z represents the image features obtained by downsampling, represents implicit semantic enhancement features, x represents the input image, Norm(x) represents the normalization of the input image, and W down represents downsampling, G (i) (z) represents the implicit gating function, α represents the scaling factor, which is used to control the scale of the prompt signal. The default value is 1. sim(z,p) represents the cosine similarity limited to the interval [0, 1]. sim(z,p) is element-wise multiplied with z and multiplied by the scaling factor α to obtain the enhanced implicit semantic enhancement feature.

[0094] The above implicit gating mechanism can effectively integrate knowledge-aware prompt information into the corresponding feature data, thereby enhancing the semantic information of the feature.

[0095] In the fully automatic branch, the downsampling layer is mainly used to reduce the spatial dimension of the input data while maintaining or increasing the representativeness of the features, which helps to reduce the computational cost while capturing higher-level abstract features; the implicit gating module can effectively incorporate knowledge-aware cue information into the corresponding feature data, thereby enhancing the semantic information of the features; the 3D convolutional layer is used to capture feature information across multiple dimensions, including spatial and temporal dimensions; the upsampling layer is used to recover the spatial details lost due to downsampling to match the needs of subsequent processing.

[0096] Correspondingly, for the relevant contents of the semi-automatic branching of the first adapter, please refer to the relevant description in step S7.

[0097] The second adapter is Figure 3 As shown in a in the figure, it includes the downsampling layer (W down ), 3D convolution layer (3D Conv) and upsampling layer (W up ), the downsampling layer is mainly used to reduce the spatial dimension of the input data while maintaining or increasing the representativeness of the features, which helps to reduce the computational cost while capturing higher-level abstract features; the 3D convolutional layer is used to capture feature information across multiple dimensions, including spatial and temporal dimensions; the upsampling layer is used to restore the spatial details lost due to downsampling to match the needs of subsequent processing.

[0098] Step S3: Based on the multi-scale adapter and the first feature, obtain a first multi-scale fusion feature of the first input image, and proceed to step S4.

[0099] In the application, the output of the improved image encoder is the input of the multi-scale adapter. Specifically, the first feature is input into the multi-scale adapter to extract the multi-scale fusion feature.

[0100] In one embodiment, step S3 specifically includes:

[0101] Inputting the first feature into the multi-scale adapter to obtain a first multi-scale feature;

[0102] The first multi-scale features are fused to obtain the first multi-scale fused features.

[0103] In the application, after the first feature is input into the multi-scale adapter, the corresponding multi-scale feature (i.e., the first multi-scale feature) is extracted through the multi-scale adapter, and then the first branch fusion feature is obtained through the top-down branch, and the second branch fusion feature is obtained through the bottom-up branch. Finally, the first branch fusion feature and the second branch fusion feature are fused to obtain the corresponding first multi-scale fusion feature.

[0104] In applications, by introducing a multi-scale adapter and using top-down and bottom-up branches to fuse multi-scale features, the detection and segmentation performance of the model can be significantly improved, and the feature expression ability and robustness of the model can be enhanced.

[0105] Step S4: Based on the mask decoder and the first multi-scale fusion feature, obtain and output a first segmentation mask of the first input image.

[0106] In the application, after the first multi-scale fusion feature is input into the mask decoder, the mask decoder adopts a dynamic progressive upsampling method (i.e., combining the multi-scale features extracted by the multi-scale adapter during the upsampling process) to gradually restore the spatial resolution of the feature map, which is beneficial to improving the upsampling quality, and finally obtains the corresponding first segmentation mask and outputs it to the user end.

[0107] It should be noted that the relevant processes in steps S3 to S4 do not include Figure 6 PromptEmbeddings in .

[0108] In the application, the user end includes but is not limited to mobile phones, tablet computers, wearable devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA) and other devices.

[0109] It can be understood that the fusion features obtained through the top-down branch and the bottom-up branch of the multi-scale adapter, and the dynamic progressive upsampling structure combined with the multi-scale features together form a U-shaped structure.

[0110] like Figure 4 As shown, the three-dimensional medical image segmentation method provided in the embodiment of the present application further includes the following steps S5 to S9:

[0111] Step S5: In the prompt mode, obtain the initialized second input image and initial prompt data, and proceed to step S6.

[0112] In the application, the initial prompt data includes but is not limited to prompt data such as points (which may be central points or random points, which are not limited here), boxes, texts, etc. It should be noted that the initial prompt data is encoded data that can be directly input into the semantic perception prompt encoder.

[0113] In the application, the process of obtaining the initialized second input image is similar to the process of obtaining the initialized first input image. For details, please refer to the relevant description in the aforementioned step S1, which will not be repeated here.

[0114] Step S6: Based on the semantic-aware prompt encoder, obtain the prompt embedding features of the initial prompt data, and proceed to step S7.

[0115] In the application, after obtaining the hint embedding features of the initial hint data, they can be introduced into the semi-automatic branch used in the first adapter, so that the explicit gating module can effectively integrate the knowledge-aware hints and the hint embedding features after projection transformation into the corresponding feature data, thereby enhancing the semantic information of the features.

[0116] In the application, the schematic diagram of the semantic-aware cue encoder is shown in Figure 5, where the class number refers to the number of categories in the classification task. When building the model, it is necessary to know clearly how many categories there are in order to correctly set the dimension of the output layer (for example, in a classification task with 10 categories, the last layer of the model usually has a 10-dimensional output feature vector); feature connection and linear transformation refer to connecting multiple feature vectors together and then generating the final cue embedding features through linear transformation.

[0117] Step S7: Based on the improved image encoder, obtain the second feature of the second input image, and proceed to step S8.

[0118] In the application, when obtaining the second feature of the second input image, the semi-automatic branch used in the first adapter, the semi-automatic branch (also referred to as the display branch) is as follows Figure 3 In the prompt mode, for the external prompt data (initial prompt data), the semantic-aware prompt encoder can be used to generate a class-balanced prompt embedding feature p e After that, p e Introducing semi-automatic branching, in order to e Projected into a latent space shared with the image semantic features, N (where N is equal to the number of transformer blocks or the number of first adapters or the number of second adapters) learnable knowledge-aware cues p can be initialized in the semi-automatic branch. Where n represents the hint embedding feature p e The dimension of the projected transformed hint is obtained by using the knowledge-aware hint p. Then, the explicit gating module can also obtain explicit semantic enhancement features based on the downsampled image features, which can be specifically expressed as:

[0119]

[0120] Among them, RP represents the projection function, Mean and Linear functions are used for dimension alignment, represents explicit semantic enhancement features, z represents the image features obtained by downsampling, represents the hint embedding feature after projection transformation, α represents the scaling factor, which is used to control the hint signal scale, and the default value is 1. represents the cosine similarity restricted to the interval [0, 1], Element-wise product with z and multiplied by the scaling factor α to obtain the enhanced explicit semantic enhancement feature

[0121] The above process embeds external cues into p e Projecting to the same dimension as the learnable knowledge-aware hint p makes cross-domain embedding possible. The correlations are re-weighted to extract the necessary features.

[0122] In the semi-automatic branch, the downsampling layer is mainly used to reduce the spatial dimension of the input data while maintaining or increasing the representativeness of the features, which helps to reduce the computational cost while capturing higher-level abstract features; the explicit gating module can effectively integrate the knowledge-aware cues and the cue embedding features after projection transformation into the corresponding feature data, thereby enhancing the semantic information of the features; the 3D convolutional layer is used to capture feature information across multiple dimensions, including spatial and temporal dimensions; the upsampling layer is used to recover the spatial details lost due to downsampling to match the needs of subsequent processing.

[0123] In one embodiment, step S7 specifically includes:

[0124] The second input image is input into the improved image encoder, and in the improved image encoder, feature extraction is performed on the second input image through the semi-automatic branch of the first adapter, the multi-head attention layer and the second adapter to obtain a second feature.

[0125] In application, the structure of the improved image encoder is shown in Figure 1Specifically, after the second input image is input into each transformer block in the improved image encoder, the first adapter combines the task-specific knowledge perception prompts and the projected prompt embedding features to obtain an enhanced feature map to enhance the feature selection and self-prompting capabilities of the model; in the residual connection and normalization layer, the enhanced feature map is residually connected, and the connected feature map is normalized to stabilize the extraction process and improve the model performance; the multi-head self-attention layer uses the multi-head attention mechanism to process the feature map to generate an attention-weighted feature map to obtain the dependency between features; the LoRA layer is introduced to reduce the number of parameters that need to be updated and improve the fine-tuning efficiency; in the second adapter, the feature map is further processed to generate an enhanced feature map; the enhanced feature map is normalized in the normalization layer to stabilize the extraction process; finally, the normalized feature map is processed through the feedforward network layer to generate the final feature, i.e., the second feature.

[0126] Step S8: Based on the multi-scale adapter and the second feature, obtain a second multi-scale fusion feature of the second input image, and proceed to step S9.

[0127] In the application, the output of the improved image encoder is the input of the multi-scale adapter. Specifically, the second feature is input into the multi-scale adapter, the multi-scale feature is first extracted, and then the corresponding multi-scale fusion feature is obtained according to the multi-scale feature.

[0128] In one embodiment, step S8 specifically includes:

[0129] Inputting the second feature into the multi-scale adapter to obtain a second multi-scale feature;

[0130] The second multi-scale features are fused to obtain the second multi-scale fused features.

[0131] In applications, such as Figure 6 As shown, after the second feature is input into the multi-scale adapter, the corresponding multi-scale feature (i.e., the second multi-scale feature) is extracted through the multi-scale adapter, and then the first branch fusion feature is obtained through the top-down branch, and the second branch fusion feature is obtained through the bottom-up branch. Finally, the first branch fusion feature and the second branch fusion feature are fused to obtain the corresponding second multi-scale fusion feature.

[0132] In applications, by introducing a multi-scale adapter and using top-down and bottom-up branches to fuse multi-scale features, the detection and segmentation performance of the model can be significantly improved, and the feature expression ability and robustness of the model can be enhanced.

[0133] Proceed to step S9, based on the mask decoder, the second multi-scale fusion feature and the hint embedding feature, obtain and output a second segmentation mask of the second input image.

[0134] In applications, such as Figure 6 As shown, after the second multi-scale fusion features and the hint embedding features are input into the mask decoder, the mask decoder adopts a dynamic progressive upsampling method (i.e., combining the multi-scale features extracted by the multi-scale adapter and the hint embedding features in the upsampling process) to gradually restore the spatial resolution of the feature map, which is beneficial to improving the upsampling quality, and finally obtains the corresponding second segmentation mask and outputs it to the user end.

[0135] It should be noted that the multi-scale adapter and the mask decoder are connected through the neck of SAM, which is not shown in the figure, that is, the multi-scale fusion features obtained by the multi-scale adapter are input into the mask decoder after passing through the neck of SAM.

[0136] The embodiment of the present application designs the original SAM model, proposes an MSPT architecture, and gives a specific structural diagram. By inserting two adapters into the image encoder of the original SAM model, the architecture can be applied to both unprompted and prompted scenarios, solving the limitation of the current SAM model in low accuracy when performing image segmentation in unprompted scenarios, and effectively improving the accuracy of image segmentation in prompted scenarios. The introduction of multi-scale adapters to extract hierarchical features at different levels can better capture information at different scales and improve the adaptability of the model.

[0137] In order to verify the effectiveness of this application, we used the BTCV dataset, prostate segmentation dataset, and spinal cord tumor dataset for training and testing, respectively, and used Dice score and Hausdorff distance (HD) as evaluation indicators:

[0138] The BTCV dataset contains 30 3D CT images covering 13 abdominal organs; each 3D CT image contains 85 to 198 slices, with a resolution of 512×512 and an in-plane resolution range of 0.54×0.54mm. 2 To 0.98×0.98mm 2 ;

[0139] The prostate segmentation dataset comes from six MRI sources (SiteA to SiteF), collected from NIC-ISBI13, I2CVB, and PROMISE12 datasets; the MRI sources contain 30, 30, 19, 13, 12, and 12 case data, respectively;

[0140] The Spinal Cord Tumor Dataset includes 3D MRI scans of 653 patients, specifically for preoperative evaluation of spinal cord tumors, without prior therapeutic intervention; the dataset is notable for its comprehensive representation of the spine, including cervical, thoracic, and lumbar regions, and covers four major types of spinal cord tumors: 1) spinal meningioma (247 cases), 2) ependymoma (203 cases), 3) astrocytoma (101 cases), and 4) hemangioblastoma (102 cases). Each patient's MRI image consists of enhanced T1-weighted sagittal (T1SC) MRI slices; the original resolution of the sagittal MRI images varies, with in-plane resolution ranging from 0.34 to 1.06 mm and slice thickness spanning from 1.5 to 8 mm; the number of slices in the sagittal MRI images ranges from 9 to 36.

[0141] For semi-automatic segmentation, we set three different initial prompt information for each test dataset, such as Figure 7 As shown, set a center point prompt (1point c / scan), a random point prompt (1point r / scan) and a randomly translated bounding box hint (1bbox rs / scan).

[0142] The final experimental results of the BTCV challenge dataset are shown in Table 1, where R.Kd: right kidney, L.Kd: left kidney, GB: gallbladder, Eso.: esophagus, IVC: inferior vena cava, AG: adrenal gland; It shows the performance difference between MSPT-I / E and SAMFull-FT under the same setting. It can be seen that the fully automatic segmentation proposed in this application always outperforms other SAM variant methods, reaching 86.1% in Dice score and 2.66mm in Hausdorff distance (HD), which is at least 0.6% higher than the Dice score of other SAM variants; the semi-automatic segmentation proposed in this application is at least 0.1% higher than other SAM variants.

[0143]

[0144] Table 1

[0145] The final experimental results on the prostate segmentation dataset are shown in Tables 2 and 3 below, compared with other SAM variants on the full-motion segmentation task and semi-automatic segmentation task on the prostate dataset (all SAM variants use the ViT-B backbone to ensure fair comparison). Indicates the performance difference between MSPT-I / E and SAMFull-FT under the same settings. It can be seen that the fully automatic segmentation proposed in this application has a maximum improvement of 3.0% in Dice score and an average Dice improvement of 2.6% compared with SAMFull-FT; the semi-automatic segmentation proposed in this application has a maximum improvement of 0.2% in Dice score compared with SAMFull-FT.

[0146]

[0147] Table 2

[0148]

[0149] Table 3

[0150] The final experimental results of the spinal cord tumor dataset are shown in Table 4 below, where a 5-fold cross-validation comparison is performed on the spinal cord tumor dataset between the self-prompting adapter proposed in this application and the MA-SAM method (where all SAM variants use the ViT-B backbone to ensure fair comparison); It shows the performance difference between the self-prompting adapter of this application and the basic adapter in the MA-SAM framework under the same setting. It can be seen that the fully automatic segmentation proposed in this application has a maximum improvement of 0.7% in Dice score compared with MA-SAM.

[0151]

[0152] Table 4

[0153] In summary, the MSPT architecture provided in the embodiment of the present application can effectively improve the accuracy of image segmentation in both non-cue scenes and cue scenes compared with other existing SAM methods.

[0154] In addition, we evaluate the MSPT framework on the BTCV dataset by evaluating the impact of the following factors one by one:

[0155] (1) Validity of the first adapter (self-prompting adapter):

[0156] We first demonstrate that the model architecture can focus on task-relevant signals through self-hint tuning. Table 5 below shows the superior performance of the self-hint adapter when replacing the first standard adapter (before multi-head attention) in the image encoder. The proposed self-hint adapter combines the learnable knowledge-aware hint p in the standard adapter and achieves better adaptability under the implicit / explicit gating module, with at least 1.3% improvement in Dice score compared to the baseline adapter.

[0157]

[0158] Table 5

[0159] (2) Performance of single-scale and multi-scale frameworks in the Progressive Efficient Transfer Learning (PETL) method:

[0160] We demonstrate the necessity of the proposed multi-scale framework in enhancing the performance of PETL methods while keeping the number of tuning parameters limited. As shown in Table 6 below, different PETL methods that combine multi-scale features achieve significant improvements in Dice scores. Specifically, the improvements of LoRA, first adapter (fully automatic branch), and first adapter (semi-automatic branch) are 1.4%, 0.8%, and 0.4%, respectively.

[0161]

[0162] Table 6

[0163] (3) Variants of MSPT-E:

[0164] To evaluate the ability of the image encoder to directly share latent information with task-specific prompt features, we tested different methods of integrating prompts into the image encoder. Table 7 below shows the results of different methods of introducing external prompts into the MSPT-E branch. Among them, "SAM prompt encoder" refers to the original setting of the first adapter that does not introduce additional point prompt features into the image encoder. In contrast, "Feature Embed" refers to the feature representation of external prompts (such as points), and "Learned Embed" refers to the learned weights of point embedding. It can be concluded that incorporating learned embedding into the self-prompt adapter produces better results than feature embedding.

[0165] Tip Encoder Design Dice(%)↑ HD(mm)↓ SAM prompt encoder 88.7 2.24 MSPT-E (Feature Embed) 88.5 3.04 MSPT-E (Learned Embed) 88.8 2.14

[0166] Table 7

[0167] (4) Prompt method design:

[0168] Table 8 below shows the training / test results for different hint positions in the explicit branch of the image encoder. It can be concluded that when the hints in the training and test settings are consistent, the performance tends to be higher, for example, using the center point hint, the Dice score can reach 89.2%, and using the random point hint, the Dice score can reach 88.8%.

[0169]

[0170] Table 8

[0171] In addition, the results of parameter efficiency analysis of each method on the BTCV dataset are shown in Table 9 below. The results show that the advantages of adding learnable knowledge-aware cues and multi-scale representations to the ViT architecture in the adapter can achieve a balance between model performance and tuning parameters.

[0172]

[0173] Table 9

[0174] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0175] Embodiment 2

[0176] The present application also provides a 3D medical image segmentation device for executing the method steps in the above 3D medical image segmentation method embodiment. The device can be a virtual appliance in a terminal device, which is run by a processor of the terminal device, or it can be the terminal device itself.

[0177] like Figure 8 As shown, the three-dimensional medical image segmentation device 100 provided in the embodiment of the present application includes a data processing module 101, a feature acquisition module 102, a feature fusion module 103, and a mask decoding module 104.

[0178] The data processing module 101 is used to obtain an initialized first input image in a silent mode;

[0179] A feature acquisition module 102, configured to acquire a first feature of the first input image based on an improved image encoder;

[0180] A feature fusion module 103, configured to obtain a first multiscale fusion feature of the first input image based on a multiscale adapter and the first feature;

[0181] The mask decoding module 104 is used to obtain and output a first segmentation mask of the first input image based on a mask decoder and the first multi-scale fusion feature.

[0182] In one embodiment, the data processing module 101 is further configured to:

[0183] In the prompt mode, obtaining an initialized second input image and initial prompt data;

[0184] In one embodiment, the 3D medical image segmentation apparatus 100 further includes a prompt feature acquisition module 105, which is specifically used to:

[0185] Obtaining a prompt embedding feature of the initial prompt data based on a semantic-aware prompt encoder;

[0186] In one embodiment, the feature acquisition module 102 is further used to:

[0187] Based on the improved image encoder, obtaining a second feature of the second input image;

[0188] In one embodiment, the feature fusion module 103 is further used to:

[0189] Based on the multi-scale adapter and the second feature, obtaining a second multi-scale fusion feature of the second input image;

[0190] In one embodiment, the mask decoding module 104 is further configured to:

[0191] Based on the mask decoder, the second multi-scale fusion feature and the hint embedding feature, a second segmentation mask of the second input image is obtained and output.

[0192] In one embodiment, the improved image encoder includes a first adapter inserted before a multi-head attention layer of each transformer block in the image encoder, and a second adapter inserted after the multi-head attention layer;

[0193] The first adapter includes a fully automatic branch and a semi-automatic branch;

[0194] The fully automatic branch includes an implicit gating module, a downsampling layer, a 3D convolution layer, and an upsampling layer;

[0195] The semi-automatic branch includes an explicit gating module, a downsampling layer, a 3D convolution layer, and an upsampling layer;

[0196] The second adapter includes a downsampling layer, a 3D convolution layer, and an upsampling layer.

[0197] In one embodiment, the feature acquisition module 102 is further used to:

[0198] The first input image is input into the improved image encoder, and in the improved image encoder, feature extraction is performed on the first input image through the fully automatic branch of the first adapter, the multi-head attention layer and the second adapter to obtain a first feature.

[0199] In one embodiment, the feature acquisition module 102 is further used to:

[0200] The second input image is input into the improved image encoder, and in the improved image encoder, feature extraction is performed on the second input image through the semi-automatic branch of the first adapter, the multi-head attention layer and the second adapter to obtain a second feature.

[0201] In one embodiment, the feature fusion module 103 is further used to:

[0202] Inputting the first feature into the multi-scale adapter to obtain a first multi-scale feature;

[0203] The first multi-scale features are fused to obtain the first multi-scale fused features.

[0204] In one embodiment, the feature fusion module 103 is further used to:

[0205] Inputting the second feature into the multi-scale adapter to obtain a second multi-scale feature;

[0206] The second multi-scale features are fused to obtain the second multi-scale fused features.

[0207] In application, each unit in the above device may be a software program module, or may be implemented by different logic circuits integrated in a processor or independent physical components connected to a processor, or may be implemented by multiple distributed processors.

[0208] Embodiment 3

[0209] like Fig. 9 As shown, the embodiment of the present application further provides a terminal device 200, including: at least one processor 201 ( Fig. 9 Only one processor is shown in the figure), memory 202, and a computer program 203 stored in the memory 202 and executable on at least one processor 201. When the processor 201 executes the computer program 203, the steps in the above-mentioned method embodiments are implemented.

[0210] In the application, the terminal device includes but is not limited to the processor, memory, Fig. 9 It is only an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, such as human-computer interaction devices, input and output devices, network access devices, etc. The network access device may include a communication module for the terminal device to communicate with the user terminal.

[0211] In applications, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. For example, the processor may be a timing controller (TCON). A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0212] In applications, the memory may be an internal storage unit of a terminal device in some embodiments, for example, a hard disk or memory of the terminal device. The memory may also be an external storage device of the terminal device in other embodiments, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the terminal device. The memory may also include both an internal storage unit of the terminal device and an external storage device. The memory is used to store operating systems, applications, boot loaders, data, and other programs, such as program codes of computer programs. The memory may also be used to temporarily store data that has been output or is to be output.

[0213] In the application, the communication module can be set as any device that can directly or indirectly perform long-distance wired or wireless communication with the user terminal according to actual needs. For example, the communication module can provide communication solutions including wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth, Zigbee, mobile communication networks, global navigation satellite systems (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc., which are applied to network devices. The communication module may include an antenna, and the antenna may have only one array element or an antenna array including multiple array elements. The communication module can receive electromagnetic waves through the antenna, frequency modulate and filter the electromagnetic wave signals, and send the processed signals to the processor. The communication module can also receive the signal to be sent from the processor, frequency modulate and amplify it, and convert it into electromagnetic waves for radiation through the antenna.

[0214] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0215] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The functional modules in the embodiment can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. In addition, the specific names of the functional modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0216] The embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0217] An embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned various method embodiments.

[0218] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include at least: any entity or device capable of carrying the computer program code to the terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0219] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0220] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0221] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0222] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0223] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A three-dimensional medical image segmentation method, characterized in that: The three-dimensional medical image segmentation method comprises: In silent mode, obtain the initialized first input image; Based on the improved image encoder, obtaining a first feature of the first input image; Based on the multi-scale adapter and the first feature, obtaining a first multi-scale fusion feature of the first input image; Based on the mask decoder and the first multi-scale fusion feature, a first segmentation mask of the first input image is obtained and output.

2. The three-dimensional medical image segmentation method according to claim 1, characterized in that: Also includes: In the prompt mode, obtaining an initialized second input image and initial prompt data; Obtaining a prompt embedding feature of the initial prompt data based on a semantic-aware prompt encoder; Based on the improved image encoder, obtaining a second feature of the second input image; Based on the multi-scale adapter and the second feature, obtaining a second multi-scale fusion feature of the second input image; Based on the mask decoder, the second multi-scale fusion feature and the hint embedding feature, a second segmentation mask of the second input image is obtained and output.

3. The three-dimensional medical image segmentation method according to claim 2, characterized in that: The improved image encoder comprises a first adapter inserted before a multi-head attention layer of each transformer block in the image encoder, and a second adapter inserted after the multi-head attention layer; The first adapter includes a fully automatic branch and a semi-automatic branch; The fully automatic branch includes an implicit gating module, a downsampling layer, a 3D convolution layer, and an upsampling layer; The semi-automatic branch includes an explicit gating module, a downsampling layer, a 3D convolution layer, and an upsampling layer; The second adapter includes a downsampling layer, a 3D convolution layer, and an upsampling layer.

4. The three-dimensional medical image segmentation method according to claim 3, characterized in that: The obtaining of the first feature of the first input image comprises: The first input image is input into the improved image encoder, and in the improved image encoder, feature extraction is performed on the first input image through the fully automatic branch of the first adapter, the multi-head attention layer and the second adapter to obtain a first feature.

5. The three-dimensional medical image segmentation method according to claim 3, characterized in that: The obtaining a second feature of the second input image comprises: The second input image is input into the improved image encoder, and in the improved image encoder, feature extraction is performed on the second input image through the semi-automatic branch of the first adapter, the multi-head attention layer and the second adapter to obtain a second feature.

6. The three-dimensional medical image segmentation method according to any one of claims 1 to 5, characterized in that: The acquiring a first multi-scale fusion feature of the first input image based on the multi-scale adapter and the first feature comprises: Inputting the first feature into the multi-scale adapter to obtain a first multi-scale feature; The first multi-scale features are fused to obtain the first multi-scale fused features.

7. The three-dimensional medical image segmentation method according to any one of claims 2 to 5, characterized in that: The step of acquiring a second multiscale fusion feature of the second input image based on the multiscale adapter and the second feature includes: Inputting the second feature into the multi-scale adapter to obtain a second multi-scale feature; The second multi-scale features are fused to obtain the second multi-scale fused features.

8. A three-dimensional medical image segmentation device, characterized in that: include: A data processing module, used for obtaining an initialized first input image in a silent mode; A feature acquisition module, configured to acquire a first feature of the first input image based on an improved image encoder; a feature fusion module, configured to obtain a first multiscale fusion feature of the first input image based on a multiscale adapter and the first feature; A mask decoding module is used to obtain and output a first segmentation mask of the first input image based on a mask decoder and the first multi-scale fusion feature.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the three-dimensional medical image segmentation method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the three-dimensional medical image segmentation method according to any one of claims 1 to 7 are implemented.

Images