A hint-based interactive brain tumor segmentation method and system

Abstract

The present invention provides a prompt-based interactive brain tumor segmentation method and system, which relate to the technical field of image segmentation. The method comprises: obtaining a brain image containing multiple slices and annotating tumors; selecting a slice with the most tumor pixels as a prompt image; performing brightness enhancement processing on the tumor area in the prompt image and extracting two-dimensional features; using a three-dimensional convolutional neural network to extract three-dimensional features of the brain image, splicing and fusing the two-dimensional and three-dimensional features, and obtaining slice fine segmentation features through an upsampling layer; extracting annotated image features, and performing average mask pooling on the prompt information of the enhanced prompt image and the annotated image features to construct a first prototype feature; calculating the similarity between the three-dimensional feature and the first prototype feature, and constructing a second prototype feature through pooling fusion; and performing feature matching based on the first prototype, the second prototype, and the slice fine segmentation features to obtain a final segmentation result of the tumor and non-tumor areas.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and in particular to a prompt-based interactive brain tumor segmentation method and system. Background Art

[0002] Brain tumors are serious diseases caused by the abnormal growth of brain cells, and their treatment and diagnosis have always been one of the challenges in the medical field. With the development of medical imaging technology, the early detection and accurate segmentation of brain tumors have become increasingly important. Accurate segmentation of brain tumors is crucial for subsequent treatment plan design, tumor monitoring, and patient prognosis assessment. Currently, brain tumor segmentation methods based on deep learning are developing rapidly and have made significant progress to a certain extent. However, although these methods have been widely studied and applied in clinical practice, many technical difficulties still exist, especially in the processing of complex tumor morphology, blurred boundaries, image noise, and diverse data.

[0003] Currently, automatic brain tumor segmentation methods are primarily based on three deep learning architectures: convolutional neural networks (CNNs), generative adversarial networks (GANs), and transformers. CNN-based segmentation methods employ local image patch classification to transform the image segmentation problem into a small-scale image patch classification task. This method can implicitly extract features by learning from training data, significantly improving segmentation accuracy. GAN-based brain tumor segmentation methods utilize the adversarial mechanism between the generator and the discriminator to produce high-quality segmentation results. Transformers excel in processing large-scale image data and can effectively capture long-range dependencies in images. They have been gradually applied to brain tumor segmentation tasks, demonstrating promising results.

[0004] However, existing methods are not ideal for handling certain difficult samples, especially when faced with images with complex tumor morphology or blurred boundaries, resulting in inaccurate segmentation results. Due to the scarcity and diversity of brain tumor data and the subjectivity of data annotation, training data is insufficient and of varying quality, making it difficult to effectively guarantee model accuracy and stability. Summary of the Invention

[0005] To address the unsatisfactory performance of existing methods when processing difficult samples, particularly when dealing with images with complex tumor morphology or blurred boundaries, resulting in inaccurate segmentation results. Due to the scarcity and diversity of brain tumor data and the subjectivity of data annotation, which leads to insufficient training data and uneven quality, this presents a technical problem that effectively guarantees model accuracy and stability. This paper proposes a prompt-based interactive brain tumor segmentation method and system.

[0006] The technical solutions provided by the embodiments of the present invention are as follows:

[0007] First aspect:

[0008] An embodiment of the present invention provides a hint-based interactive brain tumor segmentation method, comprising:

[0009] S1: Acquire brain images containing multiple slice samples;

[0010] S2: marking the tumor on each of the slice samples to form a marked image;

[0011] S3: Calculate the number of tumor pixels in each labeled slice sample, and select the slice sample with the largest number of tumor pixels as the prompt image;

[0012] S4: performing brightness enhancement processing on the tumor area in the prompt image to obtain an enhanced prompt image;

[0013] S5: extracting two-dimensional features from the enhanced prompt image through a two-dimensional convolutional neural network;

[0014] S6: extracting three-dimensional features of the brain image through a three-dimensional convolutional neural network;

[0015] S7: concatenating and fusing the two-dimensional features with the three-dimensional features, and restoring the spatial resolution of the fused features through the upsampling layer of the three-dimensional convolutional neural network to obtain slice fine segmentation features;

[0016] S8: extracting the annotated image features of the annotated image through the three-dimensional convolutional neural network;

[0017] S9: performing average mask pooling calculation on the prompt information of the enhanced prompt image and the annotated image features to construct first prototype features of tumor and non-tumor areas;

[0018] S10: calculating the similarity between the three-dimensional features of the brain image and the first prototype features, and performing feature fusion through average mask pooling to construct a second prototype feature of the brain image;

[0019] S11: performing feature matching on the brain image by combining the first prototype feature, the second prototype feature, and the slice fine segmentation feature to obtain a final segmentation result of the tumor and non-tumor areas.

[0020] Second aspect:

[0021] An embodiment of the present invention provides a prompt-based interactive brain tumor segmentation system, comprising:

[0022] processor;

[0023] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the prompt-based interactive brain tumor segmentation method according to the first aspect is implemented.

[0024] The third aspect:

[0025] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the prompt-based interactive brain tumor segmentation method as described in the first aspect is implemented.

[0026] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0027] (1) In the embodiments of the present invention, brightness enhancement is used to highlight the tumor region, thereby improving the segmentation recognizability. The features extracted by the two-dimensional convolutional neural network enhance the local details of the tumor region. At the same time, the fusion of two-dimensional and three-dimensional features and the restoration of spatial resolution make the details of the tumor region clearer, especially in complex or blurred images. These processes can significantly improve the segmentation effect in difficult samples.

[0028] (2) In this embodiment of the present invention, the slice sample with the most tumor pixels is selected as the prompt image to ensure that the most representative samples are selected for training. At the same time, the prototype features are constructed through average mask pooling calculation, which reduces the impact of annotation subjectivity and enhances the generalization ability of the model. Feature fusion further improves the adaptability and stability of the model to different data samples, thereby improving the accuracy and reliability of the model in the face of data scarcity, diversity and annotation issues. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 A schematic diagram of a flow chart of a hint-based interactive brain tumor segmentation method provided by an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of the structure of a prompt-based interactive brain tumor segmentation system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0033] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0034] In the embodiments of the present invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same. The terms "of," "corresponding," and "corresponding" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same.

[0035] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0036] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0037] Reference Manual Figure 1 , which shows a flow chart of a hint-based interactive brain tumor segmentation method provided by an embodiment of the present invention.

[0038] An embodiment of the present invention provides a prompt-based interactive brain tumor segmentation method. This method can be implemented by a prompt-based interactive brain tumor segmentation device, which can be a terminal or a server. The processing flow of the prompt-based interactive brain tumor segmentation method may include the following steps:

[0039] S1: Acquire a brain image containing multiple slice samples.

[0040] Slice samples are two-dimensional images obtained by slicing three-dimensional brain image data (such as MRI or CT scans) along a specific plane. These slices are typically presented as a series of parallel, continuous images, each representing a cross-section of the brain along that plane. Slice samples are widely used in medical image analysis, particularly in the detection and segmentation of brain tumors.

[0041] It should be noted that each slice represents a two-dimensional image of the brain on a certain plane.

[0042] In the present invention, converting brain images into slice samples can more effectively extract local information and improve the accuracy of segmentation and detection.

[0043] S2: Annotate each slice sample for the tumor to form an annotated image.

[0044] Annotated images are images that have specific regions (such as tumors, lesions, and tissue structures) marked on the original image, either manually or automatically. Annotated images are commonly used for supervised learning tasks in medical image analysis, serving as training data to guide deep learning models, enabling them to learn how to identify and segment specific regions.

[0045] In the present invention, accurately labeled images can help the model better understand the morphology, size, and position of the target area. Especially in medical image segmentation tasks such as tumors, precise labeling helps the model provide more refined segmentation results in practical applications.

[0046] S3: Calculate the number of tumor pixels in each labeled slice sample, and select the slice sample with the largest number of tumor pixels as the prompt image.

[0047] The number of tumor pixels refers to the total number of pixels occupied by the tumor region in medical images (such as MRI and CT scans). Each medical image is typically composed of millions of pixels, each representing a small area within the image and possessing a specific grayscale value or color information. In brain images, pixels in tumor regions often exhibit different grayscale values than those in normal tissue, making it possible to identify tumor regions through image processing techniques.

[0048] In a possible implementation, S3 specifically includes:

[0049] The number of tumor pixels in each labeled slice sample is calculated using the following formula, and the slice sample with the largest number of tumor pixels is selected as the prompt image:

[0050]

[0051] Among them, N i represents the number of tumor pixels in the i-th slice, S i1 Indicates the number of pixels with the label 1 in the i-th slice position, S i2 represents the number of pixels with the label 2 in the i-th slice position, S i4 Indicates the number of pixels with label position 4 in the i-th slice, max indicates the maximum value, and i indicates the index of the slice.

[0052] Specifically, the screening method is as follows: 、 、 Respectively represent the first The number of pixels labeled 1, 2, and 4 on each slice, For the The total number of pixels of the three types of labels on the slice is found in slices 0-154. , you can locate the slice The slice with the largest number of tumor pixels was selected.

[0053] In this invention, by selecting slices with the highest number of tumor pixels as prompt images, we ensure that the most representative samples are selected. These samples typically contain key features of the tumor region, providing more precise guidance for subsequent segmentation tasks, thereby improving the quality of model training. Furthermore, by selecting slices with the highest number of tumor pixels, the model can adapt to more typical and representative tumor features. This will make the training results more generalizable to new or unlabeled images, improving the stability of segmentation and diagnosis.

[0054] S4: Perform brightness enhancement processing on the tumor area in the prompt image to obtain an enhanced prompt image.

[0055] Brightness enhancement is a technique used in image preprocessing that aims to enhance the brightness or contrast of specific areas in an image, making the target area (such as a tumor, lesion, or other region of interest) more prominent. This processing method improves image legibility by changing the grayscale value (i.e., brightness) of pixels in the image. Brightness enhancement can significantly improve the visibility of the target area, especially when the contrast between the target area and the background is low.

[0056] In a possible implementation, S4 specifically includes:

[0057] The grayscale value of the position labeled 1 in the prompt image is magnified by a first preset multiple, the grayscale value of the position labeled 2 is magnified by a second preset multiple, and the grayscale value of the position labeled 4 is magnified by a third preset multiple to form an enhanced prompt image.

[0058] Optionally, the first preset multiple is 1.5, the second preset multiple is 1.25, and the third preset multiple is 0.7.

[0059] Specifically, then, the first The slice is the "hint" we selected, and its serial number reflects the spatial location information of the tumor, while the pixel value and range reflect the edge information of the tumor. According to the prompt information we just set, the brightness adjustment operation is performed on the tumor area in the sample image with a relatively rich number of pixels. According to the scaling factor, the original image of the four modes is For each slice, the grayscale values of pixels at position labels 1, 2, and 4 are increased to 1.5, 1.25, and 0.7 times of the original values, respectively.

[0060] In the present invention, by enhancing the brightness of pixels at different label positions, the tumor area can be made more prominent, especially when the contrast between the tumor area and the background is low, the brightness enhancement can significantly improve the visibility of the target area. This helps the automated system to identify tumors more easily. At the same time, performing different brightness enhancement processing on different labels (such as areas with labels 1, 2, and 4) (for example, amplifying the grayscale value of label 1 by 1.5 times, amplifying the grayscale value of label 2 by 1.25 times, and reducing the grayscale value of label 4 to 0.7 times) can help the model better distinguish different parts of the tumor or different types of tumors. In this way, the model can process the tumor area more accurately according to the different labels, thereby improving the segmentation accuracy.

[0061] S5: Extract 2D features from the enhanced hint image through a 2D convolutional neural network.

[0062] Among them, 2D CNN (Convolutional Neural Network) is a deep learning model that is widely used in the field of image processing, especially in tasks such as image classification, object detection, and image segmentation.

[0063] In a possible implementation, S5 includes:

[0064] S501: Extract features from the enhanced prompt image using the following formula to obtain preliminary features:

[0065]

[0066] in, Represents preliminary features, f represents composite function operation, L represents the total number of feature map layers, Represents the output feature map calculation function of the Lth layer in the two-dimensional convolutional neural network, Represents the output feature map calculation function of the L-1 layer in the two-dimensional convolutional neural network, Represents the output feature map calculation function of the first layer in the two-dimensional convolutional neural network, Represents the first two-dimensional convolutional neural network The calculation function of the layer output feature map, represents the pooling operation, represents the activation function, Represents the output feature map of the l-1th layer in the two-dimensional convolutional neural network, represents the convolution kernel weight of the l-1th layer in the two-dimensional convolutional neural network, Represents the bias term of the lth layer in a two-dimensional convolutional neural network.

[0067] S502: Using the attention mechanism, the injected prompt information is combined with the preliminary features to obtain two-dimensional features:

[0068]

[0069] in, represents two-dimensional features, represents the activation function, Represents preliminary features, Indicates prompt information. T represents the transpose operation, Indicates the dimension of the prompt information.

[0070] The attention mechanism, which mimics the human visual attention process, is widely used in deep learning, particularly in natural language processing (NLP) and computer vision (CV). Its core idea is to enable the model to dynamically select and focus on specific parts of the input, rather than processing all input data equally. This process helps improve model efficiency and accuracy, especially when processing long sequences or complex data.

[0071] Optionally, a reassembly operation is performed on the two-dimensional features. This process is based on the shape requirements set by the subsequent splicing conditions to ensure that they can meet the geometric shape and data format requirements for splicing and fusion with other features.

[0072] In this invention, a two-dimensional convolutional neural network (2D CNN) is used to extract preliminary features from the enhanced cue image, effectively extracting spatial structural information (such as edges, texture, and shape) from the image, thereby providing a richer representation for subsequent feature processing. Furthermore, the introduction of an attention mechanism allows the model to guide the learning process based on cue information (such as the spatial location of the tumor or the focus on a specific area), thereby improving the model's accuracy when handling specific tasks. For example, in the brain tumor segmentation task, by focusing attention on the tumor area, the model can more accurately detect and segment the tumor, reducing background interference and errors.

[0073] Furthermore, the reorganization operation ensures that the 2D features meet the shape requirements of subsequent feature concatenation and fusion. This step ensures the consistency of features in dimension, shape, and data format, laying the foundation for subsequent feature fusion and avoiding computational issues caused by mismatched feature sizes.

[0074] S6: Extract 3D features of brain images through 3D convolutional neural network.

[0075] 3D Convolutional Neural Networks (3D CNNs) are deep learning models widely used to process data with three-dimensional structures, such as medical imaging (such as MRI and CT scans), video processing, and 3D object recognition. Unlike 2D CNNs, which process two-dimensional images, 3D CNNs process three-dimensional data, capturing spatial depth information within the data and effectively analyzing complex three-dimensional structures.

[0076] In a possible implementation, S6 specifically includes:

[0077] The three-dimensional features of the brain image are extracted using the following formula:

[0078]

[0079] in, represents three-dimensional features, f represents composite function operation, L represents the total number of feature map layers, Represents the output feature map calculation function of the Lth layer in the three-dimensional convolutional neural network, Represents the output feature map calculation function of the L-1 layer in the three-dimensional convolutional neural network, Represents the output feature map calculation function of the first layer in the three-dimensional convolutional neural network, Represents the first The calculation function of the layer output feature map, represents the pooling operation, represents the activation function, Represents the output feature map of the l-1th layer in the three-dimensional convolutional neural network, represents the convolution kernel weight of the l-1th layer in the three-dimensional convolutional neural network, Represents the bias term of the lth layer in the 3D convolutional neural network.

[0080] In the present invention, through three-dimensional convolution operations, 3D CNN can better identify and learn complex spatial structures in images. For example, tumors are usually manifested in multiple slices, and three-dimensional convolution can combine the information of these slices to form a more complete description of the tumor area, improving the richness of feature expression. At the same time, in medical image analysis, especially the analysis of brain images, structures such as tumors and brain tissues often have complex morphology and spatial relationships. By using 3D convolution, the model can learn these complex three-dimensional structures, thereby improving segmentation accuracy and reducing the occurrence of mis-segmentation or missed segmentation.

[0081] S7: The two-dimensional features are concatenated and fused with the three-dimensional features, and the spatial resolution of the fused features is restored through the upsampling layer of the three-dimensional convolutional neural network to obtain the slice fine segmentation features.

[0082] The upsampling layer is an operation in a convolutional neural network (CNN) that increases the spatial resolution of the input data, thereby restoring the image size or enhancing the image details. It is commonly used in tasks such as image segmentation, generative models (such as generative adversarial networks), and autoencoders, especially when it is necessary to restore image details or generate high-resolution images.

[0083] Among them, slice fine segmentation features are features extracted from medical imaging data, which are used to describe the specific morphology, position, boundary and other information of the target area (such as a tumor or organ) in each image slice (usually referring to a two-dimensional slice image of an MRI or CT scan).

[0084] In a possible implementation, S7 specifically includes:

[0085] S701: Copy the two-dimensional feature D times in the depth dimension to expand the two-dimensional feature to keep the shape consistent with the 3D feature.

[0086] In the present invention, by replicating the 2D features in the depth dimension ( S701 ), the spatial dimensions of the 2D features can be aligned with the 3D features, thereby ensuring shape consistency when the features are stitched and fused. This process avoids computational issues caused by feature size mismatch.

[0087] S702: The three-dimensional features are combined with the expanded enhanced features to obtain fused features:

[0088]

[0089] Among them, d represents the depth dimension index, h represents the height dimension index, w represents the width dimension index, c represents the channel dimension index, and D represents the total number of layers in the depth dimension. represents the fusion feature, represents the concatenation operation along the channel dimension, Represents three-dimensional features, Represents the expanded two-dimensional features.

[0090] In the present invention, the splicing and fusing of 2D and 3D features ( S702 ) enables the model to simultaneously utilize information from different dimensions. 2D features generally better capture local details, while 3D features provide global spatial information. By fusing these two features, the model can learn richer and more comprehensive features, improving its ability to recognize complex structures in brain images (such as tumors and brain tissue).

[0091] S703: Restore the spatial resolution of the fused features through the upsampling layer of the 3D convolutional neural network to obtain the slice fine segmentation features:

[0092]

[0093] in, represents the slice fine segmentation feature, Indicates the function compound executed from right to left, L indicates the total number of feature map layers, represents ordinary 3D convolution, represents the feature concatenation with the connection, represents the upsampling operation of 3D transposed convolution, Indicates fusion features.

[0094] In this paper, the upsampling layer of a 3D convolutional neural network effectively restores the spatial resolution of the fused features, enabling the model to recover more detail from low-resolution feature maps. This is particularly important in image segmentation tasks, where detail recovery is crucial for accurately segmenting tumors or other small structures. Upsampling allows the model to perform finer image segmentation at higher resolutions, thereby improving segmentation accuracy.

[0095] S8: Extracting annotation image features of labeled images via 3D convolutional neural networks.

[0096] S9: Perform average mask pooling calculation on the prompt information of the enhanced prompt image and the annotated image features to construct the first prototype features of the tumor and non-tumor areas.

[0097] It should be noted that the prompt information is specifically the shape, size and location of the tumor. The first prototype feature is specifically to further introduce prompt information to define the prototype of the tumor and non-tumor area after completing the annotation of image features.

[0098] Among them, Average Mask Pooling is a special pooling method that is often used in deep learning models, especially in tasks such as image segmentation and feature extraction. It is a pooling operation that reduces the spatial dimension by calculating the average value of pixels or features in a specific area while maintaining meaningful information.

[0099] In a possible embodiment, the first prototype features of the tumor and non-tumor regions are specifically:

[0100]

[0101] in, represents the first prototype feature, represents the average mask pooling calculation, Indicates prompt information. Represents the labeled image features.

[0102] In the present invention, by combining the prompt information with the annotated image features and performing average mask pooling calculations, the model can be helped to focus on the key features of the tumor and non-tumor areas. The average mask pooling calculation helps to combine this information with the annotated image features, thereby enhancing the model's focus on the tumor area and avoiding interference from background noise or irrelevant areas. At the same time, through the average mask pooling calculation, the features of the tumor and non-tumor areas can be more clearly distinguished, thereby helping the model to better understand the morphology, size and location of the tumor. Incorporating the prompt information into the calculation helps to further accurately define the prototypes of the tumor and non-tumor areas, thereby improving the accuracy of segmentation and detection.

[0103] S10: Calculating the similarity between the three-dimensional features of the brain image and the prototype features, and performing feature fusion through average mask pooling to construct a second prototype feature of the brain image.

[0104] Similarity is a metric that measures the degree of similarity between two objects or data. In fields such as deep learning, data analysis, information retrieval, image processing, and natural language processing, similarity is often used to compare the similarities between two data objects (such as images, text, and feature vectors) to facilitate decision-making or optimization.

[0105] It should be noted that the second prototype features are constructed based on the first prototype features by combining additional feature information (such as brain image features or other prior knowledge). This process typically involves comparing, weighting, and fusing features, with the goal of further refining the model's learning and recognition capabilities based on the first prototype features.

[0106] In a possible implementation, S10 specifically includes:

[0107] S1001: Calculate the similarity between the three-dimensional features of the brain image and the first prototype features using the following formula:

[0108]

[0109] Where M represents the similarity between the brain image features and the first prototype features, represents the activation function, Indicates cosine similarity calculation, represents the first prototype feature, Represents the three-dimensional features of brain images.

[0110] In this paper, by calculating the similarity between brain image features and the features of the first prototype, the model can accurately measure the relationship between the input image and the predefined prototype. This process enables the model to make more precise feature selection based on similarity, helping to improve the accuracy of image analysis tasks such as tumor segmentation or disease diagnosis.

[0111] S1002: Based on the similarity, feature fusion is performed using the average mask pooling calculation method to construct the second prototype feature of the brain image:

[0112]

[0113] in, represents the second prototype feature, represents the average mask pooling calculation, M represents the similarity between the brain image features and the first prototype features, Represents brain image features.

[0114] In this paper, feature fusion is performed through average mask pooling. The model not only combines brain image features with the first prototype features, but also optimizes the feature fusion process through similarity weighting. This fusion method can better preserve important image information while reducing the interference of irrelevant information, making the final second prototype features more representative and discriminative.

[0115] S11: Feature matching is performed on the brain image by combining the first prototype feature, the second prototype feature and the slice fine segmentation feature to obtain the final segmentation results of the tumor and non-tumor areas.

[0116] In a possible implementation, the final segmentation results of the tumor and non-tumor regions are specifically as follows:

[0117]

[0118] in, Represents the final segmentation results of tumor and non-tumor areas, represents the activation function, Indicates cosine similarity calculation, represents the concatenation operation along the channel dimension, represents the first prototype feature, represents the second prototype feature, Represents brain image features.

[0119] In this paper, by combining multiple feature sources (such as the first and second prototype features and the refined segmentation features) for feature matching, and using similarity calculation and the Softmax activation function, the model can more accurately identify and segment tumor and non-tumor areas. This method not only improves segmentation accuracy and enhances the model's discriminative ability, but also optimizes computational efficiency and enhances the model's generalization ability, enabling it to perform well on different types of brain images.

[0120] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0121] (1) In the embodiments of the present invention, brightness enhancement is used to highlight the tumor region, thereby improving the segmentation recognizability. The features extracted by the two-dimensional convolutional neural network enhance the local details of the tumor region. At the same time, the fusion of two-dimensional and three-dimensional features and the restoration of spatial resolution make the details of the tumor region clearer, especially in complex or blurred images. These processes can significantly improve the segmentation effect in difficult samples.

[0122] (2) In this embodiment of the present invention, the slice sample with the most tumor pixels is selected as the prompt image to ensure that the most representative samples are selected for training. At the same time, the prototype features are constructed through average mask pooling calculation, which reduces the impact of annotation subjectivity and enhances the generalization ability of the model. Feature fusion further improves the adaptability and stability of the model to different data samples, thereby improving the accuracy and reliability of the model in the face of data scarcity, diversity and annotation issues.

[0123] Reference Manual Figure 2 , showing a structural schematic diagram of a prompt-based interactive brain tumor segmentation system provided by the present invention.

[0124] The present invention further provides a prompt-based interactive brain tumor segmentation system 20, which is applied to the above-mentioned prompt-based interactive brain tumor segmentation method, comprising:

[0125] Processor 201.

[0126] The memory 202 stores computer-readable instructions. When the computer-readable instructions are executed by the processor 201 , the prompt-based interactive brain tumor segmentation method according to the method embodiment is implemented.

[0127] The prompt-based interactive brain tumor segmentation system 20 provided by the present invention can execute the above-mentioned prompt-based interactive brain tumor segmentation method and achieve the same or similar technical effects. To avoid repetition, the present invention will not describe them in detail.

[0128] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0129] (1) In the embodiments of the present invention, brightness enhancement is used to highlight the tumor region, thereby improving the segmentation recognizability. The features extracted by the two-dimensional convolutional neural network enhance the local details of the tumor region. At the same time, the fusion of two-dimensional and three-dimensional features and the restoration of spatial resolution make the details of the tumor region clearer, especially in complex or blurred images. These processes can significantly improve the segmentation effect in difficult samples.

[0130] (2) In this embodiment of the present invention, the slice sample with the most tumor pixels is selected as the prompt image to ensure that the most representative samples are selected for training. At the same time, the prototype features are constructed through average mask pooling calculation, which reduces the impact of annotation subjectivity and enhances the generalization ability of the model. Feature fusion further improves the adaptability and stability of the model to different data samples, thereby improving the accuracy and reliability of the model in the face of data scarcity, diversity and annotation issues.

[0131] It should be understood that the processor in the embodiments of the present invention may be a central processing unit (CPU), but may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0132] It should also be understood that the memory in the embodiments of the present invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0133] The above embodiments can be implemented in whole or in part via software, hardware (e.g., circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0134] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0135] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0136] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0137] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0138] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0139] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0140] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0141] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0142] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical disks.

[0143] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the prompt-based interactive brain tumor segmentation method as described in the method embodiment is implemented.

[0144] The computer-readable storage medium provided by the present invention can implement the steps and effects of the prompt-based interactive brain tumor segmentation method of the above method embodiment. To avoid repetition, the present invention will not describe them in detail.

[0145] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0146] (1) In the embodiments of the present invention, brightness enhancement is used to highlight the tumor region, thereby improving the segmentation recognizability. The features extracted by the two-dimensional convolutional neural network enhance the local details of the tumor region. At the same time, the fusion of two-dimensional and three-dimensional features and the restoration of spatial resolution make the details of the tumor region clearer, especially in complex or blurred images. These processes can significantly improve the segmentation effect in difficult samples.

[0147] (2) In this embodiment of the present invention, the slice sample with the most tumor pixels is selected as the prompt image to ensure that the most representative samples are selected for training. At the same time, the prototype features are constructed through average mask pooling calculation, which reduces the impact of annotation subjectivity and enhances the generalization ability of the model. Feature fusion further improves the adaptability and stability of the model to different data samples, thereby improving the accuracy and reliability of the model in the face of data scarcity, diversity and annotation issues.

[0148] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

[0149] There are a few points to note:

[0150] (1) The drawings of the embodiments of the present invention only relate to the structures related to the embodiments of the present invention. Other structures may refer to conventional designs.

[0151] (2) For the sake of clarity, the thickness of layers or regions in the drawings used to describe the embodiments of the present invention are exaggerated or reduced, that is, these drawings are not drawn to scale. It is understood that when an element such as a layer, film, region, or substrate is referred to as being "on" or "under" another element, the element may be "directly on" or "under" the other element or intervening elements may be present.

[0152] (3) In the absence of conflict, the embodiments of the present invention and the features therein may be combined with each other to form new embodiments.

[0153] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A hint-based interactive brain tumor segmentation method, characterized in that include: S1: Acquire brain images containing multiple slice samples; S2: marking tumors on each of the slice samples to form a marked image; S3: Calculate the number of tumor pixels in each labeled slice sample, and select the slice sample with the largest number of tumor pixels as the prompt image; S4: performing brightness enhancement processing on the tumor area in the prompt image to obtain an enhanced prompt image; S5: extracting two-dimensional features from the enhanced prompt image through a two-dimensional convolutional neural network; S6: extracting three-dimensional features of the brain image through a three-dimensional convolutional neural network; S7: concatenating and fusing the two-dimensional features with the three-dimensional features, and restoring the spatial resolution of the fused features through the upsampling layer of the three-dimensional convolutional neural network to obtain slice fine segmentation features; S8: extracting the annotated image features of the annotated image through the three-dimensional convolutional neural network; S9: performing average mask pooling calculation on the prompt information of the enhanced prompt image and the annotated image features to construct first prototype features of tumor and non-tumor areas; S10: calculating the similarity between the three-dimensional features of the brain image and the first prototype features, and performing feature fusion through average mask pooling to construct a second prototype feature of the brain image; S11: performing feature matching on the brain image by combining the first prototype feature, the second prototype feature, and the slice fine segmentation feature to obtain a final segmentation result of the tumor and non-tumor areas; The final segmentation results of the tumor and non-tumor areas are specifically as follows: ； in, Represents the final segmentation results of tumor and non-tumor areas, represents the activation function, Indicates cosine similarity calculation, represents the concatenation operation along the channel dimension, represents the first prototype feature, represents the second prototype feature, Represents the slice fine segmentation feature.

2. The hint-based interactive brain tumor segmentation method according to claim 1, characterized in that The S3 specifically includes: The number of tumor pixels in each labeled slice sample is calculated using the following formula, and the slice sample with the largest number of tumor pixels is selected as the prompt image: ； in, N i Indicates the i The number of tumor pixels in each slice, S i1 Indicates the i The number of pixels with position label 1 in each slice, S i2 Indicates the i The number of pixels in the slice position label is 2, S i4 Indicates the i The number of pixels in the slice label position 4, max represents the maximum value, i Represents an index into a slice.

3. The hint-based interactive brain tumor segmentation method according to claim 1, wherein The S4 specifically includes: The grayscale value of the position labeled 1 in the prompt image is magnified by a first preset multiple, the grayscale value of the position labeled 2 is magnified by a second preset multiple, and the grayscale value of the position labeled 4 is magnified by a third preset multiple to form the enhanced prompt image.

4. The hint-based interactive brain tumor segmentation method according to claim 1, wherein The S5 includes: S501: Extract features of the enhanced prompt image using the following formula to obtain preliminary features: ； in, Represents preliminary features, f Represents composite function operations, L Indicates the total number of feature map layers, Represents the first two-dimensional convolutional neural network L The output feature map calculation function of the layer, Represents the first two-dimensional convolutional neural network L -1 layer output feature map calculation function, Represents the output feature map calculation function of the first layer in the two-dimensional convolutional neural network, Represents the first two-dimensional convolutional neural network The calculation function of the layer output feature map, represents the pooling operation, represents the activation function, Represents the first two-dimensional convolutional neural network l -1 layer output feature map, Represents the first two-dimensional convolutional neural network l -1 layer convolution kernel weight, Represents the first two-dimensional convolutional neural network l The bias term of the layer; S502: Using the attention mechanism, the injected prompt information is combined with the preliminary features to obtain the two-dimensional features: ； in, represents two-dimensional features, represents the activation function, Represents preliminary features, Indicates prompt information. T represents the transpose operation, Indicates the dimension of the prompt information.

5. The hint-based interactive brain tumor segmentation method according to claim 1, wherein The S6 is specifically: The three-dimensional features of the brain image are extracted using the following formula: ； in, Represents three-dimensional features, f Represents composite function operations, L Indicates the total number of feature map layers, Represents the first L The output feature map calculation function of the layer, Represents the first L -1 layer output feature map calculation function, Represents the output feature map calculation function of the first layer in the three-dimensional convolutional neural network, Represents the first The calculation function of the layer output feature map, represents the pooling operation, represents the activation function, Represents the first l -1 layer output feature map, Represents the first l -1 layer convolution kernel weight, Represents the first l The bias term of the layer.

6. The hint-based interactive brain tumor segmentation method according to claim 1, wherein The S7 specifically includes: S701: Copy the two-dimensional feature D times in the depth dimension to expand the shape of the two-dimensional feature to be consistent with the shape of the 3D feature; S702: The three-dimensional feature is combined with the expanded enhanced feature to obtain a fused feature: ； in, d represents the depth dimension index, h represents the height dimension index, w Represents the width dimension index, c represents the channel dimension index, D Indicates the total number of layers in the depth dimension, represents the fusion feature, represents the concatenation operation along the channel dimension, Represents three-dimensional features, Represents the expanded two-dimensional features; S703: Restoring the spatial resolution of the fusion feature through the upsampling layer of the three-dimensional convolutional neural network to obtain a slice fine segmentation feature: ； in, represents the slice fine segmentation feature, Indicates function composition executed from right to left, L Indicates the total number of feature map layers, represents ordinary 3D convolution, represents the feature concatenation with the connection, represents the upsampling operation of 3D transposed convolution, Indicates fusion features.

7. The hint-based interactive brain tumor segmentation method according to claim 1, wherein: The first prototype features of the tumor and non-tumor regions are specifically: ； in, represents the first prototype feature, represents the average mask pooling calculation, Indicates prompt information. Represents the labeled image features.

8. The hint-based interactive brain tumor segmentation method according to claim 1, wherein: The S10 specifically includes: S1001: Calculate the similarity between the three-dimensional features of the brain image and the first prototype features using the following formula: ； in, M Represents the similarity between the three-dimensional features of the brain image and the first prototype features, represents the activation function, Indicates cosine similarity calculation, represents the first prototype feature, Representing the three-dimensional features of brain images; S1002: Based on the similarity, feature fusion is performed using an average mask pooling calculation method to construct a second prototype feature of the brain image: ； in, represents the second prototype feature, represents the average mask pooling calculation, M Represents the similarity between the three-dimensional features of the brain image and the first prototype features, Represents the three-dimensional features of brain images.

9. A prompt-based interactive brain tumor segmentation system, characterized in that include: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the prompt-based interactive brain tumor segmentation method according to any one of claims 1 to 8 is implemented.

Images