Image segmentation method, image segmentation model construction method, device and medium

Abstract

The application discloses an image segmentation method, an image segmentation model construction method, an image segmentation model construction device and a medium, and relates to the technical field of image segmentation. The method comprises the following steps: acquiring a to-be-processed medical image; pre-processing the to-be-processed medical image; inputting the pre-processed to-be-processed medical image into a plurality of segmentation models constructed, obtaining a pre-segmentation result corresponding to each segmentation model, and embedding a self-attention mechanism module in each segmentation model; fusing the pre-segmentation result corresponding to each segmentation model to obtain a segmentation result of the to-be-processed medical image. The self-attention mechanism module embedded in the application is used to calculate the weight representation of each feature in the to-be-processed medical image, and then the weight representation is provided to the backbone structure in the neural network model, so that the neural network model can realize global learning between the features of the to-be-processed medical image according to the weight representation, and then realize high-precision segmentation of the to-be-processed medical image, and finally improve the stability and reliability of the neural network recognition model.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and more specifically, to an image segmentation method, an image segmentation model construction method, an apparatus, and a medium. Background Technology

[0002] Convolutional Neural Networks (CNNs) have achieved great success in image segmentation. This led to the development of Fully Convolutional Neural Networks (FCNs), a neural network for pixel-level image classification that solves the semantic segmentation problem with input data of arbitrary size. The U-Net framework, based on FCNs, emerged, integrating shallow and deep features, and is widely used in medical image recognition and segmentation.

[0003] Building upon this foundation, the nnU-Net framework, with its superior pre-processing and post-processing capabilities, was proposed and quickly demonstrated its advantages over U-Net. The core of the nnU-Net architecture is a 3D U-Net with an encoder-decoder structure and a SkipConnection to connect the two paths.

[0004] The nnU-Net structure described above relies on simple feature extraction based on convolution operations, which results in lower accuracy for the final medical image segmentation. Summary of the Invention

[0005] The problem addressed by this invention is how to improve the segmentation accuracy of images.

[0006] To address the above problems, in a first aspect, the present invention provides an image segmentation method, the method comprising:

[0007] Acquire medical images to be processed;

[0008] Preprocess the acquired medical images to be processed;

[0009] The preprocessed medical image to be processed is input into multiple constructed segmentation models to obtain the pre-segmentation result corresponding to each segmentation model. Each segmentation model is constructed based on nnU-Net, and each segmentation model embeds a self-attention mechanism module. The self-attention mechanism module is used to calculate the weight representation of the features in the medical image to be processed. The weight representation is used to characterize the importance of the features in the medical image to be processed.

[0010] The pre-segmentation results corresponding to each segmentation model are fused to obtain the segmentation result of the medical image to be processed. The segmentation result is used to characterize the location of at least one target region included in the medical image to be processed. Optionally, in the image segmentation method provided by the present invention, each segmentation model includes an encoder, a decoder, and a bottleneck layer. The bottleneck layer is used to connect the encoder and the decoder. The encoder includes multiple convolutional layers, and a self-attention mechanism module is embedded in the bottleneck layer and / or the convolutional layers of the encoder.

[0011] Optionally, the image segmentation method provided by the present invention uses a multimodal brain MRI image as the medical image to be processed, the target region is a brain tumor region, the segmentation result indicates the location of the brain tumor region in the brain MRI image, and the segmentation result also includes at least one brain tumor type corresponding to the brain tumor region, the brain tumor type includes edema area, gangrene and / or enhanced tumor core.

[0012] Optionally, in the image segmentation method provided by the present invention, when the self-attention mechanism module is embedded in the bottleneck layer, the preprocessed medical image to be processed is input into the constructed segmentation model to obtain the pre-segmentation result, including:

[0013] The preprocessed brain MRI images are input into multiple convolutional layers in the encoder for downsampling, and the bottom convolutional layer outputs a slice sequence of the brain MRI images.

[0014] The slice sequence is input into the self-attention mechanism module in the bottleneck layer to calculate the weighted representation of each feature in the brain MRI image and output it in a multi-channel manner. The features of the brain MRI image include pixel brightness, region contour and / or region location.

[0015] The multi-channel output of the self-attention mechanism module is passed through the bottleneck layer and input into multiple deconvolution layers in the encoder for upsampling operations, and the pre-segmentation result is output.

[0016] Optionally, in the image segmentation method provided by this invention, when the self-attention mechanism module is embedded in the convolutional layer of the encoder, the preprocessed medical image to be processed is input into the constructed segmentation model to obtain the pre-segmentation result, including:

[0017] The slice sequence of the brain MRI image output from the first convolutional layer is input into the self-attention mechanism module embedded in the second convolutional layer to calculate the weight representation of each feature in the brain MRI image and output it in a multi-channel manner. The features of the brain MRI image include pixel brightness, region contour and / or region location.

[0018] The multi-channel output of the self-attention mechanism module is input into the second convolutional layer, and the output of the second convolutional layer is input into the self-attention mechanism module of the third convolutional layer, until the bottom convolutional layer outputs the downsampled result of the brain MRI image; wherein, the first convolutional layer is the top convolutional layer in the encoder, the second convolutional layer is the next convolutional layer adjacent to the first convolutional layer, and the third convolutional layer is the next convolutional layer adjacent to the second convolutional layer;

[0019] The downsampling result is input into the bottleneck layer, and the encoder performs upsampling operations to output the pre-segmentation result.

[0020] Optionally, the image segmentation method provided by the present invention calculates the weight representation of each feature in a brain MRI image, including:

[0021] Normalize the slice sequence of the input brain MRI image;

[0022] A linear transformation is performed on the normalized slice sequence to obtain the transformation matrix, and the similarity between each row in the transformation matrix is ​​calculated as the weight of each feature in the brain MRI image.

[0023] The weights are then weighted to obtain a weighted sum of the features in the brain MRI image;

[0024] The weighted slice sequence is residually joined with the input to obtain a multi-channel output, and the multi-channel output is then normalized.

[0025] The normalized multi-channel output is enhanced to obtain a weighted representation of features in the brain MRI image.

[0026] Optionally, in the image segmentation method provided by the present invention, the number of filters in the next convolutional layer of the encoder is twice the number of filters in the previous convolutional layer.

[0027] In a second aspect, the present invention provides an image segmentation model construction method for constructing a segmentation model, the segmentation model being used for image segmentation as described in the first aspect, the image segmentation model construction method comprising:

[0028] Obtain a sample dataset, which includes medical image data;

[0029] The nnU-Net neural network structure with embedded self-attention mechanism module is used to train on sample datasets and construct multiple segmentation models. The self-attention mechanism module is used to calculate the weight representation of features in the medical image to be processed. The weight representation is used to characterize the importance of features in the medical image to be processed.

[0030] Thirdly, the present invention provides an image segmentation apparatus, the image segmentation apparatus comprising:

[0031] The acquisition module is used to acquire the medical images to be processed.

[0032] The preprocessing module is used to preprocess the acquired medical images to be processed;

[0033] The segmentation module is used to input the preprocessed medical image to be processed into at least one constructed segmentation model to obtain the pre-segmentation result corresponding to each segmentation model. Each segmentation model is constructed based on nnU-Net, and each segmentation model embeds a self-attention mechanism module. The self-attention mechanism module is used to calculate the weight representation of the features in the medical image to be processed. The weight representation is used to characterize the importance of the features in the medical image to be processed.

[0034] The fusion module is used to fuse the pre-segmentation results corresponding to each segmentation model to obtain the segmentation result of the medical image to be processed. The segmentation result is used to characterize the position of at least one target region included in the medical image to be processed.

[0035] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon for implementing the method described in the first aspect.

[0036] The image segmentation method, image segmentation model construction method, apparatus, and medium provided by this invention construct an improved neural network structure model, namely, constructing multiple segmentation models with nnU-Net as the backbone structure and embedding self-attention mechanism modules within it. In practical use, the acquired medical image to be processed is preprocessed and then input into the multiple improved neural network structure models of this invention. During the process of identifying and segmenting the target region in the medical image, the embedded self-attention mechanism module can calculate the weight representation of each feature in the medical image to obtain a new representation of the feature's importance in the medical image. This weight representation is then provided to the backbone structure of the neural network model, enabling it to focus on all features of the medical image based on the feature weight representation, and to perform global learning of the relationships between features. This allows it to grasp the correlation between features in the target region and other regions of the medical image as a whole, ultimately accurately identifying and segmenting the target region in the medical image and accurately outputting the segmentation result representing the location of the target region in the medical image. This achieves high-precision segmentation of the medical image and improves the stability and reliability of the neural network recognition model. Attached Figure Description

[0037] Figure 1 This is a schematic diagram illustrating the application framework of the image segmentation method provided in some embodiments of the present invention;

[0038] Figure 2 A schematic flowchart of the image segmentation method provided in an embodiment of the present invention;

[0039] Figure 3 This is a schematic diagram of the structure of nnU-Net provided in an embodiment of the present invention;

[0040] Figure 4 A schematic diagram of the improved nnU-Net structure provided in an embodiment of the present invention;

[0041] Figure 5 A schematic diagram of the structure of the improved nnU-Net provided in some embodiments of the present invention;

[0042] Figure 6 This is a flowchart illustrating an image segmentation method provided in some embodiments of the present invention;

[0043] Figure 7 This is a flowchart illustrating an image segmentation method provided in some embodiments of the present invention;

[0044] Figure 8 This is a schematic diagram of the structure of the self-attention mechanism module provided in an embodiment of the present invention;

[0045] Figure 9 This is a flowchart illustrating an image segmentation method provided in some embodiments of the present invention;

[0046] Figure 10 A schematic diagram of the segmentation results of a brain morphology image provided in an embodiment of the present invention;

[0047] Figure 11 This is a schematic diagram of the image segmentation device provided in an embodiment of the present invention;

[0048] Figure 12 This is a schematic diagram of the structure of a computer system provided in an embodiment of the present invention. Detailed Implementation

[0049] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0050] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0051] It is understandable that medical images acquired through medical equipment, such as those generated by non-invasive imaging techniques like MRI or CT, usually need to be segmented. Clinicians need to use image segmentation techniques to manually or semi-automatically segment the acquired medical images in order to identify key areas of interest, such as lesions, and realize the practical significance of the medical images.

[0052] For example, MRI technology can generate multimodal images that reflect the morphology of human organs and tissues, such as brain, lung, or bone morphology. Specifically, it can generate multimodal image data including T1-weighted imaging, T2-weighted imaging, T1ce imaging, and free water suppression sequences (FLAIR). The complementary information from these various image modalities increases the necessary information for image segmentation. However, it also inevitably adds a large amount of unnecessary information, increasing the difficulty of recognition and segmentation. Furthermore, because there are a large number of MRI images, only a small percentage accurately reflect the target area, such as the lesion area, making the actual scientific significance of MRI images insufficient and leading to significant subjective variability.

[0053] It is also understandable that, for medical images, the relationships between the various features they contain have a significant impact on the segmentation results. For example, in brain tumor morphology images generated by MRI technology, the shape, brightness, and spatial location of the lesion area are related to the shape, brightness, and spatial location of other areas.

[0054] In related technologies, specifically deep neural network (DNN) based segmentation methods, the nnU-Net network is frequently used. However, this nnU-Net network structure, because it extracts features solely based on convolutional operations and ignores the relationships between features, results in low final segmentation accuracy.

[0055] Therefore, the image segmentation method provided in this invention, in order to improve the segmentation accuracy of target regions in medical images, uses nnU-Net as the backbone structure and embeds an attention mechanism (Transformer) module into the nnU-Net backbone structure. This enables reliable and high-precision segmentation of medical images using the improved nnU-Net network model. Specifically, during the segmentation of the medical image to be processed, the embedded Transformer module can assist the nnU-Net backbone structure in globally learning all features of the medical image to be processed by calculating the weight representation of features, i.e., a new representation of the importance of features. This effectively learns and captures the correlation between features in the image, providing a reliable and accurate basis for image segmentation, and ultimately effectively improving the accuracy of medical image segmentation.

[0056] It is understood that the image segmentation method in the embodiments of the present invention can be executed by a computer device with data processing capabilities.

[0057] During execution, in some scenarios, such as Figure 1 As shown, the computer device can acquire the medical images to be processed and the dataset used to build the neural network model from the Internet via a wireless network. Alternatively, in other scenarios, the medical images to be processed and the dataset used to build the neural network model can be directly stored locally on the computer device. This embodiment of the invention does not impose any limitations on these aspects.

[0058] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0059] Figure 2 This is a flowchart illustrating the image segmentation method according to an embodiment of the present invention. The method specifically includes:

[0060] S110, acquire the medical image to be processed.

[0061] S120, preprocess the acquired medical image to be processed.

[0062] S130, the preprocessed medical image to be processed is input into at least one constructed segmentation model to obtain the pre-segmentation result corresponding to each segmentation model. Each segmentation model is constructed based on nnU-Net, and each segmentation model is embedded with a self-attention mechanism module. The self-attention mechanism module is used to calculate the weight representation of the features in the medical image to be processed. The weight representation is used to characterize the importance of the features in the medical image to be processed.

[0063] S140, the pre-segmentation results corresponding to each segmentation model are fused to obtain the segmentation result of the medical image to be processed. The segmentation result is used to characterize the position of at least one target region included in the medical image to be processed in the medical image to be processed.

[0064] Specifically, in this embodiment of the invention, after acquiring multimodal medical images through devices such as MRI or CT, the acquired medical images are used as medical images to be processed and preprocessed.

[0065] For example, acquired brain morphology images, or lung morphology images, etc. This invention does not limit this.

[0066] Furthermore, after preprocessing the acquired medical image to be processed, the preprocessed medical image can be input into multiple pre-constructed segmentation models to perform target region identification and segmentation processing on the medical image to be processed, so that each segmentation model outputs a corresponding pre-segmentation result. The pre-segmentation result output by each segmentation model represents the location of the target region in the medical image to be processed by that segmentation model, reflecting the location of the target region in the medical image to be processed.

[0067] For example, the pre-segmentation result output by each segmentation model may include the location information of at least one target region, which may be coordinate information or visualized image information. This embodiment of the invention does not impose any limitations on this.

[0068] It is understood that the improved neural network model in this embodiment, i.e., the segmentation model, is based on nnU-Net and incorporates a self-attention mechanism module. This embedded self-attention mechanism module can calculate the weight representations of each feature in the medical image to be processed, effectively assisting the backbone structure of the neural network model so that it can perform global learning of all features, i.e., learn the global relationships between each feature.

[0069] In other words, the self-attention mechanism module can calculate the weight representation of features in the medical image to be processed, and then use these weight representations of features as a new representation of the importance of the corresponding features in the medical image. This weight representation is then provided to the backbone structure of the neural network. When the backbone structure identifies and filters features in the target region, it can identify the relationship between features in the target region and features in other regions based on the calculation results of the self-attention mechanism module, i.e., the weight representation of each feature. It can quickly and accurately capture the key features in the medical image to be processed, i.e., the features corresponding to the target region, and finally achieve accurate identification and segmentation of the target region in the medical image to be processed.

[0070] Finally, the pre-segmentation results corresponding to each segmentation model can be fused to obtain the final segmentation result corresponding to the medical image to be processed.

[0071] Correspondingly, after fusing the pre-segmentation results output by each segmentation model, the output segmentation result of the medical image to be processed indicates the position of at least one target region in the medical image to be processed. Specifically, it may include the coordinate position information of at least one target region or visualized image information to characterize the location of at least one target region in the medical image to be processed.

[0072] For example, the actual output segmentation result may include a visualized image, in which the location and outline of the target region are marked by different colors or lines, thus achieving the segmentation of the medical image to be processed.

[0073] For example, in the context of MRI medical images of brain morphology, the target region to be identified and segmented could be a brain tumor lesion. The corresponding segmentation result could specifically include the location information of the brain tumor lesion, such as outputting a visualized segmentation result that segments the area where the brain tumor lesion is located in the brain MRI image.

[0074] It is understood that the image segmentation method in this embodiment of the invention employs an improved neural network structure, namely, embedding a self-attention mechanism module in the nnU-Net model structure, to form a novel neural network model structure that can focus on the correlation between features in the medical image to be processed. This enables the self-attention mechanism module to re-represent features through weights when identifying and segmenting target regions in the medical image to be processed, allowing the segmentation model to perform global learning of the correlation between features to extract more semantic features. This provides a reliable basis for identifying target regions in the medical image to be processed, ultimately improving the segmentation accuracy of the image and enhancing model performance.

[0075] Optionally, in some embodiments, the multiple segmentation models in S130 built with nnU-Net as the backbone have the following specific structures: Figure 3 As shown, it includes an encoder, a decoder, and a bottleneck layer connecting the encoder and decoder. The left half of the U-shaped structure is the encoder, and the right half is the decoder.

[0076] The encoder can include multiple convolutional layers, such as five convolutional layers, which is the downsampling process of the medical image to be processed. Similarly, the decoder can include multiple deconvolutional layers, such as a five-layer structure, which is the upsampling process of the medical image to be processed.

[0077] The encoder on the left and the horizontal arrow represent a 3×3×3 convolution operation. After each convolution, instance normalization and a Leaky ReLU (lReLU) activation function with a slope of 0.01 are applied, with a stride of [1,1,1]. The downward arrow on the left represents a downsampling operation, and the upward arrow on the right represents an upsampling operation, which can be either a 3×3×3 convolution or deconvolution operation. Similarly, after each convolution, instance normalization and an lReLU activation function with a slope of 0.01 are applied, with a stride of [2,2,2].

[0078] It is understood that, in order to extract key features of the medical image to be processed more effectively and improve the accuracy of image segmentation, a self-attention mechanism module is embedded in the above-mentioned nnU-Net backbone structure to identify and learn the correlation between features.

[0079] Optional, such as Figure 4 As shown, in some embodiments of the present invention, in order to balance the computational load of data processing and ensure image segmentation accuracy, the self-attention mechanism module can be embedded in, for example, Figure 4 The bottleneck layer of the nnU-Net backbone structure shown can be improved by replacing the bottleneck layer module in the nnU-Net neural network with a bottleneck layer carrying a self-attention mechanism module.

[0080] It is understandable that the reason for adding a self-attention mechanism module to the improved neural network structure model, that is, the bottleneck layer, which is the bottom layer of the U-shaped structure of nnU-Net, is because the output data of the medical image data to be processed after passing through the encoder has the richest number of channels and the most refined feature information. Therefore, adding a self-attention mechanism module to the bottleneck layer can most efficiently establish the dependency between feature channels.

[0081] Optional, such as Figure 5 As shown, in other embodiments of the present invention, the self-attention mechanism module may also be embedded into each or part of the convolutional layers of the encoder.

[0082] It is understood that in this embodiment, since a self-attention mechanism module is embedded in each convolutional layer, when using the improved neural network structure to identify and segment the target region in the medical image to be processed, it is possible to continuously learn associated features, that is, to continuously capture and optimize key features, and ultimately provide a reliable basis for the identification and processing of the target region of the image, assisting the backbone structure in the neural network structure to achieve high-precision identification and segmentation of the target region.

[0083] It is understood that MRI images have multiple modalities, including T1-weighted imaging, T2-weighted imaging, T1ce imaging, and free water suppression sequences (FLAIR), etc. The complementary information from these multiple modalities increases the necessary information for segmentation. In this embodiment of the invention, accurate segmentation can be achieved through an embedded self-attention mechanism module.

[0084] To better understand the medical image segmentation method provided in the embodiments of the present invention, a detailed explanation will be given in the scenario where the medical image to be processed is a brain MRI image.

[0085] In this scenario, one embodiment of the present invention, specifically when the self-attention mechanism module is embedded in the bottleneck layer, such as... Figure 6As shown, the specific process of segmenting medical images using the segmentation model in S130 to obtain pre-segmentation results includes:

[0086] S131, the preprocessed brain MRI image is input into multiple convolutional layers in the encoder for downsampling, and the slice sequence of the brain MRI image is output in the lowest convolutional layer.

[0087] S132, the slice sequence is input into the self-attention mechanism module in the bottleneck layer to calculate the weight representation of each feature in the brain MRI image and output it in a multi-channel manner. The features of the brain MRI image include pixel brightness, region contour and / or region location, etc.

[0088] S133, the multi-channel output of the self-attention mechanism module is passed through the bottleneck layer and input into multiple deconvolution layers in the encoder for upsampling operation, and the pre-segmentation result is output.

[0089] Specifically, such as Figure 4 As shown, when the self-attention mechanism module is embedded in the bottleneck layer, the brain MRI image is input into each convolutional layer of the encoder for downsampling. After each convolutional layer, instance normalization and an lReLU activation function with a slope of 0.01 are applied, with a stride of [1,1,1], to output a slice sequence. For example, a 512-channel slice (patch) is output.

[0090] Furthermore, the slice sequence output from the lowest convolutional layer is input into the self-attention mechanism module, which calculates the weight representation of each feature of the input MRI image. Then, it is input into the deconvolution layer in the decoder through the bottleneck layer. After each deconvolution, instance normalization and lReLU activation function with a slope of 0.01 can be used for processing, with a step size of [2,2,2].

[0091] It is understood that in this embodiment, since the self-attention mechanism module is embedded in the finest part of the neural network structure, that is, in the bottleneck layer where the output data of the medical image data to be processed after passing through the encoder has the richest number of channels and the most refined feature information, the entire network model can establish the dependency between feature channels in the most efficient way.

[0092] Alternatively, in other embodiments, when the self-attention mechanism module is embedded in a convolutional layer in the encoder, such as Figure 7 As shown, in S130, the preprocessed medical image to be processed is input into the constructed segmentation model to obtain the pre-segmentation result, which specifically includes:

[0093] S134, the slice sequence of the brain MRI image output from the first convolutional layer is input into the self-attention mechanism module embedded in the second convolutional layer to calculate the weight representation of each feature in the brain MRI image and output it in a multi-channel manner. The features include pixel brightness, region contour and / or region location, etc.

[0094] S135, the multi-channel output of the self-attention mechanism module is input to the second convolutional layer, and the output of the second convolutional layer is input to the self-attention mechanism module of the third convolutional layer, until the bottom convolutional layer outputs the downsampling result of the brain MRI image; wherein, the first convolutional layer is the top convolutional layer in the encoder, the second convolutional layer is the next convolutional layer adjacent to the first convolutional layer, and the third convolutional layer is the next convolutional layer adjacent to the second convolutional layer.

[0095] S136, the downsampling result is passed through the bottleneck layer into the decoder for upsampling, and the pre-segmentation result is output.

[0096] Specifically, in this embodiment, a self-attention mechanism module is embedded before multiple convolutional layers of the encoder, such as in all convolutional layers below the first convolutional layer. Alternatively, this can be selectively configured, and the present invention does not limit this.

[0097] In this neural network structure, after the first convolutional layer outputs the downsampling result, the result is output to the self-attention mechanism module of the next layer to calculate the weights of the features in the brain MRI image, and then input into the convolutional layer. The same operation is performed in each convolutional layer with an embedded self-attention mechanism module until the bottom convolutional layer outputs the downsampling result of the brain MRI image.

[0098] For example, such as Figure 5 The structure shown embeds a self-attention mechanism module in all four convolutional layers from the second to the fifth layer to improve the accuracy of image segmentation.

[0099] It is understood that the deconvolution operation of the encoder is similar to that in the above embodiments, and will not be described again here.

[0100] It is understood that in this embodiment, since self-attention mechanism modules are embedded in multiple convolutional layers, the feature processing of brain MRI images is maximized, that is, the global relationship of all features can be learned more comprehensively, ultimately improving the segmentation accuracy.

[0101] Optionally, in some embodiments, the self-attention mechanism module can be embedded in each convolutional layer and the bottleneck layer simultaneously, or selectively in some convolutional layers, or only in the bottleneck layer. The embodiments of the present invention do not limit the specific embedding location.

[0102] Optional, such as Figure 8 As shown, in some embodiments of the present invention, such as Figure 4 and Figure 5 In the illustrated embodiment, the embedded Transformer module may specifically include: an instance normalization layer, a multi-head attention layer, and a feedforward neural network layer. By mapping the input multi-channel patch sequence to a new three-dimensional space, the dependencies between features captured by the encoder are established, making the network more flexible.

[0103] Alternatively, in other embodiments, each Transformer module may also include two instance normalization layers, a multi-head attention layer, and a feedforward neural network layer.

[0104] It is understood that in the embodiments of the present invention, the improved neural network structure with the nnU-Net neural network as the backbone may specifically embed only one layer of Transformer module, or may embed multiple layers of Transformer module. The embodiments of the present invention do not limit this.

[0105] For example, such as Figure 4 and Figure 5 As shown, the embedded three-layer continuous self-attention mechanism module is used to improve the stability and accuracy of the network model.

[0106] Specifically, it can be seen in the corresponding diagram as follows: Figure 3 The network structure shown can be improved as follows: Figure 4 As shown, a continuous three-layer Transformer module can be embedded in the bottleneck layer, which means that the bottleneck layer module in the nnU-Net neural network can be improved into a bottleneck layer module with three continuous Transformer layers.

[0107] Combination Figure 4 Each bottleneck layer module with three consecutive Transformer layers includes: three Transformer attention mechanism modules connected in this way, a convolutional layer with a kernel size of 3×3×3 and a stride of 1×1×1, an instance normalization layer, and a Leaky ReLU layer.

[0108] It is understandable that by using an improved neural network structure with the nnU-Net neural network as the backbone, and embedding Transformer modules in the convolutional layers and / or bottleneck layers, the various units in the Transformer modules can be used to map the input multi-channel patch sequence to a new three-dimensional space when segmenting the target region of the medical image to be processed. This establishes the dependency relationship between the features captured by the encoder, thereby providing a reliable basis for high-precision image segmentation and ultimately making the neural network model more flexible.

[0109] It is understandable that, for brain MRI images, the corresponding target region can be the lesion area of ​​a brain tumor. Therefore, in the acquired medical image to be processed, the regional morphology, pixel brightness, and spatial location of the target region are correlated with corresponding features in other regions. Thus, this self-attention mechanism module can be used to calculate the weighted representation of pixel brightness, the shape of each region, and / or the weighted representation of the location of each region in the MRI brain morphology image.

[0110] That is, in the above S132 and S134 in this scenario, the calculation of the weighted representation of features in the brain MRI image, such as Figure 9 As shown, the specific steps include the following:

[0111] S01, normalize the slice sequence of the input brain MRI image.

[0112] S02, the similarity between rows in the transformation matrix, serves as the weight of each feature of the brain MRI image, which includes pixel brightness, region contour, and / or region location.

[0113] S03, the weights are weighted to obtain the weighted sum of the brain MRI image features.

[0114] S04, perform a residual connection between the weighted and input slice sequence to obtain a multi-channel output, and then normalize the multi-channel output.

[0115] S05, the normalized multi-channel output is enhanced to obtain the weighted representation of the features in the brain MRI image.

[0116] Specifically, after the brain MRI images are downsampled to obtain slice sequences, they are input into the Transformer module for processing, such as three consecutive Transformer layers. Each Transformer layer includes two instance normalization layers, one multi-head attention layer, and one feedforward neural network layer.

[0117] In each Transformer layer, the neural network structure first takes a brain MRI image and performs downsampling convolution to obtain a slice sequence, such as 512 patches, which are then fed into the first instance normalization layer. The values ​​at specific locations are then normalized by channel. This instance normalization operation can normalize normally distributed values ​​to a standard normal distribution, thereby improving the model's training performance.

[0118] Furthermore, after normalization, the result of the normalization operation is input into the first multi-head attention layer. The multi-head attention layer can obtain three matrices, such as Q, K, and V, from the input through multiple independent linear transformations. The specific parameters of the linear transformations are learned by the network.

[0119] Then, the similarity of any row in matrices Q and K is calculated, that is, for each q i The value (the i-th row of the Q matrix), calculate its relationship with all k j The attention weight A is obtained by comparing the similarity of the values ​​(the j-th column of the K matrix). i,j Use these weights on v j We obtain O by performing a weighted summation. i Ultimately, O i The features are concatenated together as a weighted sum, and then a linear transformation is performed to obtain the final output.

[0120] It can be understood that the three matrix elements above represent feature values ​​in a brain MRI image, such as pixels, shape, or location. Attention weight A i,j This can represent the weight value of a feature in a brain MRI image.

[0121] Furthermore, a residual connection is performed between the output of the multi-head attention layer and the input of the Transformer layer, namely the slice sequence of brain MRI images output by the convolutional layer. The residual connection obtains a multi-channel output, such as a 512-channel output, by adding the input of the Transformer layer to the output of the multi-head attention layer.

[0122] Furthermore, the output data after the residual concatenation is normalized through a second instance normalization layer. The normalized data is then input into the feedforward neural network layer.

[0123] The feedforward neural network first performs a linear transformation on the input. The transformed data is then activated using the GELU function, followed by a dropout layer to enhance the features. This dropout layer randomly deactivates some neurons from the linear transformation, improving the model's generalization ability. The input then undergoes another linear transformation and dropout layer to obtain the feature-enhanced output, which is the weighted representation of the features—a new representation that reflects the importance of the features.

[0124] Finally, the multi-channel output obtained above is residually concatenated with the output of the dropout layer to obtain the output of a Transformer module.

[0125] It is understandable that when the self-attention mechanism module in a neural network model includes multiple Transformer modules, they can all be operated according to the above process, and the parameters of all linear transformations are learnable.

[0126] It's understandable that in the scenario of identifying brain tumor regions in brain MRI images—where tumors in MRI images often have irregular shapes that may be related to the shapes of other regions—a self-attention mechanism allows the segmentation model to utilize the obtained weighted sums to identify the shapes of target regions and learn their correlations with the shapes of other regions.

[0127] Furthermore, regarding pixel brightness, tumors in MRI images typically exhibit different brightness levels compared to surrounding areas. Segmentation models can learn the differences between the target region (the tumor region) and other regions by weighting and calculating the brightness features using a self-attention mechanism, thereby identifying areas associated with the tumor.

[0128] Finally, regarding spatial location, a specific region in an MRI image may be related to the location of its surrounding regions. The segmentation model can learn the spatial relationship between regions in the medical image to be processed through the computation results of the self-attention mechanism, thereby better capturing the spatial information of the tumor region.

[0129] It is understandable that by learning the relationships between the above features through the self-attention mechanism module, the segmentation model can better understand MRI images, thereby improving the accuracy of MRI image segmentation.

[0130] Furthermore, in the scenario of brain MRI image segmentation, after obtaining the pre-segmentation results of each segmentation model, the pre-segmentation results corresponding to each segmentation model can be fused to obtain the final segmentation result corresponding to the brain MRI image.

[0131] That is, after fusing the pre-segmentation results output by each segmentation model, the segmentation result of the medical image to be processed can include the coordinate location information of at least one brain tumor lesion region, or visualized image information, to characterize the location of at least one target region of the medical image to be processed.

[0132] For example, the actual output segmentation result may include a visualized image, in which the outline of the brain tumor lesion area is marked with lines, thus achieving the segmentation of the medical image to be processed.

[0133] The corresponding segmentation result can specifically include the location information of the brain tumor lesion area in the brain MRI image. For example, the output visualization segmentation result indicates that the brain tumor lesion area has been segmented in the brain MRI image.

[0134] Furthermore, in some embodiments, the segmentation results may also include the brain tumor type corresponding to each region. For example, the output may also include label information for edema (ED), non-enhancing tumor (NET), and enhancing tumor core (ET).

[0135] Understandably, in practice, after obtaining the above segmentation results, namely the three non-overlapping tumor regions, the regions of the tumor core and the whole tumor are calculated, namely: Tumor core (TC) and Whole tumor (WT), where TC = ET + NET and WT = ED + ET + NET.

[0136] Furthermore, in some embodiments of the present invention, in order to improve the stability of the improved neural network model, increase the number of parameters and representation capabilities of the model, better capture detailed information in the input data, and ultimately improve the accuracy of image segmentation, the number of filters in the network model, i.e. the parameters of the convolutional layers, can be adjusted to maximize the amount of data processed.

[0137] In specific implementations, in some embodiments, the number of filters in the next convolutional layer of the encoder can be set to twice the number of filters in the previous convolutional layer.

[0138] For example, with increased data volume, the original nnU-Net feature limit of 320 can easily lead to the loss of a large amount of detailed information, while increasing the maximum number of filters to 512 can help the model better retain this information, thereby improving the model's performance.

[0139] It is understood that the backbone network model in this embodiment, namely the nnU-Net network model, has 2D and 3D U-Net models, as well as a cascaded 3D U-Net model. 2D U-Net performs segmentation independently on each image slice, while 3D U-Net utilizes the 3D context of the entire volume for segmentation. Therefore, 2D U-Net is better suited for handling elongated and planar structures, while 3D U-Net is better suited for handling three-dimensional structures such as spheres and organs. Cascaded 3D U-Net utilizes the outputs of multiple U-Net models in a cascaded manner to better handle images with multiple segmentation levels.

[0140] Correspondingly, in some embodiments of the present invention, in order to improve the processing accuracy of medical images and expand the application scenarios of the model to process images of various shapes, the constructed multiple models may include a two-dimensional neural network model, a three-dimensional neural network model and / or two cascaded three-dimensional neural network models.

[0141] Specifically, in the embodiments of the present invention, the improved neural network model provided may include multiple segmentation models trained separately, such as a two-dimensional neural network model (i.e., 2D TransU-Net), a three-dimensional neural network model (i.e., 3D TransU-Net), and / or two cascaded three-dimensional neural network models (i.e., 3D TransU-Net cascaded model).

[0142] The 2D TransU-Net is a standard TransU-Net; the 3D TransU-Net can operate on an entire image's pixels. In the cascaded 3D TransU-Net model, the first network operates on the downsampled image to obtain a low-resolution image segmentation result, while the second network adjusts the result from the first network across the entire image's pixels to obtain a further refined result. Padding convolutions are used during the training of the two cascaded 3D TransU-Net models to achieve the same image output and input shapes, and Leaky ReLU can be used as the activation function.

[0143] For example, 2D TransU-Net can be used for tasks that require processing slender structures; 3D TransU-Net can be used for tasks that require processing three-dimensional structures; and cascaded 3D TransU-Net can be used for tasks that require processing multi-layer segmentation.

[0144] In the context of brain MRI images, the improved neural network structure of this invention, considering the stereoscopic nature of MRI imaging, makes 3D TransU-Net more suitable for segmenting brain tumor regions in brain MRI images when used as a single model. The cascaded models of 2D TransU-Net and 3D TransU-Net mainly contribute to model integration.

[0145] It is understood that the specific segmentation model included in the constructed neural network model can be flexibly selected according to actual needs, and the embodiments of the present invention do not limit this.

[0146] Furthermore, based on this, specifically in S140, the fusion of pre-segmentation results from multiple models can be achieved through a voting mechanism to obtain the final segmentation result, thereby enabling the segmentation result to accurately reflect the location of the target region in the medical image to be processed.

[0147] First, the label value output by each segmentation model can be determined. This label value is the assigned value to each pixel in the medical image to indicate whether the region is background or a lesion. The label value can be any character information; for example, "0" can represent the background color, while other different characters can represent different tumor types. After obtaining the label value from each pre-segmentation result, all the label values ​​can be statistically integrated to obtain the final segmentation result.

[0148] The method provided in this invention, which centrally processes data using multiple segmentation models, addresses the issue that the results of a single segmentation model may fluctuate significantly when the dataset contains noise or interference. By integrating the results of multiple models, the volatility can be reduced, thereby enhancing the robustness of the entire neural network model, reducing the bias and variance of each model, and ultimately improving the overall model accuracy.

[0149] It is understood that, in this embodiment of the invention, before inputting the acquired medical image to be processed into the constructed neural network model for target region recognition and segmentation, the neural network model can be trained and constructed using sample data. That is, the method of this embodiment of the invention further includes:

[0150] S01, Obtain the image sample dataset, which includes the training set and the test set.

[0151] S02, preprocess the images in the image sample dataset.

[0152] S03, Based on the nnU-Net neural network structure with the self-attention mechanism module embedded, the sample dataset is trained to construct multiple segmentation models. The self-attention mechanism module is used to calculate the weight representation of the features in the medical image to be processed. The weight representation is used to characterize the importance of the feature in the medical image to be processed.

[0153] Specifically, after obtaining the image sample dataset, such as brain MRI images, a five-fold cross-validation method can be used to determine the training and test sets from the sample data. Then, the constructed neural network model, namely a model with nnU-Net as the backbone structure and embedded with a self-attention mechanism module, is used to train the training set to construct the aforementioned 2D TransU-Net, 3D TransU-Net, and two cascaded 3D TransU-Net segmentation models.

[0154] It is understandable that the above-mentioned segmentation models can be configured, designed, and trained independently of each other.

[0155] Optionally, in some embodiments of the present invention, for the above-described improved neural network model, the calculation of the loss function can consider two types of loss: binary cross-entropy loss (BCE) and dice loss (Dice), and the overall loss can be expressed as:

[0156] Loss = L_BCE + L_Dice

[0157] Furthermore, regarding the evaluation metric, the dice similarity coefficient (DCS) is used, and the specific formula is as follows:

[0158] DSC = 2|X∩Y| / (|X|+|Y|)

[0159] In the semantic segmentation task, X represents the ground truth mask image, Y represents the segmented image from the algorithm, |X∩Y| represents the number of pixels at the intersection of the ground truth mask image and the segmented image from the algorithm, and |X|+|Y| represents the total number of pixels in the ground truth mask image and the segmented image from the algorithm.

[0160] According to the above formula, the closer the segmented image is to the real mask image, the closer the DSC value is to 1. The segmentation results are evaluated by calculating the dice similarity coefficients of ET, TC, and WT using the formula.

[0161] Furthermore, in order to better understand the image segmentation method provided in the embodiments of the present invention, the following will take the segmentation of brain tumors in brain morphology images obtained by MRI imaging technology as an example, that is, the area corresponding to the brain tumor in the brain morphology image is the target area, and will be described in detail.

[0162] In this example, the BraTS 2021 dataset is selected as an illustration to verify the effectiveness of the proposed method for glioma segmentation. Data preprocessing is performed, and a five-fold cross-validation strategy is used to divide the dataset into a training set and an internal validation set. Training is then conducted using the training set. The loss functions used are DICE loss and BCE loss. Each epoch is set to 250 batches. For the first stage of the 3D UNet and UNet cascade, the DICE loss is calculated separately for each sample, and then averaged over a batch. For other networks, the DICE loss for the entire batch is calculated. The Adam optimizer is selected with a learning rate of 3×10⁻⁶. -4 The learning rate adjustment strategy involves calculating the exponential moving average loss of the training and validation sets. If the exponential moving average loss of the training set decreases by less than 5 × 10⁻⁶ within 30 epochs... -3 If the learning rate decays by a factor of 5, and the validation set exponential moving average loss decreases by less than 5 × 10⁻⁶ within 60 epochs... -3 Or the learning rate is less than 1×10 -6If the training stops, the corresponding training model, i.e., the segmentation model, is obtained.

[0163] Four modalities of brain MRI images were input into multiple pre-trained segmentation models, including 2D TransU-Net, 3D TransU-Net, and two cascaded 3D TransU-Net models. The sliding window method was used to predict the test images in the test set, resulting in multiple brain tumor segmentation results. The prediction results from multiple models were integrated using a voting mechanism to obtain the final glioma segmentation result.

[0164] Experimental results are as follows Figure 10 As shown, the horizontal plane in the anatomy was selected for display. The first to fourth columns are: FLAIR modal image, T1 modal image, T1ce modal image and T2 modal image, respectively. The fifth column is the ground truth label, the sixth column is the segmentation result of this example, and the seventh column is the visualization of the segmentation result. (a)-(c) are: the 90th slice of BraTS2021_00003, the 70th slice of BraTS2021_00108, and the 77th slice of BraTS2021_01458, respectively.

[0165] Depend on Figure 10 As shown, the brain tumor MRI image segmentation method based on improved nnU-Net multimodal feature fusion established in this embodiment can effectively segment different regions of brain tumors and achieve excellent segmentation metrics. The segmented images are also closer to the true ground-level label values. This indicates that the brain tumor MRI image segmentation method based on improved nnU-Net multimodal feature fusion established in this embodiment provides an accurate implementation method for segmenting different regions of brain tumor MRI images and has certain practical applications.

[0166] It is understood that in this embodiment of the invention, by constructing an improved neural network structure model, namely, constructing multiple segmentation models with nnU-Net as the backbone structure and embedding self-attention mechanism modules therein, in actual use, i.e., when performing target region recognition and segmentation processing on multimodal medical images to be processed, the acquired medical images to be processed are preprocessed and then input into the multiple improved neural network structure models provided in this embodiment of the invention. In the process of recognizing and segmenting the target region in the medical image to be processed, the embedded self-attention mechanism module can be used to calculate the weights of each feature in the medical image to be processed, and through the weighted sum of features, a new representation of the features is made. This allows the neural network to learn the global relationship between the features of the medical image to be processed based on the weighted sum of features, thereby being able to grasp the correlation between the features of the target region and other regions in the medical image to be processed as a whole, and thus accurately locate the target region in the medical image to be processed, and accurately output the segmentation result representing the location of the target region in the medical image to be processed, achieving high-precision segmentation of the medical image to be processed, and improving the stability and reliability of the neural network recognition model.

[0167] It can also be understood that the actual processing of the medical images to be processed, or the model building process, namely S120 and S02, includes the preprocessing of the acquired medical images to be processed, which may specifically include the previous data augmentation processing, as well as the image cropping, resampling and / or standardization after augmentation processing.

[0168] Specifically, image cropping involves cropping 3D medical images to non-zero regions. This operation is performed on both the images and labels in the dataset. First, a 3D non-zero mask is generated from the 4D image data to indicate which regions in the image are non-zero. The non-zero mask for the entire 4D image is the union of the non-zero masks for each modality. Then, based on the generated non-zero mask, the size and position of the bounding box used for cropping are determined. Finally, each modality of the image is cropped sequentially based on the bounding box, and then the images are reassembled. Brain MRI image datasets often have a relatively large amount of background; image cropping can reduce image size, avoid unnecessary computation, and improve computational efficiency.

[0169] Resampling aims to address the inconsistency in the actual spatial size represented by individual voxels across different images in some 3D medical image datasets. Resampling first requires determining the size of the target space, which is often achieved by using the median of different spacings across the images in the dataset.

[0170] However, in anisotropic datasets, the spacing value at the 10th percentile of the dataset is taken as the target space size for the maximum spacing coordinate. For anisotropy, the criterion is whether, in the median spacing, one dimension's spacing is more than three times that of another dimension's spacing, and the median size of that dimension is less than one-third of the median size of the other dimension. Next, since the product between spacing and shape is a constant for each image, the target size for each image can be determined based on the target space size. Finally, the resize function in the skimage library is called to adjust the size of each image, using different resize strategies depending on whether anisotropy exists. If there is no anisotropy, 3rd-order spline interpolation is performed on the entire 3D image. If the image is anisotropic, assuming the dimension with the larger spacing is the z-axis, 3rd-order spline interpolation is performed only in the xy-plane, while nearest-neighbor interpolation is used on the z-axis. For segmented labeled images, nearest-neighbor interpolation is used in all three dimensions, regardless of anisotropy.

[0171] Standardization involves z-scoring each 3D image using only its own mean and standard deviation. If the image size is reduced by 1 / 4 or more during data cropping, normalization is only performed on the non-zero regions from the cropping stage. Standardization ensures that the grayscale values ​​of each image in the dataset have the same distribution, preventing outliers from negatively impacting the training model.

[0172] It is understood that the above-mentioned preprocessing methods can be executed sequentially or selectively in practice. This embodiment of the invention does not limit the specific method or the order of execution.

[0173] Optionally, in some embodiments, blockchain can be used to store data for each node during the training and use of the model.

[0174] As can be understood, blockchain is a novel application of computer technologies such as distributed data storage, peer-to-peer transmission, and consensus mechanisms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information and generate the next block. In this embodiment, the relevant data information obtained from cross-validation and the generated brain tumor segmentation model are stored in the blockchain network, improving data processing speed and facilitating rapid querying and retrieval of the target model and data.

[0175] On the other hand, such as Figure 11 This invention also provides an image segmentation apparatus 200, comprising:

[0176] The first acquisition module 210 is used to acquire the medical image to be processed;

[0177] The preprocessing module 220 is used to preprocess the acquired medical image to be processed;

[0178] The segmentation module 230 is used to input the preprocessed medical image to be processed into at least one constructed segmentation model to obtain the pre-segmentation result corresponding to each segmentation model. Each segmentation model is constructed based on nnU-Net, and each segmentation model embeds at least one layer of self-attention mechanism module. The self-attention mechanism module is used to calculate the weight representation of the features in the medical image to be processed. The weight representation is used to characterize the importance of the feature in the medical image to be processed.

[0179] The fusion module 240 is used to fuse the pre-segmentation results corresponding to each segmentation model to obtain the segmentation result of the medical image to be processed. The segmentation result is used to characterize the position of at least one target region included in the medical image to be processed in the medical image to be processed.

[0180] Optionally, in the image segmentation apparatus provided in the embodiments of the present invention, each recognition model includes an encoder, a decoder and a bottleneck layer. The bottleneck layer is used to connect the encoder and the decoder. The encoder includes multiple convolutional layers, and the self-attention mechanism module is embedded in the bottleneck layer and / or the convolutional layers of the encoder.

[0181] Optionally, in the image segmentation apparatus provided in this embodiment of the invention, the medical image to be processed is a multimodal brain MRI image, the target region is a brain tumor region, and the segmentation result indicates the location of the brain tumor region in the brain MRI image.

[0182] Optionally, in the image segmentation apparatus provided in this embodiment of the invention, when the self-attention mechanism module is embedded in the bottleneck layer, the segmentation module is specifically used for:

[0183] The preprocessed brain MRI image is input into multiple convolutional layers in the encoder for downsampling, and the slice sequence of the brain MRI image is output in the bottom convolutional layer.

[0184] The slice sequence is input into the self-attention mechanism module in the bottleneck layer to calculate the weight representation of each feature in the brain MRI image and output it in a multi-channel manner.

[0185] The multi-channel output of the self-attention mechanism module is passed through the bottleneck layer and input into multiple deconvolution layers in the encoder for upsampling operations, and the pre-segmentation result is output.

[0186] Optionally, in the image segmentation apparatus provided in this embodiment of the invention, when the self-attention mechanism module is embedded in the convolutional layer of the encoder, the segmentation module is specifically used for:

[0187] The slice sequence of the brain MRI image output from the first convolutional layer is input into the self-attention mechanism module embedded in the second convolutional layer to calculate the weight representation of each feature in the brain MRI image and output it in a multi-channel manner.

[0188] The multi-channel output of the self-attention mechanism module is input into the second convolutional layer, and the output of the second convolutional layer is input into the self-attention mechanism module of the third convolutional layer, until the bottom convolutional layer outputs the downsampling result of the brain MRI image; wherein, the first convolutional layer is the top convolutional layer in the encoder, the second convolutional layer is the next convolutional layer adjacent to the first convolutional layer, and the third convolutional layer is the next convolutional layer adjacent to the second convolutional layer;

[0189] The downsampling result is input into the bottleneck layer and the encoder for upsampling, and the pre-segmentation result is output.

[0190] Optionally, in the image segmentation apparatus provided in this embodiment of the invention, the segmentation module is specifically used for:

[0191] Normalize the slice sequence of the input brain MRI image;

[0192] A linear transformation is performed on the normalized slice sequence to obtain a transformation matrix, and the similarity between each row in the transformation matrix is ​​calculated as the weight of each feature of the brain MRI image. The features of the brain MRI image include pixel brightness, region contour and / or region location.

[0193] The weights are then applied to obtain a weighted sum of the features of the brain MRI image.

[0194] The weighted slice sequence is residually connected to the input to obtain a multi-channel output, which is then normalized.

[0195] The normalized multi-channel output is enhanced to obtain the weighted representation of the features in the brain MRI image.

[0196] Optionally, the image segmentation apparatus provided in this embodiment of the invention includes a two-dimensional neural network model, a three-dimensional neural network model, and two cascaded three-dimensional neural network models.

[0197] Optionally, the image segmentation apparatus provided in this embodiment of the invention further includes at least one brain tumor type corresponding to a brain tumor region, wherein the brain tumor type includes an edematous area, gangrene, and / or an enhanced tumor core.

[0198] Optionally, in the image segmentation apparatus provided in this embodiment of the invention, the number of filters in the next convolutional layer is twice the number of filters in the previous convolutional layer.

[0199] On the other hand, embodiments of the present invention also provide an image segmentation model construction apparatus, the image segmentation model construction apparatus comprising:

[0200] The second acquisition module is used to acquire a sample dataset, which includes medical image data.

[0201] The module is used to train the sample dataset based on the nnU-Net neural network structure with the self-attention mechanism module embedded in it, and to build multiple segmentation models. The self-attention mechanism module is used to calculate the weight representation of the features in the medical image to be processed. The weight representation is used to characterize the importance of the feature in the medical image to be processed.

[0202] On the other hand, the computer device provided in the embodiments of the present invention further includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image method as described above.

[0203] The following is for reference. Figure 12 , Figure 12 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention.

[0204] like Figure 12As shown, the electronic device includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage section 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304. In some embodiments, the following components are connected to the I / O interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A driver 310 is also connected to the I / O interface 305 as needed. Removable media 311, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are mounted on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed. In particular, according to embodiments of the invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the invention include a computer program product comprising a computer program carried on a machine-readable medium containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable media 311. When the computer program is executed by central processing unit (CPU) 301, the functions defined in the electronic device of the invention are performed.

[0205] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. For example, a computer-readable storage medium can be an electronic device, apparatus, or device that is electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an electronic device, apparatus, or device that executes instructions. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit programs for use by or in connection with an electronic device, apparatus, or device whose instructions are executed. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0206] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of electronic devices, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based electronic device that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0207] The units or modules described in the embodiments of the present invention can be implemented in software or hardware. The described units or modules can also be housed in a processor; for example, they can be described as: a processor including: a first acquisition module, a preprocessing module, a segmentation module, and a fusion module. The names of these units or modules do not necessarily limit the specific unit or module itself. For example, the segmentation module can also be described as "used to input the preprocessed medical image to be processed into at least one constructed segmentation model to obtain a pre-segmentation result corresponding to each segmentation model, each segmentation model being constructed based on nnU-Net, and each segmentation model embedding at least one layer of self-attention mechanism module, the self-attention mechanism module being used to calculate the weight representation of features in the medical image to be processed, the weight representation being used to characterize the importance of features in the medical image to be processed."

[0208] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable storage medium stores one or more computer programs that are used by one or more processors to execute the image segmentation method described in the above embodiments of the present invention; or to execute the image segmentation model construction method described in the above embodiments of the present invention.

[0209] In summary, the image segmentation method, image segmentation model construction method, apparatus, and medium provided by this invention utilize an embedded self-attention mechanism module to calculate the weight representation of each feature in the medical image to be processed, and then provide it to the backbone structure of the neural network model, enabling it to achieve global learning between the features of the medical image to be processed based on the weight representation, thereby achieving high-precision segmentation of the medical image to be processed, and ultimately improving the stability and reliability of the neural network recognition model.

[0210] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. An image segmentation method, characterized in that, The image segmentation method includes: Acquire medical images to be processed; The acquired medical images to be processed are preprocessed; The preprocessed medical image to be processed is input into at least one constructed segmentation model to obtain a pre-segmentation result corresponding to each segmentation model. Each segmentation model is constructed based on nnU-Net and each segmentation model embeds a self-attention mechanism module. The self-attention mechanism module is used to calculate the weight representation of the features in the medical image to be processed. The weight representation is used to characterize the importance of the features in the medical image to be processed. Each segmentation model includes an encoder, a decoder, and a bottleneck layer. The bottleneck layer is used to connect the encoder and the decoder. The encoder includes multiple convolutional layers. The self-attention mechanism module is embedded in the bottleneck layer and / or the convolutional layers in the encoder. The pre-segmentation results corresponding to each segmentation model are fused to obtain the segmentation result of the medical image to be processed. The segmentation result is used to characterize the position of at least one target region included in the medical image to be processed in the medical image to be processed. The medical image to be processed is a brain MRI image. When the self-attention mechanism module is embedded in the bottleneck layer, the step of inputting the preprocessed medical image to be processed into at least one constructed segmentation model to obtain the pre-segmentation result corresponding to each segmentation model includes: The preprocessed brain MRI image is input into multiple convolutional layers in the encoder for downsampling, and a slice sequence of the brain MRI image is output from the bottom convolutional layer. The slice sequence is input into the self-attention mechanism module in the bottleneck layer to calculate the weight representation of each feature in the brain MRI image and output it in a multi-channel manner. The features of the brain MRI image include pixel brightness, region shape and / or region location. The multi-channel output result of the self-attention mechanism module is input into multiple deconvolutional layers in the encoder through the bottleneck layer for upsampling, and the pre-segmentation result is output.

2. The image segmentation method according to claim 1, characterized in that, The target region is a brain tumor region, and the segmentation result indicates the location of the brain tumor region in the brain MRI image. The segmentation result also includes at least one brain tumor type corresponding to the brain tumor region, and the brain tumor type includes edema, gangrene and / or enhanced tumor core.

3. The image segmentation method according to claim 2, characterized in that, When the self-attention mechanism module is embedded in the convolutional layer of the encoder, the step of inputting the preprocessed medical image to be processed into at least one constructed segmentation model to obtain the pre-segmentation result corresponding to each segmentation model includes: The slice sequence of the brain MRI image output from the first convolutional layer is input into the self-attention mechanism module embedded in the second convolutional layer to calculate the weight representation of each feature in the brain MRI image and output it in a multi-channel manner. The features of the brain MRI image include pixel brightness, region shape and / or region location. The multi-channel output of the self-attention mechanism module is input to the second convolutional layer, and the output of the second convolutional layer is input to the self-attention mechanism module of the third convolutional layer, until the bottom convolutional layer outputs the downsampling result of the brain MRI image; wherein, the first convolutional layer is the top convolutional layer in the encoder, the second convolutional layer is the next convolutional layer adjacent to the first convolutional layer, and the third convolutional layer is the next convolutional layer adjacent to the second convolutional layer; The downsampling result is input into the bottleneck layer and the encoder for upsampling, and the pre-segmentation result is output.

4. The image segmentation method according to claim 3, characterized in that, The weighting representation of each feature in the brain MRI image includes: The slice sequence of the input brain MRI image is normalized. A linear transformation is performed on the normalized slice sequence to obtain a transformation matrix, and the similarity between each row in the transformation matrix is ​​calculated as the weight of each feature in the brain MRI image. The weights are then weighted to obtain a weighted sum of the brain MRI image features; The weighted and input slice sequence is residually connected to obtain a multi-channel output, and the multi-channel output is normalized. The normalized multi-channel output is subjected to feature enhancement processing to obtain the weighted representation of each feature in the brain MRI image.

5. The image segmentation method according to claim 3, characterized in that, The number of filters in the next convolutional layer of the encoder is twice the number of filters in the previous convolutional layer.

6. A method for constructing an image segmentation model, characterized in that, A method for constructing a segmentation model, wherein the segmentation model is used in the image segmentation method as described in any one of claims 1-5, the image segmentation model construction method comprising: Obtain a sample dataset, which includes medical images; The nnU-Net neural network structure with embedded self-attention mechanism module is used to train the sample dataset and construct multiple segmentation models. The self-attention mechanism module is used to calculate the weight representation of the features in the medical image, and the weight representation is used to characterize the importance of the features in the medical image to be processed.

7. An image segmentation apparatus, characterized in that, The image segmentation device includes: The acquisition module is used to acquire the medical images to be processed. A preprocessing module is used to preprocess the acquired medical image to be processed; A segmentation module is used to input the preprocessed medical image to be processed into at least one constructed segmentation model to obtain a pre-segmentation result corresponding to each segmentation model. Each segmentation model is constructed based on nnU-Net, and each segmentation model embeds a self-attention mechanism module. The self-attention mechanism module is used to calculate the weight representation of the features in the medical image to be processed. The weight representation is used to characterize the importance of the features in the medical image to be processed. Each segmentation model includes an encoder, a decoder, and a bottleneck layer. The bottleneck layer is used to connect the encoder and the decoder. The encoder includes multiple convolutional layers. The self-attention mechanism module is embedded in the bottleneck layer and / or the convolutional layers in the encoder. A fusion module is used to fuse the pre-segmentation results corresponding to each segmentation model to obtain the segmentation result of the medical image to be processed. The segmentation result is used to characterize the position of at least one target region included in the medical image to be processed in the medical image to be processed. The medical image to be processed is a brain MRI image. When the self-attention mechanism module is embedded in the bottleneck layer, the step of inputting the preprocessed medical image to be processed into at least one constructed segmentation model to obtain the pre-segmentation result corresponding to each segmentation model includes: The preprocessed brain MRI image is input into multiple convolutional layers in the encoder for downsampling, and a slice sequence of the brain MRI image is output from the bottom convolutional layer. The slice sequence is input into the self-attention mechanism module in the bottleneck layer to calculate the weight representation of each feature in the brain MRI image and output it in a multi-channel manner. The features of the brain MRI image include pixel brightness, region shape and / or region location. The multi-channel output result of the self-attention mechanism module is input into multiple deconvolutional layers in the encoder through the bottleneck layer for upsampling, and the pre-segmentation result is output.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is used to implement the method as described in any one of claims 1-6.

Images