Medical image segmentation method, device, equipment and storage medium based on improved U-Net

Abstract

Disclosed are a medical image segmentation method, apparatus, device, and storage medium based on an improved U-Net, belonging to the field of image segmentation technology. The method includes: obtaining a trained first U-Net network, which is trained using a first type of medical image in a training set; adding a feature filtering module between the encoder and decoder of each layer of the first U-Net network to obtain a second U-Net network; fine-tuning the feature filtering module in the second U-Net network using a second type of medical image in the training set, where the second type of medical image belongs to a different domain than the first type of medical image; and segmenting the second type of medical image in a test set based on the fine-tuned second U-Net network. This method can improve the versatility of the U-Net network.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image segmentation, and in particular to a medical image segmentation method, apparatus, device, and storage medium based on an improved U-Net. Background Art

[0002] Medical images are used to represent images produced by various medical instruments. Medical image segmentation refers to the act of segmenting parts of a medical image according to a specific goal. For example, if the goal is to segment an organ with a lesion, medical image segmentation is used to segment the organ with the lesion from the medical image.

[0003] Medical images of different categories or modalities are also medical images of different domains. Segmenting medical images of different domains is cross-domain medical image segmentation.

[0004] In related technologies, U-Net networks are commonly used for cross-domain medical image segmentation. U-Net consists of an encoder and a decoder. The encoder gradually reduces the spatial dimensions of the input image through a series of convolutional and max-pooling layers, extracting and learning important features from the image in the process. The decoder gradually restores the spatial dimensions of the image through upsampling operations and convolutional layers. Skip connections are also included between the encoder and decoder. Skip connections directly connect the feature maps in the encoder to the corresponding layers of the decoder. These skip connections allow the decoder to reference earlier features during the upsampling process, thereby compensating for details that traditional deep learning models may overlook.

[0005] However, the generalization ability and adaptability of the U-Net network are poor. Summary of the Invention

[0006] This disclosure provides a medical image segmentation method, apparatus, device, and storage medium based on an improved U-Net, which can improve the versatility of the U-Net network. The technical solution includes at least the following solutions:

[0007] In a first aspect, a medical image segmentation method based on an improved U-Net is provided, including: obtaining a trained first U-Net network, wherein the first U-Net network is trained using the first type of medical image in the training set; adding a feature filtering module between the encoder and the decoder of each layer of the first U-Net network to obtain a second U-Net network, wherein the first encoder, the first feature filtering module and the first decoder are connected in sequence, the first encoder is used to input the output coding features into the first feature filtering module and the second encoder, the first feature filtering module is used to filter the background in the coding features output by the first encoder using a threshold method, and input the coding features after filtering the background into the first decoder, the first encoder is any encoder in the second U-Net network, and the second encoder is an encoder in the layer below the layer where the first encoder is located; fine-tuning the feature filtering module in the second U-Net network using the second type of medical image in the training set, wherein the second type of medical image and the first type of medical image belong to different domains; and segmenting the second type of medical image in the test set based on the fine-tuned second U-Net network.

[0008] Optionally, the first feature filtering module is used to filter the background of the coding feature output by the first encoder using a threshold method, including: inputting the first coding feature into an adaptive convolutional neural network to obtain a first convolution feature of the first coding feature, where the first coding feature is the coding feature output by the first encoder; performing global average pooling on the first convolution feature to obtain a first global representation of the first convolution feature; using a cosine metric function to determine a first cosine distance between the first convolution feature and the first global representation, where the first cosine distance includes the cosine distance of each pixel in the first feature map; setting the mask of pixels in the first cosine distance whose cosine distance is less than a distance threshold to 0 to obtain a mask of the first coding feature after filtering the background; multiplying the mask of the first coding feature after filtering the background by the first convolution feature to obtain the first coding feature after filtering the background.

[0009] Optionally, the second U-Net network has a total of layer, is a positive integer, the first decoder is the first The decoder in the layer, is a positive integer, The value range is 1 to ,

[0010] exist Pick In the case of , the first decoder is used to output the decoded feature of the first coding feature after filtering the background using the following formula:

[0011]

[0012] in, For the The decoded features of the first encoded features output by the decoder in the layer, For the said The decoder in the layer, For the said the first coded feature after filtering the background in the layer;

[0013] exist Take 1 to In the case of any value of , the first decoder is used to output the decoded feature of the first coding feature after filtering the background using the following formula:

[0014]

[0015] in, For the The decoded features of the first encoded features output by the decoder in the layer, For the said The decoder in the layer, For the said the first coded feature after filtering the background in the layer; For the The decoded features output by the decoder in the layer.

[0016] Optionally, the use of the second type of medical images in the training set to fine-tune the feature filtering module in the second U-Net network includes: inputting the second type of medical images in the training set into the second U-Net network in sequence to obtain multiple segmentation masks output by the second U-Net network; for any segmentation mask, calculating the segmentation loss of the segmentation mask to obtain the segmentation loss of the segmentation mask; based on the segmentation loss, using an optimization algorithm to optimize the parameters in the feature filtering module of the second U-Net network.

[0017] In the second aspect, a medical image segmentation device based on an improved U-Net is also provided, including: a first acquisition module for acquiring a trained first U-Net network, wherein the first U-Net network is trained using the first type of medical image in the training set; a second acquisition module for adding a feature filtering module between the encoder and the decoder of each layer of the first U-Net network to obtain a second U-Net network, wherein the first encoder, the first feature filtering module and the first decoder are connected in sequence, the first encoder is used to input the output coding features into the first feature filtering module and the second encoder, the first feature filtering module is used to filter the background in the coding features output by the first encoder using a threshold method, and input the coding features after filtering the background into the first decoder, the first encoder is any encoder in the second U-Net network, and the second encoder is an encoder in the next layer of the layer where the first encoder is located; a fine-tuning module for fine-tuning the feature filtering module in the second U-Net network using the second type of medical image in the training set, wherein the second type of medical image and the first type of medical image belong to different domains; a testing module for segmenting the second type of medical image in the test set based on the fine-tuned second U-Net network.

[0018] Optionally, the second acquisition module is also used to: input the first coding feature into an adaptive convolutional neural network to obtain a first convolution feature of the first coding feature, where the first coding feature is the coding feature output by the first encoder; perform global average pooling on the first convolution feature to obtain a first global representation of the first convolution feature; use a cosine metric function to determine a first cosine distance between the first convolution feature and the first global representation, where the first cosine distance includes the cosine distance of each pixel in the first feature map; set the mask of pixels in the first cosine distance whose cosine distance is less than the distance threshold to 0 to obtain a mask of the first coding feature after filtering the background; multiply the mask of the first coding feature after filtering the background by the first convolution feature to obtain the first coding feature after filtering the background.

[0019] Optionally, the fine-tuning module is also used to: input the second type of medical images in the training set into the second U-Net network in sequence to obtain multiple segmentation masks output by the second U-Net network; for any segmentation mask, calculate the segmentation loss of the segmentation mask to obtain the segmentation loss of the segmentation mask; based on the segmentation loss, use an optimization algorithm to optimize the parameters in the feature filtering module of the second U-Net network.

[0020] In a third aspect, a computer device is also provided, comprising: a memory and a processor, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor, thereby executing the medical image segmentation method based on the improved U-Net described in the above embodiment.

[0021] In a fourth aspect, a computer-readable storage medium is also provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor, thereby executing the medical image segmentation method based on the improved U-Net described in the above embodiment.

[0022] In a fifth aspect, a computer program product is provided, comprising a computer program / instruction, which implements the method described in the first aspect when executed by a processor.

[0023] The beneficial effects of the technical solutions provided by the embodiments of the present disclosure include at least:

[0024] In an embodiment of the present disclosure, a feature filtering module is added to the trained first U-Net network to obtain a second U-Net network, and the feature filtering module in the second U-Net network is fine-tuned using a second type of medical image, so that the fine-tuned second U-Net network can accurately segment medical images belonging to the same domain as the second type of medical image. Compared with a traditional U-Net network (such as the first U-Net network), the method in the embodiment of the present disclosure can increase the versatility of the U-Net network at a lower cost (fine-tuning) without retraining the U-Net network, saving costs. When faced with complex scenarios, cross-domain tasks or sparse data sets, the method in the embodiment of the present disclosure exhibits greater robustness and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 A flowchart of a medical image segmentation method based on an improved U-Net provided by an exemplary embodiment of the present disclosure is shown;

[0027] Figure 2 A flowchart of a medical image segmentation method based on an improved U-Net provided by another exemplary embodiment of the present disclosure is shown;

[0028] Figure 3 It is a structural diagram of the second U-Net network;

[0029] Figure 4 It is a structural diagram of the feature filtering module;

[0030] Figure 5 A schematic structural diagram of a medical image segmentation device based on an improved U-Net provided by an exemplary embodiment of the present disclosure is shown;

[0031] Figure 6 It is a structural diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning understood by persons of ordinary skill in the field to which the present disclosure belongs. The words "first", "second", "third" and similar terms used in the patent application specification and claims of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one" or "a" do not indicate a quantity limitation, but rather indicate the existence of at least one. Words such as "include" or "comprising" mean that the elements or objects appearing before "include" or "comprising" cover the elements or objects listed after "include" or "comprising" and their equivalents, and do not exclude other elements or objects. Words such as "connected" or "connected" and similar terms are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0033] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0034] To facilitate understanding of the embodiments of the present disclosure, the following first explains the concepts involved in the embodiments of the present disclosure:

[0035] Domain of medical images: Medical images of the same category and modality are considered to be in the same domain. Here, the category can be an organ category. For example, if multiple medical images are of the same organ, then these images are of the same category and in the same domain.

[0036] A modality can be an instrument that generates medical images. For example, medical images generated by the same medical instrument are in the same domain.

[0037] If the medical images in the training set are images of the same organ on different medical instruments, then the modalities of these medical images are different domains, and they are considered cross-domain medical images. Alternatively, if the medical images in the training set are images of different organs on the same medical instrument, then the categories of these medical images are different domains, and they are considered cross-domain medical images. Alternatively, if the medical images in the training set are images of different organs on different medical instruments, then they are considered cross-domain medical images. The number of domains involved in the training set is related to the number of medical instruments and organ categories in the medical images.

[0038] After training, conventional U-Net networks can typically only accurately segment medical images within the same domain as the medical images in the training set. For example, if the training set includes medical images from five medical instruments and seven organs (that is, it includes seven categories and five modalities), the U-Net network trained using this training set can accurately segment medical images within the domains of these five medical instruments and seven organs. However, it cannot accurately segment medical images in domains other than these, resulting in poor generalization and adaptability of the U-Net network. If segmentation of medical images in other domains is required, the U-Net network must be retrained, which is costly.

[0039] Here, the above-mentioned domain types are only examples. In actual application, other types of domains are also included. The embodiments of the present disclosure do not limit the types of domains.

[0040] Figure 1 A flowchart of a medical image segmentation method based on an improved U-Net provided by an exemplary embodiment of the present disclosure is shown, and the method can be executed by a computer device. Figure 1 , the method comprising:

[0041] In step 101, a first U-Net network that has completed training is obtained.

[0042] The first U-Net network is trained using the first category of medical images in the training set.

[0043] In some embodiments, the first U-Net network may be a U-Net network that has been trained and has known parameters.

[0044] Before executing step 101 , it is necessary to first obtain a medical image dataset, which includes a first category of medical images and a second category of medical images, wherein the second category of medical images and the first category of medical images belong to different domains.

[0045] The medical image dataset can then be divided into a training set and test set ,The training set includes the first and second category of medical images, and the ,testing set includes the second category of images.

[0046] There are a total of medical images, including Class 1 medical images. And A second type of medical image, , 、 are integers and 、 All less than There are a total of A second-category medical image. 、 All are positive integers.

[0047] , represents the first There are three channels in a medical image, and the length is And the width is ; , represents the first The true segmentation mask of the medical image exists categories, length And the width is In the true segmentation mask, the value of the mask at the position where the object to be segmented is located is 1, and the value of the mask at the position where there is no object to be segmented is 0. is an integer, The value range is 1 to .

[0048] Similarly, , which represents the first The second type of medical image has 3 channels, with a length of And the width is ; , which represents the first The true segmentation mask of the second type of medical image exists categories, length And the width is . is an integer, The value range is 1 to .

[0049] In the embodiment of the present disclosure, the structure of the first U-Net network is the same as that of the traditional U-Net network.

[0050] The traditional U-Net network has a symmetrical U-shaped structure, with the encoder on the left and the decoder on the right. The U-shaped structure consists of multiple layers, each of which includes an encoder and decoder. The encoder outputs the encoded features, and the decoder outputs the decoded features.

[0051] After a medical image is input into the U-Net network, it is first input into the encoder of the first layer. The encoder downsamples the medical image to obtain the encoding features of the first layer. The encoding features of the first layer are then input into the encoder of the second layer and the decoder of the first layer. This process is repeated until the encoding features of the last layer are obtained. The encoding features of the last layer are directly input into the decoder of the last layer. The decoder of the last layer then upsamples the encoding features of the last layer to obtain the decoding features of the last layer. The decoding features of the last layer are then input into the decoder of the second-to-last layer. The decoder of the second-to-last layer upsamples the decoding features of the last layer and the encoding features of the second-to-last layer. This process is repeated until the decoding features of the first layer are obtained. The decoding features of the first layer are then input into the segmentation head, which then outputs the segmentation result of the medical image.

[0052] There are many related technologies for implementing the traditional U-Net network, so detailed description is omitted here.

[0053] Optionally, the first U-Net network has a total of layer, is a positive integer. In this case, step 101 includes the following three steps:

[0054] In the first step, the first type of medical images in the training set are sequentially input into the first U-Net network to obtain multiple segmentation masks output by the first U-Net network.

[0055] In the When the first type of medical image is input into the first U-Net network, the The encoding features output by the encoder in the layer are expressed using formula (1). The value range is 1 to , is the number of medical images of the first category in the training set.

[0056] (1)

[0057] In formula (1), For the The first type of medical image in the first U-Net network The encoded features output by the encoder in the layer, For this The encoder in the layer, For the The first type of medical image in the first U-Net network The encoded features output by the encoder in the layer, in addition, when When it is equal to 1, For the A first-class medical image. is a positive integer, The value range is 1 to . , Indicates the The dimensions of the encoded and decoded features in the layer; Indicates the The resolution of the encoded and decoded features in the layer, For the The height of the encoding features and decoding features in the layer, For the The width of the encoded and decoded features in the layer.

[0058] exist Pick In the case of The decoded features output by the decoder in the layer are expressed using formula (2).

[0059] (2)

[0060] In formula (2), For the The first type of medical image in the first U-Net network The decoded features output by the decoder in the layer, For the The first type of medical image in the first U-Net network The encoded features output by the encoder in the layer, For this Decoder in the layer. The meanings of other parameters in formula (2) are the same as those in formula (1), and their detailed description is omitted here.

[0061] exist Take 1 to In the case of any value in The decoded features output by the decoder in the layer are expressed using formula (3).

[0062] (3)

[0063] In formula (2), For the The first type of medical image in the first U-Net network The decoded features output by the decoder in the layer, For the The first type of medical image in the first U-Net network The encoded features output by the encoder in the layer, For this The decoder in the layer, For the The first type of medical image in the first U-Net network The decoded features output by the decoder in the layer.

[0064] In the After the first type of medical image passes through the decoder in the first layer of the first U-Net network, the decoder of the first layer will output the decoding features of the first layer. , and then input the decoded features of the first layer into the segmentation head to obtain the predicted The segmentation mask of the first-class medical image is obtained. This process can be expressed using formula (4).

[0065] (4)

[0066] In formula (4), For the predicted Segmentation mask of the first-class medical image, For the split head, The decoded features output by the decoder in the first layer of the first U-Net network.

[0067] Here, the encoding module in the first U-Net network can be used To express, Including respectively located in The encoder in the layer The encoder in the layer is represented as The decoding module in the first U-Net network can be used To express, Including respectively located in The decoder in the layer The encoder in the layer is represented as . 、 These are the parameters that need to be learned in the first U-Net network.

[0068] In the second step, for any segmentation mask, the segmentation loss of the segmentation mask is calculated to obtain the segmentation loss of the segmentation mask.

[0069] Here, after obtaining a segmentation mask, the segmentation loss of the segmentation mask can be calculated.

[0070] Exemplarily, the segmentation loss is the mean square error. The segmentation loss of the segmentation mask of the first-class medical image is calculated using formula (5).

[0071] (5)

[0072] In formula (5), For the The segmentation loss of the segmentation mask of the first-class medical image, Express and Calculate the mean square error, For the The true segmentation mask of the first-class medical image. The meanings of other parameters in formula (5) are the same as those in formula (4), and their detailed description is omitted here.

[0073] In the third step, based on the segmentation loss, an optimization algorithm is used to optimize the parameters in the feature filtering module of the first U-Net network.

[0074] Optionally, the SGD (Stochastic Gradient Descent) optimization algorithm is used to optimize the parameters in the first U-Net network. This process can be expressed by formula (6).

[0075] (6)

[0076] In formula (6), is the learning rate, which can be set to 0.01 in the initial state; for 、 The gradient of 、 are the parameters that need to be learned in the first U-Net network; 、 For the optimized 、 The meanings of other parameters in formula (6) are the same as those in formula (5), and their detailed description is omitted here.

[0077] In passing the After the segmentation loss of the first type of medical image is used to optimize the parameters in the first U-Net network, the first type of medical image in the training set can be continuously input to optimize the parameters in the first U-Net network until Each of the first-category medical images is input into the first U-Net network to complete one iterative training.

[0078] Repeat the first to third steps for multiple iterations until the maximum number of iterations of the first U-Net network is reached. The model with the smallest segmentation loss among multiple iterations is the first U-Net network. The parameters of the model with the smallest segmentation loss are 、 , that is, the encoder in the first U-Net network is , the decoder in the first U-Net network is .

[0079] Exemplarily, the maximum number of iterations of the first U-Net network is 50, that is, epoch1=50.

[0080] In step 102, a feature filtering module is added between the encoder and decoder of each layer of the first U-Net network to obtain a second U-Net network.

[0081] In the second U-Net network, the first encoder, the first feature filtering module and the first decoder are connected in sequence. The first encoder is used to input the output coding features into the first feature filtering module and the second encoder. The first feature filtering module is used to filter the background in the coding features output by the first encoder using a threshold method, and input the coding features after filtering the background into the first decoder. The first encoder is any encoder in the second U-Net network, and the second encoder is an encoder in the layer below the layer where the first encoder is located.

[0082] In step 103, the second type of medical images in the training set are used to fine-tune the feature filtering module in the second U-Net network.

[0083] In step 104, the second type of medical images in the test set are segmented based on the fine-tuned second U-Net network.

[0084] In an embodiment of the present disclosure, a feature filtering module is added to the trained first U-Net network to obtain a second U-Net network, and the feature filtering module in the second U-Net network is fine-tuned using a second type of medical image, so that the fine-tuned second U-Net network can accurately segment medical images belonging to the same domain as the second type of medical image. Compared with a traditional U-Net network (such as the first U-Net network), the method in the embodiment of the present disclosure can increase the versatility of the U-Net network at a lower cost (fine-tuning) without retraining the U-Net network, saving costs. When faced with complex scenarios, cross-domain tasks or sparse data sets, the method in the embodiment of the present disclosure exhibits greater robustness and stability.

[0085] Taking a sparse data set as an example, when faced with a sparse data set, the method in the embodiment of the present disclosure is adopted. There is no need to retrain the U-Net network. Instead, the sparse data set only needs to be treated as the second type of medical image to fine-tune the trained U-Net network. The model training cost is low and the trained model has a better segmentation effect on the sparse data set.

[0086] Figure 2 A flowchart of a medical image segmentation method based on an improved U-Net provided by another exemplary embodiment of the present disclosure is shown, and the method can be executed by a computer device. Figure 2 , the method comprising:

[0087] In step 201, a first U-Net network that has completed training is obtained.

[0088] The relevant contents of step 201 refer to the aforementioned step 101, and detailed description is omitted here.

[0089] In step 202, a feature filtering module is added between the encoder and decoder of each layer of the first U-Net network to obtain a second U-Net network.

[0090] In the second U-Net network, the first encoder, the first feature filtering module and the first decoder are connected in sequence. The first encoder is used to input the output coding features into the first feature filtering module and the second encoder. The first feature filtering module is used to filter the background in the coding features output by the first encoder using a threshold method, and input the coding features after filtering the background into the first decoder. The first encoder is any encoder in the second U-Net network, and the second encoder is an encoder in the layer below the layer where the first encoder is located.

[0091] The encoding module and decoding module in the second U-Net network are the same as those in the first U-Net network, that is, the encoder in the second U-Net network is , the decoder in the second U-Net network is The new feature filtering module in the second U-Net network can be expressed as ,in Parameters that need to be fine-tuned for the feature filtering module.

[0092] Figure 3 It is a schematic diagram of the structure of the second U-Net network. Figure 3 As shown, the second U-Net network has a total of four layers, each layer includes an encoder, a feature filtering module and a decoder. The first encoder is the encoder in the first layer. , the first feature filtering module is the feature filtering module in the first layer, for example, the first decoder is the decoder in the first layer When a medical image is input to the first encoder After that, the first encoder The medical image is downsampled to obtain the encoding features of the medical image in the first layer, and then the encoding features of the medical image in the first layer are input to the first feature filtering module and the encoder in the second layer respectively. The first feature filtering module performs background filtering on the encoding features in the first layer to obtain the encoding features after filtering the background in the first layer and input the encoding features after filtering the background in the first layer to the first decoder; the encoder in the second layer The encoded features in the first layer are downsampled, and so on until the encoder in the fourth layer After the coding features in the fourth layer are output, the coding features in the fourth layer are directly input into the feature filtering module in the fourth layer, and then the feature filtering module in the fourth layer performs background filtering on the coding features in the fourth layer to obtain the coding features after background filtering in the fourth layer and input the coding features after background filtering in the fourth layer into the decoder in the fourth layer. The decoder in the fourth layer The encoded features after filtering the background in the fourth layer are upsampled to obtain the decoded features of the fourth layer, and then the decoded features of the fourth layer are spliced ​​with the encoded features after filtering the background in the third layer, and the spliced ​​features are input into the encoder in the third layer. , and so on until the encoder in the first layer After outputting the decoded features in the first layer, the decoded features in the first layer are input to the segmentation head, thereby obtaining the segmentation mask of the medical image output by the second U-Net network.

[0093] Optionally, the feature filtering module includes: an adaptive convolutional neural network, an average pooling module, a cosine metric module and a threshold comparison module.

[0094] Figure 4 This is a structural diagram of the feature filtering module. Figure 4 The feature filtering module is described. In the feature filtering module, the adaptive convolutional neural network Conv is connected to the average pooling module and the cosine metric module, the average pooling module is connected to the cosine metric module, and the cosine metric module is connected to the threshold comparison module.

[0095] The adaptive convolutional neural network (Conv) is used to convolve the encoding feature X input to the feature filtering module to obtain the convolution feature of the encoding feature X. The average pooling module is used to perform average pooling on the convolution feature of the encoding feature X to obtain a global representation of the convolution feature. The cosine metric module is used to determine the cosine distance between the convolution feature of the encoding feature X and the global representation of the convolution feature using the cosine metric function. This cosine distance includes the cosine distance of each pixel in the encoding feature X. The threshold comparison module is used to set the mask of pixels whose cosine distance is less than the distance threshold to 0 to obtain the mask of the encoding feature X after background filtering. Finally, the background-filtered mask of the encoding feature X is multiplied by the convolution feature of the encoding feature X to obtain the output of the feature filtering module, which is the encoding feature X after background filtering.

[0096] In this case, optionally, the feature filtering module is configured to implement filtering the background in the coding features output by the first encoder by using a threshold method by adopting the following steps ae.

[0097] Step a: input the first coding feature into the adaptive convolutional neural network to obtain a first convolution feature of the first coding feature.

[0098] The first encoding feature is the encoding feature output by the first encoder.

[0099] Optionally, the second U-Net network has a total of layer, is a positive integer, the first decoder is the first The decoder in the layer, is a positive integer, The value range is 1 to , in this case, step a is expressed using formula (7).

[0100] (7)

[0101] In formula (7), The first The second type of medical image in the second U-Net network The encoded features output by the encoder in the layer are also the first encoded features. represents the adaptive convolutional neural network, This is the first convolution feature. is a positive integer, The value range is 1 to , is the number of the second type of medical images in the training set. The meanings of other parameters in formula (7) are the same as those in formula (1), and their detailed description is omitted here.

[0102] Step b: performing global average pooling on the first convolutional feature to obtain a first global representation of the first convolutional feature.

[0103] Optionally, step b is expressed using formula (8).

[0104] (8)

[0105] In formula (8), The first The second type of medical image in the second U-Net network The global representation in the layer is also the first global representation of the first convolutional feature. Indicates that a global pooling operation is performed. The meanings of other parameters in formula (8) are the same as those in formula (7), and their detailed description is omitted here.

[0106] Step c: using a cosine metric function to determine a first cosine distance between the first convolution feature and the first global representation.

[0107] The first cosine distance includes a cosine distance of each pixel of the first feature map.

[0108] Optionally, step c is expressed using formula (9).

[0109] (9)

[0110] In formula (9), The first The second type of medical image in the second U-Net network The cosine distance in the layer, that is, the first cosine distance, is the cosine metric function. The meanings of other parameters in formula (9) are the same as those in formula (7) and formula (8), and their detailed description is omitted here.

[0111] Step d: setting the mask of pixels whose cosine distance in the first cosine distance is less than the distance threshold to 0, thereby obtaining the mask of the first coding feature after filtering the background.

[0112] Optionally, step d is expressed using formula (10).

[0113] (10)

[0114] In formula (10), The first The second type of medical image in the second U-Net network The mask after filtering the background in the layer, that is, the mask of the first encoded feature after filtering the background, is a threshold filter function, which means that the mask of pixels whose cosine distance is less than the distance threshold is set to 0. The meanings of other parameters in formula (10) are the same as those in formula (9), and their detailed description is omitted here.

[0115] Optionally, the distance threshold is an empirical value, and the embodiment of the present disclosure does not limit the size of the distance threshold.

[0116] In step e, the mask of the first coding feature after background filtering is multiplied by the first convolution feature to obtain the first coding feature after background filtering.

[0117] Optionally, step e is expressed using formula (11).

[0118] (11)

[0119] In formula (11), The first The second type of medical image in the second U-Net network The coded feature after background filtering in the layer is also the first coded feature after background filtering. The meanings of other parameters in formula (11) are the same as those in formula (7) and formula (10), and their detailed description is omitted here.

[0120] In the above steps ae, step a corresponds to the adaptive convolutional neural network in the feature filtering module, step b corresponds to the average pooling module in the feature filtering module, step c corresponds to the cosine measurement module in the feature filtering module, and step d corresponds to the threshold comparison module in the feature filtering module.

[0121] The working process of the above feature filtering module (steps ae) can be uniformly described as formula (12).

[0122] (12)

[0123] In formula (12), Represents the new feature filtering module in the second U-Net network. is the parameter that needs to be fine-tuned for the feature filtering module. The meanings of other parameters in formula (12) are the same as those in formula (7) and formula (11), and their detailed description is omitted here.

[0124] In step 203, the feature filtering module in the second U-Net network is fine-tuned using the second type of medical images in the training set.

[0125] Optionally, step 203 includes the following three steps:

[0126] In the first step, the second type of medical images in the training set are sequentially input into the second U-Net network to obtain multiple segmentation masks output by the second U-Net network.

[0127] Optionally, in the training set When the first type of medical image is input into the first U-Net network, the The encoded features output by the encoder in the layer are expressed using formula (13).

[0128] (13)

[0129] In formula (13), For the training set The second type of medical image in the second U-Net network The encoding features output by the encoder in the layer. The meanings of other parameters in formula (13) are the same as those in formula (7) and formula (1), and their detailed description is omitted here.

[0130] Optionally, in Pick In the case of , the first decoder is used to output the decoded features of the first encoded features after filtering the background using formula (14).

[0131] (14)

[0132] In formula (14), For the training set The second type of medical image in the second U-Net network The decoded features output by the decoder in the layer, that is, Pick Time The decoded features of the first encoded features output by the decoder in the layer. For the training set The second type of medical image in the second U-Net network The encoded features after filtering the background in the layer, that is, Pick Time The first encoded feature output by the encoder in the layer. The meanings of other parameters in formula (14) are the same as those in formula (2) and are not described in detail here.

[0133] exist Take 1 to In the case of any value of , the first decoder is used to output the decoded features of the first encoded features after filtering the background using formula (15).

[0134] (15)

[0135] in, For the training set The second type of medical image in the second U-Net network The decoded features output by the decoder in the layer, that is, Pick Time The decoded features of the first encoded features output by the decoder in the layer, For the The decoding features output by the decoder in the layer. The meanings of other parameters in formula (15) are the same as those in formula (3) and formula (11), and their detailed description is omitted here.

[0136] In the training set After the second type of medical image passes through the decoder in the first layer of the second U-Net network, the decoder in the second layer will output the decoding features of the first layer. , and then input the decoded features of the first layer into the segmentation head to obtain the predicted The segmentation mask of the second type of medical image is obtained. This process can be expressed using formula (16).

[0137] (16)

[0138] In formula (16), The first The segmentation mask of the second type of medical image, is the decoded feature output by the decoder in the first layer of the second U-Net network. The meanings of other parameters in formula (16) are the same as those in formula (4) and are not described in detail here.

[0139] In the second step, for any segmentation mask, the segmentation loss of the segmentation mask is calculated to obtain the segmentation loss of the segmentation mask.

[0140] Here, after obtaining a segmentation mask, the segmentation loss of the segmentation mask can be calculated.

[0141] Exemplarily, the segmentation loss is the mean square error. The segmentation loss of the segmentation mask of the first-class medical image is calculated using formula (17).

[0142] (17)

[0143] In formula (5), The first The segmentation loss of the segmentation mask of the second type of medical image, Express and Calculate the mean square error, The first The true segmentation mask of the second-category medical image. The meanings of other parameters in formula (17) are the same as those in formula (16), and their detailed description is omitted here.

[0144] In the third step, based on the segmentation loss, an optimization algorithm is used to optimize the parameters in the feature filtering module of the second U-Net network.

[0145] Optionally, the SGD optimization algorithm is used to optimize the parameters of the feature filtering module in the second U-Net network. This process can be expressed by formula (18).

[0146] (18)

[0147] In formula (18), for The gradient of These are the parameters that need to be learned in the feature filtering module of the second U-Net network; For the optimized The meanings of other parameters in formula (18) are the same as those in formula (6) and formula (17), and their detailed description is omitted here.

[0148] In the training set After the segmentation loss of the second type of medical image is used to optimize the parameters in the feature filtering module in the second U-Net network, the second type of medical image in the training set can be continuously input to optimize the parameters in the feature filtering module until Each of the second-category medical images is input into the second U-Net network to complete one iterative training.

[0149] Repeat the first to third steps for multiple iterations until the maximum number of iterations of the second U-Net network is reached. The model with the smallest segmentation loss in multiple iterations is the optimal second U-Net network. The parameters of the model with the smallest segmentation loss are , that is, the encoder in the second U-Net network is , the decoder in the second U-Net network is And the feature filtering module in the second U-Net network is .

[0150] Optionally, the maximum number of iterations of the second U-Net network is much smaller than the maximum number of iterations of the first U-Net network. For example, if the maximum number of iterations of the first U-Net network, epoch1, is 50, then the maximum number of iterations of the second U-Net network, epoch2, can be 10, 15, etc.

[0151] Here, since only the parameters in the feature filtering module are trained and the number of iterations is much smaller than the number of iterations when training the first U-Net network, this process can be regarded as fine-tuning the second U-Net network.

[0152] In step 204, the second type of medical images in the test set are segmented based on the fine-tuned second U-Net network.

[0153] The test set verifies the performance of the fine-tuned second U-Net network, that is, inputs the second type of medical images in the test set into the fine-tuned second U-Net network, and judges the fine-tuned second U-Net network based on the predicted segmentation mask output by the fine-tuned second U-Net network.

[0154] There are many related technologies on how to use test sets to verify model performance, so detailed description is omitted here.

[0155] After the fine-tuned second U-Net network is verified to be qualified, the fine-tuned second U-Net network can be used for medical image segmentation. The fine-tuned second U-Net network also has good accuracy when segmenting medical images in the same domain as the second type of medical images.

[0156] In an embodiment of the present disclosure, a feature filtering module is added to the trained first U-Net network to obtain a second U-Net network, and the feature filtering module in the second U-Net network is fine-tuned using a second type of medical image, so that the fine-tuned second U-Net network can accurately segment medical images belonging to the same domain as the second type of medical image. Compared with a traditional U-Net network (such as the first U-Net network), the method in the embodiment of the present disclosure can increase the versatility of the U-Net network at a lower cost (fine-tuning) without retraining the U-Net network, saving costs. When faced with complex scenarios, cross-domain tasks or sparse data sets, the method in the embodiment of the present disclosure exhibits greater robustness and stability.

[0157] Taking a sparse data set as an example, when faced with a sparse data set, the method in the embodiment of the present disclosure is adopted. There is no need to retrain the U-Net network. Instead, the sparse data set only needs to be treated as the second type of medical image to fine-tune the trained U-Net network. The model training cost is low and the trained model has a better segmentation effect on the sparse data set.

[0158] The following are device embodiments of the present application. For details not described in detail in the device embodiments, reference may be made to the above method embodiments.

[0159] Figure 5FIG2 shows a schematic diagram of the structure of a medical image segmentation device based on an improved U-Net provided by an exemplary embodiment of the present disclosure. Figure 5 The medical image segmentation device 500 based on the improved U-Net includes: a first acquisition module 501, a second acquisition module 502, a fine-tuning module 503 and a testing module 504.

[0160] The first acquisition module 501 is used to acquire a first U-Net network that has been trained. The first U-Net network is trained using the first type of medical images in the training set.

[0161] The second acquisition module 502 is used to add a feature filtering module between the encoder and decoder of each layer of the first U-Net network to obtain a second U-Net network. In the second U-Net network, the first encoder, the first feature filtering module and the first decoder are connected in sequence. The first encoder is used to input the output coding features into the first feature filtering module and the second encoder. The first feature filtering module is used to filter the background in the coding features output by the first encoder using a threshold method, and input the coding features after filtering the background into the first decoder. The first encoder is any encoder in the second U-Net network, and the second encoder is an encoder in the layer below the layer where the first encoder is located.

[0162] The fine-tuning module 503 is used to fine-tune the feature filtering module in the second U-Net network using the second type of medical images in the training set, where the second type of medical images and the first type of medical images belong to different domains.

[0163] The testing module 504 is used to segment the second type of medical images in the test set based on the fine-tuned second U-Net network.

[0164] Optionally, the second acquisition module 502 is also used to: input the first coding feature into the adaptive convolutional neural network to obtain a first convolution feature of the first coding feature, where the first coding feature is the coding feature output by the first encoder; perform global average pooling on the first convolution feature to obtain a first global representation of the first convolution feature; use a cosine metric function to determine a first cosine distance between the first convolution feature and the first global representation, where the first cosine distance includes the cosine distance of each pixel in the first feature map; set the mask of pixels in the first cosine distance whose cosine distance is less than the distance threshold to 0 to obtain a mask of the first coding feature after filtering the background; multiply the mask of the first coding feature after filtering the background by the first convolution feature to obtain the first coding feature after filtering the background.

[0165] Optionally, the fine-tuning module 503 is also used to: input the second type of medical images in the training set into the second U-Net network in sequence to obtain multiple segmentation masks output by the second U-Net network; for any segmentation mask, calculate the segmentation loss of the segmentation mask to obtain the segmentation loss of the segmentation mask; based on the segmentation loss, use an optimization algorithm to optimize the parameters in the feature filtering module of the second U-Net network.

[0166] It should be noted that the above-mentioned embodiments provide an example of the division of the above-mentioned functional modules when performing medical image segmentation using the improved U-Net-based medical image segmentation device. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the above-mentioned embodiments provide an improved U-Net-based medical image segmentation device and an improved U-Net-based medical image segmentation method embodiment, which are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0167] The division of modules in the embodiments of the present disclosure is illustrative and represents only a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the present disclosure may be integrated into a single processor, exist physically as separate modules, or be integrated into a single module. The integrated modules may be implemented in either hardware or software functional modules.

[0168] If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a terminal device (which can be a personal computer, mobile phone, or communication device, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program code.

[0169] Figure 6 Schematic diagram of the structure of the computer device provided by the embodiment of the present disclosure. Figure 6 As shown, the computer device 600 includes a processor 601 and a memory 602 .

[0170] Processor 601 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 601 may be implemented in hardware using at least one of the following: a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), or a PLA (Programmable Logic Array). Processor 601 may also include a main processor and a coprocessor. The main processor is used to process data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing content displayed on the display screen. In some embodiments, processor 601 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0171] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one instruction, which is executed by the processor 601 to implement the medical image segmentation method based on the improved U-Net provided in the embodiments of the present disclosure.

[0172] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the computer device 600, and the computer device 600 may include more or fewer components than shown in the figure, or combine some components, or adopt a different arrangement of components.

[0173] The embodiments of the present disclosure also provide a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by a processor of a computer device, the computer device is enabled to execute the medical image segmentation method based on the improved U-Net provided in the embodiments of the present disclosure.

[0174] The embodiments of the present disclosure also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the medical image segmentation method based on the improved U-Net provided in the embodiments of the present disclosure.

[0175] The above description is merely an optional embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A medical image segmentation method based on improved U-Net, characterized in that: The method comprises: Obtaining a trained first U-Net network, where the first U-Net network is trained using a first type of medical image in a training set; A feature filtering module is added between the encoder and decoder of each layer of the first U-Net network to obtain a second U-Net network. In the second U-Net network, the first encoder, the first feature filtering module and the first decoder are connected in sequence. The first encoder is used to input the output coding features into the first feature filtering module and the second encoder. The first feature filtering module is used to filter the background in the coding features output by the first encoder using a threshold method, and input the coding features after filtering the background into the first decoder. The first encoder is any encoder in the second U-Net network, and the second encoder is an encoder in the layer below the layer where the first encoder is located. fine-tuning a feature filtering module in the second U-Net network using a second type of medical image in the training set, where the second type of medical image and the first type of medical image belong to a different domain; Based on the fine-tuned second U-Net network, the second type of medical images in the test set are segmented.

2. The method according to claim 1, characterized in that The first feature filtering module is configured to filter the background of the encoding feature output by the first encoder using a threshold method, including: Inputting the first coding feature into an adaptive convolutional neural network to obtain a first convolution feature of the first coding feature, where the first coding feature is the coding feature output by the first encoder; Performing global average pooling on the first convolutional features to obtain a first global representation of the first convolutional features; Determining a first cosine distance between the first convolutional feature and the first global representation using a cosine metric function, where the first cosine distance includes a cosine distance of each pixel of the first feature map; Setting the mask of pixels whose cosine distance in the first cosine distance is less than the distance threshold to 0 to obtain the mask of the first coding feature after filtering the background; The mask of the first coded feature after background filtering is multiplied by the first convolution feature to obtain the first coded feature after background filtering.

3. The method according to claim 2, wherein The second U-Net network has a total of layer, is a positive integer, the first decoder is the first The decoder in the layer, is a positive integer, The value range is 1 to , exist Pick In the case of , the first decoder is used to output the decoded feature of the first coding feature after filtering the background using the following formula: in, For the The decoded features of the first encoded features output by the decoder in the layer, For the said The decoder in the layer, For the said the first coded feature after filtering the background in the layer; exist Take 1 to In the case of any value of , the first decoder is used to output the decoded feature of the first coding feature after filtering the background using the following formula: in, For the The decoded features of the first encoded features output by the decoder in the layer, For the said The decoder in the layer, For the said the first coded feature after filtering the background in the layer; For the The decoded features output by the decoder in the layer.

4. The method according to any one of claims 1 to 3, characterized in that The fine-tuning of the feature filtering module in the second U-Net network using the second type of medical images in the training set includes: Inputting the second type of medical images in the training set into the second U-Net network in sequence to obtain multiple segmentation masks output by the second U-Net network; For any segmentation mask, calculating the segmentation loss of the segmentation mask to obtain the segmentation loss of the segmentation mask; Based on the segmentation loss, an optimization algorithm is used to optimize the parameters in the feature filtering module of the second U-Net network.

5. A medical image segmentation device based on improved U-Net, characterized in that: The device comprises: A first acquisition module is used to acquire a trained first U-Net network, where the first U-Net network is trained using a first type of medical image in a training set; A second acquisition module is used to add a feature filtering module between the encoder and decoder of each layer of the first U-Net network to obtain a second U-Net network. In the second U-Net network, the first encoder, the first feature filtering module and the first decoder are connected in sequence. The first encoder is used to input the output coding features into the first feature filtering module and the second encoder. The first feature filtering module is used to filter the background in the coding features output by the first encoder using a threshold method, and input the coding features after filtering the background into the first decoder. The first encoder is any encoder in the second U-Net network, and the second encoder is an encoder in the layer below the layer where the first encoder is located; a fine-tuning module, configured to fine-tune a feature filtering module in the second U-Net network using a second type of medical image in the training set, where the second type of medical image belongs to a different domain than the first type of medical image; The testing module is used to segment the second type of medical images in the test set based on the fine-tuned second U-Net network.

6. The device according to claim 5, characterized in that The second acquisition module is further configured to: input the first coding feature into an adaptive convolutional neural network to obtain a first convolution feature of the first coding feature, where the first coding feature is a coding feature output by the first encoder; Performing global average pooling on the first convolutional features to obtain a first global representation of the first convolutional features; Determining a first cosine distance between the first convolutional feature and the first global representation using a cosine metric function, where the first cosine distance includes a cosine distance of each pixel of the first feature map; Setting the mask of pixels whose cosine distance in the first cosine distance is less than the distance threshold to 0 to obtain the mask of the first coding feature after filtering the background; The mask of the first coded feature after background filtering is multiplied by the first convolution feature to obtain the first coded feature after background filtering.

7. The device according to claim 5 or 6, characterized in that The fine-tuning module is further configured to: sequentially input the second type of medical images in the training set into the second U-Net network to obtain a plurality of segmentation masks output by the second U-Net network; For any segmentation mask, calculating the segmentation loss of the segmentation mask to obtain the segmentation loss of the segmentation mask; Based on the segmentation loss, an optimization algorithm is used to optimize the parameters in the feature filtering module of the second U-Net network.

8. A computer device, characterized in that: The computer device includes: a memory and a processor, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor to implement the method according to any one of claims 1 to 4.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images