Medical image segmentation method and device based on improved U-Net, equipment and storage medium

Abstract

The invention discloses a medical image segmentation method, device and equipment based on improved U-Net, and a storage medium, and belongs to the technical field of image segmentation, the method comprises the steps that a trained first U-Net network is acquired, and the first U-Net network is formed by training a first type of medical images in a training set; adding a feature filtering module between an encoder and a decoder of each layer of the first U-Net network to obtain a second U-Net network; finely adjusting a feature filtering module in the second U-Net network by adopting a second type of medical images in the training set, wherein the second type of medical images and the first type of medical images belong to different domains; and based on the fine-tuned second U-Net network, segmenting the second type of medical images in the test set. According to the method, the universality of the U-Net network can be improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image segmentation, and in particular to a medical image segmentation method, apparatus, device and storage medium based on an improved U-Net. Background Art

[0002] Medical images are used to indicate images produced by various medical instruments. Medical image segmentation refers to the act of segmenting parts of a medical image according to a certain goal. For example, if the goal is to segment an organ with a lesion, then medical image segmentation is used to segment the organ with the lesion from the medical image.

[0003] Medical images of different categories or modalities are also medical images of different domains. Segmenting medical images of different domains is called cross-domain medical image segmentation.

[0004] In the related art, the U-Net network is usually used for cross-domain medical image segmentation. U-Net includes an encoder and a decoder. The encoder gradually reduces the spatial dimension of the input image through a series of convolutional layers and maximum pooling layers, and extracts and learns important features in the image in this process. The decoder gradually restores the spatial dimension of the image through upsampling operations and convolutional layers. There is also a jump connection between the encoder and the decoder. The jump connection directly connects the feature map in the encoder to the corresponding layer of the decoder. Through the jump connection, the decoder can refer to the early features during the upsampling process, thereby making up for the details that may be ignored by the traditional deep learning model.

[0005] However, the generalization ability and adaptability of the U-Net network are poor. Summary of the invention

[0006] The present disclosure provides a medical image segmentation method, apparatus, device and storage medium based on an improved U-Net, which can improve the versatility of the U-Net network. The technical solution at least includes the following solutions: In a first aspect, a medical image segmentation method based on an improved U-Net is provided, comprising: obtaining a trained first U-Net network, wherein the first U-Net network is trained using a first type of medical image in a training set; adding a feature filtering module between an encoder and a decoder of each layer of the first U-Net network to obtain a second U-Net network, wherein a first encoder, a first feature filtering module and a first decoder are connected in sequence, wherein the first encoder is used to input the outputted encoded features into the first feature filtering module and the second encoder, wherein the first feature filtering module is used to filter the background in the encoded features outputted by the first encoder using a threshold method, and input the encoded features after filtering the background into the first decoder, wherein the first encoder is any encoder in the second U-Net network, and the second encoder is an encoder in a layer below the layer where the first encoder is located; fine-tuning the feature filtering module in the second U-Net network using the second type of medical image in the training set, wherein the second type of medical image and the first type of medical image belong to different domains; and segmenting the second type of medical image in a test set based on the fine-tuned second U-Net network.

[0007] Optionally, the first feature filtering module is used to filter the background of the coding feature output by the first encoder using a threshold method, including: inputting the first coding feature into an adaptive convolutional neural network to obtain a first convolution feature of the first coding feature, where the first coding feature is the coding feature output by the first encoder; performing global average pooling on the first convolution feature to obtain a first global representation of the first convolution feature; using a cosine metric function to determine a first cosine distance between the first convolution feature and the first global representation, where the first cosine distance includes a cosine distance of each pixel of the first feature map; setting the mask of pixels in the first cosine distance whose cosine distance is less than a distance threshold to 0 to obtain a mask of the first coding feature after filtering the background; multiplying the mask of the first coding feature after filtering the background by the first convolution feature to obtain the first coding feature after filtering the background.

[0008] Optionally, the second U-Net network has a total of layer, is a positive integer, the first decoder is the first The decoder in the layer, is a positive integer, The value range is 1 to , exist Pick In the case of, the first decoder is used to output the decoded feature of the first encoding feature after filtering the background using the following formula:

[0009] in, For the The decoded features of the first encoded features output by the decoder in the layer, For the said The decoder in the layer, For the said the first coded feature after filtering background in the layer; exist Take 1 to In the case of any value of , the first decoder is used to output the decoded feature of the first encoding feature after filtering the background using the following formula:

[0010] in, For the The decoded features of the first encoded features output by the decoder in the layer, For the said The decoder in the layer, For the said the first coded feature after filtering background in the layer; For the The decoded features output by the decoder in the layer.

[0011] Optionally, the use of the second category of medical images in the training set to fine-tune the feature filtering module in the second U-Net network includes: inputting the second category of medical images in the training set into the second U-Net network in sequence to obtain multiple segmentation masks output by the second U-Net network; for any segmentation mask, calculating the segmentation loss of the segmentation mask to obtain the segmentation loss of the segmentation mask; based on the segmentation loss, using an optimization algorithm to optimize the parameters in the feature filtering module of the second U-Net network.

[0012] In the second aspect, a medical image segmentation device based on an improved U-Net is also provided, including: a first acquisition module, used to acquire a trained first U-Net network, wherein the first U-Net network is trained using the first type of medical images in the training set; a second acquisition module, used to add a feature filtering module between the encoder and the decoder of each layer of the first U-Net network to acquire a second U-Net network, wherein the first encoder, the first feature filtering module and the first decoder are connected in sequence, wherein the first encoder is used to input the output encoding features into the first feature filtering module and the second encoder, wherein the first feature filtering module is used to filter the background in the encoding features output by the first encoder using a threshold method, and input the encoding features after filtering the background into the first decoder, wherein the first encoder is any encoder in the second U-Net network, and the second encoder is an encoder in the next layer of the layer where the first encoder is located; a fine-tuning module, used to fine-tune the feature filtering module in the second U-Net network using the second type of medical images in the training set, wherein the second type of medical images and the first type of medical images belong to different domains; and a testing module, used to segment the second type of medical images in the test set based on the fine-tuned second U-Net network.

[0013] Optionally, the second acquisition module is also used to: input the first coding feature into an adaptive convolutional neural network to obtain a first convolution feature of the first coding feature, where the first coding feature is the coding feature output by the first encoder; perform global average pooling on the first convolution feature to obtain a first global representation of the first convolution feature; use a cosine metric function to determine a first cosine distance between the first convolution feature and the first global representation, where the first cosine distance includes the cosine distance of each pixel in the first feature map; set the mask of pixels in the first cosine distance whose cosine distance is less than a distance threshold to 0 to obtain a mask of the first coding feature after filtering the background; multiply the mask of the first coding feature after filtering the background by the first convolution feature to obtain the first coding feature after filtering the background.

[0014] Optionally, the fine-tuning module is also used to: input the second type of medical images in the training set into the second U-Net network in sequence to obtain multiple segmentation masks output by the second U-Net network; for any segmentation mask, calculate the segmentation loss of the segmentation mask to obtain the segmentation loss of the segmentation mask; based on the segmentation loss, use an optimization algorithm to optimize the parameters in the feature filtering module of the second U-Net network.

[0015] In a third aspect, a computer device is also provided, comprising: a memory and a processor, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor, thereby executing the medical image segmentation method based on the improved U-Net described in the above embodiment.

[0016] In a fourth aspect, a computer-readable storage medium is also provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to execute the medical image segmentation method based on the improved U-Net described in the above embodiment.

[0017] In a fifth aspect, a computer program product is provided, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method described in the first aspect is implemented.

[0018] The beneficial effects brought by the technical solution provided by the embodiments of the present disclosure include at least: In the embodiment of the present disclosure, a feature filtering module is added to the trained first U-Net network to obtain a second U-Net network, and the feature filtering module in the second U-Net network is fine-tuned using the second type of medical image, so that the fine-tuned second U-Net network can accurately segment medical images belonging to the same domain as the second type of medical image. Compared with the traditional U-Net network (such as the first U-Net network), the method in the embodiment of the present disclosure can increase the versatility of the U-Net network at a relatively low cost (fine-tuning) without retraining the U-Net network, thus saving costs. When faced with complex scenarios, cross-domain tasks or sparse data sets, the method in the embodiment of the present disclosure exhibits greater robustness and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flowchart of a medical image segmentation method based on an improved U-Net provided by an exemplary embodiment of the present disclosure is shown; Figure 2 A flowchart of a medical image segmentation method based on an improved U-Net provided by another exemplary embodiment of the present disclosure is shown; Figure 3 It is a schematic diagram of the structure of the second U-Net network; Figure 4 It is a structural diagram of the feature filtering module; Figure 5 A schematic diagram of the structure of a medical image segmentation device based on an improved U-Net provided by an exemplary embodiment of the present disclosure is shown; Figure 6 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] Unless otherwise defined, the technical or scientific terms used herein shall have the usual meanings understood by persons of ordinary skill in the field to which the present disclosure belongs. The words "first", "second", "third" and similar words used in the patent application specification and claims of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one" or "one" do not indicate a quantitative limitation, but rather indicate the presence of at least one. Words such as "include" or "comprise" and similar words mean that the elements or objects appearing before "include" or "comprise" cover the elements or objects listed after "include" or "comprise" and their equivalents, and do not exclude other elements or objects. Words such as "connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0022] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0023] In order to facilitate understanding of the embodiments of the present disclosure, the concepts involved in the embodiments of the present disclosure are first explained below: Domain of medical images: Medical images of the same category and modality can be considered to be in the same domain. Here, the category can be an organ category. For example, if multiple medical images are medical images of the same organ, then these medical images are of the same category and in the same domain.

[0024] A modality can be an instrument that produces medical images. For example, medical images produced by the same medical instrument are in the same domain.

[0025] If the medical images in the training set are images of the same organ of different medical instruments, then the modalities of these medical images are different domains, and they belong to a type of cross-domain medical images. Alternatively, if the medical images in the training set are images of different organs of the same medical instrument, then the categories of these medical images are different domains, and they belong to a type of cross-domain medical images. Alternatively, if the medical images in the training set are images of different organs of different medical instruments, they also belong to cross-domain medical images. The number of domains involved in the training set is related to the number of medical instruments and the number of organ categories in the medical images.

[0026] After training, conventional U-Net networks can usually only accurately segment medical images in the same domain as the medical images in the training set. For example, if the training set includes medical images from five medical instruments and seven organs (that is, it includes domains in seven categories and domains in five modalities), then the U-Net network trained with this training set can accurately segment medical images in the domains of these five medical instruments and seven organs, but cannot accurately segment medical images in domains other than these domains, resulting in poor generalization and adaptability of the U-Net network. If medical images in other domains need to be segmented, the U-Net network needs to be retrained, which is costly.

[0027] Here, the above-mentioned domain types are only examples. In actual application, other types of domains are also included. The embodiments of the present disclosure do not limit the types of domains.

[0028] Figure 1 A flowchart of a medical image segmentation method based on an improved U-Net provided by an exemplary embodiment of the present disclosure is shown, and the method can be executed by a computer device. Figure 1 , the method comprising: In step 101, a first U-Net network that has completed training is obtained.

[0029] The first U-Net network is trained using the first type of medical images in the training set.

[0030] In some embodiments, the first U-Net network may be a U-Net network that has been trained and has known parameters.

[0031] Before executing step 101, it is necessary to first obtain a medical image dataset, where the medical image dataset includes a first type of medical images and a second type of medical images, wherein the second type of medical images and the first type of medical images belong to different domains.

[0032] The medical image dataset can then be divided into a training set and test set ,The training set includes the first category of medical images and the second category of medical images, and the testing set includes the second category of images.

[0033] In the training set, there are a total of medical images, including Class I medical images. And The second type of medical image, , , are integers and , All less than There are a total of A second type of medical image. , All are positive integers.

[0034] , which represents the first A medical image has 3 channels, with a length of And the width is ; , which represents the first The true segmentation mask of the medical image exists categories, length And the width is In the real segmentation mask, the value of the mask at the position where the object to be segmented is 1, and the value of the mask at the position where there is no object to be segmented is 0. is an integer, The value range is 1 to .

[0035] Similarly, , which means the first The second type of medical image has 3 channels and is And the width is ; , which means the first The true segmentation mask of the second class medical image exists categories, length And the width is . is an integer, The value range is 1 to .

[0036] In the embodiment of the present disclosure, the structure of the first U-Net network is the same as the structure of the traditional U-Net network.

[0037] The traditional U-Net network is a symmetrical U-shaped structure, with the encoder on the left and the decoder on the right. The U-shaped structure consists of multiple layers, each of which includes an encoder and a decoder. The output of the encoder is the encoding feature, and the output of the decoder is the decoding feature.

[0038] After a medical image is input into the U-Net network, it is first input into the encoder of the first layer. The encoder downsamples the medical image to obtain the encoding features of the first layer, and then inputs the encoding features of the first layer into the encoder of the second layer and the decoder of the first layer respectively. This process is repeated until the encoding features of the last layer are obtained. The encoding features of the last layer are directly input into the decoder of the last layer, and then the decoder of the last layer upsamples the encoding features of the last layer to obtain the decoding features of the last layer, and then inputs the decoding features of the last layer into the decoder of the penultimate layer; the decoder of the penultimate layer upsamples the decoding features of the last layer and the encoding features of the penultimate layer, and so on until the decoding features of the first layer are obtained, and then the decoding features of the first layer are input into the segmentation head, which can output the segmentation result of the medical image.

[0039] There are many related technologies for implementing the traditional U-Net network, so detailed description is omitted here.

[0040] Optionally, the first U-Net network has a total of layer, is a positive integer. In this case, step 101 includes the following three steps: In the first step, the first type of medical images in the training set are sequentially input into the first U-Net network to obtain multiple segmentation masks output by the first U-Net network.

[0041] In the When the first type of medical image is input into the first U-Net network, the The encoding features output by the encoder in the layer are expressed using formula (1). The value range is 1 to , is the number of the first category of medical images in the training set.

[0042] (1) In formula (1), For the The first type of medical image in the first U-Net network The encoded features output by the encoder in the layer, For this The encoder in the layer, For the The first type of medical image in the first U-Net network The encoded features output by the encoder in the layer, in addition, when When it is equal to 1, For the A first-class medical image. is a positive integer, The value range is 1 to . , Indicates The dimensions of the encoded and decoded features in the layer; Indicates The resolution of the encoded and decoded features in the layer, For the The height of the encoded features and decoded features in the layer, For the The width of the encoded and decoded features in the layer.

[0043] exist Pick In the case of The decoded features output by the decoder in the layer are expressed using formula (2).

[0044] (2) In formula (2), For the The first type of medical image in the first U-Net network The decoded features output by the decoder in the layer, For the The first type of medical image in the first U-Net network The encoded features output by the encoder in the layer, For this Decoder in the layer. The meanings of other parameters in formula (2) are the same as those in formula (1), and their detailed description is omitted here.

[0045] exist Take 1 to For any value in The decoded features output by the decoder in the layer are expressed using formula (3).

[0046] (3) In formula (2), For the The first type of medical image in the first U-Net network The decoded features output by the decoder in the layer, For the The first type of medical image in the first U-Net network The encoded features output by the encoder in the layer, For this The decoder in the layer, For the The first type of medical image in the first U-Net network The decoded features output by the decoder in the layer.

[0047] In the After the first type of medical image passes through the decoder in the first layer of the first U-Net network, the decoder of the first layer will output the decoding features of the first layer. , and then input the decoded features of the first layer into the segmentation head to obtain the predicted The segmentation mask of the first-class medical image is obtained. This process can be expressed by formula (4).

[0048] (4) In formula (4), For the predicted Segmentation mask of the first class medical image, For the split head, The decoded features output by the decoder in layer 1 of the first U-Net network.

[0049] Here, the encoding module in the first U-Net network can be used To express, Including located in The encoder in the layer The encoder in the layer is represented as The decoding module in the first U-Net network can be used To express, Including located in The decoder in the layer The encoder in the layer is represented as . , These are the parameters that need to be learned in the first U-Net network.

[0050] In the second step, for any segmentation mask, the segmentation loss of the segmentation mask is calculated to obtain the segmentation loss of the segmentation mask.

[0051] Here, after obtaining a segmentation mask, the segmentation loss of the segmentation mask can be calculated.

[0052] Exemplarily, the segmentation loss is the mean square error. The segmentation loss of the segmentation mask of the first-class medical image is calculated using formula (5).

[0053] (5) In formula (5), For the The segmentation loss of the segmentation mask of the first-class medical image, Express and Calculate the mean square error. For the The true segmentation mask of the first-class medical image. The meanings of other parameters in formula (5) are the same as those in formula (4), and their detailed description is omitted here.

[0054] In the third step, based on the segmentation loss, an optimization algorithm is used to optimize the parameters in the feature filtering module of the first U-Net network.

[0055] Optionally, the SGD (Stochastic Gradient Descent) optimization algorithm is used to optimize the parameters in the first U-Net network. This process can be expressed by formula (6).

[0056] (6) In formula (6), is the learning rate, which can be set to 0.01 in the initial state; for , The gradient of , are the parameters that need to be learned in the first U-Net network; , For the optimized , The meanings of other parameters in formula (6) are the same as those in formula (5), and their detailed description is omitted here.

[0057] In passing the After the segmentation loss of the first type of medical image is used to optimize the parameters in the first U-Net network, the first type of medical image in the training set can be continuously input to optimize the parameters in the first U-Net network until Each of the first-category medical images is input into the first U-Net network to complete one iteration of training.

[0058] Repeat the first to third steps for multiple iterations until the maximum number of iterations of the first U-Net network is reached. The model with the smallest segmentation loss in multiple iterations is the first U-Net network. The parameters of the model with the smallest segmentation loss are , , that is, the encoder in the first U-Net network is , the decoder in the first U-Net network is .

[0059] Exemplarily, the maximum number of iterations of the first U-Net network is 50, that is, epoch1=50.

[0060] In step 102, a feature filtering module is added between the encoder and the decoder of each layer of the first U-Net network to obtain a second U-Net network.

[0061] In the second U-Net network, the first encoder, the first feature filtering module and the first decoder are connected in sequence. The first encoder is used to input the output coding features into the first feature filtering module and the second encoder. The first feature filtering module is used to filter the background in the coding features output by the first encoder using a threshold method, and input the coding features after filtering the background into the first decoder. The first encoder is any encoder in the second U-Net network, and the second encoder is an encoder in the next layer of the layer where the first encoder is located.

[0062] In step 103, the second type of medical images in the training set are used to fine-tune the feature filtering module in the second U-Net network.

[0063] In step 104, the second type of medical images in the test set are segmented based on the fine-tuned second U-Net network.

[0064] In the embodiment of the present disclosure, a feature filtering module is added to the trained first U-Net network to obtain a second U-Net network, and the feature filtering module in the second U-Net network is fine-tuned using the second type of medical image, so that the fine-tuned second U-Net network can accurately segment medical images belonging to the same domain as the second type of medical image. Compared with the traditional U-Net network (such as the first U-Net network), the method in the embodiment of the present disclosure can increase the versatility of the U-Net network at a relatively low cost (fine-tuning) without retraining the U-Net network, thus saving costs. When faced with complex scenarios, cross-domain tasks or sparse data sets, the method in the embodiment of the present disclosure exhibits greater robustness and stability.

[0065] Taking a sparse data set as an example, when faced with a sparse data set, the method in the disclosed embodiment is adopted. There is no need to retrain the U-Net network. Instead, the sparse data set only needs to be used as the second type of medical image to fine-tune the trained U-Net network. The model training cost is low and the trained model has a better segmentation effect on the sparse data set.

[0066] Figure 2 A flowchart of a medical image segmentation method based on an improved U-Net provided by another exemplary embodiment of the present disclosure is shown, and the method can be executed by a computer device. Figure 2 , the method comprising: In step 201, a first U-Net network that has completed training is obtained.

[0067] The relevant contents of step 201 refer to the aforementioned step 101, and the detailed description is omitted here.

[0068] In step 202, a feature filtering module is added between the encoder and the decoder of each layer of the first U-Net network to obtain a second U-Net network.

[0069] In the second U-Net network, the first encoder, the first feature filtering module and the first decoder are connected in sequence. The first encoder is used to input the output coding features into the first feature filtering module and the second encoder. The first feature filtering module is used to filter the background in the coding features output by the first encoder using a threshold method, and input the coding features after filtering the background into the first decoder. The first encoder is any encoder in the second U-Net network, and the second encoder is an encoder in the next layer of the layer where the first encoder is located.

[0070] The encoding module and decoding module in the second U-Net network are the same as those in the first U-Net network, that is, the encoder in the second U-Net network is , the decoder in the second U-Net network is The newly added feature filtering module in the second U-Net network can be expressed as ,in Parameters that need to be fine-tuned for the feature filtering module.

[0071] Figure 3 It is a schematic diagram of the structure of the second U-Net network. Figure 3 As shown, the second U-Net network has a total of four layers, each of which includes an encoder, a feature filtering module, and a decoder. The first encoder is the encoder in the first layer. , the first feature filtering module is the feature filtering module in the first layer, and the first decoder is the decoder in the first layer. When a medical image is input to the first encoder After that, the first encoder The medical image is downsampled to obtain the encoded features of the medical image in the first layer, and then the encoded features of the medical image in the first layer are input into the first feature filtering module and the encoder in the second layer respectively. The first feature filtering module performs background filtering on the encoded features in the first layer to obtain the encoded features after filtering the background in the first layer and input the encoded features after filtering the background in the first layer into the first decoder; the encoder in the second layer The encoded features in the first layer are downsampled, and so on until the encoder in the fourth layer After the coding features in the fourth layer are output, the coding features in the fourth layer are directly input into the feature filtering module in the fourth layer, and then the feature filtering module in the fourth layer performs background filtering on the coding features in the fourth layer to obtain the coding features after filtering the background in the fourth layer and input the coding features after filtering the background in the fourth layer into the decoder in the fourth layer. The decoder in the fourth layer The encoded features after filtering the background in the fourth layer are upsampled to obtain the decoded features of the fourth layer, and then the decoded features of the fourth layer and the encoded features after filtering the background in the third layer are concatenated, and the concatenated features are input into the encoder in the third layer. , and so on, until the encoder in the first layer After outputting the decoded features in the first layer, the decoded features in the first layer are input to the segmentation head, thereby obtaining the segmentation mask of the medical image output by the second U-Net network.

[0072] Optionally, the feature filtering module includes: an adaptive convolutional neural network, an average pooling module, a cosine metric module and a threshold comparison module.

[0073] Figure 4 This is a schematic diagram of the structure of the feature filtering module. Figure 4 The feature filtering module is described. In the feature filtering module, the adaptive convolutional neural network Conv is respectively connected to the average pooling module and the cosine measurement module, the average pooling module is connected to the cosine measurement module, and the cosine measurement module is connected to the threshold comparison module.

[0074] The adaptive convolutional neural network Conv is used to convolve the coding feature X input to the feature filtering module to obtain the convolution feature of the coding feature X; the average pooling module is used to perform average pooling on the convolution feature of the coding feature X to obtain the global representation of the convolution feature; the cosine metric module is used to determine the cosine distance between the convolution feature of the coding feature X and the global representation of the convolution feature using the cosine metric function, and the cosine distance includes the cosine distance of each pixel in the coding feature X. The threshold comparison module is used to set the mask of the pixel whose cosine distance is less than the distance threshold to 0 to obtain the mask of the coding feature X after filtering the background; finally, the mask of the coding feature X after filtering the background is multiplied by the convolution feature of the coding feature X to obtain the output of the feature filtering module, that is, the coding feature X after filtering the background.

[0075] In this case, optionally, the feature filtering module is used to implement filtering the background in the encoding features output by the first encoder by using the threshold method by adopting the following steps ae.

[0076] Step a: input the first coding feature into an adaptive convolutional neural network to obtain a first convolution feature of the first coding feature.

[0077] The first encoding feature is an encoding feature output by the first encoder.

[0078] Optionally, the second U-Net network has a total of layer, is a positive integer, the first decoder is the first The decoder in the layer, is a positive integer, The value range is 1 to , in this case, step a is expressed using formula (7).

[0079] (7) In formula (7), In the training set The second type of medical image in the second U-Net network The encoded features output by the encoder in the layer are also the first encoded features. represents an adaptive convolutional neural network, This is the first convolution feature. is a positive integer, The value range is 1 to , is the number of the second type of medical images in the training set. The meanings of other parameters in formula (7) are the same as those in formula (1), and their detailed description is omitted here.

[0080] Step b: performing global average pooling on the first convolutional feature to obtain a first global representation of the first convolutional feature.

[0081] Optionally, step b is expressed using formula (8).

[0082] (8) In formula (8), In the training set The second type of medical image in the second U-Net network The global representation in the layer is also the first global representation of the first convolutional feature. Indicates that a global pooling operation is performed. The meanings of other parameters in formula (8) are the same as those in formula (7), and their detailed description is omitted here.

[0083] Step c: using a cosine metric function to determine a first cosine distance between the first convolution feature and the first global representation.

[0084] The first cosine distance includes a cosine distance of each pixel of the first feature map.

[0085] Optionally, step c is expressed using formula (9).

[0086] (9) In formula (9), In the training set The second type of medical image in the second U-Net network The cosine distance in the layer, that is, the first cosine distance, is the cosine metric function. The meanings of other parameters in formula (9) are the same as those in formula (7) and formula (8), and their detailed description is omitted here.

[0087] Step d: setting the mask of pixels whose cosine distance in the first cosine distance is less than the distance threshold to 0, thereby obtaining the mask of the first coding feature after filtering the background.

[0088] Optionally, step d is expressed using formula (10).

[0089] (10) In formula (10), In the training set The second type of medical image in the second U-Net network The mask after filtering the background in the layer, that is, the mask of the first encoded feature after filtering the background, is a threshold filtering function, that is, the mask of pixels whose cosine distance is less than the distance threshold is set to 0. The meanings of other parameters in formula (10) are the same as those in formula (9), and detailed description is omitted here.

[0090] Optionally, the distance threshold is an empirical value, and the embodiment of the present disclosure does not limit the size of the distance threshold.

[0091] Step e: multiplying the mask of the first coded feature after filtering the background by the first convolution feature to obtain the first coded feature after filtering the background.

[0092] Optionally, step e is expressed using formula (11).

[0093] (11) In formula (11), In the training set The second type of medical image in the second U-Net network The coded feature after filtering the background in the layer is also the first coded feature after filtering the background. The meanings of other parameters in formula (11) are the same as those in formula (7) and formula (10), and detailed description is omitted here.

[0094] In the above steps ae, step a corresponds to the adaptive convolutional neural network in the feature filtering module, step b corresponds to the average pooling module in the feature filtering module, step c corresponds to the cosine measurement module in the feature filtering module, and step d corresponds to the threshold comparison module in the feature filtering module.

[0095] The working process of the above feature filtering module (steps ae) can be uniformly described as formula (12).

[0096] (12) In formula (12), Represents the newly added feature filtering module in the second U-Net network. is the parameter that needs to be fine-tuned in the feature filtering module. The meanings of other parameters in formula (12) are the same as those in formula (7) and formula (11), and their detailed description is omitted here.

[0097] In step 203, the second type of medical images in the training set are used to fine-tune the feature filtering module in the second U-Net network.

[0098] Optionally, step 203 includes the following three steps: In the first step, the second type of medical images in the training set are sequentially input into the second U-Net network to obtain multiple segmentation masks output by the second U-Net network.

[0099] Optionally, in the training set When the first type of medical image is input into the first U-Net network, the The encoded features output by the encoder in the layer are expressed using formula (13).

[0100] (13) In formula (13), For the training set The second type of medical image in the second U-Net network The encoding features output by the encoder in the layer. The meanings of other parameters in formula (13) are the same as those in formula (7) and formula (1), and their detailed description is omitted here.

[0101] Optionally, in Pick In the case of , the first decoder is used to output the decoded features of the first encoded features after filtering the background using formula (14).

[0102] (14) In formula (14), For the training set The second type of medical image in the second U-Net network The decoded features output by the decoder in the layer, that is, Pick Time The decoded features of the first encoded features output by the decoder in the layer. For the training set The second type of medical image in the second U-Net network The encoded features after filtering the background in the layer, that is, in Pick Time The first encoded feature output by the encoder in the layer. The meanings of other parameters in formula (14) are the same as those in formula (2), and their detailed description is omitted here.

[0103] exist Take 1 to In the case of any value of , the first decoder is used to output the decoded features of the first encoded features after filtering the background using formula (15).

[0104] (15) in, For the training set The second type of medical image in the second U-Net network The decoded features output by the decoder in the layer, that is, Pick Time The decoded features of the first encoded features output by the decoder in the layer, For the The decoding features output by the decoder in the layer. The meanings of other parameters in formula (15) are the same as those in formula (3) and formula (11), and their detailed description is omitted here.

[0105] In the training set After the second type of medical image passes through the decoder in the first layer of the second U-Net network, the decoder in the second layer will output the decoding features of the first layer. , and then input the decoded features of the first layer into the segmentation head to obtain the predicted The segmentation mask of the second type of medical image is obtained. This process can be expressed by formula (16).

[0106] (16) In formula (16), is the first The segmentation mask of the second type of medical image, is the decoded feature output by the decoder in the first layer of the second U-Net network. The meanings of other parameters in formula (16) are the same as those in formula (4), and their detailed description is omitted here.

[0107] In the second step, for any segmentation mask, the segmentation loss of the segmentation mask is calculated to obtain the segmentation loss of the segmentation mask.

[0108] Here, after obtaining a segmentation mask, the segmentation loss of the segmentation mask can be calculated.

[0109] Exemplarily, the segmentation loss is the mean square error. The segmentation loss of the segmentation mask of the first-class medical image is calculated using formula (17).

[0110] (17) In formula (5), In the training set The segmentation loss of the segmentation mask of the second class medical image, Express and Calculate the mean square error. In the training set The true segmentation mask of the second type of medical image. The meanings of other parameters in formula (17) are the same as those in formula (16), and their detailed description is omitted here.

[0111] In the third step, based on the segmentation loss, an optimization algorithm is used to optimize the parameters in the feature filtering module of the second U-Net network.

[0112] Optionally, the SGD optimization algorithm is used to optimize the parameters of the feature filtering module in the second U-Net network. This process can be expressed by formula (18).

[0113] (18) In formula (18), for The gradient of The parameters that need to be learned in the feature filtering module of the second U-Net network; For the optimized The meanings of other parameters in formula (18) are the same as those in formula (6) and formula (17), and their detailed description is omitted here.

[0114] In the training set After optimizing the parameters in the feature filtering module in the second U-Net network by the segmentation loss of the second type of medical image, the second type of medical image in the training set can be continuously input to optimize the parameters in the feature filtering module until Each of the second-category medical images is input into the second U-Net network to complete one iteration of training.

[0115] Repeat the first to third steps for multiple iterations until the maximum number of iterations of the second U-Net network is reached. The model with the smallest segmentation loss in multiple iterations is the optimal second U-Net network. The parameters of the model with the smallest segmentation loss are , that is, the encoder in the second U-Net network is , the decoder in the second U-Net network is And the feature filtering module in the second U-Net network is .

[0116] Optionally, the maximum number of iterations of the second U-Net network is much smaller than the maximum number of iterations of the first U-Net network. For example, if the maximum number of iterations of the first U-Net network, epoch1, is 50, then the maximum number of iterations of the second U-Net network, epoch2, can be 10, 15, etc.

[0117] Here, since only the parameters in the feature filtering module are trained and the number of iterations is much smaller than the number of iterations when the first U-Net network is trained, this process can be regarded as fine-tuning the second U-Net network.

[0118] In step 204, the second type of medical images in the test set are segmented based on the fine-tuned second U-Net network.

[0119] The test set verifies the performance of the fine-tuned second U-Net network, that is, inputs the second type of medical images in the test set into the fine-tuned second U-Net network, and judges the fine-tuned second U-Net network according to the predicted segmentation mask output by the fine-tuned second U-Net network.

[0120] There are many related technologies on how to use a test set to verify model performance, so detailed description is omitted here.

[0121] After the fine-tuned second U-Net network is verified to be qualified, the fine-tuned second U-Net network can be used for medical image segmentation. And the fine-tuned second U-Net network has good accuracy when segmenting medical images in the same domain as the second type of medical images.

[0122] In the embodiment of the present disclosure, a feature filtering module is added to the trained first U-Net network to obtain a second U-Net network, and the feature filtering module in the second U-Net network is fine-tuned using the second type of medical image, so that the fine-tuned second U-Net network can accurately segment medical images belonging to the same domain as the second type of medical image. Compared with the traditional U-Net network (such as the first U-Net network), the method in the embodiment of the present disclosure can increase the versatility of the U-Net network at a relatively low cost (fine-tuning) without retraining the U-Net network, thus saving costs. When faced with complex scenarios, cross-domain tasks or sparse data sets, the method in the embodiment of the present disclosure exhibits greater robustness and stability.

[0123] Taking a sparse data set as an example, when faced with a sparse data set, the method in the disclosed embodiment is adopted. There is no need to retrain the U-Net network. Instead, the sparse data set only needs to be used as the second type of medical image to fine-tune the trained U-Net network. The model training cost is low and the trained model has a better segmentation effect on the sparse data set.

[0124] The following are device embodiments of the present application. For details not described in detail in the device embodiments, reference may be made to the above method embodiments.

[0125] Figure 5 FIG. 1 is a schematic diagram showing a structure of a medical image segmentation device based on an improved U-Net provided by an exemplary embodiment of the present disclosure. Figure 5 The medical image segmentation device 500 based on improved U-Net includes: a first acquisition module 501, a second acquisition module 502, a fine-tuning module 503 and a testing module 504.

[0126] The first acquisition module 501 is used to acquire a first U-Net network that has been trained. The first U-Net network is trained using the first type of medical images in the training set.

[0127] The second acquisition module 502 is used to add a feature filtering module between the encoder and the decoder of each layer of the first U-Net network to obtain the second U-Net network. In the second U-Net network, the first encoder, the first feature filtering module and the first decoder are connected in sequence. The first encoder is used to input the output coding features into the first feature filtering module and the second encoder. The first feature filtering module is used to filter the background in the coding features output by the first encoder using a threshold method, and input the coding features after filtering the background into the first decoder. The first encoder is any encoder in the second U-Net network, and the second encoder is an encoder in the layer below the layer where the first encoder is located.

[0128] The fine-tuning module 503 is used to fine-tune the feature filtering module in the second U-Net network using the second type of medical images in the training set, and the second type of medical images and the first type of medical images belong to different domains.

[0129] The testing module 504 is used to segment the second type of medical images in the test set based on the fine-tuned second U-Net network.

[0130] Optionally, the second acquisition module 502 is also used to: input the first coding feature into the adaptive convolutional neural network to obtain a first convolution feature of the first coding feature, where the first coding feature is the coding feature output by the first encoder; perform global average pooling on the first convolution feature to obtain a first global representation of the first convolution feature; use a cosine metric function to determine a first cosine distance between the first convolution feature and the first global representation, where the first cosine distance includes the cosine distance of each pixel in the first feature map; set the mask of pixels in the first cosine distance whose cosine distance is less than the distance threshold to 0 to obtain a mask of the first coding feature after filtering the background; multiply the mask of the first coding feature after filtering the background by the first convolution feature to obtain the first coding feature after filtering the background.

[0131] Optionally, the fine-tuning module 503 is also used to: input the second type of medical images in the training set into the second U-Net network in sequence to obtain multiple segmentation masks output by the second U-Net network; for any segmentation mask, calculate the segmentation loss of the segmentation mask to obtain the segmentation loss of the segmentation mask; based on the segmentation loss, use an optimization algorithm to optimize the parameters in the feature filtering module of the second U-Net network.

[0132] It should be noted that: when the medical image segmentation device based on the improved U-Net provided in the above embodiment performs medical image segmentation, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the medical image segmentation device based on the improved U-Net provided in the above embodiment and the medical image segmentation method embodiment based on the improved U-Net belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0133] The division of modules in the embodiments of the present disclosure is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present disclosure may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0134] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a terminal device (which can be a personal computer, mobile phone, or communication device, etc.) or a processor (processor) to execute all or part of the steps of the method of each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0135] Figure 6 Schematic diagram of the structure of the computer device provided by the embodiment of the present disclosure. Figure 6 As shown, the computer device 600 includes: a processor 601 and a memory 602 .

[0136] The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0137] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one instruction, which is used to be executed by the processor 601 to implement the medical image segmentation method based on the improved U-Net provided in the embodiments of the present disclosure.

[0138] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the computer device 600, and the computer device 600 may include more or less components than those shown in the figure, or combine some components, or adopt a different arrangement of components.

[0139] The embodiments of the present disclosure also provide a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by a processor of a computer device, the computer device is enabled to execute the medical image segmentation method based on the improved U-Net provided in the embodiments of the present disclosure.

[0140] The present disclosure also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the medical image segmentation method based on the improved U-Net provided in the present disclosure.

[0141] The above description is only an optional embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A medical image segmentation method based on improved U-Net, characterized in that: The method comprises: Obtaining a first U-Net network that has been trained, where the first U-Net network is trained using a first type of medical image in a training set; A feature filtering module is added between the encoder and the decoder of each layer of the first U-Net network to obtain a second U-Net network. In the second U-Net network, the first encoder, the first feature filtering module and the first decoder are connected in sequence. The first encoder is used to input the output coding features into the first feature filtering module and the second encoder. The first feature filtering module is used to filter the background in the coding features output by the first encoder using a threshold method, and input the coding features after filtering the background into the first decoder. The first encoder is any encoder in the second U-Net network, and the second encoder is an encoder in a layer below the layer where the first encoder is located. fine-tuning a feature filtering module in the second U-Net network using a second type of medical image in the training set, wherein the second type of medical image and the first type of medical image belong to a different domain; Based on the fine-tuned second U-Net network, the second type of medical images in the test set are segmented.

2. The method according to claim 1, characterized in that The first feature filtering module is used to filter the background of the encoding feature output by the first encoder by using a threshold method, including: Inputting a first coding feature into an adaptive convolutional neural network to obtain a first convolutional feature of the first coding feature, where the first coding feature is a coding feature output by the first encoder; Performing global average pooling on the first convolutional feature to obtain a first global representation of the first convolutional feature; Determine a first cosine distance between the first convolutional feature and the first global representation using a cosine metric function, where the first cosine distance includes a cosine distance of each pixel of the first feature map; Setting the mask of pixels whose cosine distance in the first cosine distance is less than the distance threshold to 0, thereby obtaining the mask of the first coding feature after filtering the background; The mask of the first coded feature after filtering the background is multiplied by the first convolution feature to obtain the first coded feature after filtering the background.

3. The method according to claim 2, characterized in that: The second U-Net network has a total of layer, is a positive integer, the first decoder is the first The decoder in the layer, is a positive integer, The value range is 1 to , exist Pick In the case of, the first decoder is used to output the decoded feature of the first encoding feature after filtering the background using the following formula: in, For the The decoded features of the first encoded features output by the decoder in the layer, For the The decoder in the layer, For the the first coded feature after filtering background in the layer; exist Take 1 to In the case of any value of , the first decoder is used to output the decoded feature of the first encoding feature after filtering the background using the following formula: in, For the The decoded features of the first encoded features output by the decoder in the layer, For the The decoder in the layer, For the the first coded feature after filtering background in the layer; For the The decoded features output by the decoder in the layer.

4. The method according to any one of claims 1 to 3, characterized in that: The step of fine-tuning the feature filtering module in the second U-Net network using the second type of medical images in the training set includes: Inputting the second type of medical images in the training set into the second U-Net network in sequence to obtain a plurality of segmentation masks output by the second U-Net network; For any segmentation mask, calculating the segmentation loss of the segmentation mask to obtain the segmentation loss of the segmentation mask; Based on the segmentation loss, an optimization algorithm is used to optimize the parameters in the feature filtering module of the second U-Net network.

5. A medical image segmentation device based on improved U-Net, characterized in that: The device comprises: A first acquisition module is used to acquire a first U-Net network that has been trained, where the first U-Net network is trained using a first type of medical image in a training set; A second acquisition module is used to add a feature filtering module between the encoder and the decoder of each layer of the first U-Net network to obtain a second U-Net network, in which the first encoder, the first feature filtering module and the first decoder are connected in sequence, the first encoder is used to input the output coding features into the first feature filtering module and the second encoder, the first feature filtering module is used to filter the background in the coding features output by the first encoder by using a threshold method, and input the coding features after filtering the background into the first decoder, the first encoder is any encoder in the second U-Net network, and the second encoder is an encoder in a layer below the layer where the first encoder is located; A fine-tuning module, configured to fine-tune a feature filtering module in the second U-Net network using a second type of medical image in the training set, wherein the second type of medical image and the first type of medical image belong to a different domain; The testing module is used to segment the second type of medical images in the test set based on the fine-tuned second U-Net network.

6. The device according to claim 5, characterized in that The second acquisition module is further used to: input the first coding feature into the adaptive convolutional neural network to obtain a first convolution feature of the first coding feature, where the first coding feature is the coding feature output by the first encoder; Performing global average pooling on the first convolutional feature to obtain a first global representation of the first convolutional feature; Determine a first cosine distance between the first convolutional feature and the first global representation using a cosine metric function, where the first cosine distance includes a cosine distance of each pixel of the first feature map; Setting the mask of pixels whose cosine distance in the first cosine distance is less than the distance threshold to 0, thereby obtaining the mask of the first coding feature after filtering the background; The mask of the first coded feature after filtering the background is multiplied by the first convolution feature to obtain the first coded feature after filtering the background.

7. The device according to claim 5 or 6, characterized in that The fine-tuning module is also used to: input the second type of medical images in the training set into the second U-Net network in sequence to obtain multiple segmentation masks output by the second U-Net network; For any segmentation mask, calculating the segmentation loss of the segmentation mask to obtain the segmentation loss of the segmentation mask; Based on the segmentation loss, an optimization algorithm is used to optimize the parameters in the feature filtering module of the second U-Net network.

8. A computer device, characterized in that: The computer device comprises: a memory and a processor, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor to implement the method according to any one of claims 1 to 4.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images