A liver tumor segmentation method, device and equipment based on a cavity convolution

Abstract

The application provides a liver tumor segmentation method, device and equipment based on a cavity convolution. The method comprises the following steps: acquiring liver images with different lesion degrees; inputting the liver images into a first-level segmentation network to acquire a coarse segmentation image and an edge extraction result image of the whole image; inputting the coarse segmentation image, the edge extraction result image of the whole image and a liver image to be segmented into a second-level segmentation network; performing convolution processing on the coarse segmentation image and the edge extraction result image of the whole image to extract features, and performing cavity convolution processing on the liver image to be segmented to extract features; fusing the extracted features; and analyzing the fused features to acquire a final segmentation result image of the liver tumor image. The method provided by the application improves the image segmentation accuracy and the edge detail information of the medical image segmentation result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a liver tumor segmentation method, device and equipment based on a dilated convolution. BACKGROUND

[0002] The development of medical imaging technology greatly improves the accuracy of disease diagnosis, and also promotes the development of medical technology from another aspect. It fundamentally improves the level of medical treatment and further strengthens the basic guarantee of people's life.

[0003] By scanning the organs of the human body, and presenting the scanned results in the form of images, it is helpful for doctors to diagnose diseases. The liver tumor of the human body is a relatively common tumor, but the tumor does not have a fixed shape and distribution. Therefore, it is extremely difficult to segment the liver tumor from the image by using the traditional image segmentation algorithm. Thanks to the development of deep learning technology, image segmentation technology has entered the stage of semantic level segmentation. However, the existing liver tumor segmentation technology still has two problems: (1) due to the difference between individuals, the tumor lesion process and distribution on the image are completely different, which also leads to a small amount of data samples that can be used for training, and it is difficult to achieve accurate segmentation; (2) the existing image segmentation algorithm will lose part of the image edge detail information in the multiple encoding and decoding process, resulting in poor edge of the segmented result. At the same time, improving the edge detail information of the medical image segmentation result can provide more help for medical diagnosis and analysis, so improving the edge detail of the medical image segmentation result is an important task. SUMMARY

[0004] The purpose of the present application is to solve the defects of the prior art, and to provide a liver tumor segmentation method, device and equipment based on a dilated convolution.

[0005] A liver tumor segmentation method based on a dilated convolution, comprising:

[0006] Obtaining liver images of different lesion degrees;

[0007] Inputting the liver images into a first-level segmentation network to obtain a rough segmentation image for the liver images and an edge extraction result image of the whole image, the rough segmentation image containing tumor distribution information;

[0008] Inputting the rough segmentation image, the edge extraction result image of the whole image and a liver image to be segmented into a second-level segmentation network, performing convolution processing on the rough segmentation image and the edge extraction result image of the whole image through the second-level segmentation network, and performing dilated convolution processing on the liver image to be segmented through the second-level segmentation network;

[0009] Fusing all the extracted features;

[0010] The fusion features are analyzed to obtain a final segmentation result image of the liver tumor image.

[0011] Further, in the first segmentation network of the liver tumor segmentation method based on the dilated convolution, a loss function corresponding to the rough segmentation image is:

[0012]

[0013]

[0014]

[0015] wherein I p represents a segmentation result output by the deeplab neural network, I t represents a corresponding label, Loss represents a loss of the rough segmentation image in the training process of the first segmentation network, i and j respectively represent coordinates of pixel points in the image, and β is a constant close to zero.

[0016] Further, in the liver tumor segmentation method based on the dilated convolution, the second segmentation network adopts an encoder-decoder structure.

[0017] Further, in the liver tumor segmentation method based on the dilated convolution, a final segmentation result image output by the second segmentation network is expressed by the following formula:

[0018] I=aF-(1-a)B (4)

[0019] wherein I represents a final generated segmentation result image, a represents a generated new segmentation result to be fused, F is a generated foreground image, and B represents data to be enhanced.

[0020] Further, in the liver tumor segmentation method based on the dilated convolution, the encoder of the second segmentation network includes twice up-sampling and down-sampling processes.

[0021] Further, in the first segmentation network of the liver tumor segmentation method based on the dilated convolution, a loss function corresponding to the edge extraction result image of the whole image is:

[0022]

[0023]

[0024]

[0025] wherein I pr represents edge information of the generated image, Ipt denotes the label corresponding to the edge information, (i, j) denotes the position of the pixel in the image, Loss edge denotes the loss of the edge extraction part in the training process of the primary segmentation network.

[0026] Further, in the liver tumor segmentation method based on the dilated convolution as described above, the feature extraction by the secondary segmentation network on the liver image to be segmented includes: extracting the feature by setting different dilated rates.

[0027] A liver tumor segmentation device based on dilated convolution, comprising:

[0028] An acquisition unit is configured to acquire liver images with different lesion degrees.

[0029] A segmentation unit is configured to input the liver image into a primary segmentation network, acquire a coarse segmentation image for the liver image and an edge extraction result image of the whole image, and the coarse segmentation image contains tumor distribution information.

[0030] A feature extraction unit is configured to input the coarse segmentation image, the edge extraction result image of the whole image, and the liver image to be segmented into a secondary segmentation network, extract features by performing convolution processing on the coarse segmentation image and the edge extraction result image of the whole image respectively by the secondary segmentation network, and extract features by performing dilated convolution processing on the liver image to be segmented by the secondary segmentation network.

[0031] A fusion unit is configured to fuse all the extracted features.

[0032] An analysis unit is configured to analyze the fused features and acquire a final segmentation result image of the liver tumor image.

[0033] An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the liver tumor segmentation method based on dilated convolution as described above when executing the program.

[0034] A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the liver tumor segmentation method based on dilated convolution as described above.

[0035] A computer program product comprising a computer program, wherein the computer program is executable by a processor to implement the liver tumor segmentation method based on dilated convolution as described above.

[0036] The application provides a liver tumor segmentation method, device and equipment based on a cavity convolution, which is characterized in that: on one hand, a first segmentation network is used to coarsely segment liver images with different lesion degrees, and a coarse segmentation image, an edge extraction result image of the whole image and a liver image to be segmented are input into a second segmentation network to realize accurate segmentation of the liver image to be segmented; on the other hand, the edge extraction result image of the whole image is output by the first segmentation network, and the edge extraction result image of the whole image and the liver image to be segmented are taken as inputs of the second segmentation network, so that the edge detail information of the medical image segmentation result is improved. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 A liver tumor segmentation method based on a cavity convolution is provided.

[0038] Figure 2 A principle block diagram of a liver tumor segmentation device based on a cavity convolution is provided.

[0039] Figure 3 A structure schematic diagram of an electronic device is provided. DETAILED DESCRIPTION

[0040] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application are described clearly and completely below. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0041] Figure 1 A liver tumor segmentation method based on a cavity convolution is provided. Figure 1 As shown in the figure, the method comprises the following steps:

[0042] Step 101: obtaining liver images with different lesion degrees;

[0043] Step 102: inputting the liver images into a first segmentation network to obtain a coarse segmentation image for the liver images and an edge extraction result image of the whole image, wherein the coarse segmentation image contains tumor distribution information;

[0044] Step 103: inputting the coarse segmentation image, the edge extraction result image of the whole image and a liver image to be segmented into a second segmentation network, performing convolution processing on the coarse segmentation image and the edge extraction result image of the whole image by the second segmentation network, and performing cavity convolution processing on the liver image to be segmented by the second segmentation network;

[0045] Step 104: fuse all the extracted features;

[0046] Step 105: analyze the fused features to obtain the final segmentation result map of the liver tumor image.

[0047] Specifically, the present application is a liver tumor segmentation method based on a cavity convolution. The method realizes the purpose of data enhancement through a first-level segmentation network and a second-level segmentation network. For the first-level segmentation network, the target surface data input by the first-level segmentation network is used to extract lesion information. The target surface data is original liver data with different degrees of lesion information, and the lesion information refers to the distribution of tumors on the liver. That is, the input of the first-level segmentation network is a liver organ image with tumors, and the output is a distribution information map of liver tumors. For the second-level segmentation network, the second-level information fusion network is used to fuse the extracted lesion information and the original liver image input by the second-level segmentation network, that is, to fuse the lesion information into the data sample to be enhanced, and finally to obtain enhanced data. The input of the second-level segmentation network includes three parts: 1, the output of the first-level network; 2, a liver organ image without carrying other lesion information; and 3, condition information for controlling the generation position of lesion data. Since medical data is different from other data, ethical review is required before using these data for research. However, in actual application research, a large number of samples are often required. Therefore, data enhancement is needed, but the data generated by data enhancement needs to meet the real situation. Therefore, when generating, it is necessary to cooperate with relevant medical experts, and to constrain and realize data enhancement according to common lesion positions and other information. The liver organ image without carrying other lesion information is the single variable principle, that is, for a certain disease, other interference needs to be excluded. The first-level segmentation network adopts a deeplab network structure, and its loss function is defined as follows.

[0048]

[0049]

[0050]

[0051] In the above formula, I p represents the segmentation result output by the deeplab neural network, and I tThe corresponding label is represented, and Loss represents the loss in the training process. i and j represent the coordinates of the pixels in the image, respectively. The formula (3) is used to process the case where the input data does not contain lesion information. If the data does not contain lesion information during the training process, the denominator of the loss function shown in the above formula (3) may be zero, which may cause problems in the training. The data without lesion information is added in the training to enhance the diversity of the samples and learn a neural network model with stronger extraction capability. The beta is a constant close to zero, which can be adjusted according to the effect of the finally generated data in practice.

[0052] The secondary segmentation network is trained in the manner of conditional generative adversarial network, and the generator part of the network adopts the refiner network. The encoder-decoder structure is still adopted. The primary segmentation network has generated the lesion information, so the secondary segmentation network only needs to further refine the lesion information and generate a new lesion segmentation result and foreground F according to the input data. With the new lesion segmentation result and foreground F, the finally enhanced data can be generated according to the following formula (4).

[0053] I=aF-(1-a)B (4)

[0054] In formula (4), I represents the finally generated image, a represents the generated new segmentation result to be fused, F is the generated foreground image, and B represents the data to be enhanced.

[0055] The method provided by the application performs coarse segmentation on the liver images with different lesion degrees through the primary segmentation network, calculates the segmentation result of the coarse segmentation network, extracts features of the input liver image to be segmented through the dilated convolution of the secondary segmentation network, increases the receptive field in the image segmentation process, so that more information about the image edge and structure is obtained, and finally, the coarse segmentation result is further refined according to the features extracted by the dilated convolution, and a refined segmentation result image is obtained.

[0056] The first-level segmentation network takes U-net as a basic network, and the input is a liver image with different lesion degrees, and the output is a rough segmentation image. Two main features affecting image segmentation are local features and global features. In actual segmentation tasks, it corresponds to the receptive field of the image. Compared with the U-net algorithm, the PsPnet algorithm represents different receptive fields through multi-scale information, and fuses these multi-scale information through the image pyramid for image segmentation tasks. The Deeplab series algorithm proposes to obtain local features and global features of the image by setting different hole rates, and generates the final segmentation result according to the analysis of these features. The common point of these algorithms is that there is only one encoder and decoder, so in order to achieve better segmentation effect, the depth of the encoder and decoder in the algorithm design will be deep, and the number of down-sampling will be more. The information of the smallest feature map in each dimension will be reduced to less than 1 / 32 of the original, and the feature map is down-sampled to such a small size. It is difficult to recover all the edge detail information through up-sampling, which inevitably leads to the problem that the edge information of part of the data is not clear. This is the problem that image segmentation by a first-level segmentation network will face.

[0057] On the other hand, most of the pixels in the image are background and foreground. That is, the pixel distribution of the same object is very close. Therefore, under the condition of not requiring edge details, it is relatively simple to do rough segmentation of the image. Therefore, under such a background, two-level segmentation networks are set, the first-level segmentation network does rough segmentation, and the second-level segmentation network perfects the edge and detail information of the image. In practical applications, the first-level segmentation network adopts u-net, which is a relatively simple network structure, and the network depth and network complexity are less than those of PsPnet and depplab series algorithms, which can effectively save the calculation time. The difference between the u-net algorithm and the first-level segmentation network provided by the application is that the rough segmentation result output by the first-level segmentation network not only contains a rough segmentation image, but also outputs an edge extraction result image of the whole image. That is, an output is added to the u-net network, which outputs the edge information of the image at the same time as the segmentation result.

[0058] The second-level segmentation network adopts the PAPnet algorithm provided by the application. The PAPnet algorithm is different from the PsPnet algorithm, and the PAPnet algorithm is an optimized algorithm of the application. The PsPnet algorithm extracts features through a general convolutional neural network, and the pooling operation in the convolutional neural network causes local information loss. The PAPnet provided by the application uses a hole convolution instead of the pooling operation, which can effectively solve the problem of local information loss caused by the pooling operation.

[0059] The secondary segmentation network provided in the application is divided into three layers, i.e. upper, middle and lower layers, and corresponds to three inputs, i.e. a rough segmentation image of the primary network and an edge extraction result image of the whole image, and a liver image to be segmented. The rough segmentation image and the edge extraction result of the primary network are not extracted by the dilated convolution to extract the convolution features under different dilated rates, but only simple convolution processing is performed, and the features of different receptive fields are extracted by using the dilated convolution on the liver image to be segmented. The reason is that the secondary network is to refine the edge details of segmentation, and the rough segmentation image and the edge information generated by the primary network are processed and cannot represent the image features under different receptive fields. Finally, a feature fusion layer is used to fuse the three features together for common analysis, and then the final segmentation result is output. In the refinement analysis, the information of the three branches is jointly used to generate the final segmentation result.

[0060] For the loss function of the neural network, the loss function definition of the primary rough segmentation network is as follows. The loss function of the image segmentation part is consistent with formula (3) in the above, and is represented by Loss seg1 The loss of the edge extraction part is defined as follows, and is represented by Loss edge

[0061]

[0062]

[0063]

[0064] As described above, I pr represents the edge information of the generated image, I pt represents the label corresponding to the edge information, and (i, j) represents the position of the pixel in the image. The loss function of the secondary segmentation network is also consistent with formula (3) in the above, and is represented by Loss seg2 .

[0065] The final loss function is shown in formula (8) as follows:

[0066] Loss=Loss seg1 +Loss seg2 +Loss edge (8)

[0067] Loss edge represents the loss of the edge in the primary segmentation network.

[0068] ​The PAPnet network provided in this invention utilizes edge extraction results and the liver image to be segmented to analyze edge details in the image, and further refines the segmentation results output by the first-level network. Since the size of the tumor is not fixed, it is necessary to fuse information from different receptive fields in the image to jointly analyze and refine the edge details. In PAPnet, dilated convolution is used to obtain feature information from the receptive field, and different dilation rates are set to extract features.

[0069] The PAPnet encoding and decoding process only includes two upsampling and downsampling processes. This is because the first-level segmentation network has already obtained a coarse segmentation result. PAPnet's role is to analyze edge characteristics, requiring edge features under different receptive fields. Therefore, the number of upsampling and downsampling operations cannot be too high. To ensure feature extraction and analysis capabilities, two additional convolutional layers are added after upsampling to analyze edge characteristics.

[0070] To verify the segmentation performance of the segmentation network of this invention, it was compared with other algorithms on a public dataset. The experiment used a Linux system and a Titan V hardware platform with 12GB of GPU memory. The IOU scores are shown in Table 1.

[0071] Table 1

[0072] Algorithm IOU score Algorithm of the present patent 0.963 Deeplabv3+ 0.943 PSPnet 0.891 u-net 0.832

[0073] As can be seen from Table 1, the method provided by this invention has more accurate segmentation precision.

[0074] The dilated convolution-based liver tumor segmentation device provided by the present invention will be described below. The dilated convolution-based liver tumor segmentation device described below can be referred to in correspondence with the dilated convolution-based liver tumor segmentation method described above.

[0075] Figure 2 A schematic diagram of a liver tumor segmentation device based on dilated convolution provided by the present invention is shown below. Figure 2 As shown, the device includes:

[0076] The acquisition unit 201 is used to acquire liver images of different degrees of lesion.

[0077] The segmentation unit 202 is used to input the liver image into a first-level segmentation network to obtain a coarse segmentation image of the liver image and an edge extraction result image of the entire image. The coarse segmentation image contains tumor distribution information.

[0078] The feature extraction unit 203 is configured to input the rough segmentation image, the edge extraction result image of the whole image, and the liver image to be segmented into a secondary segmentation network, extract features from the rough segmentation image and the edge extraction result image of the whole image by performing convolution processing on the rough segmentation image and the edge extraction result image of the whole image respectively through the secondary segmentation network, and extract features from the liver image to be segmented by performing hole convolution processing on the liver image to be segmented through the secondary segmentation network.

[0079] The fusion unit 204 is configured to fuse all the extracted features.

[0080] The analysis unit 205 is configured to analyze the fused features to obtain a final segmentation result image of the liver tumor image.

[0081] Figure 3 An example of an entity structure diagram of an electronic device is shown in Figure 3 As shown in the figure, the electronic device can include a processor 310, a communications interface 320, a memory 330, and a communications bus 340, wherein the processor 310, the communications interface 320, and the memory 330 can communicate with each other through the communications bus 340. The processor 310 can invoke a logical instruction in the memory 330 to execute a liver tumor segmentation method based on hole convolution, which includes:

[0082] Obtaining liver images of different lesion degrees;

[0083] Inputting the liver images into a primary segmentation network to obtain a rough segmentation image of the liver images and an edge extraction result image of the whole image, wherein the rough segmentation image contains tumor distribution information;

[0084] Inputting the rough segmentation image, the edge extraction result image of the whole image, and the liver image to be segmented into a secondary segmentation network, extracting features from the rough segmentation image and the edge extraction result image of the whole image by performing convolution processing on the rough segmentation image and the edge extraction result image of the whole image respectively through the secondary segmentation network, and extracting features from the liver image to be segmented by performing hole convolution processing on the liver image to be segmented through the secondary segmentation network;

[0085] Fusing all the extracted features;

[0086] Analyzing the fused features to obtain a final segmentation result image of the liver tumor image.

[0087] In addition, the logic instructions in the memory 330 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0088] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the liver tumor segmentation method based on the cavity convolution provided by the above-mentioned methods.

[0089] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the liver tumor segmentation method based on the cavity convolution provided by the above-mentioned methods.

[0090] The device embodiments described above are only schematic, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0091] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary universal hardware platforms, and of course, can also be realized by hardware. Based on such understanding, the above technical solutions essentially or the parts that contribute to the prior art can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0092] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features therein can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A liver tumor segmentation method based on dilated convolution, characterized in that, include: Obtain liver images of different degrees of lesion; The liver image is input into a first-level segmentation network to obtain a coarse segmentation image of the liver image and an edge extraction result image of the entire image. The coarse segmentation image contains tumor distribution information. The coarse segmented image, the edge extraction result image of the whole image, and the liver image to be segmented are input into a two-level segmentation network. The two-level segmentation network performs convolution processing on the coarse segmented image and the edge extraction result image of the whole image to extract features, and performs dilated convolution processing on the liver image to be segmented to extract features. All extracted features are fused; The fusion features were analyzed to obtain the final segmentation result of the liver tumor image.

2. The liver tumor segmentation method based on dilated convolution according to claim 1, characterized in that, In the first-level segmentation network, the loss function corresponding to the coarse segmentation of the image is: Among them, I p I represents the segmentation result output by the DeepLab neural network. t The label represents the corresponding label, Loss represents the loss of coarse segmentation of the image during the training of the first-level segmentation network, i and j represent the coordinates of the pixels in the image, and β is a constant close to zero.

3. The liver tumor segmentation method based on dilated convolution according to claim 1, characterized in that, The two-level segmentation network adopts an encoder-decoder structure.

4. The liver tumor segmentation method based on dilated convolution according to claim 1, characterized in that, The final segmentation result image output by the two-level segmentation network is represented by the following formula: I=aF–(1-a)B (4) Where I represents the final segmentation result image, a represents the new segmentation result to be fused, F is the generated foreground image, and B represents the data to be enhanced.

5. The liver tumor segmentation method based on dilated convolution according to claim 3, characterized in that, The encoder of the two-level segmentation network includes two upsampling and downsampling processes.

6. The liver tumor segmentation method based on dilated convolution according to claim 1, characterized in that, In the first-level segmentation network, the loss function corresponding to the edge extraction result map of the entire image is: Among them, I pr I represents the edge information of the generated image. pt The label represents the edge information, and (i, j) represents the position of the pixel in the image. Loss edge This represents the loss of the edge extraction part during the training of the first-level segmentation network.

7. The liver tumor segmentation method based on dilated convolution according to claim 1, characterized in that, The step of extracting features by performing dilated convolution processing on the liver image to be segmented through the secondary segmentation network includes: extracting features by setting different dilation rates.

8. A liver tumor segmentation device based on dilated convolution, characterized in that, include: The acquisition unit is used to acquire liver images of different degrees of lesion. The segmentation unit is used to input the liver image into a first-level segmentation network to obtain a coarse segmentation image of the liver image and an edge extraction result image of the entire image. The coarse segmentation image contains tumor distribution information. The feature extraction unit is used to input the coarse segmented image, the edge extraction result image of the whole image, and the liver image to be segmented into a two-level segmentation network. The two-level segmentation network performs convolution processing on the coarse segmented image and the edge extraction result image of the whole image to extract features, and performs dilated convolution processing on the liver image to be segmented to extract features. The fusion unit is used to fuse all the extracted features; The analysis unit is used to analyze the fusion features and obtain the final segmentation result of the liver tumor image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the liver tumor segmentation method based on dilated convolution as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the liver tumor segmentation method based on dilated convolution as described in any one of claims 1 to 7.

Images