Image Segmentation Method, Apparatus, Electronic Device and Readable Medium

Abstract

The present disclosure provides an image segmentation method, apparatus, electronic device, and readable medium. The image segmentation method includes: performing a chunking process on the image to be segmented to obtain chunked images; classifying and labeling the chunked images according to the proportion of the target image in the chunked images; inputting the chunked images after classification and labeling into a trained convolutional neural network to obtain a segmented image; and guiding a machine learning algorithm with prior knowledge to perform edge correction on the image output by the convolutional neural network to obtain a corrected segmented image. Through the embodiments of the present disclosure, the accuracy, reliability, and image processing speed of obtaining the segmented image are improved.

Description

Background Art

[0002] Currently, lesion analysis and organ contour detection in medical images require accurate segmentation of the target image to facilitate doctors to make accurate judgments.

[0003] In related technologies, machine vision is widely used in medical image processing scenarios. Machine vision products convert the captured target into an image signal and transmit it to a dedicated image processing system to obtain the morphological information of the captured target. The information is then converted into a digital signal based on pixel distribution, brightness, color and other information.

[0004] However, since the colors between the target area and the background area in most medical images are not resolved, and there are often problems such as unclear edges, discontinuities, and missing edges, the image segmentation of medical images has poor reliability, low accuracy, and slow speed.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0006] The present disclosure aims to provide an image segmentation method, apparatus, electronic device, and readable medium, which are used to at least to some extent overcome the problem of poor image segmentation accuracy caused by limitations and defects of related technologies.

[0007] According to a first aspect of an embodiment of the present disclosure, there is provided an image segmentation method, comprising: performing block processing on an image to be segmented to obtain block images; classifying and labeling the block images according to the proportion of a target image in the block images; inputting the classified and labeled block images into a trained convolutional neural network to obtain a segmented image; and performing edge correction on an image output by the convolutional neural network using a machine learning algorithm guided by prior knowledge to obtain a corrected segmented image.

[0008] In an exemplary embodiment of the present disclosure, the convolutional neural network includes a feature extraction layer and a deconvolution layer connected in sequence, the feature extraction layer includes an initial convolution layer, a maximum pooling layer and multiple convolution layers connected in sequence, the feature extraction layer is used to extract the feature vector of the block image, and the deconvolution layer is used to upsample the feature vector to obtain the feature matrix of the block image.

[0009] In an exemplary embodiment of the present disclosure, the convolution kernel size of the initial convolution layer is 7×7, the sampling interval of the initial convolution layer is 2, and the number of channels of the initial convolution layer is 64.

[0010] In an exemplary embodiment of the present disclosure, the sampling interval of the max pooling layer is 2.

[0011] In an exemplary embodiment of the present disclosure, the feature extraction layer includes a max pooling layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, and a fourth convolutional layer. The output result of the max pooling layer is input to the first convolutional layer. The first convolutional layer includes two bottleneck layers. The convolutional kernel size of the first convolutional layer is 3×3. The sampling interval of the first convolutional layer is 1. The number of channels of the first convolutional layer is 64. The output result of the first convolutional layer is input to the second convolutional layer. The second convolutional layer includes two bottleneck layers. The convolutional kernel size of the second convolutional layer is 3×3. The sampling interval of the second convolutional layer is 2. The number of channels of the second convolutional layer is 128. The output result of the second convolutional layer is input to the third convolutional layer. The third convolutional layer includes two bottleneck layers. The convolutional kernel size of the third convolutional layer is 3×3. The sampling interval of the third convolutional layer is 1. The number of channels of the third convolutional layer is 256. The output result of the third convolutional layer is input to the fourth convolutional layer. The fourth convolutional layer includes two bottleneck layers. The convolutional kernel size of the fourth convolutional layer is 3×3. The sampling interval of the fourth convolutional layer is 2. The number of channels of the fourth convolutional layer is 512.

[0012] In an exemplary embodiment of the present disclosure, the transposed convolution layer includes a first transposed convolution layer, a second transposed convolution layer, a third transposed convolution layer, a fourth transposed convolution layer, and a fifth transposed convolution layer. The output result of the feature extraction layer is input to the first transposed convolution layer. The first transposed convolution layer includes two bottleneck layers. The convolution kernel size of the first transposed convolution layer is 3×3. The sampling interval of the first transposed convolution layer is 2. The number of channels of the first transposed convolution layer is 256. The output result of the first transposed convolution layer is input to the second transposed convolution layer. The second transposed convolution layer includes two bottleneck layers. The convolution kernel size of the second transposed convolution layer is 3×3. The sampling interval of the second transposed convolution layer is 2. The number of channels of the second transposed convolution layer is 128. The output result of the second transposed convolution layer is input to the third transposed convolution layer. The third transposed convolution layer includes two bottleneck layers. The convolution kernel size of the third transposed convolution layer is 3×3. The sampling interval of the third transposed convolution layer is 2. The number of channels of the third transposed convolution layer is 64. The output result of the third transposed convolution layer is input to the fourth transposed convolution layer. The fourth transposed convolution layer includes two bottleneck layers. The convolution kernel size of the fourth transposed convolution layer is 3×3. The sampling interval of the fourth transposed convolution layer is 2. The number of channels of the third transposed convolution layer is 64. The output result of the fourth transposed convolution layer is input to the fifth transposed convolution layer. The fifth transposed convolution layer includes two bottleneck layers. The convolution kernel size of the fifth transposed convolution layer is 3×3. The sampling interval of the fifth transposed convolution layer is 2. The number of channels of the fifth transposed convolution layer is 1.

[0013] In an exemplary embodiment of the present disclosure, before performing the block processing on the image to be segmented, it further includes:

[0014] Inputting the feature matrix sample of the classified and labeled block image into the fully connected layer of the convolutional neural network to obtain the feature vector sample output by the fully connected layer of the convolutional neural network;

[0015] Constructing a triplet loss function according to the inter-class distance of the feature vector samples output by the fully connected layer.

[0016] In an exemplary embodiment of the present disclosure, before performing the block processing on the image to be segmented, it further includes: constructing a multi-layer perceptron for each of the transposed convolution layers. The multi-layer perceptron includes an input layer, a hidden layer, and an output layer connected in sequence. The feature vector sample of the block image is input to the input layer. The hidden layer is used to complete the deep extraction of the feature vector sample. The output layer is used to perform the classification of the feature vector sample after the deep extraction.

[0017] In an exemplary embodiment of the present disclosure, before performing the block processing on the image to be segmented, it further includes: performing a tiling operation on the feature matrix sample output by the deconvolution layer to obtain a two-dimensional vector sample; inputting the two-dimensional vector sample into the input layer of the multi-layer perceptron, and the multi-layer perceptron outputs a classification result of the two-dimensional vector sample; determining a classification loss function corresponding to the classification result through a cross-entropy algorithm.

[0018] In an exemplary embodiment of the present disclosure, before performing the block processing on the image to be segmented, it further includes: inputting a block image sample with classification annotations into a convolutional neural network; determining a regression loss function according to the output result sample of the convolutional neural network and the block image sample.

[0019] According to a second aspect of the embodiments of the present disclosure, there is provided an image segmentation apparatus, including: a block module configured to perform block processing on an image to be segmented to obtain a block image; a labeling module configured to perform classification labeling on the block image according to the proportion of the target image in the block image; a neural network module configured to input the block image after classification labeling into a trained convolutional neural network to obtain a segmented image; a correction module configured to perform edge correction on the image output by the convolutional neural network through a machine learning algorithm guided by prior knowledge to obtain a corrected segmented image.

[0020] According to a third aspect of the present disclosure, there is provided an electronic device, including: a memory; and a processor coupled to the memory, the processor being configured to execute the method according to any one of the above based on instructions stored in the memory.

[0021] According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, it implements the image segmentation method according to any one of the above.

[0022] In the embodiments of the present disclosure, the convolutional neural network is used to identify the block image with classification annotations to obtain a segmented image, and further, a machine learning algorithm is combined to perform edge correction on the image output by the neural network to obtain a corrected segmented image, thereby improving the accuracy, reliability, and image processing speed of obtaining the segmented image.

[0023] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure. Obviously, the accompanying drawings in the following description are only some embodiments of the present disclosure, and for those of ordinary skill in the art, other accompanying drawings can be obtained based on these drawings without creative efforts.

[0025] Figure 1 Shows a flowchart of an image segmentation method in an exemplary embodiment of the present disclosure;

[0026] Figure 2 Shows a flowchart of an image segmentation method in another exemplary embodiment of the present disclosure;

[0027] Figure 3 Shows a flowchart of an image segmentation method in another exemplary embodiment of the present disclosure;

[0028] Figure 4 Shows a flowchart of an image segmentation method in another exemplary embodiment of the present disclosure;

[0029] Figure 5 Shows a flowchart of an image segmentation method in another exemplary embodiment of the present disclosure;

[0030] Figure 6 Shows a schematic diagram of a convolutional layer in an image segmentation scheme in an exemplary embodiment of the present disclosure;

[0031] Figure 7 Shows a schematic diagram of a fully connected layer in an image segmentation scheme in another exemplary embodiment of the present disclosure;

[0032] Figure 8 Shows a schematic diagram of an mlp layer in an image segmentation scheme in another exemplary embodiment of the present disclosure;

[0033] Figure 9 Shows a schematic diagram of an improved unet network in an image segmentation scheme in another exemplary embodiment of the present disclosure;

[0034] Figure 10 Shows a schematic diagram of an image processed by an image segmentation scheme in an exemplary embodiment of the present disclosure;

[0035] Figure 11 Shows a schematic diagram of an image processed by an image segmentation scheme in another exemplary embodiment of the present disclosure;

[0036] Figure 12 Shows a schematic diagram of an image processed by an image segmentation scheme in another exemplary embodiment of the present disclosure;

[0037] Figure 13 A schematic diagram of an image processed by an image segmentation scheme in another exemplary embodiment of the present disclosure is shown;

[0038] Figure 14 A block diagram of an image segmentation device in an exemplary embodiment of the present disclosure is shown;

[0039] Figure 15 A block diagram of an electronic device in an exemplary embodiment of the present disclosure is shown. Detailed implementation manners

[0040] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the embodiments of the present disclosure. However, those skilled in the art will realize that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be employed in other cases. Well-known technical solutions are not shown or described in detail to avoid obscuring aspects of the present disclosure.

[0041] In addition, the accompanying drawings are only schematic illustrations of the present disclosure, and the same reference numerals in the drawings denote the same or similar parts, so repeated descriptions thereof will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0042] The example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0043] Figure 1 is a flowchart of an image segmentation method in an exemplary embodiment of the present disclosure.

[0044] Referring to Figure 1 , the image segmentation method may include:

[0045] Step S102, performing a blocking process on the image to be segmented to obtain a blocked image.

[0046] Step S104, classifying and labeling the blocked image according to the proportion of the target image in the blocked image.

[0047] Step S106: Input the classified and labeled segmented images into the trained convolutional neural network to obtain the segmented image.

[0048] Step S108: Use prior knowledge to guide the machine learning algorithm to perform edge correction on the image output by the convolutional neural network to obtain the corrected segmented image.

[0049] In the embodiment of the present disclosure, the convolutional neural network is used to identify the classified and labeled segmented images to obtain the segmented image, and further, the machine learning algorithm is combined to perform edge correction on the image output by the neural network to obtain the corrected segmented image, improving the accuracy, reliability, and image processing speed of obtaining the segmented image.

[0050] In an exemplary embodiment of the present disclosure, the machine learning algorithm can adopt the watershed algorithm, use the watershed algorithm based on the labeled image, and use prior knowledge to guide the watershed algorithm to obtain a better image segmentation effect.

[0051] In an exemplary embodiment of the present disclosure, the segmented image is denoted as image, and the segmented image obtained after being processed by the convolutional neural network is denoted as segImage (this image is a binary image). Perform a morphological erosion operation on segImage to obtain the binary image segImage_erode. segImage_eorde is the labeled image in the watershed algorithm. After correcting the segmented image through the watershed algorithm, the accurately segmented image is output.

[0052] The following will elaborate on each step of the image segmentation method in detail.

[0053] In an exemplary embodiment of the present disclosure, the convolutional neural network includes a feature extraction layer and a deconvolution layer connected in sequence. The feature extraction layer includes an initial convolutional layer, a max pooling layer, and multiple convolutional layers connected in sequence. The feature extraction layer is used to extract the feature vectors of the segmented images, and the deconvolution layer is used to perform upsampling processing on the feature vectors to obtain the feature matrix of the segmented images.

[0054] In an exemplary embodiment of the present disclosure, the convolutional kernel size of the initial convolutional layer is 7×7, the sampling interval of the initial convolutional layer is 2, and the number of channels of the initial convolutional layer is 64.

[0055] In an exemplary embodiment of the present disclosure, the sampling interval of the max pooling layer is 2.

[0056] In an exemplary embodiment of the present disclosure, the feature extraction layer includes a max pooling layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer. The output result of the max pooling layer is input to the first convolutional layer. The first convolutional layer includes two bottleneck layers, the convolutional kernel size of the first convolutional layer is 3×3, the sampling interval of the first convolutional layer is 1, the number of channels of the first convolutional layer is 64. The output result of the first convolutional layer is input to the second convolutional layer. The second convolutional layer includes two bottleneck layers, the convolutional kernel size of the second convolutional layer is 3×3, the sampling interval of the second convolutional layer is 2, the number of channels of the second convolutional layer is 128. The output result of the second convolutional layer is input to the third convolutional layer. The third convolutional layer includes two bottleneck layers, the convolutional kernel size of the third convolutional layer is 3×3, the sampling interval of the third convolutional layer is 1, the number of channels of the third convolutional layer is 256. The output result of the third convolutional layer is input to the fourth convolutional layer. The fourth convolutional layer includes two bottleneck layers, the convolutional kernel size of the fourth convolutional layer is 3×3, the sampling interval of the fourth convolutional layer is 2, the number of channels of the fourth convolutional layer is 512.

[0057] In an exemplary embodiment of the present disclosure, the transposed convolution layer includes a first transposed convolution layer, a second transposed convolution layer, a third transposed convolution layer, a fourth transposed convolution layer, and a fifth convolution layer. The output result of the feature extraction layer is input into the first transposed convolution layer. The first transposed convolution layer includes two bottleneck layers. The convolution kernel size of the first transposed convolution layer is 3×3. The sampling interval of the first transposed convolution layer is 2. The number of channels of the first transposed convolution layer is 256. The output result of the first transposed convolution layer is input into the second transposed convolution layer. The second transposed convolution layer includes two bottleneck layers. The convolution kernel size of the second transposed convolution layer is 3×3. The sampling interval of the second transposed convolution layer is 2. The number of channels of the second transposed convolution layer is 128. The output result of the second transposed convolution layer is input into the third transposed convolution layer. The third transposed convolution layer includes two bottleneck layers. The convolution kernel size of the third transposed convolution layer is 3×3. The sampling interval of the third transposed convolution layer is 2. The number of channels of the third transposed convolution layer is 64. The output result of the third transposed convolution layer is input into the fourth transposed convolution layer. The fourth transposed convolution layer includes two bottleneck layers. The convolution kernel size of the fourth transposed convolution layer is 3×3. The sampling interval of the fourth transposed convolution layer is 2. The number of channels of the third transposed convolution layer is 64. The output result of the fourth transposed convolution layer is input into the fifth transposed convolution layer. The fifth transposed convolution layer includes two bottleneck layers. The convolution kernel size of the fifth transposed convolution layer is 3×3. The sampling interval of the fifth transposed convolution layer is 2. The number of channels of the fifth transposed convolution layer is 1.

[0058] In an exemplary embodiment of the present disclosure, as Figure 6 shown, in the bottleneck layer 600, NC represents the number of channels of the data, relu is the activation function, and 3x3 is the convolution kernel size.

[0059] In an exemplary embodiment of the present disclosure, as Figure 2 shown, before performing the block processing on the image to be segmented, it further includes:

[0060] Step S202: Input the feature matrix sample of the classified and labeled block image into the fully connected layer of the convolutional neural network to obtain the feature vector sample output by the fully connected layer of the convolutional neural network.

[0061] Step S204: Construct a triplet loss function according to the inter-class distance of the feature vector sample output by the fully connected layer.

[0062] In an exemplary embodiment of the present disclosure, as Figure 3 shown, before performing the block processing on the image to be segmented, it further includes:

[0063] Step S302: Construct a multi-layer perceptron for each of the deconvolution layers. The multi-layer perceptron includes an input layer, a hidden layer, and an output layer connected in sequence. The feature vector samples of the segmented images are input into the input layer. The hidden layer is used to perform deep extraction of the feature vector samples, and the output layer is used to classify the feature vector samples after the deep extraction.

[0064] In an exemplary embodiment of the present disclosure, as Figure 8 shown, the multi-layer perceptron (Multilayer Perceptron, abbreviated as mlp) module. The circles represent individual neurons. Each neuron in the input layer 802 is connected to each neuron in the hidden layer 804, and each neuron in the hidden layer 804 is connected to each neuron in the output layer 806. That is, each neuron in adjacent layers influences each other. The hidden layer 804 completes the deep extraction of features, and the output layer 806 completes the classification of categories.

[0065] In an exemplary embodiment of the present disclosure, as Figure 4 shown, before performing the segmentation processing on the image to be segmented, it further includes:

[0066] Step S402: Perform a tiling operation on the feature matrix samples output by the deconvolution layer to obtain two-dimensional vector samples.

[0067] Step S404: Input the two-dimensional vector samples into the input layer of the multi-layer perceptron, and the multi-layer perceptron outputs the classification results of the two-dimensional vector samples.

[0068] Step S406: Determine the classification loss function corresponding to the classification results through the cross-entropy algorithm.

[0069] In an exemplary embodiment of the present disclosure, as Figure 5 shown, before performing the segmentation processing on the image to be segmented, it further includes:

[0070] Step S502: Input the segmented image samples with classification annotations into the convolutional neural network.

[0071] Step S504: Determine the regression loss function according to the output result samples of the convolutional neural network and the segmented image samples.

[0072] In an exemplary embodiment of the present disclosure, as Figure 6 、 Figure 7 、 Figure 8 and Figure 9As shown in the figure, during the decoding process, the first feature matrix D1 700 is passed through a fully connected layer (FC) module to obtain a normalized feature vector V1. Similarly, the second feature matrix D2, the third feature matrix D3, and the fourth feature matrix D4 are respectively passed through a fully connected layer to output their corresponding second feature vectors V2, V3, and V4. "flatten" represents flattening the feature matrix, "batchnorm" represents batch normalization, "Fully Connected" represents a fully connected layer, and "normalization" represents normalization of a single feature vector.

[0073] In an exemplary embodiment of the present disclosure, Figure 9 As shown in the figure, the convolutional neural network adopts the improved unet network, that is, a fully connected layer FC is added to the deconvolution layer of the unet network, and the deconvolution layer is constructed as shown in the figure. Figure 8 The multi-layer perceptron mlp shown.

[0074] like Figure 9 As shown, the network input scales the fixed-size image block 900 from a resolution of 1024×1024 to a resolution of 224×224. The process of training the convolutional neural network includes:

[0075] (1) Network input: RGB (red, green, and blue) images with a size of 224x224.

[0076] (2) Extract features of different scales through convolutional neural networks. The specific structure is as follows:

[0077] (2.1) 1 convolutional layer, the input is a 224x224x3 image, the convolution kernel size is 7x7, the sampling interval is 2, the number of channels is 64, and the output is recorded as E1.

[0078] (2.2) A maxpool layer, with input E1, sampling interval 2, and output denoted as maxpool

[0079] (2.3) 2 bottlenetck layers, input is maxpool, convolution kernel size is 3×3, sampling interval is 1, number of channels is 64, and output is recorded as E2.

[0080] (2.4) 2 bottlenetck layers, input is E2, convolution kernel size is 3×3, sampling interval is 2, number of channels is 128, and output is recorded as E3.

[0081] (2.5) 2 bottleneck layers, input is E3, convolution kernel size is 3x3, sampling interval is 1, number of channels is 256, and output is recorded as E4.

[0082] (2.6) Two bottleneck layers with an input of E4, a convolution kernel size of 3x3, a sampling interval of 2, 512 channels, and the output denoted as E5.

[0083] (3) The transposed convolution layer of the convolutional neural network has the following specific structure:

[0084] (3.1) Image upsampling, two bottleneck layers with an input of E5, a convolution kernel size of 3x3, a sampling interval of 2, 256 channels, and the output denoted as D1.

[0085] (3.2) Image upsampling, two bottleneck layers with an input of D1, a convolution kernel size of 3x3, a sampling interval of 2, 128 channels, and the output denoted as D2.

[0086] (3.3) Image upsampling, two bottleneck layers with an input of D2, a convolution kernel size of 3x3, a sampling interval of 2, 64 channels, and the output denoted as D3.

[0087] (3.4) Image upsampling, two bottleneck layers with an input of D3, a convolution kernel size of 3x3, a sampling interval of 2, 64 channels, and the output denoted as D4.

[0088] (3.5) Image upsampling, two bottleneck layers with an input of D4, a convolution kernel size of 3x3, a sampling interval of 2, 1 channel, and the output denoted as output.

[0089] As Figure 9 shown, the classification of categories is completed for four feature modules of different scales, which has a certain robustness for the classification of target images of different scales, and is specifically described as follows:

[0090] (1) Construct the first mlp module, flatten the feature matrix D1 to obtain a two-dimensional vector VF1. Use VF1 as the input of the first mlp module, and the output is the number of classification categories. Use CrossEntropy (cross entropy) as the loss function, and this classifier is denoted as CLS1.

[0091] (2) Construct the second mlp module, flatten the feature matrix D2 to obtain a two-dimensional vector VF2. Use VF2 as the input of the second mlp module, and the output is the number of classification categories. Use CrossEntropy (cross entropy) as the loss function, and this classifier is denoted as CLS2.

[0092] (3) Construct the third MLP module and flatten the feature matrix D3 to obtain a two-dimensional vector VF3. VF3 is used as the input of the third MLP module, and the output is the number of classifications. CrossEntropy is used as the loss function, and the classifier is denoted as CLS3.

[0093] (4) Construct the fourth MLP module and perform a flattening operation on the feature matrix D4 to obtain a two-dimensional vector VF4. VF4 is used as the input of the fourth MLP module, and the output is the number of classifications. CrossEntropy is used as the loss function, and the classifier is denoted as CLS4.

[0094] Based on the improved structure of the above convolutional neural network, the output vector of the image with label 1 is recorded as V1, the output vector of the image with label 0 is recorded as V0, and the feature vector of the reference sample is V anchor (The reference sample is the output vector with label 1). The principle of ternary loss is to maximize the distance between classes and minimize the distance within classes. The calculation method of triplet loss is shown in formula (1):

[0095] , formula (1).

[0096] in, Indicates the feature vector with label 1, Indicates the feature vector whose label is 0.

[0097] For the feature vectors obtained in (1), the corresponding ternary losses are calculated respectively, and the average of the ternary loss values corresponding to the four feature vectors is used as the final ternary loss value. The calculation method of the ternary loss value AveTri Loss is shown in the following formula (2):

[0098] , formula (2).

[0099] Based on the improved structure of the above convolutional neural network, the calculation method of classification loss is shown in formula (3):

[0100] , formula (3).

[0101] like Figure 10 The glomerular cross-sectional image 1000 shown includes a glomerular region 1002 , a non-glomerular region 1004 , and a background region 1006 of the medical image.

[0102] like Figure 11 and ​As shown, the glomerular section diagram 1000 is segmented to obtain a non-glomerular section segmented image 1100 and a glomerular section segmented image 1200. The glomerular section segmented image 1200 shows a glomerular region 1202.

[0103] As ​ shown, the glomerular region 1302 of the glomerular section segmented image 1300 is labeled, and the unlabeled region 1304 is not labeled, resulting in a labeled image 1300.

[0104] Based on the improved structure of the above convolutional neural network, denoting output as the network output and mask as the labeled image 1300 (binary image), the calculation method of the regression loss regression Loss is shown in Formula (IV):

[0105] , Formula (IV).

[0106] According to the triplet loss, classification loss, and regression loss, the total loss of the convolutional neural network is obtained. The calculation method of the total loss Loss is shown in Formula (V):

[0107] , Formula (V).

[0108] Among them, Wt is the triplet loss weight, Wc is the classification error weight, and Wr is the regression loss weight.

[0109] Corresponding to the above method embodiments, the present disclosure also provides an image segmentation device, which can be used to execute the above method embodiments.

[0110] ​ is a block diagram of an image segmentation device in an exemplary embodiment of the present disclosure.

[0111] Referring to ​ , the image segmentation device 1400 may include:

[0112] A segmentation module 1402, configured to segment the image to be segmented to obtain a segmented image.

[0113] A labeling module 1404, configured to classify and label the segmented image according to the proportion of the target image in the segmented image.

[0114] A neural network module 1406, configured to input the segmented image after classification and labeling into a trained convolutional neural network to obtain a segmented image.

[0115] A correction module 1408, configured to perform edge correction on the image output by the convolutional neural network through a machine learning algorithm guided by prior knowledge to obtain a corrected segmented image.

[0116] In an exemplary embodiment of the present disclosure, the convolutional neural network includes a feature extraction layer and a deconvolution layer connected in sequence. The feature extraction layer includes an initial convolutional layer, a max pooling layer, and multiple convolutional layers connected in sequence. The feature extraction layer is used to extract the feature vectors of the segmented images, and the deconvolution layer is used to perform upsampling processing on the feature vectors to obtain the feature matrices of the segmented images.

[0117] In an exemplary embodiment of the present disclosure, the convolutional kernel size of the initial convolutional layer is 14×14, the sampling interval of the initial convolutional layer is 2, and the number of channels of the initial convolutional layer is 64.

[0118] In an exemplary embodiment of the present disclosure, the sampling interval of the max pooling layer is 2.

[0119] In an exemplary embodiment of the present disclosure, the feature extraction layer includes a max pooling layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer. The output result of the max pooling layer is input to the first convolutional layer. The first convolutional layer includes two bottleneck layers. The convolutional kernel size of the first convolutional layer is 3×3, the sampling interval of the first convolutional layer is 1, and the number of channels of the first convolutional layer is 64. The output result of the first convolutional layer is input to the second convolutional layer. The second convolutional layer includes two bottleneck layers. The convolutional kernel size of the second convolutional layer is 3×3, the sampling interval of the second convolutional layer is 2, and the number of channels of the second convolutional layer is 128. The output result of the second convolutional layer is input to the third convolutional layer. The third convolutional layer includes two bottleneck layers. The convolutional kernel size of the third convolutional layer is 3×3, the sampling interval of the third convolutional layer is 1, and the number of channels of the third convolutional layer is 256. The output result of the third convolutional layer is input to the fourth convolutional layer. The fourth convolutional layer includes two bottleneck layers. The convolutional kernel size of the fourth convolutional layer is 3×3, the sampling interval of the fourth convolutional layer is 2, and the number of channels of the fourth convolutional layer is 512.

[0120] In an exemplary embodiment of the present disclosure, the transposed convolution layer includes a first transposed convolution layer, a second transposed convolution layer, a third transposed convolution layer, a fourth transposed convolution layer, and a fifth convolution layer. The output result of the feature extraction layer is input to the first transposed convolution layer. The first transposed convolution layer includes two bottleneck layers. The convolution kernel size of the first transposed convolution layer is 3×3. The sampling interval of the first transposed convolution layer is 2. The number of channels of the first transposed convolution layer is 256. The output result of the first transposed convolution layer is input to the second transposed convolution layer. The second transposed convolution layer includes two bottleneck layers. The convolution kernel size of the second transposed convolution layer is 3×3. The sampling interval of the second transposed convolution layer is 2. The number of channels of the second transposed convolution layer is 128. The output result of the second transposed convolution layer is input to the third transposed convolution layer. The third transposed convolution layer includes two bottleneck layers. The convolution kernel size of the third transposed convolution layer is 3×3. The sampling interval of the third transposed convolution layer is 2. The number of channels of the third transposed convolution layer is 64. The output result of the third transposed convolution layer is input to the fourth transposed convolution layer. The fourth transposed convolution layer includes two bottleneck layers. The convolution kernel size of the fourth transposed convolution layer is 3×3. The sampling interval of the fourth transposed convolution layer is 2. The number of channels of the third transposed convolution layer is 64. The output result of the fourth transposed convolution layer is input to the fifth transposed convolution layer. The fifth transposed convolution layer includes two bottleneck layers. The convolution kernel size of the fifth transposed convolution layer is 3×3. The sampling interval of the fifth transposed convolution layer is 2. The number of channels of the fifth transposed convolution layer is 1.

[0121] In an exemplary embodiment of the present disclosure, the neural network module 1406 is further configured to: input the feature matrix sample of the segmented image with classification labels into the fully connected layer of the convolutional neural network to obtain the feature vector sample output by the fully connected layer of the convolutional neural network; construct a triplet loss function according to the inter-class distance of the feature vector samples output by the fully connected layer.

[0122] In an exemplary embodiment of the present disclosure, the neural network module 1406 is further configured to: construct a multi-layer perceptron for each of the transposed convolution layers. The multi-layer perceptron includes an input layer, a hidden layer, and an output layer connected in sequence. The feature vector sample of the segmented image is input to the input layer. The hidden layer is used to complete the deep extraction of the feature vector sample. The output layer is used to perform the classification of the feature vector sample after the deep extraction.

[0123] In an exemplary embodiment of the present disclosure, the neural network module 1406 is also used to: perform a tiling operation on the feature matrix samples output by the deconvolution layer to obtain two-dimensional vector samples; input the two-dimensional vector samples into the input layer of the multi-layer perceptron, and the multi-layer perceptron outputs the classification results of the two-dimensional vector samples; and determine the classification loss function corresponding to the classification results through a cross-entropy algorithm.

[0124] In an exemplary embodiment of the present disclosure, the neural network module 1406 is further used to: input the classified and labeled block image samples into the convolutional neural network; and determine the regression loss function based on the output result samples of the convolutional neural network and the block image samples.

[0125] Since the functions of the apparatus 1400 have been described in detail in the corresponding method embodiments, they will not be described in detail herein.

[0126] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0127] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0128] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Therefore, various aspects of the present invention may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, microcode, etc.), or in a combination of hardware and software, collectively referred to herein as "circuits," "modules," or "systems."

[0129] Refer to the following ​ An electronic device 1500 according to this embodiment of the present invention will be described. ​ The electronic device 1500 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0130] like ​ As shown, electronic device 1500 is implemented as a general-purpose computing device. Components of electronic device 1500 may include, but are not limited to, the aforementioned at least one processing unit 1510, the aforementioned at least one storage unit 1520, and a bus 1530 connecting various system components (including storage unit 1520 and processing unit 1510).

[0131] Among them, the storage unit stores program code, and the program code can be executed by the processing unit 1510, so that the processing unit 1510 executes the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of this specification. For example, the processing unit 1510 can execute the method as shown in the embodiments of the present disclosure.

[0132] The storage unit 1520 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 15201 and / or a cache storage unit 15202, and may further include a read-only storage unit (ROM) 15203.

[0133] The storage unit 1520 may also include a program / utilities 15204 having a set (at least one) of program modules 15205. Such program modules 15205 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0134] The bus 1530 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus structures.

[0135] The electronic device 1500 may also communicate with one or more external devices 1540 (such as a keyboard, a pointing device, a Bluetooth device, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1500, and / or may communicate with any device that enables the electronic device 1500 to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication may be carried out through an input / output (I / O) interface 1550. Moreover, the electronic device 1500 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 1560. As shown in the figure, the network adapter 1560 communicates with other modules of the electronic device 1500 through the bus 1530. It should be understood that although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0136] From the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described herein can be implemented by software or by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a portable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0137] In an exemplary embodiment of the present disclosure, there is also provided a computer-readable storage medium having stored thereon a program product capable of implementing the above method of this specification. In some possible implementation manners, various aspects of the present invention can also be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of this specification.

[0138] The program product for implementing the above method according to the embodiments of the present invention may employ a portable compact disc read-only memory (CD-ROM) and include program code, and can run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device.

[0139] The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0140] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which readable program code is carried. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The readable signal medium may also be any readable medium other than a readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0141] The program code contained on the readable medium can be transmitted by any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0142] The program code for performing the operations of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, executed as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or, alternatively, can be connected to an external computing device (e.g., by using an Internet service provider to connect through the Internet).

[0143] In addition, the above-mentioned drawings are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present invention, and are not for limiting purposes. It is easily understood that the processes shown in the above-mentioned drawings do not indicate or limit the chronological order of these processes. Additionally, it is also easily understood that these processes can be executed, for example, synchronously or asynchronously in multiple modules.

[0144] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and embodiments are only to be considered as exemplary, and the true scope and concept of the present disclosure are pointed out by the claims.

Claims

1. An image segmentation method, characterized in that, Including: Inputting the feature matrix samples of the segmented and labeled block images into the fully connected layer of the convolutional neural network to obtain the feature vector samples output by the fully connected layer of the convolutional neural network; Constructing a triplet loss function according to the inter-class distance of the feature vector samples output by the fully connected layer; Constructing a multi-layer perceptron for each deconvolution layer, the multi-layer perceptron including an input layer, a hidden layer and an output layer connected in sequence, the feature vector samples of the block images are input into the input layer, the hidden layer is used to complete the deep extraction of the feature vector samples, and the output layer is used to perform the classification of the feature vector samples after the deep extraction; Performing a tiling operation on the feature matrix samples output by the deconvolution layer to obtain two-dimensional vector samples; Inputting the two-dimensional vector samples into the input layer of the multi-layer perceptron, and the multi-layer perceptron outputs the classification results of the two-dimensional vector samples; Determining the classification loss function corresponding to the classification results through the cross-entropy algorithm; Performing a block processing on the image to be segmented to obtain block images; Classifying and labeling the block images according to the proportion of the target image in the block images; Inputting the classified and labeled block images into the trained convolutional neural network to obtain a segmented image; Performing edge correction on the image output by the convolutional neural network through a machine learning algorithm guided by prior knowledge to obtain a corrected segmented image, The convolutional neural network includes a feature extraction layer and a deconvolution layer connected in sequence, the feature extraction layer includes an initial convolutional layer, a max pooling layer and multiple convolutional layers connected in sequence, the feature extraction layer is used to extract the feature vectors of the block images, and the deconvolution layer is used to perform upsampling processing on the feature vectors to obtain the feature matrices of the block images.

2. The image segmentation method according to claim 1, wherein The convolutional kernel size of the initial convolutional layer is 7×7, the sampling interval of the initial convolutional layer is 2, and the number of channels of the initial convolutional layer is 64.

3. The image segmentation method according to claim 1, wherein, The sampling interval of the max pooling layer is 2.

4. The image segmentation method according to claim 1, wherein, The feature extraction layer includes a max pooling layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, and a fourth convolutional layer. The output result of the max pooling layer is input into the first convolutional layer. The first convolutional layer includes two bottleneck layers, the convolutional kernel size of the first convolutional layer is 3×3, the sampling interval of the first convolutional layer is 1, the number of channels of the first convolutional layer is 64. The output result of the first convolutional layer is input into the second convolutional layer. The second convolutional layer includes two bottleneck layers, the convolutional kernel size of the second convolutional layer is 3×3, the sampling interval of the second convolutional layer is 2, the number of channels of the second convolutional layer is 128. The output result of the second convolutional layer is input into the third convolutional layer. The third convolutional layer includes two bottleneck layers, the convolutional kernel size of the third convolutional layer is 3×3, the sampling interval of the third convolutional layer is 1, the number of channels of the third convolutional layer is 256. The output result of the third convolutional layer is input into the fourth convolutional layer. The fourth convolutional layer includes two bottleneck layers, the convolutional kernel size of the fourth convolutional layer is 3×3, the sampling interval of the fourth convolutional layer is 2, the number of channels of the fourth convolutional layer is 512.

5. The image segmentation method according to claim 1, wherein, The transposed convolution layer includes a first transposed convolutional layer, a second transposed convolutional layer, a third transposed convolutional layer, a fourth transposed convolutional layer, and a fifth transposed convolutional layer. The output result of the feature extraction layer is input into the first transposed convolutional layer. The first transposed convolutional layer includes two bottleneck layers, the convolutional kernel size of the first transposed convolutional layer is 3×3, the sampling interval of the first transposed convolutional layer is 2, the number of channels of the first transposed convolutional layer is 256. The output result of the first transposed convolutional layer is input into the second transposed convolutional layer. The second transposed convolutional layer includes two bottleneck layers, the convolutional kernel size of the second transposed convolutional layer is 3×3, the sampling interval of the second transposed convolutional layer is 2, the number of channels of the second transposed convolutional layer is 128. The output result of the second transposed convolutional layer is input into the third transposed convolutional layer. The third transposed convolutional layer includes two bottleneck layers, the convolutional kernel size of the third transposed convolutional layer is 3×3, the sampling interval of the third transposed convolutional layer is 2, the number of channels of the third transposed convolutional layer is 64. The output result of the third transposed convolutional layer is input into the fourth transposed convolutional layer. The fourth transposed convolutional layer includes two bottleneck layers, the convolutional kernel size of the fourth transposed convolutional layer is 3×3, the sampling interval of the fourth transposed convolutional layer is 2, the number of channels of the fourth transposed convolutional layer is 64. The output result of the fourth transposed convolutional layer is input into the fifth transposed convolutional layer. The fifth transposed convolutional layer includes two bottleneck layers, the convolutional kernel size of the fifth transposed convolutional layer is 3×3, the sampling interval of the fifth transposed convolutional layer is 2, the number of channels of the fifth transposed convolutional layer is 1.

6. The image segmentation method according to any one of claims 1-5, characterized in that, Before performing the block processing on the image to be segmented, it further includes: Input the classified and labeled segmented image samples into a convolutional neural network; Determine a regression loss function based on the output result samples of the convolutional neural network and the segmented image samples.

7. An image segmentation device, characterized in that, It includes: A neural network module configured to input the feature matrix samples of the classified and labeled segmented images into the fully connected layer of the convolutional neural network to obtain the feature vector samples output by the fully connected layer of the convolutional neural network; The neural network module is further configured to construct a triplet loss function according to the inter-class distance of the feature vector samples output by the fully connected layer; The neural network module is further configured to construct a multi-layer perceptron for each deconvolution layer. The multi-layer perceptron includes an input layer, a hidden layer, and an output layer connected in sequence. The feature vector samples of the segmented images are input into the input layer. The hidden layer is used to complete the deep extraction of the feature vector samples, and the output layer is used to perform the classification of the feature vector samples after the deep extraction; The neural network module is further configured to perform a tiling operation on the feature matrix samples output by the deconvolution layer to obtain two-dimensional vector samples; The neural network module is further configured to input the two-dimensional vector samples into the input layer of the multi-layer perceptron, and the multi-layer perceptron outputs the classification results of the two-dimensional vector samples; The neural network module is further configured to determine the classification loss function corresponding to the classification results through a cross-entropy algorithm; A segmentation module configured to perform segmentation processing on the image to be segmented to obtain segmented images; A labeling module configured to classify and label the segmented images according to the proportion of the target image in the segmented images; The neural network module is further configured to input the classified and labeled segmented images into the trained convolutional neural network to obtain a segmented image; A correction module configured to perform edge correction on the image output by the convolutional neural network through a machine learning algorithm guided by prior knowledge to obtain a corrected segmented image, The convolutional neural network includes a feature extraction layer and a deconvolution layer connected in sequence. The feature extraction layer includes an initial convolutional layer, a max pooling layer, and multiple convolutional layers connected in sequence. The feature extraction layer is used to extract the feature vectors of the segmented images, and the deconvolution layer is used to perform upsampling processing on the feature vectors to obtain the feature matrix of the segmented images.

8. An electronic device, characterized in that, It includes: A memory; And A processor coupled to the memory, the processor being configured to execute the image segmentation method according to any one of claims 1-6 based on instructions stored in the memory.

9. A computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, it implements the image segmentation method according to any one of claims 1-6.

Images