Image segmentation method, device, computer equipment and storage medium

Abstract

The present application relates to an image segmentation method, device, computer equipment and storage medium. The method comprises: performing pixel-level segmentation processing on an image to be segmented based on a segmentation network to obtain a segmentation result of the image to be segmented; the segmentation result comprises the segmentation category of each position of interest in the image to be segmented; the image to be segmented is input into a classification network for image-level classification processing to obtain a classification result corresponding to the image to be segmented; the classification result comprises the classification probability that the image to be segmented belongs to each candidate category; and the segmentation result of the image to be segmented is optimized according to the classification probability of belonging to each candidate category in the classification result and the segmentation category in the segmentation result. The use of this method can improve the accuracy of the segmentation result.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image segmentation method, apparatus, computer equipment and storage medium. Background Art

[0002] With the continuous development of medical imaging technology, when patients go to the hospital for physical examinations, doctors usually take medical images of the patients and perform relevant technical processing on the taken medical images to obtain image analysis results.

[0003] Usually when processing medical images of patients, most of them use deep learning segmentation networks to segment the medical images, segment the areas or points of concern to the doctor from the medical images, and then further analyze the areas or points of concern to obtain image analysis results.

[0004] However, when the above technology is used for segmentation processing, the segmentation results obtained may be inaccurate. Summary of the invention

[0005] Based on this, it is necessary to provide an image segmentation method, device, computer equipment and storage medium that can improve the accuracy of segmentation results in response to the above technical problems.

[0006] An image segmentation method, the method comprising:

[0007] Perform pixel-level segmentation processing on the image to be segmented based on the segmentation network to obtain a segmentation result of the image to be segmented; the segmentation result includes a segmentation category for each location of interest in the image to be segmented;

[0008] The image to be segmented is input into the classification network for image-level classification processing to obtain a classification result corresponding to the image to be segmented; the classification result includes the classification probability that the image to be segmented belongs to each candidate category;

[0009] The segmentation result of the image to be segmented is optimized according to the classification probability of each candidate category in the classification result and the segmentation category in the segmentation result.

[0010] In one embodiment, the classification network includes a first classification subnetwork and a second classification subnetwork, the image to be segmented includes a plurality of image slices, and the image to be segmented is input into the classification network for image-level classification processing to obtain a classification result corresponding to the image to be segmented, including:

[0011] Inputting multiple image slices into a first classification subnetwork for spatial classification processing to obtain a first classification probability and an image feature vector of each image slice belonging to each candidate category;

[0012] The image feature vectors of each image slice are input into the second classification subnetwork in time order for time classification processing, so as to obtain the second classification probability that each image slice belongs to each candidate category; the second classification subnetwork has the function of learning the time correlation between each image slice;

[0013] According to the first classification probability and the second classification probability of each image slice, the classification probability of each image slice belonging to each candidate category is obtained, and the classification probability of each image slice belonging to each candidate category is determined as the classification result corresponding to the image to be segmented.

[0014] In one embodiment, the classification network further includes a third classification subnetwork, and the classification probability of each image slice belonging to each candidate category is obtained according to the first classification probability and the second classification probability of each image slice, including:

[0015] In the third classification subnetwork, the first classification probability and the second classification probability of each image slice input are concatenated to obtain a probability union consisting of the first classification probability and the second classification probability of each image slice;

[0016] In the third classification subnetwork, the probability union is fully connected to obtain the classification probability that each image slice belongs to each candidate category; wherein the number of classification probabilities of each image slice is the same as the number of first classification probabilities and the number of second classification probabilities.

[0017] In one embodiment, the segmentation category of each position of interest in the image to be segmented includes the segmentation category of each position of interest in each image slice; and optimizing the segmentation result of the image to be segmented according to the classification probability of belonging to each candidate category in the classification result and the segmentation category in the segmentation result includes:

[0018] Obtain the probability threshold corresponding to each candidate category;

[0019] The classification probability of each image slice belonging to each candidate category is compared with the corresponding probability threshold to obtain the classification category corresponding to each image slice;

[0020] The classification category of each image slice is matched with the segmentation category of each interesting position in the corresponding image slice, and the segmentation result of the image to be segmented is optimized according to the matching result.

[0021] In one embodiment, the above-mentioned optimizing the segmentation result of the image to be segmented according to the matching result includes:

[0022] For the same image slice, if the matching result is that the segmentation category of the position of interest does not exist in the classification category of the image slice, the position of interest in the image slice is removed from the segmentation result.

[0023] In one embodiment, the above-mentioned obtaining the probability threshold corresponding to each candidate category includes:

[0024] Acquire a plurality of test sample images, each of which includes a plurality of test image slices;

[0025] Input each test image slice into the classification network to obtain the classification probability that each test image slice belongs to each candidate category;

[0026] According to the classification probability of each test image slice belonging to each candidate category, multiple classification probabilities corresponding to each candidate category are obtained;

[0027] According to the multiple classification probabilities corresponding to each candidate category, a receiver operating characteristic curve (ROC) curve corresponding to each candidate category is determined;

[0028] According to the curve characteristics of the ROC curve corresponding to each candidate category, the probability threshold corresponding to each candidate category is determined.

[0029] In one embodiment, the training method of the first classification sub-network and the second classification sub-network includes:

[0030] Acquire a plurality of first training images and a plurality of second training images; wherein each first training image includes a first labeled classification probability, and each second training image includes a second labeled classification probability;

[0031] Using a plurality of first training images as inputs of an initial first classification subnetwork, using a first labeled classification probability corresponding to each first training image as a reference output of the initial first classification subnetwork, training the initial first classification subnetwork, and obtaining a first classification subnetwork;

[0032] Based on the plurality of second training images and the first classification sub-network, the initial second classification sub-network is trained to obtain a second classification sub-network.

[0033] In one embodiment, the first classification subnetwork is a convolutional neural network (CNN), and the second classification subnetwork is a temporal network.

[0034] An image segmentation device, the device comprising:

[0035] A segmentation module is used to perform pixel-level segmentation processing on the image to be segmented based on the segmentation network to obtain a segmentation result of the image to be segmented; the segmentation result includes a segmentation category of each location of interest in the image to be segmented;

[0036] The classification module is used to input the image to be segmented into the classification network for image-level classification processing to obtain a classification result corresponding to the image to be segmented; the classification result includes the classification probability that the image to be segmented belongs to each candidate category;

[0037] The optimization module is used to optimize the segmentation result of the image to be segmented according to the classification probability of each candidate category in the classification result and the segmentation category in the segmentation result.

[0038] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0039] Perform pixel-level segmentation processing on the image to be segmented based on the segmentation network to obtain a segmentation result of the image to be segmented; the segmentation result includes a segmentation category for each location of interest in the image to be segmented;

[0040] The image to be segmented is input into the classification network for image-level classification processing to obtain a classification result corresponding to the image to be segmented; the classification result includes the classification probability that the image to be segmented belongs to the candidate category;

[0041] The segmentation result of the image to be segmented is optimized according to the classification probability of the candidate category in the classification result and the segmentation category in the segmentation result.

[0042] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0043] Perform pixel-level segmentation processing on the image to be segmented based on the segmentation network to obtain a segmentation result of the image to be segmented; the segmentation result includes a segmentation category for each location of interest in the image to be segmented;

[0044] The image to be segmented is input into the classification network for image-level classification processing to obtain a classification result corresponding to the image to be segmented; the classification result includes the classification probability that the image to be segmented belongs to the candidate category;

[0045] The segmentation result of the image to be segmented is optimized according to the classification probability of the candidate category in the classification result and the segmentation category in the segmentation result.

[0046] The above-mentioned image segmentation method, apparatus, computer equipment and storage medium perform pixel-level segmentation processing on the image to be segmented based on the segmentation network to obtain the segmentation result of the image to be segmented, input the image to be segmented into the classification network for image-level classification processing, obtain the classification result corresponding to the image to be segmented, and optimize the segmentation result of the image to be segmented according to the classification probability of belonging to the candidate category in the classification result and the segmentation category in the segmentation result; wherein the segmentation result includes the segmentation category of each position of interest in the image to be segmented, and the classification result includes the classification probability that the image to be segmented belongs to the candidate category. In this method, since the image-level classification result can be used to constrain the pixel-level segmentation result, that is, the final image segmentation result is obtained by combining the image-level classification result and the pixel-level segmentation result, the final segmentation result is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is an internal structure diagram of a computer device in one embodiment;

[0048] Figure 2 is a schematic flow chart of an image segmentation method in one embodiment;

[0049] Figure 3 is a flow chart of an image segmentation method in another embodiment;

[0050] Figure 3a A schematic diagram of the structure of a classification network in another embodiment;

[0051] Figure 4 is a flow chart of an image segmentation method in another embodiment;

[0052] Figure 5 FIG. 4 is a structural block diagram of an image segmentation device in an embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] The image segmentation method provided in the present application can be applied to a computer device, which can be a terminal or a server. Taking the computer device as a terminal as an example, its internal structure diagram can be as follows: Figure 1 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image segmentation method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0055] Those skilled in the art will understand that Figure 1The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0056] The execution subject of the embodiment of the present application may be a computer device or an image segmentation device. The following description will be made with the execution subject being a computer device.

[0057] In one embodiment, an image segmentation method is provided. This embodiment relates to a specific process of optimizing the segmentation result after obtaining the image pixel-level segmentation result. Figure 2 As shown, the method may include the following steps:

[0058] S202, performing pixel-level segmentation processing on the image to be segmented based on the segmentation network to obtain a segmentation result of the image to be segmented; the segmentation result includes a segmentation category of each position of interest in the image to be segmented.

[0059] The image to be segmented may be an NCCT image (ie, plain scan CT, plain scan Computed Tomography, plain scan electronic computer tomography), or other images.

[0060] Pixel-level segmentation refers to the process of segmenting an image using a segmentation network, based on whether each pixel on the image belongs to a location of interest. The segmentation network here can be a convolutional neural network, such as a U-net network, a V-net network, etc. It can be a 2D (two-dimensional) segmentation network or a 3D (three-dimensional) segmentation network.

[0061] Before using the segmentation network to segment the image, the segmentation network can also be pre-trained. During training, a sample image set can be first obtained, and the sample image set includes multiple training images, and each pixel on each training image is labeled with a segmentation category; multiple training images can be input into the initial segmentation network, and the labeled segmentation category of each pixel on each training image is used as the reference output (or gold standard) of the initial segmentation network, and the initial segmentation network is trained to obtain the initial segmentation network.

[0062] Taking intracranial hemorrhage as an example, the annotated segmentation category of each pixel here is the category to which each pixel belongs, such as the intracranial hemorrhage category, which may include, for example, intracerebral hemorrhage, subdural hemorrhage, epidural hemorrhage, intraventricular hemorrhage, and subarachnoid hemorrhage.

[0063] Of course, before inputting the training images into the initial segmentation network, each training image may also be preprocessed, for example, a suitable window width and window position may be set for each training image, appropriate data enhancement may be performed (rotating, translating, flipping, etc. the training images), and data normalization or standardization may also be performed on each training image.

[0064] After the segmentation network is trained, the image to be segmented can be input into the trained segmentation network to obtain the segmentation category of each pixel on the image to be segmented, and then the pixels of the same segmentation category are grouped together by dividing the connected domain to form a position of interest. Of course, a single pixel here can also be a position of interest. In addition, the segmentation results can be preliminarily optimized through morphological operations, or the sensitivity of the segmentation results can be increased by using the method of region growing, etc. In short, the segmentation results of the position of interest can be obtained in the end. Here, the image to be segmented can include multiple image slices, so the segmentation results of the position of interest can include the segmentation results of the region of interest on each image slice.

[0065] In addition, the position of interest here refers to a region of interest or a point of interest on the image to be segmented, and of course may also include the position of the region of interest or the position of the point of interest.

[0066] After obtaining the location of interest, the segmentation category of the pixels contained therein can be used as the segmentation category of the location of interest, that is, the segmentation category of the location of interest is obtained. In addition, the segmentation category of the location of interest is just a name distinction here, which is actually the category to which the location of interest belongs. For example, the segmentation category of a location of interest can be any one of the above-mentioned intracerebral hemorrhage, subdural hemorrhage, epidural hemorrhage, intraventricular hemorrhage and subarachnoid hemorrhage.

[0067] S204, inputting the image to be segmented into a classification network for image-level classification processing to obtain a classification result corresponding to the image to be segmented; the classification result includes the classification probability that the image to be segmented belongs to each candidate category.

[0068] In this step, image-level classification processing refers to the process of classifying the entire image, and the classification result obtained is also the probability that the entire image belongs to each candidate category. The classification network here can be a neural network, which can be a 2D (two-dimensional) classification network or a 3D (three-dimensional) classification network. In addition, the classification network here can include one or more classification subnetworks. When multiple classification subnetworks are included, the multiple classification subnetworks can be the same or different.

[0069] Before using the classification network to classify the image to be segmented, the classification network may be trained first. During training, multiple training images may be obtained first, and the multiple training images here may be the same as the training images in S202 above, and each training image is labeled with the candidate categories included. Afterwards, the multiple training images may be input into the initial classification network, and the labeled category corresponding to each training image may be used as the reference output of the initial classification network, and the initial classification network may be trained to obtain the classification network.

[0070] Of course, before inputting the training images into the initial classification network, each training image may also be preprocessed. For example, a suitable window width and window position may be set for each training image, and appropriate data enhancement may be performed (rotating, translating, flipping, etc. the training images). Data normalization or standardization may also be performed on each training image.

[0071] After the classification network is trained, the above-mentioned image to be segmented can be input into the trained classification network to obtain the probability that the image to be segmented belongs to each candidate category, which is recorded as the classification probability of belonging to each candidate category.

[0072] For example, taking intracranial hemorrhage in the brain as an example, assuming that there are a total of 5 types of hemorrhage categories, namely, intracerebral hemorrhage, subdural hemorrhage, epidural hemorrhage, intraventricular hemorrhage and subarachnoid hemorrhage, the 5 types of hemorrhage categories here can be considered as candidate categories. The segmentation category of each location of interest in the above S202 can also be any of the 5 types of hemorrhage categories, then in this step, the image to be segmented can be input into the classification network, and the classification probability of the image to be segmented belonging to the 5 types of hemorrhage categories can be obtained, assuming that they are 80%, 85%, 75%, 20%, and 90% respectively.

[0073] S206, optimizing the segmentation result of the image to be segmented according to the classification probability of each candidate category in the classification result and the segmentation category in the segmentation result.

[0074] After obtaining the classification probabilities that the image to be segmented as a whole belongs to each candidate category as described above, the candidate categories included in the image to be segmented as a whole can be obtained by analyzing each classification probability.

[0075] Similarly, after obtaining the segmentation category of each interested position on the image to be segmented, it is also possible to obtain how many segmentation categories the image to be segmented as a whole includes, that is, how many candidate categories it includes.

[0076] Then, the candidate categories obtained by the classification process and the candidate categories obtained by the segmentation process can be compared to suppress the segmentation result, which can also be called optimization or constraint. Optionally, if the candidate categories included in the image to be segmented as a whole in the segmentation result and the candidate categories included in the classification result are exactly the same, then the segmentation result can be directly used as the final segmentation result. If the candidate categories included in the image to be segmented as a whole in the segmentation result are more than the candidate categories included in the classification result, assuming that the segmentation result includes three candidate categories A, B, and C, and the classification result includes two candidate categories A and B, it can be seen that there is no candidate category C in the classification result, that is, the position of interest corresponding to the candidate category C in the segmentation result is an erroneous segmentation result, then the position of interest belonging to the candidate category C in the segmentation result can be deleted from the segmentation result to obtain the final segmentation result. For the case where the candidate categories included in the image to be segmented as a whole in the segmentation result are less than the candidate categories included in the classification result, no specific limitation is made here.

[0077] For example, continuing with the example of S204 above, assuming that the segmentation categories of each position of interest on the image to be segmented are combined in S202 above to obtain five segmentation categories, that is, the five candidate categories mentioned above are obtained. Through the classification results in S204 above, assuming that four candidate categories are finally obtained, namely, intracerebral hemorrhage, subdural hemorrhage, epidural hemorrhage, and subarachnoid hemorrhage, then the positions of interest in the segmentation results that belong to the intraventricular hemorrhage category can be removed, and the remaining positions of interest and their segmentation categories are used as the final segmentation results.

[0078] It should be noted that the above-mentioned segmentation network is actually a pixel-level classification, and this embodiment combines the image-level classification results and uses the image-level classification results to constrain the segmentation results obtained by the segmentation network. This can consider the image as a whole to avoid some over-segmentation problems as much as possible, thereby making the final segmentation result more accurate.

[0079] In the above-mentioned image segmentation method, the image to be segmented is segmented at the pixel level based on the segmentation network to obtain the segmentation result of the image to be segmented, and the image to be segmented is input into the classification network for image-level classification processing to obtain the classification result corresponding to the image to be segmented, and the segmentation result of the image to be segmented is optimized according to the classification probability of belonging to the candidate category in the classification result and the segmentation category in the segmentation result; wherein the segmentation result includes the segmentation category of each position of interest in the image to be segmented, and the classification result includes the classification probability that the image to be segmented belongs to the candidate category. In this method, since the image-level classification result can be used to constrain the pixel-level segmentation result, that is, the final image segmentation result is obtained by combining the image-level classification result and the pixel-level segmentation result, the final segmentation result is more accurate.

[0080] In another embodiment, another image segmentation method is provided. This embodiment involves the above-mentioned classification network including a first classification subnetwork and a second classification subnetwork, and the image to be segmented includes multiple image slices. Then, how to classify the image to be segmented and obtain the specific process of the classification result. Figure 3 As shown, the above S204 may include the following steps:

[0081] S302, inputting multiple image slices into a first classification subnetwork for spatial classification processing, and obtaining a first classification probability and an image feature vector of each image slice belonging to each candidate category.

[0082] The above-mentioned image to be segmented is an image sequence, and the multiple image slices included therein have time correlation, that is, the image slices are arranged in time sequence.

[0083] Of course, spatial classification processing can also be performed on multiple image sequences.

[0084] In this step, before the multiple image slices are input into the first classification sub-network, the order of the multiple image slices may be disrupted, or may not be disrupted.

[0085] Optionally, the first classification subnetwork here is a convolutional neural network CNN (Convolution Neural Network), for example, it can be a Resnet residual network, an efficient-net effective network, and of course it can also be other neural networks.

[0086] The spatial classification processing in this step is essentially classification processing, but the classification processing here refers to the classification processing of each image slice separately. When classifying each image slice, other image slices do not need to participate, that is, each image slice is classified separately from its own image space, which can be considered as spatial classification processing, that is, the classification results of each image slice are obtained from the image slice level.

[0087] Then, when the first classification sub-network is used to classify each image slice, each image slice can be input into the first classification sub-network, and after convolution, pooling and other processing, a feature vector corresponding to each image slice can be obtained, which is recorded as the image feature vector corresponding to each image slice. It can be the intermediate output result of the first classification sub-network.

[0088] After obtaining the image feature vector corresponding to each image slice, each image feature vector can be processed by convolution, full connection, etc., to obtain the classification probability of each image slice belonging to each candidate category, which is recorded as the first classification probability.

[0089] S304, inputting the image feature vectors of each image slice into the second classification subnetwork in time order for time classification processing, and obtaining the second classification probability that each image slice belongs to each candidate category; the second classification subnetwork has the function of learning the time correlation between each image slice.

[0090] In this step, optionally, the second classification subnetwork is a temporal network, such as LSTM (Long-Short Term Memory), RNN (Recurrent Neural Network), GRU (Gated Recurrent Unit), etc.

[0091] When classifying each image slice here, the image slices can be arranged in chronological order to form an image sequence, and then the image sequence is input into the second classification subnetwork for classification. Of course, the image slices can also be input into the second classification subnetwork in chronological order, and arranged and combined into an image sequence in the second classification subnetwork.

[0092] When classifying an image sequence, since the image slices in the image sequence are temporally correlated and arranged in chronological order, the second classification subnetwork can fuse the temporal correlation information between the image slices during classification. In this way, the image slices can be classified based on the temporal correlation to obtain the classification results.

[0093] That is to say, the image feature vectors corresponding to each image slice can be combined in chronological order and input into the second classification subnetwork for convolution, pooling, full connection and other processing to obtain the classification probability of each image slice belonging to each candidate category, which is recorded here as the second classification probability.

[0094] S306, obtaining the classification probability of each image slice belonging to each candidate category according to the first classification probability and the second classification probability of each image slice, and determining the classification probability of each image slice belonging to each candidate category as the classification result corresponding to the image to be segmented.

[0095] On the basis that the above-mentioned classification network includes the first classification subnetwork and the second classification subnetwork, the above-mentioned classification subnetwork may further include a third classification subnetwork. Then, when obtaining the classification probability that each image slice belongs to a candidate category in this step, optionally, in the above-mentioned third classification subnetwork, the first classification probability and the second classification probability of each input image slice may be concatenated to obtain a probability union consisting of the first classification probability and the second classification probability of each image slice; and the probability union may be fully connected in the above-mentioned third classification subnetwork to obtain the classification probability that each image slice belongs to each candidate category; wherein the number of classification probabilities of each image slice is the same as the number of first classification probabilities and the number of second classification probabilities.

[0096] The probability union here can be the probability union obtained by directly splicing the first classification probability and the second classification probability of each image slice; or it can be the probability union obtained by subtracting the first classification probability and the second classification probability of each image slice, and then splicing each probability difference with the first classification probability or the second classification probability to obtain the probability union.

[0097] Optionally, the third classification subnetwork here can be a temporal network, or of course other networks. Optionally, the third classification subnetwork includes a fully connected layer, and when the third classification subnetwork is input, the input first classification probability and second classification probability can be concatenated.

[0098] Furthermore, before using the first classification sub-network and the second classification sub-network for classification, the two sub-networks may be trained first. Optionally, the following steps A1-A3 may be used for training:

[0099] A1, obtaining a plurality of first training images and a plurality of second training images; wherein each first training image includes a first labeled classification probability, and each second training image includes a second labeled classification probability.

[0100] A2, taking multiple first training images as input of an initial first classification sub-network, taking the first labeled classification probability corresponding to each first training image as a reference output of the initial first classification sub-network, training the initial first classification sub-network, and obtaining a first classification sub-network.

[0101] A3, based on the plurality of second training images and the first classification sub-network, training the initial second classification sub-network to obtain a second classification sub-network.

[0102] Here, the first training image may be the same as or different from the second training image. Optionally, the number of first training images is equal to the number of second training images, and the number of two-dimensional image slices included in each first training image and each second training image may also be equal. Then, each two-dimensional image slice of each first training image may include a first labeled classification probability, and each two-dimensional image slice of each second training image may include a second labeled classification probability.

[0103] The following is a specific network structure to illustrate the training and application process of the above two sub-networks. Figure 3a As shown, the specific training process is as follows:

[0104] 1) Taking the first training image and the second training image as NCCT images, i.e., NCCT data, as an example, after obtaining the first training image and the second training image, each first training image and each second training image can be preprocessed respectively, and the preprocessing can be the same as the preprocessing method of the above S204, which will not be repeated here.

[0105] 2) Shuffle the two-dimensional image slices of each first training image, and then input each two-dimensional image slice into the initial CNN network (i.e., the CNN classification network in the figure). In the CNN classification network, feature extraction is performed on each two-dimensional image slice, and downsampling is performed an appropriate number of times to reduce the resolution. Finally, a low-resolution feature map is obtained, whose size is C*W*H, where C is the number of channels, W is the width of the feature map, and H is the height of the feature map. Then a global pooling operation (for example: average pooling, maximum pooling, etc., average pooling is used as an example in the figure) is performed on the feature map to generate a one-dimensional feature vector, denoted as E, with a length of C. Then, a fully connected layer is used to output the classification probability of the two-dimensional image slice, denoted as the first classification probability, assuming it is P 1ik , i is the 2D image slice index in the first training image, k is the candidate category identifier, assuming there are 10 2D image slices, then i is 1 to 10, assuming there are 5 categories, then k is 1 to 5.

[0106] 3) The first labeled classification probability included in each two-dimensional image slice in the first training image is denoted as P 1ik '. Then the loss between the output probability of each 2D image slice and the corresponding first labeled classification probability (i.e., the classification target in the figure) can be calculated, i.e., P 1ik and P 1ik 'The loss between them is calculated, and the initial CNN network is trained using the calculated loss to obtain a trained CNN network.

[0107] By disrupting the two-dimensional image slices in the first training image, it is ensured that the correlation between the two-dimensional image slices input into the initial CNN network is weak, which can prevent the CNN network training process from oscillating due to the strong correlation between the input two-dimensional image slices, thereby making it easier for the CNN network to learn information, accelerating the convergence of the CNN network, and thus making the performance of the finally trained CNN network better.

[0108] 4) After the CNN network is trained, each two-dimensional image slice of the second training image can be input into the trained CNN network to obtain the feature vector and the first classification probability corresponding to each two-dimensional image slice, which is recorded as I j ={E j ,P 2jk}, j is the 2D image slice index in the second training image, k is the candidate category identifier, and then the 2D image slices belonging to the same second training image are arranged in chronological order and grouped into one group to obtain the input data set of the initial temporal network, denoted as I g ={E g ,P 2g}, where g is the serial number of the divided group, that is, the serial number of each second training image, E g is a set of feature vectors of each two-dimensional image slice in each second training image, P 2g is a set consisting of the first classification probabilities of each two-dimensional image slice in each second training image.

[0109] 5) Here, the time series network takes the LSTM classification network as an example. g The length is T*3C, where T is the time step of the LSTM classification network and C is the number of channels mentioned above, i.e., the feature vector E generated by the CNN network. g The initial LSTM classification network consists of two bidirectional LSTM layers and two densely connected layers (i.e., Dense Layer in the figure).

[0110] Then the feature vector set E generated in step 5) can be g Input into the initial LSTM classification network (i.e., the vertically stacked blocks at the top of the LSTM classification network in the figure), in the initial LSTM classification network, through two bidirectional LSTM layers and two densely connected layers, each feature vector in the input feature vector set is subjected to feature extraction, full connection and other processing, and a set of classification probabilities corresponding to each set of feature vectors can be obtained, which is recorded as P 2g’ .

[0111] 6) The classification probability obtained by inputting each second training image into the CNN classification network, that is, P obtained in 4) 2g and the corresponding P obtained in 5)2g’ The images are spliced ​​and processed by a full connection (i.e., processed in the third classification subnetwork) to obtain the classification probability of the second training image output by the LSTM classification network. The loss between the output classification probability and the second labeled classification probability corresponding to the second training image (i.e., the classification target next to the LSTM classification network in the figure) is then calculated. The initial LSTM classification network is trained using the calculated loss to obtain a trained LSTM classification network.

[0112] After the CNN network and LSTM network are trained, the image to be segmented can be classified. For example, continuing with the above intracranial hemorrhage, assuming that the image to be segmented has 5 image slices, there are a total of 5 types of bleeding, that is, there are 5 candidate categories, then after the image to be segmented is input into the CNN network, 5 image feature vectors can be obtained, as well as the probability of belonging to these 5 types of bleeding on each image slice, recorded as the first classification probability of each image slice belonging to the candidate category, a total of 25 classification probabilities will be obtained. Then the image feature vectors of each image slice are input into the LSTM, and the probability of belonging to these 5 types of bleeding on each image slice will be obtained, recorded as the second classification probability of each image slice belonging to the candidate category, a total of 25 classification probabilities will be obtained, and then the 25 classification probabilities here and the 25 classification probabilities output by CNN are spliced ​​and fully processed, and finally 25 classification probabilities will be output, that is, the classification probability of each image slice on the image to be segmented belonging to these 5 candidate categories. After that, the classification probability can be used to optimize the segmentation result.

[0113] The image segmentation method provided by the present embodiment, the above-mentioned classification network includes a first classification subnetwork and a second classification subnetwork, the above-mentioned image to be segmented includes multiple image slices, the first classification subnetwork can be used to perform spatial classification processing on each image slice to obtain a first classification probability and feature vector of each image slice belonging to a candidate category, each feature vector is input into the second classification subnetwork for temporal classification processing, and the second classification probability of each image slice belonging to the candidate category is obtained, based on the first classification probability and the second classification probability, the classification probability of each image slice belonging to the candidate category is obtained, that is, the classification probability of the image to be segmented belonging to the candidate category is obtained. Through the method of this embodiment, the two classification subnetworks can be combined to perform spatial and temporal classification processing on the image to be segmented, the outputs of different networks can be combined, and the image to be segmented can be classified from different levels, so that the final classification result can be more accurate.

[0114] In another embodiment, another image segmentation method is provided. This embodiment relates to the segmentation category of each position of interest in the above-mentioned image to be segmented, including the segmentation category of each position of interest in each image slice; then how to optimize the specific process of the segmentation result according to the classification probability in the classification result and the segmentation category in the segmentation result. Based on the above embodiment, Figure 4 As shown, the above S206 may include the following steps:

[0115] S402: Obtain a probability threshold corresponding to each candidate category.

[0116] In this step, optionally, the following steps B1-B5 may be used:

[0117] B1, obtaining a plurality of test sample images, each of which includes a plurality of test image slices.

[0118] B2, input each test image slice into the classification network to obtain the classification probability that each test image slice belongs to each candidate category.

[0119] B3, according to the classification probability of each test image slice belonging to each candidate category, a plurality of classification probabilities corresponding to each candidate category are obtained.

[0120] B4, determining a receiver operating characteristic curve (ROC) curve corresponding to each candidate category according to the multiple classification probabilities corresponding to each candidate category.

[0121] B5, determining a probability threshold corresponding to each candidate category according to the curve characteristics of the ROC curve corresponding to each candidate category.

[0122] The test sample image here can be the above NCCT image, that is, NCCT data, which can be the same as or different from the above first training image and the second training image. Each test sample image includes a plurality of image slices, which are referred to as test image slices.

[0123] After the classification network is trained as described above, each test image slice of the test sample image here can be input into the trained classification network to obtain the classification probability of each test image slice outputted belonging to each candidate category.

[0124] Here, there are 5 test sample images, each of which includes 5 test image slices, and there are 5 candidate categories in total. Then, the 25 classification probabilities finally output here, among which every 5 classification probabilities belong to the same category, can draw ROC curves for the 5 probabilities of the same category, where the horizontal axis of the ROC curve represents FPR (false positive rate) and the vertical axis represents TPR (true positive rate). After drawing the ROC curve of each category, the classification probability corresponding to the point closest to the upper left of each ROC curve can be determined as the probability threshold of the category.

[0125] S404: Compare the classification probability of each image slice belonging to each candidate category with the corresponding probability threshold to obtain the classification category corresponding to each image slice.

[0126] In this step, after obtaining the probability threshold of each candidate category, the classification probabilities of various candidate categories of each image slice can be compared with the probability threshold of the corresponding candidate category to obtain the comparison result of each image slice, and the classification category of each image slice can be obtained through the comparison result of each image slice. For example, the classification probability less than the probability threshold can be removed, that is, the candidate category of this type in the image slice is removed, so that the classification category of each image slice can be obtained.

[0127] For example, assuming that an image slice obtains classification probabilities of 5 candidate categories, the 5 classification probabilities of the image slice can be compared with their corresponding probability thresholds to obtain the classification category of the image slice. Assuming that the 5 classification probabilities obtained by the image slice are: the classification probability of intracerebral hemorrhage is 70%, and its probability threshold is 50%; the classification probability of subdural hemorrhage is 82%, and its probability threshold is 55%; the classification probability of epidural hemorrhage is 35%, 48%, and its probability threshold is 60%; the classification probability of intraventricular hemorrhage is 20%, and its probability threshold is 50%; the classification probability of subarachnoid hemorrhage is 90%, and its probability threshold is 65%. Then, by comparison, it can be seen that the classification probabilities of epidural hemorrhage and intraventricular hemorrhage are both less than their corresponding probability thresholds, so these two candidate categories can be removed, and the classification categories corresponding to the image slice are obtained as intracerebral hemorrhage, subdural hemorrhage, and subarachnoid hemorrhage.

[0128] All image slices can be compared according to this step, so that the classification category corresponding to each image slice can be obtained.

[0129] S406 , matching the classification category of each image slice with the segmentation category of each interesting position in the corresponding image slice, and optimizing the segmentation result of the image to be segmented according to the matching result.

[0130] In this step, the segmentation results are mainly optimized according to the image slices, that is, the segmentation results of the image slices are optimized according to the classification categories of each image slice.

[0131] After obtaining the classification category of each image slice, the segmentation category of each position of interest in each image slice can also be obtained, which is recorded as the segmentation result of each image slice. Then, the classification category of each image slice can be matched with its corresponding segmentation result. Optionally, for the same image slice, if the matching result is that the segmentation category of the position of interest does not exist in the classification category of the image slice, the position of interest in the image slice is removed from the segmentation result.

[0132] That is to say, taking an image slice as an example, assuming that the classification categories of the image slice include intracerebral hemorrhage, subdural hemorrhage, and subarachnoid hemorrhage, and the segmentation result corresponding to the image slice includes intracerebral hemorrhage, subdural hemorrhage, subarachnoid hemorrhage, and intraventricular hemorrhage, it can be seen from this comparison that the classification category of the image slice does not include the intraventricular hemorrhage category in its segmentation result, so all the locations of interest corresponding to the intraventricular hemorrhage can be removed from its segmentation result. These removed locations of interest are the results of mis-segmentation. If the classification category of the image slice is the same as the segmentation category in its segmentation result, then its segmentation result will not be processed; if the classification category of the image slice is more than the segmentation category in its segmentation result, its segmentation result will not be processed here.

[0133] All image slices can be matched according to this method, so the segmentation result of each image slice can be optimized, that is, the segmentation result of each image slice is optimized. In this way, the final segmentation result of the image to be segmented can be obtained.

[0134] The image segmentation method provided in this embodiment can obtain the probability threshold corresponding to each candidate category, compare the classification probability of each image slice belonging to each candidate category with the corresponding probability threshold, obtain the classification category corresponding to each image slice, match the classification category of each image slice with the segmentation category of each interesting position in the corresponding image slice, and optimize the segmentation result of the image to be segmented according to the matching result. Through the method of this embodiment, the classification category of the image to be segmented can be determined by the probability threshold of each category, and the process is relatively simple, fast, and accurate, so that the classification category of the image to be segmented obtained in the end can be relatively accurate, so that when the classification category is used to optimize the segmentation result, the final optimized segmentation result can be made more accurate.

[0135] In order to explain the technical solution of the present application in more detail, a specific embodiment is described below. The method may include the following steps:

[0136] S1, performing pixel-level segmentation processing on the image to be segmented based on the segmentation network to obtain a segmentation result of the image to be segmented; the segmentation result includes a segmentation category of each position of interest in the image to be segmented, and the image to be segmented includes multiple image slices.

[0137] S2, inputting the multiple image slices into the first classification subnetwork for spatial classification processing, and obtaining the first classification probability and image feature vector of each image slice belonging to the candidate category.

[0138] S3, inputting the image feature vectors of each image slice into the second classification subnetwork in chronological order for temporal classification processing, and obtaining the second classification probability that each image slice belongs to a candidate category; the second classification subnetwork has the function of learning the temporal correlation between each image slice.

[0139] S4, in the third classification subnetwork, the first classification probability and the second classification probability of each input image slice are concatenated to obtain a probability union consisting of the first classification probability and the second classification probability of each image slice.

[0140] S5, performing full connection processing on the probability union in the third classification subnetwork to obtain the classification probability that each image slice belongs to the candidate category; wherein the number of classification probabilities of each image slice is the same as the number of first classification probabilities and the number of second classification probabilities.

[0141] S6, determining the classification probability of each image slice belonging to the candidate category as the classification result corresponding to the image to be segmented.

[0142] S7, obtaining multiple classification probabilities corresponding to each candidate category.

[0143] S8, determining a receiver operating characteristic (ROC) curve corresponding to each candidate category according to multiple classification probabilities corresponding to each candidate category.

[0144] S9, determining a probability threshold corresponding to each candidate category according to the curve characteristics of the ROC curve corresponding to each candidate category.

[0145] S10, comparing the classification probability of each image slice belonging to a candidate category with the probability threshold of the corresponding candidate category, and determining the classification category corresponding to each image slice according to the comparison result.

[0146] S11, for the same image slice, if the matching result is that the segmentation category of the position of interest does not exist in the classification category of the image slice, then the position of interest in the image slice is removed from the segmentation result.

[0147] It should be understood that although Figure 2-4The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-4 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0148] In one embodiment, Figure 5 As shown, an image segmentation device is provided, comprising: a segmentation module 10, a classification module 11 and an optimization module 12, wherein:

[0149] The segmentation module 10 is used to perform pixel-level segmentation processing on the image to be segmented based on the segmentation network to obtain a segmentation result of the image to be segmented; the segmentation result includes a segmentation category of each position of interest in the image to be segmented;

[0150] The classification module 11 is used to input the image to be segmented into the classification network for image-level classification processing to obtain a classification result corresponding to the image to be segmented; the classification result includes the classification probability that the image to be segmented belongs to each candidate category;

[0151] The optimization module 12 is used to optimize the segmentation result of the image to be segmented according to the classification probability of each candidate category in the classification result and the segmentation category in the segmentation result.

[0152] For the specific definition of the image segmentation device, please refer to the definition of the image segmentation method above, which will not be repeated here.

[0153] In another embodiment, another image segmentation device is provided, the classification network includes a first classification subnetwork and a second classification subnetwork, the image to be segmented includes a plurality of image slices, and based on the above embodiment, the classification module 11 may include a spatial classification unit, a temporal classification unit and a determination unit, wherein:

[0154] A spatial classification unit, used for inputting a plurality of image slices into a first classification subnetwork for spatial classification processing, and obtaining a first classification probability and an image feature vector of each image slice belonging to each candidate category;

[0155] A time classification unit is used to input the image feature vectors of each image slice into the second classification subnetwork in time order for time classification processing, so as to obtain the second classification probability that each image slice belongs to each candidate category; the second classification subnetwork has the function of learning the time correlation between each image slice;

[0156] The determination unit is used to obtain the classification probability of each image slice belonging to each candidate category according to the first classification probability and the second classification probability of each image slice, and determine the classification probability of each image slice belonging to each candidate category as the classification result corresponding to the image to be segmented.

[0157] Optionally, the first classification subnetwork is a convolutional neural network (CNN), and the second classification subnetwork is a temporal network.

[0158] Optionally, the classification network further includes a third classification subnetwork, and the determination unit may include a concatenation subunit and a fully connected subunit, wherein:

[0159] A splicing subunit, configured to perform splicing processing on the first classification probability and the second classification probability of each image slice input in the third classification subnetwork to obtain a probability union consisting of the first classification probability and the second classification probability of each image slice;

[0160] The fully connected subunit is used to perform fully connected processing on the probability union in the third classification subnetwork to obtain the classification probability that each image slice belongs to each candidate category; wherein the number of classification probabilities of each image slice is the same as the number of first classification probabilities and the number of second classification probabilities.

[0161] In another embodiment, another image segmentation device is provided, wherein the segmentation category of each position of interest in the image to be segmented includes the segmentation category of each position of interest in each image slice; based on the above embodiment, the optimization module 12 may include a threshold determination unit, a comparison unit and a matching unit, wherein:

[0162] A threshold acquisition unit, used to acquire a probability threshold corresponding to each candidate category;

[0163] A comparison unit, used to compare the classification probability of each image slice belonging to each candidate category with the corresponding probability threshold, to obtain the classification category corresponding to each image slice;

[0164] The matching unit is used to match the classification category of each image slice with the segmentation category of each interesting position in the corresponding image slice, and optimize the segmentation result of the image to be segmented according to the matching result.

[0165] Optionally, the matching unit is further configured to, for the same image slice, remove the interested position in the image slice from the segmentation result if the matching result is that the segmentation category of the interested position does not exist in the classification category of the image slice.

[0166] Optionally, the threshold acquisition unit is also used to acquire multiple test sample images, each test sample image includes multiple test image slices; input each test image slice into the classification network to obtain the classification probability of each test image slice belonging to each candidate category; based on the classification probability of each test image slice belonging to each candidate category, obtain multiple classification probabilities corresponding to each candidate category; based on the multiple classification probabilities corresponding to each candidate category, determine the receiver operating characteristic curve (ROC) curve corresponding to each candidate category; based on the curve characteristics of the ROC curve corresponding to each candidate category, determine the probability threshold corresponding to each candidate category.

[0167] In another embodiment, another image segmentation device is provided. Based on the above embodiment, the above device may further include a training module, the training module includes an acquisition unit, a first training unit and a second training unit, wherein:

[0168] An acquisition unit, configured to acquire a plurality of first training images and a plurality of second training images; wherein each first training image includes a first labeled classification probability, and each second training image includes a second labeled classification probability;

[0169] A first training unit is used to use the plurality of first training images as inputs of an initial first classification sub-network, use the first labeled classification probabilities corresponding to the first training images as reference outputs of the initial first classification sub-network, train the initial first classification sub-network, and obtain a first classification sub-network;

[0170] The second training unit is used to train the initial second classification subnetwork based on the plurality of second training images and the first classification subnetwork to obtain the second classification subnetwork.

[0171] For the specific definition of the image segmentation device, please refer to the definition of the image segmentation method above, which will not be repeated here.

[0172] Each module in the above-mentioned image segmentation device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0173] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0174] Perform pixel-level segmentation processing on the image to be segmented based on the segmentation network to obtain a segmentation result of the image to be segmented; the segmentation result includes a segmentation category for each location of interest in the image to be segmented;

[0175] The image to be segmented is input into the classification network for image-level classification processing to obtain a classification result corresponding to the image to be segmented; the classification result includes the classification probability that the image to be segmented belongs to each candidate category;

[0176] The segmentation result of the image to be segmented is optimized according to the classification probability of each candidate category in the classification result and the segmentation category in the segmentation result.

[0177] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0178] Multiple image slices are input into a first classification subnetwork for spatial classification processing to obtain a first classification probability and an image feature vector of each image slice belonging to each candidate category; the image feature vectors of each image slice are input into a second classification subnetwork in chronological order for temporal classification processing to obtain a second classification probability of each image slice belonging to each candidate category; the second classification subnetwork has the function of learning the temporal correlation between each image slice; based on the first classification probability and the second classification probability of each image slice, the classification probability of each image slice belonging to each candidate category is obtained, and the classification probability of each image slice belonging to each candidate category is determined as the classification result corresponding to the image to be segmented.

[0179] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0180] In the third classification subnetwork, the first classification probability and the second classification probability of each input image slice are concatenated to obtain a probability union consisting of the first classification probability and the second classification probability of each image slice; in the third classification subnetwork, the probability union is fully connected to obtain the classification probability of each image slice belonging to each candidate category; wherein the number of classification probabilities of each image slice is the same as the number of first classification probabilities and the number of second classification probabilities.

[0181] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0182] Obtain the probability threshold corresponding to each candidate category; compare the classification probability of each image slice belonging to each candidate category with the corresponding probability threshold to obtain the classification category corresponding to each image slice; match the classification category of each image slice with the segmentation category of each interested position in the corresponding image slice, and optimize the segmentation result of the image to be segmented according to the matching result.

[0183] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0184] For the same image slice, if the matching result is that the segmentation category of the position of interest does not exist in the classification category of the image slice, the position of interest in the image slice is removed from the segmentation result.

[0185] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0186] Acquire multiple test sample images, each test sample image includes multiple test image slices; input each test image slice into a classification network to obtain the classification probability that each test image slice belongs to each candidate category; obtain multiple classification probabilities corresponding to each candidate category based on the classification probability that each test image slice belongs to each candidate category; determine a receiver operating characteristic curve (ROC) curve corresponding to each candidate category based on the multiple classification probabilities corresponding to each candidate category; determine a probability threshold corresponding to each candidate category based on the curve characteristics of the ROC curve corresponding to each candidate category.

[0187] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0188] Acquire multiple first training images and multiple second training images; wherein each first training image includes a first labeled classification probability, and each second training image includes a second labeled classification probability; use the multiple first training images as inputs of an initial first classification subnetwork, use the first labeled classification probabilities corresponding to each first training image as reference outputs of the initial first classification subnetwork, train the initial first classification subnetwork to obtain a first classification subnetwork; and train the initial second classification subnetwork based on the multiple second training images and the first classification subnetwork to obtain a second classification subnetwork.

[0189] In one embodiment, the first classification subnetwork is a convolutional neural network (CNN), and the second classification subnetwork is a temporal network.

[0190] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0191] Perform pixel-level segmentation processing on the image to be segmented based on the segmentation network to obtain a segmentation result of the image to be segmented; the segmentation result includes a segmentation category for each location of interest in the image to be segmented;

[0192] The image to be segmented is input into the classification network for image-level classification processing to obtain a classification result corresponding to the image to be segmented; the classification result includes the classification probability that the image to be segmented belongs to each candidate category;

[0193] The segmentation result of the image to be segmented is optimized according to the classification probability of each candidate category in the classification result and the segmentation category in the segmentation result.

[0194] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0195] Multiple image slices are input into a first classification subnetwork for spatial classification processing to obtain a first classification probability and an image feature vector of each image slice belonging to each candidate category; the image feature vectors of each image slice are input into a second classification subnetwork in chronological order for temporal classification processing to obtain a second classification probability of each image slice belonging to each candidate category; the second classification subnetwork has the function of learning the temporal correlation between each image slice; based on the first classification probability and the second classification probability of each image slice, the classification probability of each image slice belonging to each candidate category is obtained, and the classification probability of each image slice belonging to each candidate category is determined as the classification result corresponding to the image to be segmented.

[0196] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0197] In the third classification subnetwork, the first classification probability and the second classification probability of each input image slice are concatenated to obtain a probability union consisting of the first classification probability and the second classification probability of each image slice; in the third classification subnetwork, the probability union is fully connected to obtain the classification probability of each image slice belonging to each candidate category; wherein the number of classification probabilities of each image slice is the same as the number of first classification probabilities and the number of second classification probabilities.

[0198] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0199] Obtain the probability threshold corresponding to each candidate category; compare the classification probability of each image slice belonging to each candidate category with the corresponding probability threshold to obtain the classification category corresponding to each image slice; match the classification category of each image slice with the segmentation category of each interested position in the corresponding image slice, and optimize the segmentation result of the image to be segmented according to the matching result.

[0200] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0201] For the same image slice, if the matching result is that the segmentation category of the position of interest does not exist in the classification category of the image slice, the position of interest in the image slice is removed from the segmentation result.

[0202] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0203] Acquire multiple test sample images, each test sample image includes multiple test image slices; input each test image slice into a classification network to obtain the classification probability that each test image slice belongs to each candidate category; obtain multiple classification probabilities corresponding to each candidate category based on the classification probability that each test image slice belongs to each candidate category; determine a receiver operating characteristic curve (ROC) curve corresponding to each candidate category based on the multiple classification probabilities corresponding to each candidate category; determine a probability threshold corresponding to each candidate category based on the curve characteristics of the ROC curve corresponding to each candidate category.

[0204] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0205] Acquire multiple first training images and multiple second training images; wherein each first training image includes a first labeled classification probability, and each second training image includes a second labeled classification probability; use the multiple first training images as inputs of an initial first classification subnetwork, use the first labeled classification probabilities corresponding to each first training image as reference outputs of the initial first classification subnetwork, train the initial first classification subnetwork to obtain a first classification subnetwork; and train the initial second classification subnetwork based on the multiple second training images and the first classification subnetwork to obtain a second classification subnetwork.

[0206] In one embodiment, the first classification subnetwork is a convolutional neural network (CNN), and the second classification subnetwork is a temporal network.

[0207] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0208] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0209] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An image segmentation method, characterized in that: The method comprises: Performing pixel-level segmentation processing on the image to be segmented based on the segmentation network to obtain a segmentation result of the image to be segmented; the segmentation result includes a segmentation category of each position of interest in the image to be segmented; the image to be segmented includes a plurality of image slices; the segmentation category of each position of interest in the image to be segmented includes a segmentation category of each position of interest in each of the image slices; Inputting the image to be segmented into a classification network for image-level classification processing to obtain a classification result corresponding to the image to be segmented; the classification result includes the classification probability that the image to be segmented belongs to each candidate category; Optimizing the segmentation result of the image to be segmented according to the classification probability of each candidate category in the classification result and the segmentation category in the segmentation result; The optimizing the segmentation result of the image to be segmented according to the classification probability of each candidate category in the classification result and the segmentation category in the segmentation result includes: The classification category of each image slice is matched with the segmentation category of each interested position in the corresponding image slice. For the same image slice, if the segmentation category of the interested position does not exist in the classification category of the image slice, the interested position in the image slice is removed from the segmentation result.

2. The method according to claim 1, characterized in that The classification network includes a first classification subnetwork and a second classification subnetwork, and the image to be segmented is input into the classification network for image-level classification processing to obtain a classification result corresponding to the image to be segmented, including: Inputting the plurality of image slices into the first classification subnetwork for spatial classification processing to obtain a first classification probability and an image feature vector of each image slice belonging to each candidate category; Inputting the image feature vectors of each of the image slices into the second classification subnetwork in time order for time classification processing, and obtaining a second classification probability that each of the image slices belongs to each candidate category; the second classification subnetwork has a function of learning the time correlation between each of the image slices; According to the first classification probability and the second classification probability of each image slice, the classification probability that each image slice belongs to each candidate category is obtained, and the classification result of the image to be segmented is determined according to the classification probability that each image slice belongs to each candidate category.

3. The method according to claim 2, characterized in that The classification network further includes a third classification subnetwork, which obtains the classification probability that each image slice belongs to each candidate category according to the first classification probability and the second classification probability of each image slice, including: In the third classification subnetwork, the first classification probability and the second classification probability of each of the input image slices are spliced ​​to obtain a probability union consisting of the first classification probability and the second classification probability of each of the image slices; The probability union is fully connected in the third classification subnetwork to obtain the classification probability that each image slice belongs to each candidate category; wherein the number of classification probabilities of each image slice is the same as the number of the first classification probabilities and the number of the second classification probabilities.

4. The method according to claim 2 or 3, characterized in that: The method further comprises: Obtaining a probability threshold corresponding to each of the candidate categories; The classification probability of each image slice belonging to each candidate category is compared with the corresponding probability threshold to obtain the classification category corresponding to each image slice.

5. The method according to claim 4, characterized in that The obtaining of the probability threshold corresponding to each of the candidate categories includes: Acquire a plurality of test sample images, each of the test sample images comprising a plurality of test image slices; Inputting each of the test image slices into the classification network to obtain the classification probability that each of the test image slices belongs to each candidate category; According to the classification probability that each of the test image slices belongs to each candidate category, a plurality of classification probabilities corresponding to each candidate category are obtained; Determine a receiver operating characteristic (ROC) curve corresponding to each candidate category according to a plurality of classification probabilities corresponding to each candidate category; According to the curve characteristics of the ROC curve corresponding to each candidate category, the probability threshold corresponding to each candidate category is determined.

6. The method according to claim 2 or 3, characterized in that: The training method of the first classification sub-network and the second classification sub-network includes: Acquire a plurality of first training images and a plurality of second training images; wherein each of the first training images includes a first labeled classification probability; Using the multiple first training images as inputs of an initial first classification subnetwork, using the first labeled classification probabilities corresponding to the first training images as reference outputs of the initial first classification subnetwork, training the initial first classification subnetwork, and obtaining the first classification subnetwork; Based on the multiple second training images and the first classification sub-network, an initial second classification sub-network is trained to obtain the second classification sub-network.

7. An image segmentation device, characterized in that: The device comprises: A segmentation module, configured to perform pixel-level segmentation processing on the image to be segmented based on a segmentation network to obtain a segmentation result of the image to be segmented; the segmentation result includes a segmentation category of each position of interest in the image to be segmented; the image to be segmented includes a plurality of image slices; the segmentation category of each position of interest in the image to be segmented includes a segmentation category of each position of interest in each of the image slices; A classification module is used to input the image to be segmented into a classification network for image-level classification processing to obtain a classification result corresponding to the image to be segmented; the classification result includes the classification probability that the image to be segmented belongs to each candidate category; An optimization module, used for optimizing the segmentation result of the image to be segmented according to the classification probability of each candidate category in the classification result and the segmentation category in the segmentation result; The optimization module is used to match the classification category of each image slice with the segmentation category of each interested position in the corresponding image slice. For the same image slice, if the segmentation category of the interested position does not exist in the classification category of the image slice, the interested position in the image slice is removed from the segmentation result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images