Image segmentation method and system based on semi-supervised learning and point rendering

Abstract

The present invention proposes an image segmentation method and system based on semi-supervised learning and point rendering, comprising: constructing a dual-branch network to obtain a mask of a feature area in an input image; constructing a point rendering module to sample and interpolate boundary points in the mask to reconstruct the boundary of the feature area; and obtaining a feature area image based on the reconstructed boundary of the feature area. The present invention can effectively improve the accuracy of image segmentation.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image segmentation method and system based on semi-supervised learning and point rendering. Background Art

[0002] High-quality image data in some fields (such as medicine) is often scarce and lacks annotations. This is because annotating images requires a lot of manual work and high expertise, which leads to high annotation costs. In addition, when the number of labeled samples is insufficient, there will be problems such as uneven contours of segmented images and loss of a large amount of low-level information. Summary of the invention

[0003] In view of the above problems existing in the prior art, the present invention proposes an image segmentation method and system based on semi-supervised learning and point rendering, which mainly solves the problem that the image segmentation of the prior art easily ignores the detail features and leads to insufficient accuracy.

[0004] In order to achieve the above-mentioned purpose and other purposes, the technical solution adopted by the present invention is as follows.

[0005] An image segmentation method based on semi-supervised learning and point rendering, comprising:

[0006] Construct a two-branch network to obtain the mask of the feature area in the input image;

[0007] Constructing a point rendering module to sample and interpolate the boundary points in the mask to reconstruct the boundary of the feature area;

[0008] A feature region image is obtained based on the reconstructed feature region boundary.

[0009] Optionally, before acquiring the mask through the dual-branch network, the method further includes:

[0010] The input image is feature extracted through the backbone network, and the extracted feature map is input into the pre-built expectation maximization attention network, and the feature map is reduced in dimension and used as the input of the dual-branch network.

[0011] Optionally, the backbone network includes one of the following: vgg16, vgg19, resnet50, resnet101.

[0012] Optionally, the dual-branch network includes a semi-supervised segmentation generative adversarial network and a multi-label average teacher network;

[0013] Predicting the output probability distribution of each pixel in the input feature map through the semi-supervised segmentation generative adversarial network to obtain a mask of the feature area;

[0014] The erroneous prediction outputs of the semi-supervised segmentation generative adversarial network are eliminated through the multi-label average teacher network.

[0015] Optionally, the semi-supervised segmentation generative adversarial network includes a generator and a discriminator;

[0016] The generator generates multiple segmentation maps according to the input feature map;

[0017] The discriminator matches the segmentation map with a standard map, obtains a segmentation map that meets preset conditions, and outputs a mask corresponding to the segmentation map as a feature region mask.

[0018] Optionally, the step of reconstructing the feature area boundary by the point rendering module includes:

[0019] Selecting a plurality of pixel points whose predicted output probability satisfies a preset threshold range from the output of the dual-branch network, and extracting features of the selected pixel points;

[0020] Interpolate the selected pixel features and upsample to generate an interpolated image;

[0021] Selecting points, interpolating, and upsampling are performed again based on the interpolated image, and the iteration is repeated multiple times until the size of the generated interpolated image is equal to the size of the input image;

[0022] The category label of each pixel in the interpolated image is predicted, and the feature region boundary is reconstructed based on the category label.

[0023] Optionally, predict the category label of each pixel in the generated image, including:

[0024] The fine-grained features and the coarse-grained features of the interpolated image are extracted, and the fine-grained features and the coarse-grained features are spliced ​​and input into a classification prediction network to obtain the category labels of the corresponding pixels.

[0025] Optionally, the fine-grained features include boundary line texture features; and the coarse-grained features include pixel point category features.

[0026] Optionally, the interpolation comprises bilinear interpolation.

[0027] An image segmentation system based on semi-supervised learning and point rendering, comprising:

[0028] The mask acquisition module is used to build a dual-branch network to obtain the mask of the feature area in the input image;

[0029] The point rendering module is used to sample and interpolate the boundary points in the mask to reconstruct the feature region boundary; and obtain the feature region image based on the reconstructed feature region boundary.

[0030] As described above, the image segmentation method and system based on semi-supervised learning and point rendering of the present invention have the following beneficial effects.

[0031] The point rendering module is used to extract coarse and fine features of points that are difficult to distinguish in the feature map, and the prediction probability of each point is determined point by point, which can effectively improve the accuracy of boundary segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the structure of an image segmentation network based on semi-supervised learning and point rendering in one embodiment of the present invention.

[0033] Figure 2 Schematic diagram of the expected maximization attention network structure in one embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0035] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0036] Taking medical image processing as an example, tumor anatomical surgery in the medical field is often a complex task that requires a lot of professional foundation and practical experience. At the same time, a complete surgical process often takes a lot of manpower and time. In order to solve this problem, people are constantly exploring the use of deep learning to provide some guidance and auxiliary assistance to this complex operation process. The purpose of using neural networks for tumor segmentation tasks is to automatically segment the required tumor area from noisy CT images or MRI images to obtain the volume, shape and location information of the tumor. Manual segmentation of tumors requires highly professional medical recognition capabilities, and through machine learning technology, the tumor segmentation task can be regarded as a common semantic segmentation problem, and their models can be established on this basis.

[0037] See also Figure 1 The present invention provides an image segmentation method based on semi-supervised learning and point rendering, which specifically includes the following steps:

[0038] Step (1), collect tumor image data. Relevant high-quality original data can be obtained from hospitals or scientific research institutions. Different magnetic resonance imaging scanners are used to obtain brain tumor images of multiple modalities (T1, T2, T1-c, Flair) for high-grade gliomas (HGG) and low-grade gliomas (LGG) that meet the requirements.

[0039] The brain tumor images are converted into a data format that is easy for the program to process (such as .nii or .npy, etc.) through the program. If it is multimodal image data, it should be distinguished when saving to facilitate later processing.

[0040] Step (2), select a backbone network to extract features from the input image. The backbone network can be vgg16, vgg19, resnet50, resnet101, etc.

[0041] Step (3), build the Expectation-Maximization Attention Networks (EMANet):

[0042] See also Figure 2 , taking the feature map extracted by the backbone network as input, after a 1×1 convolution dimension reduction, the network will initialize a C-channel H×W resolution size feature map as K (K=H×W) basis, and get the attention image in the Ae process, and update a set of basis in the Am process. After several iterations, the basis and the attention image are used to reconstruct the feature to return to the input image size. High-dimensional, global information features are obtained.

[0043] Step (4), construct a dual-branch network, which includes a semi-supervised segmentation generative adversarial network (Semi-Supervised Semanic Segmentation GAN, s4GAN) and a multi-label average teacher network (Multi-LabelMean Teacher, MLMT).

[0044] In one embodiment, a mask of a feature region is obtained by predicting the output probability distribution of each pixel in an input feature map through a semi-supervised segmentation generative adversarial network;

[0045] Eliminating erroneous prediction outputs of generative adversarial networks for semi-supervised segmentation via multi-label averaging teacher networks.

[0046] In one embodiment, a semi-supervised segmentation generative adversarial network includes a generator and a discriminator. The generator generates multiple segmentation maps according to an input feature map;

[0047] The discriminator matches the segmentation map with the standard map, obtains a segmentation map that meets preset conditions, and outputs a mask corresponding to the segmentation map as a feature area mask.

[0048] Specifically, Figure 1 The backbone segmentation network S is used as a generator, and the generator is used to predict the segmentation map C (assuming there are C segmentation maps in total). The discriminator stacks the above C segmentation maps and the input image as new input. Its task is to match the distribution statistics of the predicted and actual segmented images.

[0049] The above training process uses three loss functions:

[0050] ①Cross entropy loss function

[0051]

[0052] ② Feature matching loss: (minimize the mean difference between the prediction and the true value to ensure that the predicted segmentation map has the same feature statistics as the actual segmentation map):

[0053]

[0054] in, represents a stacking operation, x l and x u Represents labeled and unlabeled inputs, y l and S(x u ) represent the actual segmentation map and the predicted segmentation map respectively.

[0055] ③Self-training loss (balanced generator and discriminator):

[0056]

[0057] Among them, the discriminator outputs D(S(x u )) is between 0 and 1, where 0 represents the predicted segmentation map and 1 represents the ground truth (actual segmentation map). The self-training loss is used as a quality assessment score to further balance the network optimization process. Therefore, L st It mainly consists of cross entropy loss, and γ is the threshold.

[0058] ④The final generated network loss is:

[0059] L s =L ce +λ fm L fm +λ st L st

[0060] In one embodiment, steps (2) to (4) together constitute a s4GAN network with low-level consistency, which is the upper branch of the overall network.

[0061] Step (5) constructs a multi-label semi-supervised classifier (MIMT network) with high-level consistency, which is the lower half of the overall network. Set up a student network G and a teacher network H, whose inputs are the original image and the perturbed image respectively. In order to ensure that the prediction of the student model is consistent with that of the teacher model, the consistency loss (mean square error) is used to train the network structure. Its loss function is as follows:

[0062]

[0063] Among them, z l is the multi-hot vector of ground truth.

[0064] Step (6), the above steps as a whole constitute a semi-supervised learning semantic segmentation network structure with high-level and low-level consistency, which is used to generate tumor shape masks and provide rough segmentation predictions.

[0065] Step (7), when entering the point rendering stage, the step of reconstructing the feature area boundary by the point rendering module includes:

[0066] Select multiple pixel points whose predicted output probability meets a preset threshold range from the dual-branch network output, and extract the features of the selected pixel points;

[0067] Interpolate the selected pixel features and upsample to generate an interpolated image;

[0068] Selecting points, interpolating, and upsampling are performed again based on the interpolated image, and the iteration is repeated multiple times until the size of the generated interpolated image is equal to the size of the input image;

[0069] The category label of each pixel in the interpolated image is predicted, and the feature region boundary is reconstructed based on the category label.

[0070] Specifically, training and prediction can adopt different operation steps.

[0071] During training: randomly select kN points from a uniform distribution in the tumor bounding box as candidate points; select the N most uncertain points (points with pixel-level prediction probabilities closer to 0.5) from the kN candidate points; randomly select (1-b)N points. Generally speaking, these difficult-to-predict points are distributed on the boundaries of the tumor mask. Use a lightweight segmentation head to extract the fine-grained features and coarse-grained features of these points, respectively, and then splice the two together. Finally, input the MLP (multi-layer perceptron with 1024 neurons) to predict the label. Among them, fine-grained features may include boundary line texture features, boundary shape features, boundary size features, etc.; coarse-grained features may include pixel point category features, etc.

[0072] During prediction: perform bilinear interpolation on the mask generated by the dual-branch network, upsample twice to select the top N most uncertain points, concatenate the coarse and fine granular features corresponding to these points, and use MLP to predict the labels of these points. Upsample this result again until the mask size is equal to the original image size, obtain the reconstructed feature region boundary, and convolve the reconstructed mask with the original image to obtain the feature region image.

[0073] Step (8): Build a network structure framework through the above steps (3) to (10), input enough multi-region tumor images for training to obtain a segmentation prediction model. This model can be used to complete the task of segmenting tumor images with a certain degree of accuracy.

[0074] Step (9), the image finally output by the above method can be provided to medical institutions to play some role in anatomical guidance, auxiliary reference and technical review in tumor dissection surgery.

[0075] This embodiment also provides an image segmentation system based on semi-supervised learning and point rendering, which is used to execute the image segmentation method based on semi-supervised learning and point rendering described in the aforementioned method embodiment. Since the technical principle of the system embodiment is similar to the technical principle of the aforementioned method embodiment, the same technical details will not be repeated.

[0076] In one embodiment, an image segmentation system based on semi-supervised learning and point rendering includes: a mask acquisition module for constructing a dual-branch network to acquire a mask of a feature region in an input image;

[0077] The point rendering module is used to sample and interpolate the boundary points in the mask to reconstruct the feature region boundary; and obtain the feature region image based on the reconstructed feature region boundary.

[0078] In summary, the present invention is an image segmentation method and system based on semi-supervised learning and point rendering. In order to solve the problem of excessive segmentation calculation, an expectation maximization attention network (EMANet) module is added to the feature extraction stage in the backbone segmentation network. The original features are reduced in dimension and the connection between pixels is modeled in the low-dimensional image to save computing resources; the point rendering technology is used to extract coarse and fine features of the difficult-to-distinguish points in the feature image in the process of continuous iterative upsampling, and the prediction probability of these points is determined point by point. Finally, the boundary details of the image segmentation mask are gradually changed from coarse to fine, until the feature image is restored to the required original size; the final output image can provide some anatomical guidance before surgery, some anatomical auxiliary references during surgery, and even some technical review functions after surgery. Therefore, the present invention effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0079] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. An image segmentation method based on semi-supervised learning and point rendering, characterized in that: include: Construct a two-branch network to obtain the mask of the feature area in the input image; The dual-branch network includes a semi-supervised segmentation generative adversarial network and a multi-label average teacher network; the semi-supervised segmentation generative adversarial network predicts the output probability distribution of each pixel in the input feature map to obtain a mask of the feature area; the multi-label average teacher network eliminates the erroneous prediction output of the semi-supervised segmentation generative adversarial network; the semi-supervised segmentation generative adversarial network includes a generator and a discriminator; the generator generates multiple segmentation maps according to the input feature map; the discriminator matches the segmentation map with the standard map to obtain a segmentation map that meets the preset conditions, and outputs the mask corresponding to the segmentation map as the feature area mask; Constructing a point rendering module to sample and interpolate the boundary points in the mask to reconstruct the boundary of the feature area; A feature region image is obtained based on the reconstructed feature region boundary.

2. The image segmentation method based on semi-supervised learning and point rendering according to claim 1, characterized in that: Before obtaining the mask through the dual-branch network, it also includes: The input image is feature extracted through the backbone network, and the extracted feature map is input into the pre-built expectation maximization attention network, and the feature map is reduced in dimension and used as the input of the dual-branch network.

3. The image segmentation method based on semi-supervised learning and point rendering according to claim 2, characterized in that: The backbone network includes one of the following: vgg16, vgg19, resnet50, resnet101.

4. The image segmentation method based on semi-supervised learning and point rendering according to claim 1, characterized in that: The step of reconstructing the feature area boundary by the point rendering module includes: Selecting a plurality of pixel points whose predicted output probability satisfies a preset threshold range from the output of the dual-branch network, and extracting features of the selected pixel points; Interpolate the selected pixel features and upsample to generate an interpolated image; Selecting points, interpolating, and upsampling are performed again based on the interpolated image, and the iteration is repeated multiple times until the size of the generated interpolated image is equal to the size of the input image; The category label of each pixel in the interpolated image is predicted, and the feature region boundary is reconstructed based on the category label.

5. The image segmentation method based on semi-supervised learning and point rendering according to claim 4, characterized in that: Predict the category label of each pixel in the generated image, including: The fine-grained features and the coarse-grained features of the interpolated image are extracted, and the fine-grained features and the coarse-grained features are spliced ​​and input into a classification prediction network to obtain the category labels of the corresponding pixels.

6. The image segmentation method based on semi-supervised learning and point rendering according to claim 5, characterized in that: The fine-grained features include boundary line texture features; the coarse-grained features include pixel point category features.

7. The image segmentation method based on semi-supervised learning and point rendering according to claim 4, characterized in that: The interpolation includes bilinear interpolation.

8. An image segmentation system based on semi-supervised learning and point rendering, characterized in that: include: The mask acquisition module is used to build a dual-branch network to obtain the mask of the feature area in the input image; The dual-branch network includes a semi-supervised segmentation generative adversarial network and a multi-label average teacher network; the semi-supervised segmentation generative adversarial network predicts the output probability distribution of each pixel in the input feature map to obtain a mask of the feature area; the multi-label average teacher network eliminates the erroneous prediction output of the semi-supervised segmentation generative adversarial network; the semi-supervised segmentation generative adversarial network includes a generator and a discriminator; the generator generates multiple segmentation maps according to the input feature map; the discriminator matches the segmentation map with the standard map to obtain a segmentation map that meets the preset conditions, and outputs the mask corresponding to the segmentation map as the feature area mask; The point rendering module is used to sample and interpolate the boundary points in the mask to reconstruct the feature region boundary; and obtain the feature region image based on the reconstructed feature region boundary.

Images