Image segmentation method of weakly supervised neural network based on a small number of sample line marking

Abstract

The application relates to an image segmentation method of a weakly supervised neural network based on a small amount of sample line marking, wherein the proposed weakly supervised neural network comprises two sub-networks, namely a main branch and an auxiliary branch, which are connected with each other; the main branch network is used to extract image features of an input image and output an image segmentation result; the auxiliary branch network is used to sequentially perform feature embedding and distance measurement on the extracted image features, and generate a pseudo-labeled image; the pseudo-labeled image is used to supervise the image segmentation result of the main branch network, optimize network parameters, and obtain a more accurate segmentation result. The application can adopt a line marking mode to mark a small amount of image samples, mark part of pixels, and on this basis, utilize the proposed weakly supervised neural network to realize accurate segmentation of a complex texture image in a small sample scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and aims at a digital image with complex texture, and provides a weakly supervised neural network image segmentation method based on a small amount of sample line marking. BACKGROUND

[0002] Image segmentation is one of the most important tasks in the field of image processing, and has been widely applied in many fields, such as medical image processing, remote sensing image analysis, conference background replacement, automatic driving, etc. Image segmentation methods can be divided into traditional image segmentation methods, machine learning-based image segmentation methods and deep learning-based image segmentation methods.

[0003] Traditional image segmentation methods are mainly divided into region-based segmentation and edge-based segmentation, and it is often difficult to obtain good segmentation results when facing complex texture images; the machine learning-based image segmentation method usually needs to extract the corresponding features of each pixel, and then classifies the pixel points one by one, and the segmentation effect depends on the performance of the used features and machine learning classifier, which has large calculation and storage consumption, and the segmentation effect is often not ideal; the deep learning-based image segmentation method uses a multi-layer neural network to segment the image, and a good segmentation model can accurately segment the specified type of image. However, in most cases, the deep learning-based image segmentation method needs a large number of image samples and very complete image annotation data for image processing model training, which consumes a lot of manpower, material and time cost. These existing deep learning methods often lose the accurate image segmentation ability in the small sample environment and in the case of lack of sufficient annotation data. SUMMARY

[0004] The present application aims to overcome the defects of the prior art and provides a weakly supervised neural network image segmentation method based on a small amount of sample line marking, which can realize accurate segmentation of complex texture images in a small sample scene.

[0005] The object of the present application can be achieved by the following technical solutions:

[0006] A weakly supervised neural network image segmentation method based on a small amount of sample line marking, comprising:

[0007] 1) Obtain a training image, and perform line marking on the training image;

[0008] 2) Data augmentation is performed on the labeled training image;

[0009] 3) input the enhanced image into a pre-constructed weakly supervised image segmentation network, the weakly supervised image segmentation network comprising a main branch network and an auxiliary branch network connected with each other;

[0010] 4) extracting image features of the input image based on the few pixels of the line marking through the main branch network, and outputting an image segmentation result;

[0011] 5) sequentially performing feature embedding and distance measurement on the image features extracted by the main branch network through the auxiliary branch network, to generate a pseudo-labeled map of the training image, the pseudo-labeled map being used to supervise the image segmentation result of the main branch network and optimize the parameters of the weakly supervised image segmentation network;

[0012] 6) iteratively optimizing the parameters of the weakly supervised image segmentation network through a loss function;

[0013] 7) using the main branch network in the weakly supervised image segmentation network after optimization to segment the image.

[0014] Further, the weakly supervised image segmentation network constructed is described as follows:

[0015] The main branch network is a Unet++ network, comprising five encoder modules and ten decoder modules, the image features being extracted through multi-layer feature combination of the Unet++ network, and the output of the Unet++ network being an image segmentation result;

[0016] The auxiliary branch network comprises three parts of feature embedding, calculation of feature cosine distance, and generation of a pseudo-labeled map.

[0017] Further, the process of extracting image features by the main branch network is specifically as follows:

[0018] The four output feature maps of deep supervision of the Unet++ network and the output feature map from the input to the first encoder module are up-sampled through bilinear interpolation, so that the sizes of the five output feature maps are the same as the size of the input image, the five output feature maps are spliced by channels, and then the feature extraction of the input image is completed, and the formula is as follows:

[0019]

[0020] In the formula, fea is the extracted training image feature, R(·) is a bilinear interpolation up-sampling operation, is a channel splicing operation, f 0 is the output feature map from the input to the first encoder module, f 1 ,f 2 ,f 3 ,f 4Respectively, four output feature maps of Unet++ network deep supervision.

[0021] Further, the description of the image features is as follows: in the training phase, the part of the pixel of the image data for training is marked by the method of marking with a line; the input image set is divided into multiple small batch data sets, and for the image features extracted for each small batch data set, the features corresponding to the marked pixels cf l The features corresponding to the unmarked pixels cf u The features corresponding to the marked pixels cf l The features corresponding to the unmarked pixels cf u Satisfy the following relationship:

[0022] fea=cfl l ∪cfu u

[0023] The features corresponding to the marked pixels cf l Is saved as mf l In the image training process of each small batch data set, cf l And cf u The mf l Of the previous small batch data set are input into the auxiliary branch network for feature embedding.

[0024] Further, the auxiliary branch network performs feature embedding on the extracted image features:

[0025] In the auxiliary branch network, cf l And cf u The mf l Of the current small batch data set and the previous small batch data set are embedded by a convolution layer with a convolution kernel size of n, so that the image features are embedded into a feature space with the same dimension size.

[0026] Further, the process of distance measurement of the auxiliary branch network on the image features after feature embedding is as follows:

[0027] The distance between cf l And cf u Is measured to obtain the average cosine distance between the unmarked pixel points and the line marking pixel points of each class, and the class with the smallest average cosine distance is selected as the class of the unmarked pixel points in the pseudo-labeled image, and then the corresponding pseudo-labeled image is generated.

[0028] Further, in the main branch network, a cross-entropy loss function is used to compare the segmentation result of the image segmentation network with the difference between the pseudo-labeling map generated in the auxiliary branch network, and based on the cross-entropy loss function, a loss function of the weakly supervised image segmentation network and a loss function of the auxiliary branch network are constructed.

[0029] Further, the prediction probability of each class is obtained by acquiring the output value of the softmax function before classification, and the prediction probability value is the confidence of each class, and the class of each pixel point in the pseudo-labeling map is the class where the maximum prediction probability is located in all classes.

[0030] In the pseudo-labeling map, only the pixels with high confidence are used to calculate the cross-entropy loss, and the pixels with low confidence do not affect the loss value, and the confidence threshold calculation formula is as follows:

[0031]

[0032] In the formula, T real represents the confidence threshold for distinguishing high-confidence pixels from low-confidence pixels, T high represents the highest confidence of all pixels in the pseudo-labeling map, T low represents the lowest confidence of all pixels in the pseudo-labeling map.

[0033] Further, the process of determining the loss function of the entire weakly supervised image segmentation network segmentation result is as follows:

[0034] The cross-entropy loss of the dashed line labeling and the segmentation result image of the image segmentation network is calculated, denoted as L rea , wherein the prediction class of the segmentation result of the pixel not labeled by the dashed line does not affect the loss value, the cross-entropy loss of the pseudo-labeling map and the segmentation result image of the image segmentation network is calculated, denoted as L pse .

[0035] Based on L rea and L pse , a composite supervision loss function L mas is defined as the loss function of the entire weakly supervised image segmentation network segmentation result, and the definition is as follows:

[0036]

[0037] Through the composite supervision loss function L mas , the parameters of the entire weakly supervised image segmentation network are continuously optimized.

[0038] Further, the loss function of the auxiliary branch network is as follows:

[0039] L aux = L cfl + L scr,

[0040] In the formula, L scr represents the cross-entropy loss between the pseudo-labeled image and the line-labeled image, L cfl represents the context feature difference loss function, L cfl The calculation formula is as follows:

[0041]

[0042] In the formula, D m represents the pixel set of the last batch of line-labeled images, |D m | represents the number of elements in D m represents the pixel set of the last batch of line-labeled images of class k; represents the number of elements in D k c represents the context feature difference loss of class k, and the calculation formula is as follows:

[0043]

[0044] In the formula, a ij represents the class relationship of pixels i and j, and is 0 if the classes are the same and is 2 if the classes are different; MSE represents the mean square error loss function; Dc represents the pixel set of the current batch of line-labeled images; |D c | represents the number of elements in D c cos (A, B) represents the cosine distance between features A and B, and A and B are image features after feature embedding of pixels i and j.

[0045] Compared with the prior art, the present application has the following beneficial effects:

[0046] The weakly supervised neural network image segmentation method based on a small amount of sample line-labeled images provided by the present application generates effective pseudo-labeled images by the main branch network and the auxiliary branch network of the weakly supervised image segmentation network and the construction of a new context feature difference loss function, thereby improving the network training effect in a small sample environment.

[0047] The present application realizes network training by user labeling part of pixels according to domain knowledge, and constructs an image segmentation model meeting the user's own requirements.

[0048] The present application can realize accurate segmentation of complex texture images in a small sample scenario, and the method has universality. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 ​​A flowchart of an image segmentation method of a weakly supervised neural network based on a small amount of sample line marking is provided for the embodiment.

[0050] Figure 2 For Figure 1 A structure diagram of the weakly supervised image segmentation network in the embodiment is shown. DETAILED DESCRIPTION

[0051] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the implementation cases of the present application, not all implementation cases. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should belong to the protection scope of the present application.

[0052] The present application provides an image segmentation method of a weakly supervised neural network based on a small amount of sample line marking, comprising: a weakly supervised image segmentation network constructed including a main branch network and an auxiliary branch network connected with each other; the main branch network is used to extract image features of an input image based on a small amount of pixel line marking, and outputs an image segmentation result; the auxiliary branch network is used to sequentially perform feature embedding and distance measurement on the image features extracted by the main branch network, and generate a pseudo-label map; the pseudo-label map is used to supervise the image segmentation result of the main branch network, and optimize the parameters of the weakly supervised image segmentation network.

[0053] The weakly supervised image segmentation network constructed has the characteristics of containing two branch networks, which are a main branch network and an auxiliary branch network:

[0054] (a) Main branch network structure:

[0055] The main branch network is a Unet++ network, which is mainly responsible for completing segmentation prediction and extracting image features, including five encoder modules and ten decoder modules. The Unet++ network extracts image features through multi-layer feature combination, and the output of the Unet++ network is an image segmentation result.

[0056] When the main branch network extracts image features, the four output feature maps of the Unet++ network deep supervision and the output feature map from the first encoder module to the output are up-sampled by bilinear interpolation, so that the size of the five output feature maps is the same as that of the input image. The five output feature maps are spliced by channels, and then the feature extraction of the input image is completed, and the formula is as follows:

[0057]

[0058] where fea is the extracted training image feature, R(·) is a bilinear interpolation up-sampling operation, is a channel-wise concatenation operation, f 0 is the output feature map of the first encoder module from input to output, 1 , f 2 , f 3 , f 4 are four output feature maps of the Unet++ network deep supervision, respectively.

[0059] In the training phase, the input image set is divided into multiple small batch data sets, and for the image features extracted for each small batch data set, the features corresponding to the pixels marked by the line are divided into the features cf l corresponding to the marked pixels and the features cf u corresponding to the unmarked pixels. l u satisfy the following relationship:

[0060] fea = cf l ∪ cf u

[0061] The features cf l corresponding to the marked pixels are saved as mf l In the image training process of each small batch data set, cf l and cf u of the current small batch data set and mf l of the previous small batch data set are input into the auxiliary branch network for feature embedding.

[0062] (b) Auxiliary branch network structure:

[0063] The auxiliary branch network receives the image features extracted by the main branch network for feature embedding and distance measurement to generate a pseudo-labeled map.

[0064] In the auxiliary branch network, cf l and cf u of the current small batch data set and mf l of the previous small batch data set are embedded by a convolution layer with a kernel size of n, so that the image features are embedded into a feature space with the same dimension size.

[0065] cf l and cf u ​The distance metric is performed to obtain the average cosine distance between the unlabeled pixel points and the features of the line marking pixel points of each category, and the category with the smallest average cosine distance is selected as the category of the unlabeled pixel points in the pseudo-labeled image, thereby generating the corresponding pseudo-labeled image. For features A and B, the cosine distance between the features is calculated according to the following formula:

[0066] d cos (A, B) = 1 - s cos (A, B),

[0067] In the formula, d cos (A, B) represents the cosine distance between features A and B, s cos (A, B) represents the cosine similarity between features A and B, s cos (A, B) is calculated according to the following formula:

[0068]

[0069] In the formula, n represents the dimension of the feature vector, A g and B g represent the elements in the feature vector A and the feature vector B, respectively. For the pixel features with line marking, the theoretical cosine distance can be determined by the categories of the two pixel points. When the categories of the two pixels are the same, the theoretical cosine distance between the two pixels is 0; when the categories of the two pixels are different, the theoretical cosine distance between the two pixels is 2.

[0070] The image segmentation method specifically includes the following steps:

[0071] 1) Line marking on the training image:

[0072] The labeling of the training image is completed by drawing multiple curves of arbitrary length on the image. In the training image, in order to distinguish the areas corresponding to different categories, multiple continuous curves are used to cover the areas where each category is located. The line marking of the continuous curve on the training image realizes the provision of range, position and category information.

[0073] 2) Data augmentation on the labeled training image:

[0074] Through the image enhancement method, including rotation, folding, cutting and brightness contrast transformation, the input image set and input label are data augmented, so that the training data is more abundant. Among them, the calculation formula of brightness contrast transformation on the image is as follows:

[0075] y = [x - 127.5 * (1 - B)] * k + 127.5 * (1 + B)

[0076] In the formula, x is the pixel value before transformation, y is the pixel value after transformation, B is a constant with a value of [-1, 1], used to transform brightness, k is used to transform contrast, and arctan(k) has a value of [1, 89]. The formula for calculating k is as follows:

[0077] k = tan((45 + 44 * c) / 180 * π)

[0078] In the formula, the value of c is [-1, 1].

[0079] 3) Input the enhanced image into the constructed weakly supervised image segmentation network.

[0080] 4) Extract features from training images using the main branch network:

[0081] The augmented training images and annotations are input into the main branch network, and the Unet++ network of the main branch is used to extract image features. For the five extracted feature maps, bilinear interpolation is used to resize the images to the size of the input training images. Then, the five feature maps are concatenated by channel as the image features extracted by the main branch.

[0082] 5) Generate pseudo-annotated images of the training images using an auxiliary branch network:

[0083] The image features extracted by the main branch network are input into the auxiliary branch network. The embedding features mf of the line annotation portion of the previous mini-batch dataset are calculated. l Embedded features of line annotations in the current mini-batch dataset images (cf) l By calculating the cosine distance between pixels and comparing the categories of pairs of pixels with underlined labels, the convolution kernel parameters used for feature embedding can be optimized.

[0084] cf for the current mini-batch dataset l and CF u Distance measurement is performed to obtain the average cosine distance between unlabeled pixels and the features of line-labeled pixels of each category. The category with the smallest average cosine distance is selected as the category of the unlabeled pixel in the pseudo-labeled map, and then the corresponding pseudo-labeled map is generated.

[0085] 6) Design a loss function to optimize the parameters of the image segmentation network:

[0086] In the main branch network, the cross-entropy loss function is used to compare the segmentation results of the image segmentation network with the pseudo-annotated maps generated in the auxiliary branch network. Based on the cross-entropy loss function, loss functions for the weakly supervised image segmentation network and the auxiliary branch network are constructed. The formula for calculating the cross-entropy loss function is as follows:

[0087]

[0088] In the formula, L represents the cross-entropy loss, and Ⅳ represents the number of high-confidence pixels in the pseudo-labeled image. i Let y represent the cross-entropy of pixel i, M represent the number of classes, and y represent the cross-entropy of pixel i. ik The sign function (0 or 1) is used; it takes the value 1 if the true class of pixel i is equal to k, and 0 otherwise. ik This represents the predicted probability that pixel i belongs to category k.

[0089] The predicted probability of each category is obtained by acquiring the output value of the softmax function in the pseudo-annotated image before classification. The predicted probability value is the confidence level of each category. The category of each pixel in the pseudo-annotated image is the category with the highest predicted probability among all categories. The formula for calculating the softmax function is as follows:

[0090]

[0091] In the formula, M represents the number of predicted categories, v is the prediction vector, and v h Let be the value of the h-th category in v, and k represent the category to be calculated. The calculation result is between 0 and 1, and the sum of the softmax values ​​of all categories is 1.

[0092] In the pseudo-annotated image, only high-confidence pixels are used to calculate the cross-entropy loss, while low-confidence pixels do not affect the loss value. The confidence threshold is calculated using the following formula:

[0093]

[0094] In the formula, T real T represents the confidence threshold used to distinguish between high-confidence pixels and low-confidence pixels. high T represents the highest confidence level among all pixels in the pseudo-annotated image. low This represents the lowest confidence level among all pixels in the pseudo-annotated image.

[0095] The process of determining the loss function for the entire weakly supervised image segmentation network segmentation result is as follows:

[0096] Calculate the cross-entropy loss between the line-marked image and the segmentation result image from the image segmentation network, using L... rea This indicates that the predicted category of the segmentation result for pixels not marked with lines does not affect the loss value. The cross-entropy loss between the pseudo-labeled image and the segmentation result image from the image segmentation network is calculated, using L... pse express,

[0097] Based on L rea and L ose Define the composite supervised loss function L mas The loss function, which serves as the segmentation result of the entire weakly supervised image segmentation network, is defined as follows:

[0098]

[0099] Through the composite supervision loss function L mas Continuously optimize the parameters of the entire weakly supervised image segmentation network.

[0100] The loss function for the auxiliary branch network is shown below:

[0101] L aux =L cfl +L scr ,

[0102] In the formula, L scr L represents the cross-entropy loss between the pseudo-annotated map and the line-marked map. cfl L represents the context feature difference loss function. cfl The calculation formula is as follows:

[0103]

[0104] In the formula, D m |D represents the set of pixels from the previous batch of line annotations. m | represents D m The number of elements in the set This represents the set of pixels of category k in the previous batch of line annotations; express The number of elements in the set; M represents the number of categories to be divided, L k The contextual feature difference loss for category k is calculated using the following formula:

[0105]

[0106] In the formula, a ij The value represents the class relationship between pixels i and j, 0 if they are in the same class, and 2 if they are in different classes; MSE represents the mean squared error loss function; Dc represents the set of pixels annotated in the current batch; |D c | represents D c The number of elements in the set; d cos (A, B) represents the cosine distance between features A and B, where A and B are the image features after embedding the features of pixels i and j.

[0107] 7) Use the main branch of the optimized image segmentation network to segment the test image:

[0108] During the testing and practical application phase, the image to be segmented is input into the image segmentation network, and only the main branch structure is used to segment the input image to obtain the segmentation result.

[0109] refer toFigure 1 and Figure 2 As shown in the accompanying drawings, the preferred embodiments of the present invention are described in detail below:

[0110] Example 1:

[0111] This invention discloses an image segmentation method based on a weakly supervised neural network with limited sample line annotations. The proposed weakly supervised neural network comprises two interconnected subnetworks: a main branch and an auxiliary branch. The main branch extracts image features from the input image and outputs the image segmentation result. The auxiliary branch performs feature embedding and distance measurement on the extracted image features sequentially, generating a pseudo-annotation map. This pseudo-annotation map supervises the image segmentation result of the main branch network, optimizing network parameters to obtain more accurate segmentation results. The image segmentation method includes the following steps: first, line annotations are applied to the training image; then, data augmentation is performed on the annotated training image; next, the augmented image is input into the constructed weakly supervised image segmentation network; next, features of the training image are extracted through the main branch network; then, a pseudo-annotation map of the training image is generated through the auxiliary branch network; then, a loss function is designed to optimize the parameters of the image segmentation network; finally, the optimized main branch of the image segmentation network is used to segment the test image.

[0112] Example 2:

[0113] This embodiment is basically the same as Embodiment 1, except that the training images are marked with lines:

[0114] A small image dataset is obtained and divided into training and testing sets. Line annotation and data augmentation are performed on the training set to obtain the input labels and input image set. Line annotation on the training set is achieved by drawing multiple curves of arbitrary length on each image. In the training set, to distinguish regions corresponding to different categories, multiple continuous curves are used to cover the regions of each category. The continuous curve annotations on the training images provide range, location, and category information.

[0115] Example 3:

[0116] This embodiment is basically the same as Embodiment 1, except that it performs data augmentation on the labeled training images:

[0117] Data augmentation is performed on the training set using image enhancement methods, including rotation, flipping, cropping, and brightness / contrast transformation, to enrich the training data for both the input image set and the input labels. The calculation formula for brightness / contrast transformation is shown below:

[0118] y=[x-127.5*(1-B)]*k+127.5*(1+B)

[0119] In the formula, x is the pixel value before transformation, y is the pixel value after transformation, B is a constant with a value of [-1, 1], used to transform brightness, k is used to transform contrast, and arctan(k) has a value of [1, 89]. The formula for calculating k is as follows:

[0120] k = tan((45 + 44 * c) / 180 * π)

[0121] In the formula, the value of c is [-1, 1].

[0122] Example 4:

[0123] This embodiment is basically the same as Embodiment 1, except that the enhanced image is input into the constructed weakly supervised image segmentation network:

[0124] A weakly supervised image segmentation network is constructed, using the input image set and input labels as inputs to train the network. The trained network is designed to accurately segment images using only the main branch. The weakly supervised image segmentation network consists of interconnected main and auxiliary branch networks. The main branch network extracts image features from the input image and outputs the image segmentation result. The auxiliary branch network performs feature embedding and distance measurement on the extracted image features, generating a pseudo-label map. This pseudo-label map supervises the image segmentation result of the main branch network, optimizing the parameters of the weakly supervised image segmentation network to ensure accurate image segmentation. As an optional implementation, the main branch network is a Unet++ network, including 5 encoder modules and 10 decoder modules. The segmentation prediction result of the main branch network is obtained from the output of the Unet++ network, and the extracted image features are obtained by combining multiple feature layers in the Unet++ network. As an optional implementation, in the auxiliary branch network, convolutional layers with kernel size n are used to perform cross-cutting on the current mini-batch. l and CF u Compared to the previous mini-batch mf l Feature embedding is performed, preferably with n=3, so that all the above image features are embedded into the same dimensional feature space, such as a 32-dimensional feature space.

[0125] Example 5:

[0126] This embodiment is basically the same as Embodiment 1, except that it extracts features from the training images through the main branch network:

[0127] When extracting image features, the main branch network upsamples the four output feature maps of the Unet++ deep supervised network and the output feature map of the first encoder module from input to output using bilinear interpolation. This ensures that the size of the five output feature maps is the same as the size of the input image. The five output feature maps are then concatenated by channel to complete the feature extraction from the input image. The formula is as follows:

[0128]

[0129] In the formula, fea represents the extracted training image features, and R(·) represents the bilinear interpolation upsampling operation. For channel-based splicing operations, f 0 f is the output feature map of the first encoder module from input to output. 1 f 2 f 3 f 4 These are the four output feature maps of the Unet++ network's deep supervision.

[0130] During the training phase, the input image set is divided into multiple mini-batches. For each mini-batch, the extracted image features are categorized into labeled features (cf) based on whether the corresponding pixels are marked with lines. l The feature cf corresponding to the unlabeled pixel u The feature cf corresponding to the labeled pixels l The feature cf corresponding to the unlabeled pixel u The following relationship must be satisfied:

[0131] fea=cf l ∪cf u

[0132] The feature cf corresponding to the labeled pixels l Save as mf l During the image training process of each mini-batch, the current mini-batch's cross-cutting vector (CF) is... l and CF u Compared to the previous mini-batch mf l All inputs are fed into the auxiliary branch network for feature embedding.

[0133] As an optional implementation, in the auxiliary branch network, the current mini-batch's cross-validation (CF) is improved by using convolutional layers with kernel size n. l and CF u Compared to the previous mini-batch mf l Feature embedding is performed, preferably with n=3, so that all the above image features are embedded into the same dimensional feature space, such as a 32-dimensional feature space.

[0134] Example 6:

[0135] This embodiment is basically the same as Embodiment 1, except that it generates pseudo-annotated images of the training images through an auxiliary branch network:

[0136] In the auxiliary branch, distance metrics are performed on the image features after feature embedding. This applies to the current mini-batch's cross-referencing (CF). l and CF u After feature embedding, distance measurement is performed to obtain the average cosine distance between unlabeled pixels and the line-labeled pixels of each category. The category with the smallest average cosine distance is selected as the category of the unlabeled pixel in the pseudo-label map, and then the corresponding pseudo-label map is generated.

[0137] To describe the similarity between features, cosine distance is used to measure the distance between features.

[0138] For features A and B, the formula for calculating the cosine distance between features is as follows:

[0139] d cos (A, B) = 1 - s cos (A, B),

[0140] In the formula, d cos (A, B) represents the cosine distance between features A and B, s cos (A, B) represents the cosine similarity between features A and B, s cos The formula for calculating (A, B) is as follows:

[0141]

[0142] In the formula, n represents the dimension of the feature vector, and A g and B g These represent the elements in eigenvector A and eigenvector B, respectively.

[0143] For pixel features with dashed labels, the theoretical cosine distance can be determined by the categories of the two pixels. When the two pixels belong to the same category, the theoretical cosine distance between them is 0; when the two pixels belong to different categories, the theoretical cosine distance between them is 2.

[0144] Example 7:

[0145] This embodiment is basically the same as Embodiment 1, except that it designs a loss function to optimize the parameters of the image segmentation network:

[0146] The generated pseudo-annotation map is used to supervise the image segmentation results of the main branch network. The loss function is determined based on the pseudo-annotation map, and the parameters of the weakly supervised image segmentation network are optimized.

[0147] The process of determining the loss function is as follows:

[0148] During training, cross-entropy loss is used in the main branch network to compare the segmentation results of the main branch network with the pseudo-labeled image; the calculation formula of the cross-entropy loss function is as follows:

[0149]

[0150] In the formula, L represents the cross-entropy loss, and Ⅳ represents the number of high-confidence pixels in the pseudo-labeled image. i Let y represent the cross-entropy of pixel i, M represent the number of classes, and y represent the cross-entropy of pixel i. ik The sign function (0 or 1) is used; it takes the value 1 if the true class of pixel i is equal to k, and 0 otherwise. ik This represents the predicted probability that pixel i belongs to category k.

[0151] For the pseudo-labeled map output by the auxiliary branch network, the predicted probability of each class is obtained by acquiring the output value of the softmax function before classification. This predicted probability value can be represented as the confidence score of each class. The class of each pixel in the pseudo-labeled map is the class with the highest predicted probability among all classes. The formula for calculating the softmax function is as follows:

[0152]

[0153] In the formula, M represents the number of predicted categories, v is the prediction vector, and v h Let be the value of the h-th category in v, and k represent the category to be calculated. The calculation result is between 0 and 1, and the sum of the softmax values ​​of all categories is 1.

[0154] In the pseudo-annotated image, only high-confidence pixels are used to calculate the cross-entropy loss, while low-confidence pixels do not affect the loss value. The confidence threshold is calculated using the following formula:

[0155]

[0156] In the formula T real T represents the confidence threshold used to distinguish between high-confidence pixels and low-confidence pixels. high T represents the highest confidence level among all pixels in the pseudo-annotated image. low This represents the lowest confidence level among all pixels in the pseudo-annotated image.

[0157] The composite supervised loss function is determined as the loss function for the entire weakly supervised image segmentation network's segmentation results:

[0158] Calculate the cross-entropy loss between the line-marked image and the segmentation result image from the image segmentation network, using L...rea This indicates that the predicted category of pixels not marked with lines does not affect the loss value. The cross-entropy loss between the pseudo-annotated image and the segmentation result image from the image segmentation network is calculated, using L... pse express.

[0159] Based on L rea and L pse Define the composite supervised loss function L mas As the loss function for the entire image segmentation network's segmentation result, it is defined as follows:

[0160]

[0161] Through the composite supervision loss function L mas We continuously optimize the parameters of the pseudo-annotated image segmentation network.

[0162] Determine the loss function for the auxiliary branch network:

[0163] During training, to optimize the neural network parameters used to generate pseudo-labeled maps in the auxiliary branch network, the loss function is defined as follows:

[0164] L aux =L cfl +L scr ,

[0165] In the formula, L scr L represents the cross-entropy loss between the pseudo-annotated map and the line-marked map. cfl The context feature difference loss function is calculated as follows:

[0166]

[0167] In the formula, D m |D represents the set of pixels from the previous batch of line annotations. m | represents D m The number of elements in the set This represents the set of pixels of category k in the previous batch of line annotations; express The number of elements in the set; M represents the number of categories to be divided, L k The contextual feature difference loss for category k is calculated using the following formula:

[0168]

[0169] In the formula, a ij The value represents the class relationship between pixels i and j; 0 indicates the class is the same, and 2 indicates the class is different. MSE represents the mean squared error loss function. D c Represents the set of pixels currently labeled with dashes; |Dc | represents D c The number of elements in the set; d cos (A, B) represents the cosine distance between features A and B, where A and B are the image features after embedding the features of pixels i and j.

[0170] Example 8:

[0171] This embodiment is basically the same as Embodiment 1, except that it uses the main branch of the optimized image segmentation network to segment the test image:

[0172] The test set is input into the trained weakly supervised image segmentation network, and the input image is segmented using only the main branch structure to generate the image segmentation result.

[0173] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. An image segmentation method based on a weakly supervised neural network with a small number of sample line annotations, characterized in that, include: 1) Obtain training images and add line annotations to them; 2) Perform data augmentation on the labeled training images; 3) The enhanced image is input into a pre-constructed weakly supervised image segmentation network, which includes interconnected main branch networks and auxiliary branch networks; 4) Extract image features from the input image based on the pixels marked with lines using the main branch network, and output the image segmentation result; 5) The image features extracted by the main branch network are sequentially embedded and distance measured by the auxiliary branch network to generate a pseudo-label map of the training image. This pseudo-label map is used to supervise the image segmentation results of the main branch network and optimize the parameters of the weakly supervised image segmentation network. 6) Iteratively optimize the parameters of the weakly supervised image segmentation network using the loss function; 7) Use the main branch network in the optimized weakly supervised image segmentation network to segment the image.

2. The image segmentation method based on a weakly supervised neural network with a small number of sample line annotations as described in claim 1, characterized in that, The constructed weakly supervised image segmentation network is described as follows: The main branch network is a Unet++ network, which includes 5 encoder modules and 10 decoder modules. Image features are extracted through multi-layer feature combination of the Unet++ network, and the output of the Unet++ network is the image segmentation result. The auxiliary branch network comprises three parts: feature embedding, calculation of feature cosine distance, and generation of pseudo-labeled graphs.

3. The image segmentation method based on a weakly supervised neural network with line annotation using a small number of samples as described in claim 2, characterized in that, The process of the main branch network extracting image features is as follows: The four output feature maps of the Unet++ deep supervised network and the output feature map of the first encoder module from input to output are upsampled using bilinear interpolation, so that the size of the five output feature maps is the same as the size of the input image. The five output feature maps are then concatenated by channel to complete the feature extraction of the input image, as shown in the following formula: In the formula, fea represents the extracted training image features, and R(·) represents the bilinear interpolation upsampling operation. For channel-based splicing operations, f 0 f is the output feature map of the first encoder module from input to output. 1 f 2 f 3 f 4 These are the four output feature maps of the Unet++ network's deep supervision.

4. The image segmentation method based on a weakly supervised neural network with line annotation using a small number of samples as described in claim 1, characterized in that, The image features are described as follows: During the training phase, some pixels in the training image data are labeled using a line-drawing method; the input image set is divided into multiple mini-batch datasets, and for the image features extracted from each mini-batch dataset, they are categorized into features corresponding to labeled pixels based on whether the corresponding pixels are labeled with lines. l The feature cf corresponding to the unlabeled pixel u The feature cf corresponding to the labeled pixels l The feature cf corresponding to the unlabeled pixel u The following relationship must be satisfied: fea=cf l ∪cf u The feature cf corresponding to the labeled pixels l Save as mf l During the image training process of each mini-batch dataset, the current mini-batch dataset's cf l and CF u mf with the previous mini-batch dataset l All inputs are fed into the auxiliary branch network for feature embedding.

5. The image segmentation method based on a weakly supervised neural network with a small number of sample line annotations according to claim 4, characterized in that, The auxiliary branch network performs feature embedding on the extracted image features: exist In the auxiliary branch network, the cross-validation (CF) of the current mini-batch dataset is performed by convolutional layers with kernel size n. l and CF u mf with the previous mini-batch dataset l Feature embedding is performed so that all image features are embedded into a feature space of the same dimension.

6. The image segmentation method based on a weakly supervised neural network with line annotation using a small number of samples according to claim 4, characterized in that, The process by which the auxiliary branch network performs distance measurement on the image features after feature embedding is as follows: cf for the current mini-batch dataset l and CF u Distance measurement is performed to obtain the average cosine distance between unlabeled pixels and the features of line-labeled pixels of each category. The category with the smallest average cosine distance is selected as the category of the unlabeled pixel in the pseudo-labeled map, and then the corresponding pseudo-labeled map is generated.

7. The image segmentation method based on a weakly supervised neural network with line annotation using a small number of samples according to claim 1, characterized in that, In the main branch network, the cross-entropy loss function is used to compare the difference between the segmentation results of the image segmentation network and the pseudo-annotated map generated in the auxiliary branch network. Based on the cross-entropy loss function, the loss functions of the weakly supervised image segmentation network and the auxiliary branch network are constructed.

8. The image segmentation method based on a weakly supervised neural network with a small number of sample line annotations according to claim 7, characterized in that, The predicted probability of each category is obtained by acquiring the output value of the softmax function of the pseudo-labeled image before classification. The predicted probability value is the confidence of each category. The category of each pixel in the pseudo-labeled image is the category with the maximum predicted probability among all categories. In the pseudo-annotated image, only high-confidence pixels are used to calculate the cross-entropy loss, while low-confidence pixels do not affect the loss value. The confidence threshold is calculated using the following formula: In the formula, T real T represents the confidence threshold used to distinguish between high-confidence pixels and low-confidence pixels. high T represents the highest confidence level among all pixels in the pseudo-annotated image. low This represents the lowest confidence level among all pixels in the pseudo-annotated image.

9. The image segmentation method based on a weakly supervised neural network with a small number of sample line annotations according to claim 7, characterized in that, The process of determining the loss function for the entire weakly supervised image segmentation network segmentation result is as follows: Calculate the cross-entropy loss between the line-marked image and the segmentation result image from the image segmentation network, using L... rea This indicates that the predicted category of the segmentation result for pixels not marked with lines does not affect the loss value. The cross-entropy loss between the pseudo-labeled image and the segmentation result image from the image segmentation network is calculated, using L... pse express, Based on L rea and L ose Define the composite supervised loss function L mas The loss function, which serves as the segmentation result of the entire weakly supervised image segmentation network, is defined as follows: Through the composite supervision loss function L mas Continuously optimize the parameters of the entire weakly supervised image segmentation network.

10. The image segmentation method based on a weakly supervised neural network with line annotation using a small number of samples according to claim 7, characterized in that, The loss function for the auxiliary branch network is shown below: L aux =L cfl +L scr , In the formula, L scr L represents the cross-entropy loss between the pseudo-annotated map and the line-marked map. cfl L represents the context feature difference loss function. cfl The calculation formula is as follows: In the formula, D m |D represents the set of pixels from the previous batch of line annotations. m | represents D m The number of elements in the set This represents the set of pixels of category k in the previous batch of line annotations; express The number of elements in the set; M represents the number of categories to be divided, L k The contextual feature difference loss for category k is calculated using the following formula: In the formula, a ij The value represents the class relationship between pixels i and j; 0 indicates the class is the same, and 2 indicates the class is different. MSE represents the mean squared error loss function. D c Represents the set of pixels currently labeled with dashes; |D c | represents D c The number of elements in the set; d cos (A, B) represents the cosine distance between features A and B, where A and B are the image features after embedding the features of pixels i and j.

Images