Image segmentation method based on semi-supervised learning and related device

Abstract

The application provides an image segmentation method and device based on semi-supervised learning, an electronic device and a storage medium. The image segmentation method based on semi-supervised learning comprises the following steps: collecting labeled images as a first image set and collecting unlabeled images as a second image set; training an image segmentation network based on the first image set to obtain a first segmentation network, and segmenting the second image set based on the first segmentation network to obtain first pseudo labels of the unlabeled images; performing voting based on the first pseudo labels to obtain a voting map of each pixel category of the corresponding unlabeled image; calculating the credibility of each pixel point in the corresponding unlabeled image based on the first pseudo labels; updating the first pseudo labels based on the credibility and the voting map to obtain second pseudo labels of the unlabeled images; and training the first segmentation network based on the first image set and the second image set with the second pseudo labels to obtain a second segmentation network. The application can improve the accuracy of the pseudo labels and thus improve the precision of image segmentation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an image segmentation method and device based on semi-supervised learning, an electronic device and a storage medium. BACKGROUND

[0002] Image segmentation is one of the basic tasks in the field of artificial intelligence and has been widely used in many fields such as digital medicine and smart city. Image segmentation under full supervision requires a large amount of manpower and resources for pixel-level manual annotation of all images. However, semi-supervised image segmentation can achieve image segmentation using only a small amount of manually annotated images and a large number of unlabeled images, which can greatly reduce the consumption of manpower and funds caused by image annotation in practical applications.

[0003] Currently, semi-supervised image segmentation usually first trains a segmentation network using labeled images, then inputs unlabeled images into the segmentation network to generate pseudo-labels, and then adds images with pseudo-labels to the training to obtain a trained segmentation network. However, the segmentation network trained by a small amount of data cannot accurately generate pseudo-labels for unlabeled images, and since there may be a large gap between labeled images and unlabeled images, the accuracy of the pseudo-labels is low, which further leads to low accuracy of image segmentation. SUMMARY

[0004] In view of the above, it is necessary to propose an image segmentation method based on semi-supervised learning and related devices to solve the technical problem of how to improve the accuracy of pseudo-labels and thus improve the accuracy of image segmentation, wherein the related devices include an image segmentation device based on semi-supervised learning, an electronic device and a storage medium.

[0005] The present application provides an image segmentation method based on semi-supervised learning, which comprises:

[0006] Collecting labeled images with label data as a first image set and collecting unlabeled images without label data as a second image set, wherein the label data includes the pixel category of each pixel point in the labeled image;

[0007] Training an image segmentation network based on the first image set to obtain a first segmentation network, and inputting the unlabeled images in the second image set into the first segmentation network to obtain a first pseudo-label for each unlabeled image;

[0008] Voting based on the first pseudo-label to obtain a voting map of the corresponding unlabeled image on each pixel category, wherein the voting map includes the voting result of the pixel category corresponding to the voting map at each pixel point in the unlabeled image;

[0009] calculate a confidence of each pixel point in the corresponding unlabeled image based on the first pseudo label;

[0010] update the first pseudo label based on the confidence and the voting map to obtain a second pseudo label of each unlabeled image;

[0011] train the first segmentation network based on the first image set and the second image set with the second pseudo label to obtain a second segmentation network, an input of the second segmentation network being a to-be-segmented image, and an output of the second segmentation network being a segmentation result of the to-be-segmented image.

[0012] In some embodiments, the first pseudo label comprises a first probability vector of each pixel point in the unlabeled image, the first probability vector comprising a first probability of the pixel point belonging to each pixel category.

[0013] In some embodiments, the voting based on the first pseudo label to obtain a voting map of the corresponding unlabeled image in each pixel category comprises:

[0014] taking any one pixel point in any one first pseudo label as a target pixel point;

[0015] voting in a neighborhood range of the target pixel point to obtain a voting result of each pixel category at the target pixel point, the voting result satisfying a relationship formula:

[0016]

[0017] wherein, ROI (x，y) is the neighborhood range of the target pixel point (x, y), N is the number of all pixel categories, a(i, n) is a voting coefficient, represents a first probability of a pixel point (x (x，y) , y * ) in the ROI * belonging to the pixel category i in the first probability vector, Num(ROI (x，y) ) represents the number of all pixel points in the ROI (x，y) , is the voting result of the pixel category n at the target pixel point (x, y), and the voting coefficient satisfies a relationship formula:

[0018] traversing all pixel points in the first pseudo label to obtain a voting result of each pixel category at each pixel point;

[0019] selecting the voting results of the same pixel category from all pixel points to construct a voting map of the corresponding unlabeled image in each pixel category corresponding to the first pseudo label.

[0020] In some embodiments, the calculating the confidence of each pixel point in the corresponding unlabeled image based on the first pseudo label comprises:

[0021] taking any one pixel point in any one first pseudo label as a target pixel point;

[0022] calculating the confidence of the target pixel point based on the first probability vector of the target pixel point, the confidence satisfying a relationship:

[0023]

[0024] wherein N is the number of all pixel categories, is the first probability of the first probability vector of the target pixel point (x, y) belonging to pixel category i, and Conf(x, y) is the confidence of the target pixel point (x, y), taking a value in the range of [0, 1];

[0025] traversing all pixel points in the first pseudo label to obtain the confidence of each pixel point in the corresponding unlabeled image.

[0026] In some embodiments, the updating the first pseudo label based on the confidence and the voting map to obtain a second pseudo label of each unlabeled image comprises:

[0027] for each unlabeled image, selecting a voting result of a pixel point to be updated from the voting map of all pixel categories as a voting result set, the pixel point to be updated being any one pixel point in the unlabeled image;

[0028] selecting a first probability vector of the pixel point to be updated from the first pseudo label of the unlabeled image;

[0029] updating the first probability vector based on the confidence of the pixel point to be updated and the voting result set to obtain a second probability vector of the pixel point to be updated, the second probability vector comprising second probabilities of the pixel point to be updated belonging to various pixel categories, the second probabilities satisfying a relationship:

[0030]

[0031] wherein Conf(x', y') is the confidence of the pixel point to be updated (x', y'), is the first probability of the first probability vector belonging to pixel category i, t i (x', y') is the voting result of pixel category i in the voting result set, is the second probability of the pixel point to be updated (x', y') belonging to pixel category i;

[0032] all the second probability vectors of the pixels in the unlabeled image as the second pseudo label of the unlabeled image.

[0033] In some embodiments, the training of the first segmentation network based on the first image set and the second image set with the second pseudo label to obtain a second segmentation network comprises:

[0034] randomly selecting a preset number of training images from the first image set and the second image set with the second pseudo label as a training batch, the training images including labeled images and unlabeled images;

[0035] inputting the training images in the training batch into the first segmentation network to obtain the segmentation results of the pixels in each training image;

[0036] calculating the value of the cost function based on the segmentation results of the pixels;

[0037] updating the first segmentation network according to the gradient descent method to reduce the value of the cost function;

[0038] continuously obtaining new training batches from the first image set and the second image set with the second pseudo label, updating the first segmentation network until the value of the cost function is less than a preset threshold, and obtaining a second segmentation network.

[0039] In some embodiments, the cost function satisfies the relationship:

[0040]

[0041] wherein Q1 and Q2 represent the first image set and the second image set respectively, N1 and N2 represent the number of training images in the training batch belonging to the first image set and the second image set respectively, W and H are the width and height of the training image, P u (x, y) and P v (x, y) represent the segmentation results of the pixel point (x, y) in the training image u and the training image v in the training batch respectively, represents the pixel category of the pixel point (x, y) in the label data of the training image u, represents the second probability vector of the pixel point (x, y) in the second pseudo label of the training image v, represents the calculation of P u (x, y) and the cross-entropy loss function, represents the calculation of P v (x, y) and the KL divergence, and Loss is the value of the cost function.

[0042] The embodiment of the present application further provides a device for image segmentation based on semi-supervised learning, which comprises:

[0043] An acquisition unit is configured to acquire a label image with label data as a first image set and acquire a non-label image without label data as a second image set, wherein the label data comprises pixel categories of each pixel point in the label image.

[0044] An acquisition unit is configured to train an image segmentation network based on the first image set to obtain a first segmentation network, and input the non-label image in the second image set into the first segmentation network to obtain a first pseudo label of each non-label image.

[0045] A voting unit is configured to vote based on the first pseudo label to obtain a voting map of the corresponding non-label image on each pixel category, wherein the voting map comprises a voting result of the pixel category corresponding to the voting map at each pixel point in the non-label image.

[0046] A calculation unit is configured to calculate a credibility of each pixel point in the corresponding non-label image based on the first pseudo label.

[0047] An update unit is configured to update the first pseudo label based on the credibility and the voting map to obtain a second pseudo label of each non-label image.

[0048] A training unit is configured to train the first segmentation network based on the first image set and the second image set with the second pseudo label to obtain a second segmentation network, wherein an input of the second segmentation network is a to-be-segmented image and an output of the second segmentation network is a segmentation result of the to-be-segmented image.

[0049] The embodiment of the present application further provides an electronic device, which comprises:

[0050] A memory is configured to store at least one instruction.

[0051] A processor is configured to execute the instruction stored in the memory to implement the image segmentation method based on semi-supervised learning.

[0052] The embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the image segmentation method based on semi-supervised learning.

[0053] In summary, the application calculates the credibility of each pixel point in the first pseudo label to evaluate the accuracy of each pixel point on the basis of the first pseudo label, and obtains the voting result of each pixel point by sensing the local information in the neighborhood range of the pixel point in the first pseudo label. Further, based on the voting result, greater disturbance is applied to the pixel point with smaller credibility in the first pseudo label, and smaller disturbance is applied to the pixel point with greater credibility in the first pseudo label, so as to realize the update of the first pseudo label, improve the accuracy of the pseudo label, finally, the first segmentation network is trained for the second time by using the updated pseudo label, and thus the precision of image segmentation is improved. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a flowchart of a preferred embodiment of the image segmentation method based on semi-supervised learning involved in the application.

[0055] Figure 2 is a schematic diagram of the correspondence relationship between the unlabeled image, the first pseudo label, the voting map of the pixel category and the credibility of the pixel point involved in the application.

[0056] Figure 3 is a function module diagram of a preferred embodiment of the image segmentation device based on semi-supervised learning involved in the application.

[0057] Figure 4 is a structural schematic diagram of an electronic device of a preferred embodiment of the image segmentation method based on semi-supervised learning involved in the application. DETAILED DESCRIPTION

[0058] In order to more clearly understand the purpose, features and advantages of the application, the application will be described in detail below in combination with the drawings and specific embodiments. It should be noted that the embodiments of the application and the features in the embodiments can be combined with each other without conflict. In the following description, a large number of specific details are set forth in order to facilitate a full understanding of the application, and the described embodiments are only part of the embodiments of the application, not all embodiments.

[0059] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein in the specification of this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0061] An embodiment of the present application provides an image segmentation method based on semi-supervised learning, which can be applied to one or more electronic devices. The electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0062] An electronic device can be any electronic product that can interact with a user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, Internet Protocol Television (IPTV), smart wearable device, etc.

[0063] The electronic device may also include a network device and / or a client device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0064] The network where the electronic device is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0065] like Figure 1 FIG. 1 is a flowchart of a preferred embodiment of the image segmentation method based on semi-supervised learning of the present invention. The order of the steps in the flowchart can be changed according to different requirements, and some steps can be omitted.

[0066] The image segmentation method based on semi-supervised learning provided in the embodiment of the present application can be applied to any scenario that requires image segmentation, and the method can be applied to products in these scenarios, such as tumor segmentation in the medical field, lane line segmentation in the smart city field, etc.

[0067] S10, collect label images with label data as a first image set and collect unlabel images without label data as a second image set, wherein the label data includes pixel categories of each pixel point in the label images.

[0068] In an optional embodiment, the number of pixel categories is at least two, the label data of the label images in the first image set is obtained by manual annotation, and the number of unlabel images in the second image set is greater than the number of label images in the first image set.

[0069] In this way, the first image set and the second image set are obtained, the first image set includes label images with label data, and data basis is provided for implementing semi-supervised image segmentation.

[0070] S11, training an image segmentation network based on the first image set to obtain a first segmentation network, and inputting unlabel images in the second image set into the first segmentation network to obtain first pseudo labels of each unlabel image.

[0071] In an optional embodiment, the training of the image segmentation network based on the first image set to obtain the first segmentation network includes:

[0072] randomly selecting a label image from the first image set without replacement and inputting the image into the image segmentation network to obtain a segmentation result;

[0073] calculating a value of a cross-entropy loss function according to the segmentation result and the label data, and updating the image segmentation network by using a gradient descent method;

[0074] continuously selecting label images from the first image set to update the image segmentation network until the value of the cross-entropy loss function is less than a preset value or all label images in the first image set are traversed, and the first segmentation network is obtained.

[0075] The image segmentation network can be an existing semantic segmentation network such as Unet, FCN, DeepLab, and the like, and the present application does not make any limitation; and the preset value is 0.001.

[0076] In an optional embodiment, the first segmentation network can learn a mapping relationship between each pixel point in an image and all pixel categories. Since the number of label images in the first image set is limited, the accuracy of the segmentation result obtained by the first segmentation network is not high, and the segmentation result output by the first segmentation network cannot be directly used for image segmentation, but the segmentation result output by the first segmentation network can reflect the mapping relationship between each pixel point in an unlabel image and different pixel categories to a certain extent.

[0077] In an optional embodiment, the first pseudo label comprises a first probability vector of each pixel point in the unlabeled image, and the first probability vector comprises a first probability of the pixel point belonging to each pixel category.

[0078] The first probability vector corresponds to each pixel point in the unlabeled image, and the size of the first probability vector is N rows by 1 column, where N is the number of all pixel categories; and the sum of all first probabilities in the first probability vector is 1.

[0079] For example, if the number of all pixel categories is 5, that is, all pixel points in the image are divided into 5 categories; and if the size of the unlabeled image is 3x3, the first pseudo label corresponding to the unlabeled image comprises 9 first probability vectors, and the size of the first probability vector is 5 rows by 1 column, and the first probability vector comprises a first probability of the corresponding pixel point belonging to 5 pixel categories.

[0080] In this way, by means of the first segmentation network trained in the first image set, the first pseudo label of each unlabeled image in the second image set is obtained, and the first pseudo label can reflect the mapping relationship between each pixel point in the unlabeled image and different pixel categories.

[0081] S12, voting based on the first pseudo label to obtain a voting map of the corresponding unlabeled image on each pixel category, the voting map comprising a voting result of the pixel category corresponding to the voting map at each pixel point in the unlabeled image.

[0082] In an optional embodiment, the voting based on the first pseudo label to obtain a voting map of the corresponding unlabeled image on each pixel category comprises:

[0083] Taking any pixel point in any first pseudo label as a target pixel point;

[0084] Voting in the neighborhood range of the target pixel point to obtain a voting result of each pixel category at the target pixel point, the voting result satisfying the relationship:

[0085]

[0086] wherein, ROI (x，y) is the neighborhood range of the target pixel point (x, y), N is the number of all pixel categories, and a(i, n) is a voting coefficient, represents the first probability of the pixel point (x (x，y) , y * ) in the first probability vector of the pixel point (x * , y (x，y) ) in the ROI (x，y)the number of all pixel points, is the voting result of the pixel category n at the target pixel point (x, y), and the voting coefficient satisfies the relationship:

[0087] traversing all pixel points in the first pseudo label to obtain the voting result of each pixel category at each pixel point;

[0088] selecting the voting results of the same pixel category from all pixel points to construct the voting map of the unlabeled image corresponding to the first pseudo label on each pixel category.

[0089] wherein the neighborhood range of the target pixel point (x, y) is a rectangular region with the target pixel point (x, y) as the center and the width and height dimensions of h x w, and h and w are pre-set; the voting result is the probability that the pixel point belongs to each pixel category obtained from the local information in the neighborhood range of the reference pixel point.

[0090] In this optional embodiment, all first pseudo labels are traversed to obtain the voting map of each unlabeled image in the second image set on each pixel category. One unlabeled image corresponds to N voting maps, N is the number of all pixel categories, the size of the voting map is the same as that of the unlabeled image, and the voting map of the pixel category n includes the voting result of each pixel point in the unlabeled image on the pixel category n.

[0091] Thus, based on the first pseudo label, the voting result of each pixel point on each pixel category is obtained, and then the voting map of each unlabeled image in the second image set on each pixel category is obtained, and the voting result is the probability value that the pixel point belongs to each pixel category obtained from the local information in the neighborhood range of the reference pixel point.

[0092] S13, calculating the credibility of each pixel point in the corresponding unlabeled image based on the first pseudo label.

[0093] In an optional embodiment, the calculating the credibility of each pixel point in the corresponding unlabeled image based on the first pseudo label comprises:

[0094] taking any one pixel point in any one first pseudo label as a target pixel point;

[0095] calculating the credibility of the target pixel point based on the first probability vector of the target pixel point, and the credibility satisfies the relationship:

[0096]

[0097] wherein N is the number of all pixel categories, a first probability of the target pixel point (x, y) belonging to a pixel category i, and Conf(x, y) is a confidence of the target pixel point (x, y) and ranges from 0 to 1.

[0098] All pixel points in the first pseudo label are traversed to obtain the confidence of each pixel point in the unlabeled image corresponding to the first pseudo label.

[0099] The confidence can reflect the accuracy of the first probability vector of the pixel point in the first pseudo label, and the greater the value of the confidence, the higher the accuracy of the first probability vector. For example, assuming that the number of all pixel categories is 4, if the first probability vector of the target pixel point (x, y) is [0.1, 0.3, 0, 0.6], then the confidence of the target pixel point (x, y) is:

[0100]

[0101] In this optional embodiment, all first pseudo labels are traversed to obtain the confidence of each pixel point in the unlabeled image corresponding to each first pseudo label.

[0102] In this way, the confidence of the first probability vector of each pixel point in the first pseudo label is accurately quantified.

[0103] S14, updating the first pseudo label based on the confidence and the voting map to obtain a second pseudo label of each unlabeled image.

[0104] In an optional embodiment, each first pseudo label corresponds to an unlabeled image, and each first pseudo label also corresponds to a voting map of each pixel category and a confidence of each pixel point. The correspondence between the unlabeled image, the first pseudo label, the voting map of the pixel category, and the confidence of the pixel point is as shown in Figure 2

[0105] In an optional embodiment, updating the first pseudo label based on the confidence and the voting map to obtain a second pseudo label of each unlabeled image includes:

[0106] For each unlabeled image, the voting results of the pixel points to be updated are selected from the voting maps of all pixel categories as a voting result set, and the pixel points to be updated are any pixel points in the unlabeled image.

[0107] The first probability vector of the pixel point to be updated is selected from the first pseudo label of the unlabeled image.

[0108] ​updating the first probability vector based on the confidence of the pixel point to be updated and the set of voting results to obtain a second probability vector of the pixel point to be updated, the second probability vector comprising a second probability that the pixel point to be updated belongs to each pixel category, the second probability satisfying a relationship:

[0109]

[0110] wherein, Conf(x', y') is the confidence of the pixel point (x', y') to be updated, is the first probability in the first probability vector that belongs to pixel category i, t i (x', y') is the voting result of pixel category i in the set of voting results, is the second probability that the pixel point (x', y') to be updated belongs to pixel category i;

[0111] traversing all pixel points in the unlabeled image to obtain a second probability vector of each pixel point, and taking the second probability vectors of all pixel points as second pseudo-labels of the unlabeled image.

[0112] In the optional embodiment, each unlabeled image in the second image set corresponds to a second pseudo-label, and the second pseudo-label comprises a second probability vector of each pixel point in the unlabeled image.

[0113] In this way, the first pseudo-label is updated based on the confidence of each pixel point in the unlabeled image and the voting map of each pixel category, and the second pseudo-label of each unlabeled image is obtained without human annotation, and the second pseudo-label comprehensively considers the local information in the neighborhood range of the pixel point and can accurately reflect the mapping relationship between the pixel point in the unlabeled image and each pixel category.

[0114] S15, training the first segmentation network based on the first image set and the second image set with the second pseudo-label to obtain a second segmentation network, the input of the second segmentation network being an image to be segmented, and the output being a segmentation result of the image to be segmented.

[0115] In an optional embodiment, after obtaining the second pseudo-label of each unlabeled image, the first segmentation network is trained a second time based on the first image set and the second image set with the second pseudo-label, and the first segmentation network learns an accurate mapping relationship between the pixel point and each pixel category according to the second pseudo-label.

[0116] In an optional embodiment, the training of the first segmentation network based on the first image set and the second image set with the second pseudo-label to obtain a second segmentation network comprises:

[0117] randomly pick a preset number of training images from the first image set and the second image set with the second pseudo label as a training batch, the training images including labeled images and unlabeled images;

[0118] input the training images in the training batch into the first segmentation network to obtain segmentation results of each pixel point in each training image;

[0119] calculate a value of a cost function based on the segmentation results of the pixel points, the cost function satisfying a relationship:

[0120]

[0121] wherein Q1 and Q2 represent the first image set and the second image set respectively, N1 and N2 represent the number of training images in the training batch belonging to the first image set and the second image set respectively, W and H are the width and height dimensions of the training images, P u (x, y) and Pv(x, y) represent the segmentation results of training image u and training image v in the training batch at pixel point (x, y), represents a pixel category of pixel point (x, y) in the label data of training image u, represents a second probability vector of pixel point (x, y) in the second pseudo label of training image v, represents calculating P u (x, y) and a cross-entropy loss function, represents calculating P v (x, y) and a KL divergence, and Loss is the value of the cost function;

[0122] update the first segmentation network according to the gradient descent method to reduce the value of the cost function;

[0123] constantly obtain new training batches from the first image set and the second image set with the second pseudo label, update the first segmentation network, and stop until the value of the cost function is less than a preset threshold, to obtain a second segmentation network.

[0124] wherein the preset threshold is 0.001, the preset number is 32, that is, the training batch includes 32 images, and the sum of N2 and N1 is equal to the preset number.

[0125] In the optional embodiment, the second segmentation network can learn the accurate mapping relationship between the pixel points in the image and the pixel categories, and the segmentation result of the image to be segmented can be obtained by inputting the image to be segmented into the second segmentation network, wherein the segmentation result includes the pixel category of each pixel point in the image to be segmented.

[0126] In this way, the first segmentation network is trained for the second time by using the labeled image with labeled data and the unlabeled image with the second pseudo label to obtain the second segmentation network, and the second segmentation network can learn the accurate mapping relationship between the pixel points in the image and the pixel categories.

[0127] From the above technical solutions, it can be seen that, based on the first pseudo label, the confidence of each pixel point in the first pseudo label is calculated to evaluate the accuracy of each pixel point, and meanwhile, the local information in the neighborhood range of the pixel point is perceived in the first pseudo label to obtain the voting result of each pixel point. Further, based on the voting result, a greater disturbance is applied to the pixel point with smaller confidence in the first pseudo label, and a smaller disturbance is applied to the pixel point with greater confidence in the first pseudo label, so as to realize the update of the first pseudo label, improve the accuracy of the pseudo label, and finally, the first segmentation network is trained for the second time by using the updated pseudo label, thereby improving the accuracy of image segmentation.

[0128] Please refer to Figure 3 , Figure 3 is a functional module diagram of a preferred embodiment of an image segmentation device based on semi-supervised learning according to the present application. The image segmentation device based on semi-supervised learning 11 includes an acquisition unit 110, an acquisition unit 111, a voting unit 112, a calculation unit 113, an update unit 114, and a training unit 115. The module / unit referred to in the present application refers to a series of computer readable instruction segments that can be executed by the processor 13 and can complete a fixed function, which is stored in the memory 12. In the present embodiment, the functions of each module / unit will be described in detail in the subsequent embodiments.

[0129] In an optional embodiment, the acquisition unit 110 is configured to acquire a labeled image with labeled data as a first image set and an unlabeled image without labeled data as a second image set, and the labeled data includes the pixel category of each pixel point in the labeled image.

[0130] In an optional embodiment, the acquisition unit 111 is configured to train an image segmentation network based on the first image set to obtain a first segmentation network, and input the unlabeled image in the second image set into the first segmentation network to obtain a first pseudo label of each unlabeled image.

[0131] In an optional embodiment, the first pseudo label comprises a first probability vector of each pixel in the unlabeled image, and the first probability vector comprises a first probability of the pixel belonging to each pixel category.

[0132] In an optional embodiment, the voting unit 112 is configured to vote based on the first pseudo label to obtain a voting map of the corresponding unlabeled image on each pixel category, and the voting map comprises a voting result of the pixel category corresponding to the voting map at each pixel in the unlabeled image.

[0133] In an optional embodiment, the voting based on the first pseudo label to obtain a voting map of the corresponding unlabeled image on each pixel category comprises:

[0134] taking any pixel in any first pseudo label as a target pixel;

[0135] voting in a neighborhood range of the target pixel to obtain a voting result of each pixel category at the target pixel, and the voting result satisfies a relationship:

[0136]

[0137] wherein, ROI (x，y) is the neighborhood range of the target pixel (x, y), N is the number of all pixel categories, and a(i, n) is a voting coefficient, represents a first probability of the pixel category i in the first probability vector of the pixel (x (x，y) , y * ) in the ROI * , and Num(ROI (x，y) ) represents the number of all pixels in the ROI (x，y) , is the voting result of the pixel category n at the target pixel (x, y), and the voting coefficient satisfies a relationship:

[0138] traversing all pixels in the first pseudo label to obtain a voting result of each pixel category at each pixel;

[0139] selecting the voting results of the same pixel category from all pixels to construct a voting map of the corresponding unlabeled image on each pixel category corresponding to the first pseudo label.

[0140] In an optional embodiment, the computing unit 113 is configured to calculate a credibility of each pixel in the corresponding unlabeled image based on the first pseudo label.

[0141] In an optional embodiment, calculating the credibility of each pixel in the corresponding unlabeled image based on the first pseudo label includes:

[0142] Take any pixel in any first pseudo label as the target pixel;

[0143] The credibility of the target pixel is calculated based on the first probability vector of the target pixel, and the credibility satisfies the relationship:

[0144]

[0145] Where N is the number of all pixel types, is the first probability of the target pixel point (x, y) belonging to pixel category i in the first probability vector, Conf(x, y) is the credibility of the target pixel point (x, y), and its value range is [0, 1];

[0146] All pixels in the first pseudo-label are traversed to obtain the credibility of each pixel in the corresponding unlabeled image.

[0147] In an optional embodiment, the updating unit 114 is configured to update the first pseudo label based on the credibility and the voting map to obtain a second pseudo label for each unlabeled image.

[0148] In an optional embodiment, updating the first pseudo label based on the credibility and the voting map to obtain a second pseudo label for each unlabeled image includes:

[0149] For each unlabeled image, select the voting results of the pixel to be updated from the voting maps of all pixel types as the voting result set, where the pixel to be updated is any pixel in the unlabeled image;

[0150] Selecting a first probability vector of the pixel to be updated from the first pseudo label of the unlabeled image;

[0151] The first probability vector is updated based on the credibility of the pixel to be updated and the voting result set to obtain a second probability vector for the pixel to be updated, wherein the second probability vector includes a second probability that the pixel to be updated belongs to each pixel type, and the second probability satisfies the relationship:

[0152]

[0153] Wherein, Conf(x′, y′) is the credibility of the pixel point (x′, y′) to be updated, is the first probability of belonging to pixel category i in the first probability vector, t i(x', y') is a voting result of a pixel category i in the voting result set, is a second probability that the pixel point (x', y') belongs to the pixel category i;

[0154] All the second probability vectors of the pixel points in the unlabeled image are taken as second pseudo labels of the unlabeled image.

[0155] In an optional embodiment, the training unit 115 is configured to train the first segmentation network based on the first image set and the second image set with the second pseudo labels to obtain a second segmentation network, wherein an input of the second segmentation network is an image to be segmented, and an output of the second segmentation network is a segmentation result of the image to be segmented.

[0156] In an optional embodiment, the training of the first segmentation network based on the first image set and the second image set with the second pseudo labels to obtain a second segmentation network comprises:

[0157] randomly selecting a preset number of training images from the first image set and the second image set with the second pseudo labels as a training batch, wherein the training images include labeled images and unlabeled images;

[0158] inputting the training images in the training batch into the first segmentation network to obtain segmentation results of each pixel point in each training image;

[0159] calculating a value of a cost function based on the segmentation results of the pixel points;

[0160] updating the first segmentation network according to the gradient descent method to reduce the value of the cost function;

[0161] continuously obtaining new training batches from the first image set and the second image set with the second pseudo labels, updating the first segmentation network, until the value of the cost function is less than a preset threshold, and obtaining a second segmentation network.

[0162] In an optional embodiment, the cost function satisfies a relationship:

[0163]

[0164] wherein Q1 and Q2 represent the first image set and the second image set respectively, N1 and N2 represent the number of training images in the training batch belonging to the first image set and the second image set respectively, W and H are the width and height of the training image, P u (x, y) and P v(x, y) respectively represent the segmentation results of training image u and training image v at pixel point (x, y) in the training batch, represents the pixel category of pixel point (x, y) in the label data of training image u, represents the second probability vector of pixel point (x, y) in the second pseudo label of training image v, represents the calculation of P u (x, y) and the cross-entropy loss function, represents the calculation of P v (x, y) and the KL divergence, and Loss is the numerical value of the cost function.

[0165] From the above technical solutions, it can be seen that the present application calculates the credibility of each pixel point in the first pseudo label on the basis of the first pseudo label to evaluate the accuracy of each pixel point, at the same time, the local information in the neighborhood range of the pixel point is perceived in the first pseudo label to obtain the voting result of each pixel point, further, based on the voting result, greater disturbance is applied to the pixel point with smaller credibility in the first pseudo label, and smaller disturbance is applied to the pixel point with greater credibility in the first pseudo label, so as to realize the updating of the first pseudo label, improve the accuracy of the pseudo label, finally, the first segmentation network is trained for the second time by using the updated pseudo label, and thus the precision of image segmentation is improved.

[0166] Please refer to Figure 4 , which is a structural schematic diagram of an electronic device provided by an embodiment of the present application. The electronic device 1 comprises a memory 12 and a processor 13. The memory 12 is used to store computer readable instructions, and the processor 13 is used to execute the computer readable instructions stored in the memory to realize the image segmentation method based on semi-supervised learning according to any one of the above embodiments.

[0167] In an optional embodiment, the electronic device 1 further comprises a bus, a computer program stored in the memory 12 and executable on the processor 13, such as an image segmentation program based on semi-supervised learning.

[0168] Figure 4 Only the electronic device 1 with the memory 12 and the processor 13 is shown, and those skilled in the art can understand that, Figure 4 the structure shown does not constitute a limitation on the electronic device 1, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0169] In combination with Figure 1 , the memory 12 in the electronic device 1 stores a plurality of computer readable instructions to realize an image segmentation method based on semi-supervised learning, and the processor 13 can execute the plurality of instructions to realize:

[0170] collecting a label image with label data as a first image set and collecting a non-label image without label data as a second image set, the label data including pixel categories of each pixel point in the label image;

[0171] training an image segmentation network based on the first image set to obtain a first segmentation network, and inputting the non-label image in the second image set into the first segmentation network to obtain a first pseudo label of each non-label image;

[0172] performing voting based on the first pseudo label to obtain a voting map of the corresponding non-label image on each pixel category, the voting map including a voting result of the pixel category corresponding to the voting map at each pixel point in the non-label image;

[0173] calculating a credibility of each pixel point in the corresponding non-label image based on the first pseudo label;

[0174] updating the first pseudo label based on the credibility and the voting map to obtain a second pseudo label of each non-label image;

[0175] training the first segmentation network based on the first image set and the second image set with the second pseudo label to obtain a second segmentation network, an input of the second segmentation network being a to-be-segmented image and an output being a segmentation result of the to-be-segmented image.

[0176] Specifically, the specific implementation method of the processor 13 to the above instructions can refer to Figure 1 The description of related steps in the corresponding embodiments will not be repeated here.

[0177] Those skilled in the art can understand that the schematic diagram is only an example of the electronic device 1 and does not constitute a limitation on the electronic device 1. The electronic device 1 can be a bus type structure or a star type structure. The electronic device 1 can also include more or less other hardware or software, or different component arrangements, for example, the electronic device 1 can also include an input / output device, a network access device, etc.

[0178] It should be noted that the electronic device 1 is only an example. Other existing or future electronic products, such as those adaptable to the present application, should also be included in the protection scope of the present application and included by reference.

[0179] The memory 12 includes at least one type of readable storage medium, which can be non-volatile or volatile. The readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. The memory 12 can be an internal storage unit of the electronic device 1 in some embodiments, such as a mobile hard disk of the electronic device 1. The memory 12 can also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the electronic device 1. The memory 12 can be used to store application software and various data installed on the electronic device 1, such as the code of the image segmentation program based on semi-supervised learning, and can also be used to temporarily store data that has been output or will be output.

[0180] The processor 13 can be composed of an integrated circuit in some embodiments, such as a single packaged integrated circuit or a plurality of packaged integrated circuits with the same or different functions, including one or more combinations of central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control core of the electronic device 1, which connects all components of the electronic device 1 through various interfaces and lines, executes programs or modules stored in the memory 12 (such as the image segmentation program based on semi-supervised learning), and calls data stored in the memory 12 to perform various functions of the electronic device 1 and process data.

[0181] The processor 13 executes the operating system of the electronic device 1 and various installed application programs. The processor 13 executes the application programs to implement the steps in each of the above image segmentation methods based on semi-supervised learning, such as Figure 1 the steps shown.

[0182] For example, the computer program can be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules / units can be a series of computer-readable instruction segments that can complete a specific function, which are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program can be divided into an acquisition unit 110, an acquisition unit 111, a voting unit 112, a calculation unit 113, an update unit 114, and a training unit 115.

[0183] The integrated units in the form of software function modules can be stored in a computer readable storage medium. The software function modules are stored in a storage medium and include a plurality of instructions for causing a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute part of the image segmentation method based on semi-supervised learning described in the embodiments of the present application.

[0184] The modules / units integrated in the electronic device 1, if realized in the form of software function modules and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiments can also be completed by a computer program to instruct related hardware devices, and the computer program can be stored in a computer readable storage medium and executed by a processor to implement the steps of each method embodiment.

[0185] The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, and other memories.

[0186] Further, the computer readable storage medium can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function, etc.; and the data storage area can store data created according to the use of the blockchain node, etc.

[0187] The blockchain referred to in the present application is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, and other computer technologies. Blockchain, in essence, is a decentralized database, which is a series of data blocks associated using cryptographic methods, each data block containing information about a batch of network transactions, for verifying the validity of the information (anti-fake) and generating the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer, etc.

[0188] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus or the like. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one arrow is shown in Figure 4

[0189] The embodiment of the present application further provides a computer readable storage medium (not shown in the figure), which stores computer readable instructions. The computer readable instructions are executed by a processor in an electronic device to implement the image segmentation method based on semi-supervised learning according to any one of the above embodiments.

[0190] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the above-described device embodiments are merely illustrative. For example, the division of the modules is merely a logical function division. There can be another division manner in actual implementation.

[0191] The modules described as separated components can or can not be physically separated, and the components displayed as modules can or can not be physical units. They can be located in one place or distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0192] In addition, each functional module in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present independently, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of hardware plus software functional modules.

[0193] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The plurality of units or devices stated in the specification can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

[0194] ​Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. An image segmentation method based on semi-supervised learning, characterized in that: The method comprises: Collecting labeled images with labeled data as a first image set and collecting unlabeled images without labeled data as a second image set, wherein the label data includes a pixel type of each pixel in the labeled image; Training an image segmentation network based on the first image set to obtain a first segmentation network, and inputting the unlabeled images in the second image set into the first segmentation network to obtain a first pseudo label for each unlabeled image; Voting based on the first pseudo-label to obtain a voting map for each pixel type of the corresponding unlabeled image, the voting map including voting results of the pixel type corresponding to the voting map at each pixel point in the unlabeled image; Calculate the credibility of each pixel in the corresponding unlabeled image based on the first pseudo label; updating the first pseudo label based on the confidence and the voting map to obtain a second pseudo label for each unlabeled image; The first segmentation network is trained based on the first image set and a second image set with a second pseudo label to obtain a second segmentation network, where the input of the second segmentation network is the image to be segmented, and the output is a segmentation result of the image to be segmented.

2. The image segmentation method based on semi-supervised learning according to claim 1, characterized in that The first pseudo label includes a first probability vector for each pixel in the unlabeled image, and the first probability vector includes a first probability that the pixel belongs to each pixel category.

3. The image segmentation method based on semi-supervised learning according to claim 1, characterized in that: Voting based on the first pseudo label to obtain a voting map of the corresponding unlabeled image on each pixel type includes: Take any pixel in any first pseudo label as the target pixel; Voting is performed within the neighborhood of the target pixel to obtain voting results for each pixel type at the target pixel, and the voting results satisfy the relationship: Among them, ROI (x,y) is the neighborhood range of the target pixel (x, y), N is the number of all pixel types, α(i, n) is the voting coefficient, Represents ROI (x,y) Inner pixel (x * ,y * ) belongs to the first probability of pixel category i in the first probability vector, Num(ROI (x,y) ) represents ROI (x,y) The number of all pixels in is the voting result of pixel type n at the target pixel point (x, y), and the voting coefficient satisfies the relationship: Traversing all pixels in the first pseudo-label to obtain voting results for each pixel type at each pixel; The voting results of the same pixel type are selected from all pixel points to construct a voting map of the unlabeled image corresponding to the first pseudo label on each pixel type.

4. The image segmentation method based on semi-supervised learning according to claim 1, characterized in that Calculating the credibility of each pixel in the corresponding unlabeled image based on the first pseudo label includes: Take any pixel in any first pseudo label as the target pixel; The credibility of the target pixel is calculated based on the first probability vector of the target pixel, and the credibility satisfies the relationship: Where N is the number of all pixel types, is the first probability of the target pixel point (x, y) belonging to pixel category i in the first probability vector, Conf(x, y) is the credibility of the target pixel point (x, y), and its value range is [0, 1]; All pixels in the first pseudo-label are traversed to obtain the credibility of each pixel in the corresponding unlabeled image.

5. The image segmentation method based on semi-supervised learning according to claim 1, characterized in that: Updating the first pseudo label based on the credibility and the voting map to obtain a second pseudo label for each unlabeled image includes: For each unlabeled image, select the voting results of the pixel to be updated from the voting maps of all pixel types as the voting result set, where the pixel to be updated is any pixel in the unlabeled image; Selecting a first probability vector of the pixel to be updated from the first pseudo label of the unlabeled image; The first probability vector is updated based on the credibility of the pixel to be updated and the voting result set to obtain a second probability vector for the pixel to be updated, wherein the second probability vector includes a second probability that the pixel to be updated belongs to each pixel type, and the second probability satisfies the relationship: Among them, Conf(x′,y′) is the credibility of the pixel point (x′,y′) to be updated, is the first probability of belonging to pixel category i in the first probability vector, t i (x′, y′) is the voting result of pixel type i in the voting result set, is the second probability that the pixel point (x′, y′) to be updated belongs to pixel category i; All pixels in the unlabeled image are traversed to obtain a second probability vector for each pixel, and the second probability vectors of all pixels are used as second pseudo labels for the unlabeled image.

6. The image segmentation method based on semi-supervised learning according to claim 1, characterized in that: The step of training the first segmentation network based on the first image set and the second image set with second pseudo labels to obtain a second segmentation network comprises: Randomly selecting a preset number of training images from the first image set and the second image set with the second pseudo label without replacement as a training batch, wherein the training images include labeled images and unlabeled images; Inputting the training images in the training batch into the first segmentation network to obtain a segmentation result for each pixel in each training image; Calculating a value of a cost function based on the segmentation result of the pixel points; Updating the first segmentation network according to a gradient descent method to reduce the value of the cost function; Continuously obtaining new training batches from the first image set and the second image set with the second pseudo-label, updating the first segmentation network, and stopping when the value of the cost function is less than a preset threshold, thereby obtaining a second segmentation network.

7. The image segmentation method based on semi-supervised learning according to claim 6, characterized in that: The cost function satisfies the relationship: Where Q1 and Q2 represent the first image set and the second image set respectively, N1 and N2 represent the number of training images in the training batch belonging to the first image set and the second image set respectively, W and H are the width and height of the training image, P u (x,y) and P v (x, y) represent the segmentation results of the training image u and the training image v at the pixel point (x, y) in the training batch, respectively. Represents the pixel type of the pixel point (x, y) in the label data of the training image u, The second probability vector representing the pixel (x, y) in the second pseudo label of the training image v, Indicates the calculation of P u (x,y) and The cross entropy loss function is Indicates the calculation of P v (x,y) and The KL divergence of , Loss is the value of the cost function.

8. An image segmentation device based on semi-supervised learning, characterized in that: The device comprises: an acquisition unit, configured to acquire labeled images with labeled data as a first image set and unlabeled images without labeled data as a second image set, wherein the label data includes a pixel type of each pixel in the labeled image; an acquiring unit, configured to train an image segmentation network based on the first image set to obtain a first segmentation network, and input the unlabeled images in the second image set into the first segmentation network to obtain a first pseudo label for each unlabeled image; a voting unit, configured to perform voting based on the first pseudo-label to obtain a voting map for each pixel type of the corresponding unlabeled image, the voting map including voting results for the pixel type corresponding to the voting map at each pixel point in the unlabeled image; a calculating unit, configured to calculate the credibility of each pixel in the corresponding unlabeled image based on the first pseudo label; an updating unit, configured to update the first pseudo label based on the credibility and the voting map to obtain a second pseudo label for each unlabeled image; A training unit is used to train the first segmentation network based on the first image set and a second image set with a second pseudo label to obtain a second segmentation network, wherein the input of the second segmentation network is the image to be segmented, and the output is the segmentation result of the image to be segmented.

9. An electronic device, characterized in that: The electronic device comprises: a memory storing computer-readable instructions; and A processor, configured to execute computer-readable instructions stored in the memory to implement the image segmentation method based on semi-supervised learning according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the image segmentation method based on semi-supervised learning according to any one of claims 1 to 7.

Images