Curve structure image segmentation semi-supervised training method and device based on optimal threshold value

Abstract

The invention discloses a curve structure image segmentation semi-supervised training method and device based on an optimal threshold, and relates to the field of computer vision, and the method comprises the steps: carrying out the prediction of unlabeled data based on a teacher model, and obtaining pseudo-label data; in a preset threshold value search space, searching according to a preset step length until a threshold value enabling KL divergence between pseudo label distribution and preset label prior distribution to be minimum is found, and determining the threshold value as an optimal threshold value; performing binarization processing on the pseudo label data based on the optimal threshold value to obtain binarized pseudo label data; calculating a total loss function by using the binarized pseudo label data and the real label of the labeled data, and updating student model parameters by using the total loss function; and updating teacher model parameters through a moving average method based on the current step parameters, wherein the teacher model parameters are obtained through index moving average of the student model parameters. According to the training method, the curve structure image segmentation performance under the condition that labeled data is limited and unlabeled data is abundant is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a semi-supervised training method and device for curve structure image segmentation based on an optimal threshold. Background Art

[0002] Image curve structure segmentation models play an important role in fields such as defect recognition and medical diagnosis. Existing advanced methods usually rely on using a large amount of labeled data to train deep learning frameworks to obtain image curve structure segmentation models. However, the collection of labeled data is not only costly, but also requires certain expertise, especially in the case of medical image data. Therefore, it is extremely urgent to provide a solution to the high cost of using labeled data to train deep learning frameworks in the prior art.

[0003] In order to solve the problem of high cost of training deep learning frameworks using labeled data in the prior art, semi-supervised learning (SSL) was proposed. Semi-supervised learning (SSL) is to train deep learning frameworks by effectively using cheap unlabeled data with pseudo labels. The unlabeled data with pseudo labels is generated through a self-training paradigm, thereby reducing the cost of obtaining labeled data, and further reducing the cost of training deep learning frameworks using labeled data. Semi-supervised learning for training image curve structure segmentation models is different from the semi-supervised learning method in classification tasks. The distribution of various labels in the classification training sample set of classification tasks is usually balanced, while in the task of training image curve structure segmentation models based on semi-supervised learning, due to the significant class imbalance between the foreground and background pixels of the unlabeled data, the distribution of various labels in the image curve structure segmentation training sample set generated by the self-training paradigm using unlabeled data is unbalanced. Further training the deep learning framework based on the image curve structure segmentation training sample set ultimately leads to low performance of the trained image curve segmentation model. In order to solve the problem of low performance of the aforementioned image curve segmentation model, some studies have proposed resampling the unlabeled data with pseudo labels and training with a training sample set of image curve structure segmentation with a balanced distribution of various labels. However, the resampling strategy is not suitable for the training of image curve structure segmentation tasks, because simply repeating or deleting pixels of unlabeled data will cause distortion of training samples. In addition, the use of repeated pixels is not realistic. Therefore, providing a semi-supervised training method for curve structure image segmentation based on the optimal threshold to improve the performance of the image curve structure segmentation model is an urgent problem to be solved. Summary of the invention

[0004] The present invention provides a semi-supervised training method and device for curve structure image segmentation based on an optimal threshold, the main purpose of which is to provide a semi-supervised training method for curve structure image segmentation based on an optimal threshold to improve the performance of an image curve structure segmentation model.

[0005] According to a first aspect of the present disclosure, a semi-supervised training method for curve structure image segmentation based on an optimal threshold is provided, which includes:

[0006] Step 101, using the teacher model to predict the unlabeled data and generate pseudo-labeled data;

[0007] Step 102, searching in a preset threshold search space with a preset step size until a threshold is found that minimizes the KL divergence between the pseudo-label distribution and the preset label prior distribution, and determining it as the optimal threshold;

[0008] Step 103, binarizing the pseudo-label data based on the optimal threshold to obtain binarized pseudo-label data;

[0009] Step 104, using the binarized pseudo-label data and the real labels of the labeled data to calculate the total loss function, and using the total loss function to train the student model to update the student model parameters;

[0010] Step 105, updating the teacher model parameters by a sliding average method based on the current step parameters, wherein the current step parameters are the student model parameters after real-time update.

[0011] Optionally, the determination of the optimal threshold in step 102 includes:

[0012] Obtaining the preset label prior distribution by performing statistics on the real labels;

[0013] Randomly selecting some pseudo labels with the highest confidence in the pseudo label data, and calculating an initial pseudo label distribution based on the some pseudo labels;

[0014] Combining the historical optimal threshold of the last training iteration, weighting the historical pseudo label distribution and the initial pseudo label distribution to generate the pseudo label distribution;

[0015] Based on the weighted pseudo-label distribution, within the preset threshold search space, a threshold is searched with the preset step size that minimizes the KL divergence between the pseudo-label distribution and the preset label prior distribution. This threshold is the optimal threshold.

[0016] Optionally, in step 104, the total loss function is calculated using the binarized pseudo-label data and the real labels of the labeled data, and the student model is trained using the total loss function to update the student model parameters, including:

[0017] Performing data enhancement on the unlabeled data and performing data enhancement on pseudo-labeled data corresponding to the unlabeled data;

[0018] Calculating a first loss function to calculate the cross entropy between a first prediction result of the student model for the labeled data and the true label to obtain a first loss value;

[0019] Calculating a second loss function to calculate the cross entropy between the binarized pseudo-label data and the second prediction result of the student model on the enhanced unlabeled data to obtain a second loss value, wherein the binarized pseudo-label data is obtained based on the enhanced pseudo-label data after binarization according to the optimal threshold value;

[0020] Calculating a total loss function to comprehensively calculate a total loss value of the student model according to the first loss value and the second loss value;

[0021] A target gradient for updating the student model parameters is calculated based on the total loss value, and the student model parameters are updated according to the target gradient.

[0022] Optionally, performing data enhancement on the unlabeled data includes:

[0023] Randomly select two images from the unlabeled data;

[0024] Cut a rectangular area at a certain position in one image and paste it to a specified position in another image to generate exchanged enhanced data;

[0025] Similarly, all image pairs in the unlabeled data are exchanged to obtain a data set consisting of enhanced unlabeled data;

[0026] The student model is used to predict the data set to obtain the second prediction result.

[0027] Optionally, performing data enhancement on the pseudo-label data corresponding to the unlabeled data includes:

[0028] Two pseudo-label data corresponding to the two images are selected from the pseudo-label data generated in step 101 , and the two pseudo-label data are respectively cut and the same rectangular regions are exchanged to obtain the enhanced pseudo-label data.

[0029] Optionally, the teacher model and the student model have the same architecture;

[0030] The updating of the teacher model parameters adopts the moving average method, and the updated teacher model parameters integrate the current step parameters of the student model.

[0031] According to a second aspect of the present disclosure, a semi-supervised training device for curve structure image segmentation based on an optimal threshold is provided, comprising:

[0032] The prediction unit is used to predict the unlabeled data using the teacher model and generate pseudo-labeled data;

[0033] A determination unit, used to search in a preset threshold search space with a preset step size until a threshold is found that minimizes the KL divergence between the pseudo-label distribution and the preset label prior distribution, and determine it as the optimal threshold;

[0034] A processing unit, configured to perform binarization processing on the pseudo-label data based on the optimal threshold value to obtain binarized pseudo-label data;

[0035] A training unit, used to calculate a total loss function using the binarized pseudo-label data and the real labels of the labeled data, and to train the student model using the total loss function to update the student model parameters;

[0036] An updating unit is used to update the teacher model parameters by a sliding average method based on the current step parameters, and the current step parameters are real-time updated parameters of the student model.

[0037] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0038] at least one processor; and

[0039] a memory communicatively connected to the at least one processor; wherein,

[0040] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0041] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0042] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method as described in the first aspect above.

[0043] The semi-supervised training method and device for curve structure image segmentation based on optimal threshold provided by the present invention predict unlabeled data based on a teacher model to obtain pseudo-label data; exhaustively search for thresholds based on a preset step size in a preset threshold search space until there is a threshold that minimizes the KL divergence between the pseudo-label distribution of the pseudo-label data and the preset label prior distribution, and then determine the one threshold as the optimal threshold; binarize the pseudo-label data based on the optimal threshold to obtain binarized pseudo-label data; calculate the total loss function using the binarized pseudo-label data and the true labels of the labeled data, and use the total loss function to train the student model to update the student model parameters; update the teacher model parameters through a sliding average method based on the current step parameters, and the current step parameters are the student model parameters after real-time update. Compared with the related art, the threshold is exhaustively searched based on a preset step size in a preset threshold search space until there is a threshold that minimizes the KL divergence between the pseudo-label distribution of the pseudo-label data and the preset label prior distribution, and then the threshold is determined to be the optimal threshold, so that when the pseudo-label data is binarized based on the optimal threshold, the distribution of various pseudo-labels in the binarized pseudo-label data is closer to the preset label prior distribution, thereby achieving a balanced distribution of various pseudo-labels in the binarized pseudo-label data, and after the student model is trained based on the binarized pseudo-label data and the true label, the performance of the trained learning model is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0045] Figure 1 A schematic flow chart of a semi-supervised training method for curve structure image segmentation based on an optimal threshold provided in an embodiment of the present disclosure;

[0046] Figure 2 A schematic diagram of a process of optimal threshold screening and binarization processing provided by an embodiment of the present disclosure;

[0047] Figure 3 A schematic diagram of an image cropping process based on the CutMix image cropping technology provided in an embodiment of the present disclosure;

[0048] Figure 4 A schematic flow chart of another semi-supervised training method for curve structure image segmentation based on an optimal threshold provided in an embodiment of the present disclosure;

[0049] Figure 5 A schematic diagram of the prediction effect of a model trained by a different learning method provided in an embodiment of the present disclosure;

[0050] Figure 6 A schematic diagram of the structure of a semi-supervised training device for curve structure image segmentation based on an optimal threshold provided by an embodiment of the present disclosure;

[0051] Figure 7 A schematic block diagram of an exemplary electronic device 300 provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0052] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present invention can be combined with each other without conflict.

[0053] The following detailed description is an exemplary description, which is intended to provide further detailed description of the present invention. Unless otherwise specified, all technical terms used in the present invention have the same meaning as those generally understood by those skilled in the art to which the present invention belongs. The terms used in the present invention are only for describing specific embodiments, and are not intended to limit the exemplary embodiments according to the present invention.

[0054] In addition, the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in sequences other than those illustrated or described herein.

[0055] The following describes the semi-supervised training method and device for curve structure image segmentation based on optimal threshold according to an embodiment of the present disclosure with reference to the accompanying drawings.

[0056] In order to at least provide a semi-supervised training method for image segmentation of curved structure based on an optimal threshold value to improve the performance of the image curved structure segmentation model, this embodiment provides a semi-supervised training method for image segmentation of curved structure based on an optimal threshold value.

[0057] Figure 1 A flow chart of a semi-supervised training method for curve structure image segmentation based on an optimal threshold provided by an embodiment of the present disclosure. Figure 1 As shown, the method comprises the following steps:

[0058] Step 101, predicting the unlabeled data based on the teacher model to generate pseudo-labeled data;

[0059] Step 102, searching in a preset threshold search space with a preset step size until a threshold is found that minimizes the KL divergence between the pseudo-label distribution and the preset label prior distribution, and determining it as the optimal threshold;

[0060] As a refinement of the above step 102, the threshold is exhaustively searched within the preset threshold search space based on the preset step size until there is a threshold that minimizes the KL divergence between the pseudo-label distribution of the pseudo-label data and the preset label prior distribution, and the threshold is determined to be the optimal threshold; the preset threshold search space is a continuous numerical range, and in this embodiment, the preset step size is preferably 0.05. If the numerical range is 0 to 255, the thresholds that can be selected according to the preset step size include 0, 0.05, 0.1, 0.15...254.95, 255. When there is a threshold among the selectable thresholds that minimizes the KL divergence between the pseudo-label distribution of the pseudo-label data and the preset label prior distribution, the threshold is the optimal threshold. The preset label prior distribution is a label distribution calculated by manually labeling some unlabeled data. It should be understood that the numerical range of 0 to 255 in this embodiment is only for exemplary purposes and does not constitute a limitation on the specific numerical range of the embodiments of the present disclosure.

[0061] Step 103, binarizing the pseudo-label data based on the optimal threshold to obtain binarized pseudo-label data;

[0062] Step 104, using the binarized pseudo-label data and the real labels of the labeled data to calculate the total loss function, and using the total loss function to train the student model to update the student model parameters;

[0063] Step 105, updating the teacher model parameters by a sliding average method based on the current step parameters, wherein the current step parameters are the student model parameters after real-time update.

[0064] The semi-supervised training method for curve structure image segmentation based on optimal threshold provided by the present invention predicts unlabeled data based on a teacher model to obtain pseudo-label data; exhaustively searches for thresholds based on a preset step size in a preset threshold search space until there is a threshold that minimizes the KL divergence between the pseudo-label distribution of the pseudo-label data and the preset label prior distribution, and then determines the one threshold as the optimal threshold; binarizes the pseudo-label data based on the optimal threshold to obtain binarized pseudo-label data; calculates a total loss function using the binarized pseudo-label data and the true labels of the labeled data, and uses the total loss function to train a student model to update the student model parameters; updates the teacher model parameters through a sliding average method based on the current step parameters, and the current step parameters are the student model parameters after real-time update. Compared with the related art, the threshold is exhaustively searched based on a preset step size in a preset threshold search space until there is a threshold that minimizes the KL divergence between the pseudo-label distribution of the pseudo-label data and the preset label prior distribution, and then the threshold is determined to be the optimal threshold, so that when the pseudo-label data is binarized based on the optimal threshold, the distribution of various pseudo-labels in the binarized pseudo-label data is closer to the preset label prior distribution, thereby achieving a balanced distribution of various pseudo-labels in the binarized pseudo-label data, and after the student model is trained based on the binarized pseudo-label data and the true label, the performance of the trained learning model is improved.

[0065] In order to facilitate understanding of the above steps 102 and 103, Figure 2 A schematic diagram of a process of optimal threshold screening and binarization processing provided by an embodiment of the present disclosure, Figure 2 τ is shown in 1, , τ2, τ3…τ n That is, for the different thresholds enumerated, the KL divergence between the pseudo-label distribution corresponding to the different thresholds and the prior distribution of the real label, that is, the preset label prior distribution, is calculated respectively, and the threshold corresponding to the minimum KL divergence is selected to binarize the pseudo-label data. Figure 2 PCT. is the percentage, Class is the label category, BG is the background pixel label, and FG is the foreground pixel label.

[0066] As a refinement of the above embodiment, when performing step 102 to search within the preset threshold search space with a preset step size until a threshold is found that minimizes the KL divergence between the pseudo-label distribution and the preset label prior distribution, and determining it as the optimal threshold, the following implementation methods may also be used but are not limited to, for example: obtaining the preset label prior distribution by counting the real labels; randomly selecting some pseudo-labels with the highest confidence in the pseudo-label data, and calculating the initial pseudo-label distribution based on the some pseudo-labels; combining the historical optimal threshold of the previous training iteration, weighting the historical pseudo-label distribution and the initial pseudo-label distribution to generate the pseudo-label distribution; based on the weighted pseudo-label distribution, in the preset threshold search space, searching for a threshold with the preset step size that minimizes the KL divergence between the pseudo-label distribution and the preset label prior distribution, and this threshold is the optimal threshold. That is, a preset number of pseudo-labels are randomly selected from the pseudo-label data, and an initial pseudo-label distribution is calculated based on the preset number of pseudo-labels; the historical pseudo-label distribution in the previous iterative training process and the initial pseudo-label distribution in this iterative training process are weighted together to obtain the pseudo-label distribution, the previous iterative training process is the previous iterative training of this iterative training process, and finally the optimal threshold is determined based on the pseudo-label distribution.

[0067] In order to more intuitively demonstrate the update process of the pseudo-label distribution each time the learning model is iteratively trained, and to more clearly demonstrate the process of determining the optimal threshold, this embodiment provides an exemplary description in conjunction with the formula, including: the pseudo-label distribution P(τ) depends on the prediction of the teacher model and is affected by the weight update. The naive method of estimating the pseudo-label distribution P(τ) includes forward passing all unlabeled data, that is, unlabeled data, before each training iteration, but this will bring extremely high computational overhead to each update step. Therefore, this embodiment uses incremental distribution estimation to update the pseudo-label distribution, specifically following the following update rules, see formula (1):

[0068]

[0069] Wherein, B represents the preset number of pseudo labels, β represents the weight, P0(τ*) represents the historical pseudo label distribution corresponding to the condition that the prediction parameter is less than τ* in the previous iteration process, P1(τ*) represents the historical pseudo label distribution corresponding to the condition that the prediction parameter is greater than or equal to τ* in the previous iteration process, τ* is the historical optimal threshold, and τ is the optimal threshold. i Represents the i-th pseudo-label sample among the preset number of pseudo-labels B. represents the total number of pseudo-label samples in B, The pseudo label of the pixel position hw generated for the i-th unlabeled image in the unlabeled data, where hw represents the pixel position in the image. and Constitute the initial pseudo label distribution.

[0070] The process of determining the optimal threshold is based on the following formula (2), formula (3) and formula (1), and is specifically as follows:

[0071]

[0072] Wherein, Q represents the preset label prior distribution, P represents the initial pseudo label distribution, i represents the subscript variable, hw represents the pixel position, h and w represent the row and column index of the pixel respectively, and y ihw is the label of the position of the i-th labeled image pixel hw, is the pseudo label of the pixel hw position generated for the i-th unlabeled image in the unlabeled data. C is the sum of the number of pixels with all labels 0 and the number of pixels with all labels 1, It is the sum of the number of pixels whose pseudo labels are less than the optimal threshold τ and the number of pixels whose pseudo labels are greater than or equal to the optimal threshold τ.

[0073] τ * =argmin τ∈[0,1] KL(Q||P(τ)) Formula (3)

[0074] Among them, τ * is the optimal threshold, P(τ) is the pseudo-label distribution calculated by formula (1), which is constrained by the optimal threshold τ, Q represents the preset label prior distribution, and formula (3) represents minimizing the KL divergence between the preset label prior distribution and the pseudo-label distribution. The specific process is to substitute the initial pseudo-label distribution P in formula (2) into formula (1) to calculate the pseudo-label distribution, and then determine the optimal threshold τ based on the pseudo-label distribution combined with formula (3) * It should be understood that as the pseudo-label distribution is continuously updated, in the process of aligning the pseudo-label distribution with the preset label prior distribution, the optimal threshold is also continuously and dynamically updated, thereby ensuring that the performance of the trained learning model is better.

[0075] As a refinement of the above embodiment, when executing step 104 to calculate the total loss function using the binarized pseudo-label data and the true label of the labeled data, and using the total loss function to train the student model to update the student model parameters, the following implementation methods may also be adopted but are not limited to, for example: data enhancement is performed on the unlabeled data and data enhancement is performed on the pseudo-label data corresponding to the unlabeled data; a first loss function is calculated to calculate the cross entropy between the first prediction result of the student model for the labeled data and the true label to obtain a first loss value; a second loss function is calculated to calculate the cross entropy between the binarized pseudo-label data and the second prediction result of the student model for the enhanced unlabeled data to obtain a second loss value, the binarized pseudo-label data being obtained after binarization according to the optimal threshold based on the enhanced pseudo-label data; a total loss function is calculated to comprehensively calculate the total loss value of the student model based on the first loss value and the second loss value; a target gradient for updating the student model parameters is calculated based on the total loss value, and the student model parameters are updated according to the target gradient. Based on calculating the cross entropy between the first prediction result of the student model on the labeled data and the true label, a first loss value is obtained; based on the second loss function, the cross entropy between the binarized pseudo-label data and the second prediction result of the student model on the enhanced unlabeled data is calculated to obtain a second loss value; based on the total loss function, the total loss value is calculated according to the first loss value and the second loss value. Based on the total loss value, the student model parameters are optimized to obtain a trained student model, that is, the target gradient is calculated through the total loss value, and the gradient of the student model is updated according to the target gradient, so as to iterate in a round-robin manner to obtain the best performance trained student model.

[0076] In order to facilitate the understanding of the above embodiment, this embodiment provides an exemplary explanation in combination with the formula, including: the final training process follows a two-step alternating update between the threshold hyperparameter and semi-supervised learning. In the training process, the optimal threshold τ is first estimated. * , and then use optimization algorithms such as gradient descent to update the model weights, taking into account both labeled and unlabeled losses. By alternating between updating the threshold and model weights, the performance of the model can be gradually improved. The calculation of the total loss value is shown in the following formula (4):

[0077]

[0078] Among them, λ is a hyperparameter, is the cross entropy loss function, i.e., the first loss function. The labeled training samples, i.e., the labeled data, are defined as D l ={(x i ,y i )|xi ∈R H×W×3 ,y i ∈{0,1} H×W} i＝1...Nl , x i is the image data in the labeled data, y i is the label data, i.e. the real label, D l is the labeled training sample data set, and the unlabeled training sample data set is The curve structure image segmentation model is represented as f(x;θ)∈[0,1] H×W , θ is the model parameter of the curve structure image segmentation model, is the third loss function, x j represents the jth image data in the unlabeled training sample dataset, Represents the jth image data x in the unlabeled training sample dataset j Corresponding to the generated pseudo-label, τ is the optimal threshold, is the second loss function.

[0079] As a refinement of the above embodiment, when performing data enhancement on the unlabeled data, the following implementation methods may also be used but are not limited to, for example: randomly selecting two images from the unlabeled data; cutting a rectangular area at a certain position of one image and pasting it to a specified position of another image to generate exchanged enhanced data; and so on, exchanging all image pairs in the unlabeled data to obtain a data set consisting of the enhanced unlabeled data; using the student model to predict the data set to obtain the second prediction result.

[0080] In order to facilitate the understanding of the above embodiment, this embodiment provides an exemplary description, randomly obtaining two images in the unlabeled data; cutting a first rectangular area at the first position of one image and cutting a second rectangular area at the second position of the other image, the two images including the one image and the other image; pasting the first rectangular area to the second position on the other image to obtain a first exchanged image and pasting the second rectangular area to the first position on the one image to obtain a second exchanged image, until the exchange between the two images in the unlabeled data is completed to obtain enhanced unlabeled data; predicting the data set constructed from the enhanced unlabeled data based on the learning model to obtain the second prediction result. The image cropping and exchange process involved in this embodiment is a CutMix image cropping technology provided in this embodiment, Figure 3A schematic diagram of an image cropping process based on the CutMix image cropping technology provided in the embodiment of the present disclosure is used to facilitate understanding of the image cropping process of the present embodiment. Since the image cropped by CutMix is ​​used to train the model, the model can generate a smoother decision boundary, which helps to improve the generalization performance of the model, and at the same time can improve the diversity of data enhancement, reduce the risk of overfitting, and enhance the input image and segmentation mask.

[0081] As a refinement of the above embodiment, when data enhancement is performed on the pseudo-label data corresponding to the unlabeled data, the following implementation method can be adopted but is not limited to, for example: two pseudo-label data corresponding to the two images are selected from the pseudo-label data generated in step 101, and the two pseudo-label data are respectively cut and the same rectangular areas are exchanged to obtain the enhanced pseudo-label data. Specifically, it includes: predicting the pseudo-label data corresponding to the unlabeled data based on the teacher model, and randomly obtaining two pseudo-label data corresponding to the two images from the pseudo-label data; cutting a third rectangular area at the third position of one pseudo-label data in the two pseudo-label data and cutting a fourth rectangular area at the fourth position of the other pseudo-label data, the two pseudo-label data include the one pseudo-label data and the other pseudo-label data; pasting the third rectangular area to the fourth position on the other pseudo-label data to obtain a first exchanged image and pasting the fourth rectangular area to the third position on the one pseudo-label data to obtain a second exchanged image, obtaining the enhanced pseudo-label data, calculating the first pseudo-label corresponding to the first exchanged pseudo-label data and calculating the second pseudo-label corresponding to the second exchanged pseudo-label data, until the exchange between the two pseudo-label data in the pseudo-label data is completed and the pseudo-labels corresponding to the exchanged pseudo-label data are calculated. Wherein, the first pseudo-label and the second pseudo-label can be calculated by weighted averaging the original pseudo-labels of the two pseudo-label data according to the area of ​​the cut rectangular area.

[0082] In some embodiments, the teacher model and the student model have the same architecture, the teacher model is a network model trained using labeled data, the student model is a network model that has not undergone any training, the teacher model parameters are updated using a moving average method, and the updated teacher model parameters incorporate the current step parameters of the student model.

[0083] In order to comprehensively demonstrate the semi-supervised training process involved in this disclosure, Figure 4 A schematic diagram of another process flow of a semi-supervised training method for curve structure image segmentation based on an optimal threshold provided by an embodiment of the present disclosure. Figure 4The purpose of the geometric transformations involved is to enhance the data. The specific processes include but are not limited to rotation, translation, scaling, etc. The exponential moving average (EMA) is used to update the parameters of the teacher model and to process labeled and unlabeled data in the student model. Figure 4 The semi-supervised training process involved in corresponds to the above embodiment. For the sake of clarity and conciseness, this embodiment will not be repeated. Please refer to the above embodiment.

[0084] The method proposed in this paper is experimented on different data sets. The experimental results show that the model trained by this method can predict more detailed features in the curve structure and maintain robust performance even with minimal labeled data. The specific effects are as follows:

[0085] The designed system is evaluated on five publicly available datasets: CrackForest, EM128, DRIVE128, STARE128, and CHASE128.

[0086] Datasets: CrackForest consists of 118 annotated road crack images, each with a resolution of 480×320 pixels. The dataset is randomly divided into 80% for training, 10% for validation, and 10% for testing. EM128 is designed for cell membrane segmentation and contains 30 labeled images divided into 16 regions of 128×128 pixels. It is split 240 / 240 for training and testing to form the derived dataset EM128. DRIVE128 comes from a diabetic retinopathy screening project, which divides 584×565 pixel images into 16 regions of 128×128 pixels, splits them 160 / 160 for training and testing, and creates the derived dataset DRIVE128. STARE128 is a fundus image database for retinal vessel segmentation, including 20 images with a resolution of 605×700 pixels. Each image is divided into 25 non-overlapping regions of 128×128 pixels. The dataset is divided into 393 / 92 images for training and testing to obtain the derived dataset STARE128.

[0087] The experiment mainly compares the model trained by the disclosed method with the models trained by the fully supervised learning method and the semi-supervised learning method using different backbone networks. The experimental results show that when there is 1% and 5% labeled data, the present invention is always better than the existing semi-supervised segmentation method. Especially when there is only 1% labeling, the performance gap with the suboptimal method (CutMix) is more obvious, which shows the importance of choosing a reliable threshold for the stability of SSL performance under extremely low labeling. In addition, the performance of the present invention under 5% labeling is close to the upper limit that can be achieved by the fully supervised method, which shows that SSL provides a very promising method to balance segmentation performance and labeling work. At the same time, when using 5% labeled data, the present invention shows a clearer overall prediction result: smaller cracks and blood vessels can be accurately identified in the details, instead of mistakenly judging them as background. For details, please refer to Figure 5 , Figure 5 A schematic diagram of the prediction effect of a model trained by different learning methods provided in an embodiment of the present disclosure, Figure 5 Where SemiDial(Ours) is the model trained by the disclosed method, and Figure 5 The numbers in the upper left corner indicate the image index in each dataset.

[0088] At the same time, the present invention also designs ablation experiments to verify the robustness of the system. The semi-supervised learning, data enhancement and distribution alignment modules of the present invention are experimented on two data sets, using the CutMix model as the initial backbone. After applying the original CutMix, the fully supervised baseline is improved from more than 40% to about 50%, proving the effectiveness of consistency-based semi-supervised learning in improving segmentation performance using unlabeled data. Secondly, the use of strong geometric transformations can also further improve segmentation performance. In addition, when using distribution alignment to estimate the optimal threshold τ, the present invention designs two methods. The results show that using the mean square error (MSE) loss for threshold estimation is better than using a fixed threshold of 0.5. Compared with the MSE loss for threshold estimation, the application of KL divergence enables the model to produce better performance.

[0089] In summary, the embodiments of the present disclosure can achieve the following effects:

[0090] 1. By exhaustively searching for thresholds based on a preset step size in a preset threshold search space until there is a threshold that minimizes the KL divergence between the pseudo-label distribution of the pseudo-label data and the preset label prior distribution, the threshold is determined to be the optimal threshold, so that when the pseudo-label data is binarized based on the optimal threshold, the distribution of various pseudo-labels in the binarized pseudo-label data is closer to the preset label prior distribution, thereby achieving a balanced distribution of various pseudo-labels in the binarized pseudo-label data, and after training the student model based on the binarized pseudo-label data and the true label, the performance of the trained learning model is improved.

[0091] Corresponding to the above-mentioned semi-supervised training method for curve structure image segmentation based on optimal threshold, the present invention also proposes a semi-supervised training device for curve structure image segmentation based on optimal threshold. Since the device embodiment of the present invention corresponds to the above-mentioned method embodiment, the details not disclosed in the device embodiment can be referred to the above-mentioned method embodiment, and will not be described in detail in the present invention.

[0092] Figure 6 A schematic diagram of a semi-supervised training device for curve structure image segmentation based on an optimal threshold provided by an embodiment of the present disclosure is shown in FIG. Figure 6 As shown, including:

[0093] The prediction unit is used to predict the unlabeled data using the teacher model and generate pseudo-labeled data;

[0094] A determination unit, used to search in a preset threshold search space with a preset step size until a threshold is found that minimizes the KL divergence between the pseudo-label distribution and the preset label prior distribution, and determine it as the optimal threshold;

[0095] A processing unit, configured to perform binarization processing on the pseudo-label data based on the optimal threshold value to obtain binarized pseudo-label data;

[0096] A training unit, used to calculate a total loss function using the binarized pseudo-label data and the real labels of the labeled data, and to train the student model using the total loss function to update the student model parameters;

[0097] An updating unit is used to update the teacher model parameters by a sliding average method based on the current step parameters, and the current step parameters are real-time updated parameters of the student model.

[0098] The semi-supervised training device for curve structure image segmentation based on optimal threshold provided by the present invention predicts unlabeled data based on a teacher model to obtain pseudo-label data; exhaustively searches for thresholds based on a preset step size in a preset threshold search space until there is a threshold that minimizes the KL divergence between the pseudo-label distribution of the pseudo-label data and the preset label prior distribution, and then determines the one threshold as the optimal threshold; binarizes the pseudo-label data based on the optimal threshold to obtain binarized pseudo-label data; calculates a total loss function using the binarized pseudo-label data and the true labels of the labeled data, and uses the total loss function to train a student model to update the student model parameters; updates the teacher model parameters through a sliding average method based on the current step parameters, and the current step parameters are the student model parameters after real-time update. Compared with the related art, the threshold is exhaustively searched based on a preset step size in a preset threshold search space until there is a threshold that minimizes the KL divergence between the pseudo-label distribution of the pseudo-label data and the preset label prior distribution, and then the threshold is determined to be the optimal threshold, so that when the pseudo-label data is binarized based on the optimal threshold, the distribution of various pseudo-labels in the binarized pseudo-label data is closer to the preset label prior distribution, thereby achieving a balanced distribution of various pseudo-labels in the binarized pseudo-label data, and after the student model is trained based on the binarized pseudo-label data and the true label, the performance of the trained learning model is improved.

[0099] It should be noted that the above explanation of the method embodiment is also applicable to the device of this embodiment, and the principle is the same, which is not limited in this embodiment.

[0100] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0101] Figure 7 A schematic block diagram of an example electronic device 300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0102] like Figure 7As shown, the device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 302 or a computer program loaded from a storage unit 308 to a RAM (Random Access Memory) 303. In the RAM 303, various programs and data required for the operation of the device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An I / O (Input / Output) interface 305 is also connected to the bus 304.

[0103] A number of components in the device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0104] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above, such as a semi-supervised training method for image segmentation of a curve structure based on an optimal threshold. For example, in some embodiments, a semi-supervised training method for image segmentation of a curve structure based on an optimal threshold may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured in any other appropriate manner (eg, by means of firmware) to execute the aforementioned optimal threshold-based semi-supervised training method for curve structure image segmentation.

[0105] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0106] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0107] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0108] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0109] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0110] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0111] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

Claims

1. A semi-supervised training method for curve structure image segmentation based on optimal threshold, characterized in that: The following steps are involved: Step 101, using the teacher model to predict the unlabeled data and generate pseudo-labeled data; Step 102, searching in a preset threshold search space with a preset step size until a threshold is found that minimizes the KL divergence between the pseudo-label distribution and the preset label prior distribution, and determining it as the optimal threshold; Step 103, binarizing the pseudo-label data based on the optimal threshold to obtain binarized pseudo-label data; Step 104, using the binarized pseudo-label data and the real labels of the labeled data to calculate the total loss function, and using the total loss function to train the student model to update the student model parameters; Step 105, updating the teacher model parameters by a sliding average method based on the current step parameters, wherein the current step parameters are the student model parameters after real-time update.

2. The method according to claim 1, characterized in that The determination of the optimal threshold in step 102 includes: Obtaining the preset label prior distribution by performing statistics on the real labels; Randomly selecting some pseudo labels with the highest confidence in the pseudo label data, and calculating an initial pseudo label distribution based on the some pseudo labels; Combining the historical optimal threshold of the last training iteration, weighting the historical pseudo label distribution and the initial pseudo label distribution to generate the pseudo label distribution; Based on the weighted pseudo-label distribution, within the preset threshold search space, a threshold is searched with the preset step size that minimizes the KL divergence between the pseudo-label distribution and the preset label prior distribution. This threshold is the optimal threshold.

3. The method according to claim 1, characterized in that In step 104, the total loss function is calculated using the binarized pseudo-label data and the real labels of the labeled data, and the student model is trained using the total loss function to update the student model parameters, including: Performing data enhancement on the unlabeled data and performing data enhancement on pseudo-labeled data corresponding to the unlabeled data; Calculating a first loss function to calculate the cross entropy between a first prediction result of the student model for the labeled data and the true label to obtain a first loss value; Calculating a second loss function to calculate the cross entropy between the binarized pseudo-label data and the second prediction result of the student model on the enhanced unlabeled data to obtain a second loss value, wherein the binarized pseudo-label data is obtained based on the enhanced pseudo-label data after binarization according to the optimal threshold; Calculating a total loss function to comprehensively calculate a total loss value of the student model according to the first loss value and the second loss value; A target gradient for updating the student model parameters is calculated based on the total loss value, and the student model parameters are updated according to the target gradient.

4. The method according to claim 3, characterized in that Performing data enhancement on the unlabeled data includes: Randomly select two images from the unlabeled data; Cut a rectangular area at a certain position in one image and paste it to a specified position in another image to generate exchanged enhanced data; Similarly, all image pairs in the unlabeled data are exchanged to obtain a data set consisting of enhanced unlabeled data; The student model is used to predict the data set to obtain the second prediction result.

5. The method according to claim 4, characterized in that Performing data enhancement on the pseudo-label data corresponding to the unlabeled data includes: Two pseudo-label data corresponding to the two images are selected from the pseudo-label data generated in step 101, and the two pseudo-label data are respectively cut and the same rectangular areas are exchanged to obtain the enhanced pseudo-label data.

6. The method according to any one of claims 1 to 5, characterized in that: The teacher model and the student model have the same architecture; The updating of the teacher model parameters adopts the moving average method, and the updated teacher model parameters integrate the current step parameters of the student model.

7. A semi-supervised training device for curve structure image segmentation based on optimal threshold, characterized in that: include: The prediction unit is used to predict the unlabeled data using the teacher model and generate pseudo-labeled data; A determination unit, used to search in a preset threshold search space with a preset step size until a threshold is found that minimizes the KL divergence between the pseudo-label distribution and the preset label prior distribution, and determine it as the optimal threshold; A processing unit, configured to perform binarization processing on the pseudo-label data based on the optimal threshold value to obtain binarized pseudo-label data; A training unit, used to calculate a total loss function using the binarized pseudo-label data and the real labels of the labeled data, and to train the student model using the total loss function to update the student model parameters; An updating unit is used to update the teacher model parameters by a sliding average method based on the current step parameters, and the current step parameters are real-time updated parameters of the student model.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images