Semi-supervised regression method and device based on pseudo label heterovariance hypothesis

Abstract

The invention discloses a semi-supervised regression method and device based on pseudo-label heterovariance hypothesis, and relates to the technical field of machine learning. Iteratively executing the model training step until a stop condition is met; constructing a double-layer optimization framework to perform joint optimization on parameters of the regression model and parameters of the pseudo-label uncertainty estimation network; predicting a pseudo label of the sample by using the regression model, performing heterovariance modeling on the pseudo label of the unmarked sample to obtain lower layer loss, and updating parameters of the regression model based on the lower layer loss; obtaining upper layer loss based on the updated regression model, and updating parameters of the pseudo label uncertainty estimation network based on the upper layer loss; inputting a to-be-processed regression task into the trained regression model, and outputting a corresponding regression result; and selecting different evaluation indexes according to different sub-data sets to evaluate the accuracy of a regression result. The problem of how to avoid how to accurately measure the uncertainty of the false label by purely depending on consistency regularization is solved.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a semi-supervised regression method and device based on pseudo-label heteroscedasticity assumption. Background Art

[0002] Deep learning has achieved remarkable performance in a variety of scenarios, especially when large amounts of labeled data are available. However, in real-world scenarios such as medical imaging (where labeled data is often scarce and expensive) or video analysis that requires frame-by-frame manual labeling, obtaining large-scale, accurately labeled datasets is often costly or difficult to achieve. To address this challenge, semi-supervised learning (SSL) has emerged as an effective learning paradigm by leveraging massive amounts of unlabeled data to improve learning performance, thereby reducing the reliance on expensive labeled data.

[0003] In classification tasks, semi-supervised learning techniques such as pseudo-labeling and consistency regularization have been widely adopted and have achieved remarkable results in various fields. However, semi-supervised regression (SSR) faces unique challenges that differ significantly from classification tasks. Unlike discrete pseudo-labels in classification tasks, which can be sharpened to improve prediction confidence, regression tasks inherently process continuous outputs, making it particularly difficult to establish reliable pseudo-labeling mechanisms. Furthermore, the lack of a clear decision boundary in regression tasks complicates the uncertainty quantification process, exacerbating the risk of propagating erroneous pseudo-labels. Due to these challenges, pseudo-labeling methods commonly used in semi-supervised classification are difficult to directly transfer to semi-supervised regression scenarios due to the continuous nature of regression outputs. Therefore, existing SSR methods primarily focus on improving the utilization of unlabeled data through consistency regularization mechanisms. These methods aim to enforce smoothness between the predictions for labeled and unlabeled data, ensuring that the model learns a consistent mapping from input features to continuous outputs.

[0004] However, simply relying on consistency regularization cannot accurately measure the uncertainty of pseudo-labels, and there is still a risk of overfitting to potential erroneous pseudo-labels. Summary of the Invention

[0005] In an embodiment of the present application, a semi-supervised regression method based on the pseudo-label heteroscedasticity assumption is provided to solve the problem of how to avoid relying solely on consistency regularization to accurately measure the uncertainty of pseudo-labels and still having the risk of overfitting potential erroneous pseudo-labels.

[0006] In the first aspect, an embodiment of the present application provides a semi-supervised regression method based on the pseudo-label heteroscedasticity hypothesis, the method comprising: constructing a semi-supervised regression dataset; wherein the semi-supervised regression dataset comprises multiple sub-datasets, each sub-dataset corresponds to a different type of regression task, and different semi-supervised settings are performed for the regression task type of each sub-dataset; according to different types of regression tasks, an adapted backbone network and a pseudo-label uncertainty estimation network are selected; the model training step is iteratively performed until a stopping condition is met to obtain a trained regression model; the model training step comprises: constructing a two-layer optimization framework to jointly optimize the parameters of the regression model and the parameters of the pseudo-label uncertainty estimation network; using the regression model to predict sample pseudo-labels, performing heteroscedasticity modeling on the pseudo-labels of unlabeled samples, obtaining the lower-layer loss, and updating the parameters of the regression model based on the lower-layer loss; obtaining the upper-layer loss based on the updated regression model, and updating the parameters of the pseudo-label uncertainty estimation network based on the upper-layer loss; after the model training is completed, the regression task to be processed is input into the trained regression model, and the corresponding regression result is output; different evaluation indicators are selected according to different sub-datasets to evaluate the accuracy of the regression results.

[0007] In one possible implementation, the pseudo-label uncertainty estimation network adopts a lightweight architecture design, specifically including: the pseudo-label uncertainty estimation network is a multi-layer perceptron consisting of only a single hidden layer, which has the ability to approximate any continuous function, and evaluates the uncertainty of the pseudo-label by mapping the pseudo-label of the unlabeled sample into an uncertainty value.

[0008] In one possible implementation, the expression for mapping the pseudo-label of an unlabeled sample to an uncertainty value is: Among them, z j represents the logarithmic variance of the uncertainty of the j-th unlabeled sample, represents the jth unlabeled sample, g φ represents the pseudo-label uncertainty estimation network, represents the uncertainty of the pseudo label of the jth unlabeled sample.

[0009] In one possible implementation, the expression for heteroscedasticity modeling of pseudo labels of unlabeled samples is: in, is the predicted value of the jth unlabeled sample, is the pseudo label of the jth unlabeled sample, σ j is the uncertainty of the pseudo label of the jth unlabeled sample, is the negative log-likelihood of the jth sample, and p is the probability density of the pseudo-label given the unlabeled sample.

[0010] In one possible implementation, the lower layer loss includes supervised loss and heteroscedastic unsupervised loss; the lower layer loss is obtained as follows: in, is the lower layer loss, To monitor losses, is the heteroskedastic unsupervised loss that takes into account the uncertainty of unlabeled samples, θ is the parameter of the regression model, and φ is the parameter of the pseudo-label uncertainty estimation network.

[0011] In one possible implementation, the upper layer loss is obtained as follows: in, is the upper layer loss, Bl is the data sampled from the labeled data set, and y k is the kth sample Tags, To update the parameters The kth sample obtained under the regression model The predicted value obtained, θ * (φ) indicates that the parameters of the regression model are updated under the influence of the parameters of the pseudo-label uncertainty estimation network.

[0012] In one possible implementation, the two-layer optimization framework is constructed by alternately updating the parameters of the regression model and the parameters of the pseudo-label uncertainty estimation network so that the uncertainty of the pseudo-labels of unlabeled samples can suppress unreliable pseudo-labels, while ensuring the learning of unlabeled samples whose pseudo-labels are correct but there is an error between the predicted value and the pseudo-label.

[0013] In one possible implementation, the expression for updating the parameters of the regression model based on the lower layer loss is: Among them, θ t+1 (φ t ) is the parameter of the regression model at the t+1th iteration, θ t is the parameter of the regression model at the tth iteration, φ t is the parameter of the pseudo-label uncertainty estimation network at the tth iteration, α is the learning rate of the regression model, α>0, Loss for the lower layer The gradient of the regression model with respect to its parameters θ.

[0014] In one possible implementation, the expression for updating the parameters of the pseudo-label uncertainty estimation network based on the upper layer loss is: Among them, φ t+1 is the parameter of the pseudo-label uncertainty estimation network at the t+1th iteration, β is the learning rate of the pseudo-label uncertainty estimation network, β>0, Upper layer loss The gradient with respect to the parameters φ of the pseudo-label uncertainty estimation network.

[0015] In a possible implementation, different evaluation indicators are selected according to different sub-datasets to evaluate the accuracy of the regression results, including: the evaluation indicators are mean absolute error, absolute coefficient and mean square error; the mean absolute error is: Among them, MAE is the mean absolute error, n is the number of samples in the batch, Represents the predicted value of the i-th sample, θ * is the parameter of the optimal regression model obtained after training, y i is the true value of the corresponding sample; the absolute coefficient is: Among them, R 2 is the absolute coefficient, is the mean of the corresponding true value; the mean square error is: Among them, MSE is the mean square error.

[0016] One or more technical solutions provided in the embodiments of this application have at least the following technical effects:

[0017] The embodiment of the present application provides a semi-supervised regression method based on the pseudo-label heteroscedasticity assumption, which learns the uncertainty of the pseudo-labels of unlabeled samples by introducing a pseudo-label uncertainty estimation network. A two-layer optimization framework is constructed in model training to jointly optimize the parameters of the regression model and the parameters of the pseudo-label uncertainty estimation network. The optimization process in the example of the present application adopts a two-layer optimization framework instead of using an end-to-end optimization method. During the training process, samples with correct pseudo-labels but large errors in model prediction values ​​can be learned, effectively reducing the error accumulation problem. At the same time, in the specific optimization process, the example of the present application assumes that the parameters of the pseudo-label uncertainty estimation network are only related to the parameters of the regression layer of the regression model, rather than the parameters of the entire regression model, which significantly reduces the computational cost and time consumption. In addition, the theoretical analysis based on gradient alignment and the empirical results of the SSR benchmark task show that this method significantly surpasses the existing solutions in terms of robustness and accuracy, effectively reducing the risk of overfitting the model to erroneous pseudo-labels and the error accumulation problem. It solves the problem of how to avoid relying solely on consistency regularization to accurately measure the uncertainty of pseudo-labels and the risk of overfitting to potential erroneous pseudo-labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments of the present application or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A flowchart of a semi-supervised regression method based on pseudo-label heteroscedasticity assumption provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of the structure of the pseudo-label uncertainty estimation network provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of a semi-supervised regression device based on pseudo-label heteroscedasticity assumption provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of a semi-supervised regression server based on the pseudo-label heteroscedasticity assumption provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0024] The following description of some of the technologies involved in the embodiments of this application is provided to facilitate understanding and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for the sake of clarity and conciseness, some descriptions of well-known functions and structures are omitted from the following description.

[0025] The present invention provides a semi-supervised regression method based on the pseudo-label heteroscedasticity hypothesis. Figure 1 As shown, the method includes steps S101 to S105. Figure 1 This is only an execution order shown in the embodiment of the present application, and does not represent the only execution order of a semi-supervised regression method based on the pseudo-label heteroscedasticity assumption. If the final result can be achieved, Figure 1 The steps shown may be performed in parallel or reversed.

[0026] S101: Construct a semi-supervised regression dataset. The semi-supervised regression dataset includes multiple sub-datasets, each of which corresponds to a different type of regression task, and different semi-supervised settings are performed for the regression task type of each sub-dataset.

[0027] S102: Select an adapted backbone network and pseudo-label uncertainty estimation network based on different types of regression tasks.

[0028] Specifically, this application first needs to construct semi-supervised regression datasets of different proportions. The semi-supervised regression method targeted by this application covers two regression tasks, namely age prediction regression task and similarity prediction regression task.

[0029] For the age prediction regression task, its core is to predict the age of an individual based on a face image. To meet the requirements of this task, this application introduces the UTKFace dataset and the IMDB-WIKI dataset. The UTKFace dataset is a database dedicated to age estimation of face images, containing a total of 23,705 face images, with labels ranging from 1 to 116 years old. Taking into account the actual application scenarios and data distribution characteristics, in this application, only samples aged 21 to 60 are screened out. After a rigorous screening and modification process, the final dataset contains 13,144 training samples, 3,287 test samples, and 2,629 validation samples, all of which are randomly selected from the original training set. IMDB-WIKI is a large-scale face age estimation dataset with more than 523,000 face images and their corresponding age labels. In this application, a screened dataset is used, of which 191.5K images are used for training and 11.0K images are used for validation and testing. The age range of this dataset is large, from 0 to 186 years old, and the number of images corresponding to each age varies significantly, ranging from 1 to 7,149.

[0030] For the similarity prediction regression task, the goal is to evaluate the similarity of sentence pairs from different sources. To this end, this application uses the STS-B dataset. STS-B is a semantic text similarity dataset that contains 7.2K sets of sentence pairs. These data are all derived from real-world texts such as news, pictures, and video subtitles, as well as natural language inference data. The target similarity score for each pair of sentences is a continuous value between 0 and 5. In this application, the dataset is divided into 5.2K sets of sentence pairs for training. At the same time, the validation set and test set are both 1K sets of sentence pairs, and the data distribution is balanced.

[0031] This application uses different backbone networks based on the type of dataset. For image datasets, such as the UTKFace dataset and the IMDB-WIKI dataset, this application uses the ResNet-50 network structure; for text datasets, such as the STS-B dataset, this application uses the BiLSTM framework.

[0032] Taking the UTKFace dataset as an example, to ensure the objectivity and accuracy of the evaluation results, the dataset of test image samples is invisible during training and is only used to evaluate the prediction accuracy of the classification model at the end. In practice, this application uses the UTKFace dataset to construct semi-supervised regression datasets of different proportions. Specifically, when the proportion of labeled data is set to 10%, 10% of the dataset is randomly sampled from the training set as the labeled dataset throughout the training process, and the remaining data is used as the unlabeled dataset.

[0033] For example, the labeled dataset Denoted as: in, is the i-th labeled training sample and its corresponding label, and N is the total number of samples in the labeled dataset. Denoted as: in, is the jth unlabeled sample, and M is the total number of samples in the unlabeled dataset. In this way, semi-supervised regression datasets of different proportions can be flexibly constructed to meet different experimental needs and scenarios.

[0034] Figure 2 Schematic diagram of the structure of the pseudo-label uncertainty estimation network provided in an embodiment of the present application. The pseudo-label uncertainty estimation network structure used by different data sets is the same. The pseudo-label uncertainty estimation network adopts a lightweight architecture design, specifically including: The pseudo-label uncertainty estimation network is a multi-layer perceptron consisting of only a single hidden layer, which has the ability to approximate any continuous function. It evaluates the uncertainty of the pseudo-label by mapping the pseudo-label of the unlabeled sample to the uncertainty value.

[0035] The expression for mapping the pseudo label of an unlabeled sample to an uncertainty value is: Among them, z j represents the logarithmic variance of the uncertainty of the j-th unlabeled sample, represents the jth unlabeled sample, g φ represents the pseudo-label uncertainty estimation network, represents the uncertainty of the pseudo label of the jth unlabeled sample.

[0036] Specifically, the logarithmic variance is used to ensure the stability of the calculation during actual implementation and to avoid the situation where the denominator is zero during the calculation process.

[0037] S103: Iteratively execute the model training steps until the stopping condition is met to obtain a trained regression model.

[0038] The model training process includes the following steps: A two-layer optimization framework is constructed to jointly optimize the parameters of the regression model and the parameters of the pseudo-label uncertainty estimation network.

[0039] The two-layer optimization framework is constructed by alternately updating the parameters of the regression model and the parameters of the pseudo-label uncertainty estimation network, so that the uncertainty of the pseudo-labels of unlabeled samples can suppress unreliable pseudo-labels, while ensuring the learning of unlabeled samples whose pseudo-labels are correct but there is an error between the predicted value and the pseudo-label.

[0040] The regression model is used to predict the sample pseudo labels, the pseudo labels of the unlabeled samples are modeled with heteroscedasticity, the lower layer loss is obtained, and the parameters of the regression model are updated based on the lower layer loss.

[0041] The lower layer losses include supervised loss and heteroskedastic unsupervised loss.

[0042] The lower layer loss is obtained as follows: in, is the lower layer loss, To monitor losses, is the heteroskedastic unsupervised loss that takes into account the uncertainty of unlabeled samples, θ is the parameter of the regression model, and φ is the parameter of the pseudo-label uncertainty estimation network.

[0043] Specifically, in semi-supervised regression tasks, the acquisition of the lower-level loss needs to comprehensively consider the information of labeled data and unlabeled data. For labeled data, the supervised loss is calculated through the regression model. Specifically, the regression model is used to predict the labeled samples, and the corresponding predicted values ​​are obtained. The mean squared error between the predicted values ​​and the true labels is then calculated as the supervised loss, which is expressed as follows: Among them, y i is the i-th labeled sample Tags, B l is a dataset obtained by sampling labeled samples. is the regression model f θ For the i-th labeled sample The predicted value of .

[0044] In the semi-supervised regression scenario, the goal is to jointly learn the true regression model using both labeled and unlabeled data. However, when pseudo-labeling algorithms are applied in semi-supervised regression, the pseudo-labels assigned to unlabeled data are often unstable. Moreover, the uncertainty or noise in the pseudo-labels is not fixed, especially when the label estimation process involves external sources (such as weak supervision, self-training, or noisy teacher models). This uncertainty can change.

[0045] To address this problem, the pseudo labels of unlabeled samples are modeled for heteroscedasticity. Specifically, it is assumed that the noise of the pseudo labels satisfies the mean of 0 and the variance is That is, for each unlabeled sample Pseudo labels Assuming that its variance depends on the input sample itself, it can be expressed as: in, Is the regression model input unlabeled samples The true predicted value, ε j is the noise term, the variance part Changes as the input changes.

[0046] The expression for modeling heteroscedasticity of pseudo labels of unlabeled samples is: in, is the predicted value of the jth unlabeled sample, is the pseudo label of the jth unlabeled sample, σ j is the uncertainty of the pseudo label of the jth unlabeled sample, is the negative log-likelihood of the jth sample, and p is the probability density of the pseudo-label given the unlabeled sample.

[0047] Furthermore, for a specific dataset B sampled from unlabeled data u , the unsupervised loss is the sum of the negative log-likelihood losses of all samples. The specific expression is as follows: in, is the heteroskedastic unsupervised loss.

[0048] If the assumption is that homoscedasticity is the case, σ is fixed to 1 and the unsupervised loss is the standard mean squared error (MSE).

[0049] In unsupervised loss, when there are unlabeled samples during training Assigned incorrect pseudo labels When , the model can increase the uncertainty of the pseudo label, that is, adjust σ j, to reduce the impact of the pseudo label on the overall loss, thereby alleviating the regression model f θ In contrast, the standard mean squared error loss strictly fits the incorrect pseudo-labels, resulting in increased confirmation bias as training progresses. In this application, appropriate uncertainty values ​​can be dynamically assigned to pseudo-labels of varying degrees of error during training, allowing the model to more flexibly and robustly handle the uncertainty of pseudo-labels.

[0050] Use the gradient descent method to update the parameters of the regression model. The expression for updating the parameters of the regression model based on the lower layer loss is: Among them, θ t+1 (φ t ) is the parameter of the regression model at the t+1th iteration, θ t is the parameter of the regression model at the tth iteration, φ t is the parameter of the pseudo-label uncertainty estimation network at the tth iteration, α is the learning rate of the regression model, α>0, Loss for the lower layer The gradient of the regression model with respect to its parameters θ.

[0051] The upper layer loss is obtained based on the updated regression model, and the parameters of the pseudo label uncertainty estimation network are updated based on the upper layer loss.

[0052] Specifically, the upper layer loss can be understood as the regression loss of labeled data. In this application, the specific goal is to generate well-calibrated uncertainty estimates to protect the regression model from incorrect pseudo-labels while ensuring good generalization to labeled data. Specifically, by continuously paying attention to The effect on the labeled data and the optimization of the parameter φ are performed.

[0053] The upper layer loss is obtained as follows: in, is the upper loss, is the data sampled from the labeled dataset, y k is the kth sample Tags, To update the parameters The kth sample obtained under the regression model The predicted value obtained, θ * (φ) indicates that the parameters of the regression model are updated under the influence of the parameters of the pseudo-label uncertainty estimation network.

[0054] It needs to be explained that From the labeled dataset The samples obtained by sampling are different from the samples used to calculate the supervision loss of the lower layer loss in one iteration, that is,

[0055] The purpose of the framework for obtaining the upper layer loss is to ensure that by optimizing the parameter φ, the regression network optimized by the lower layer network can perform well on unbiased and trustworthy labeled data.

[0056] Specifically, in the semi-supervised regression task, the parameter update of the pseudo-label uncertainty estimation network plays a key role in improving the overall performance of the model. φ The parameter update mechanism of the proposed method is innovatively designed to improve computational efficiency and effectively utilize the information in the training data. The traditional pseudo-label uncertainty estimation network directly maps each unlabeled sample to its logarithmic variance z j , but this method has the problem of low computational efficiency. The pseudo-label uncertainty estimation network g in this application φ A more efficient input mapping method is used. It does not directly process the unlabeled samples themselves, but converts the predicted values ​​of the regression model into and its corresponding pseudo labels As input. This mapping can be expressed as: z j Represents the logarithmic variance of the uncertainty of the j-th unlabeled sample.

[0057] The architecture of the pseudo-label uncertainty estimation network is a multi-layer perceptron (MLP) with a single hidden layer. Despite its simple structure, such a network can theoretically approximate any continuous function defined on a compact set according to the universal approximation theorem. It should be noted that although r j and They all come from the same regression model, but they are not exactly the same in this work. This difference is due to the different random data augmentation methods used during training. Generate pseudo labels And r j comes from the same input of the strongly augmented version.

[0058] The expression for updating the parameters of the pseudo-label uncertainty estimation network based on the upper layer loss is:

[0059] Among them, φ t+1 is the parameter of the pseudo-label uncertainty estimation network at the t+1th iteration, β is the learning rate of the pseudo-label uncertainty estimation network, β>0,

[0060] Upper layer loss The gradient with respect to the parameters φ of the pseudo-label uncertainty estimation network.

[0061] Since the above formula The second-order gradient calculation is introduced, so the entire regression model needs to be expanded by the second-order derivative. This process is costly and inefficient for deep models. This application proposes a more efficient approximation algorithm: assuming that φ is only related to the parameters of the regression head (a fully connected layer), this assumption means that in the above formula, only the second-order derivative of the regression head needs to be expanded, and the entire regression network does not need to be calculated. Since the parameters of the regression head are far less than the parameters of the entire regression model, compared with the traditional two-layer optimization algorithm, this application has significantly improved computational efficiency.

[0062] Furthermore, the stopping condition in this application is that the model training reaches a preset number of iterations, that is, the number of model training steps executed reaches a preset number of iterations. The preset number of iterations is 30 epochs, and the preset number of iterations per epoch is related to the number of batches of unlabeled samples. If the stopping condition is not met, the regression model parameters will continue to be updated.

[0063] Specifically, in the semi-supervised regression task, in order to continuously optimize the model performance, it is necessary to use the labeled data set and the unlabeled data set obtained by sampling the new batch to advance the learning process of the regression task. Subsequently, the model training step can be performed again. These steps together constitute a conventional two-layer optimization framework for iteratively updating the parameters θ of the regression model and the parameters φ of the pseudo-label uncertainty estimation network. Theoretically, it is an intuitive and simple strategy to jointly optimize the parameters θ of the regression model and the parameters φ of the pseudo-label uncertainty estimation network in an end-to-end training process. However, this strategy has fundamental limitations, mainly reflected in its inability to effectively distinguish between the following two different situations. 1. Difficult but correct samples. In this case, the pseudo-label is correct, but due to the high complexity or noise interference of the sample itself, the model prediction value There is a large error between the pseudo-label and the label, which results in a large square error. Although such samples are more difficult to learn, they are crucial for improving the generalization ability and performance of the model because they often contain key information that the model needs to learn in complex scenarios. 2. Simple but wrong samples: The simplicity mentioned here refers to the characteristics of the sample itself, that is, the characteristics of the sample are clearer and easier to identify. However, pseudo labels It is wrong, which may be caused by errors in the pseudo-label generation process or other factors. It is close to the true label, but due to the error of the pseudo label, it will also lead to a large square error. In the end-to-end joint optimization strategy, for the above two cases, the optimization objective will prompt σ jThis is because a larger square error will be regarded as a sample with higher uncertainty, thus increasing σ j To suppress the impact of these samples on model training. However, this approach will have a serious negative impact. It will inhibit the learning of difficult but correct samples, making it impossible for the model to obtain sufficient information from these samples with important learning value, thereby limiting the improvement of model performance.

[0064] In order to overcome the limitations of end-to-end joint optimization, this application proposes an innovative two-layer optimization framework. The core idea of ​​this framework is to use the mechanism of two-layer optimization to make the uncertainty estimate σ j It can selectively suppress unreliable pseudo labels while still promoting learning of difficult but effective samples, and can better reduce error accumulation and overfitting of the model to erroneous pseudo labels.

[0065] S104: After the model training is completed, the regression task to be processed is input into the trained regression model, and the corresponding regression result is output.

[0066] Specifically, in this application, when the model training meets the stopping condition, the model training step is stopped, and the parameters of the regression model that performs best on the validation set during the training process are used as the parameters of the trained regression model.

[0067] S105: Select different evaluation indicators according to different sub-datasets to evaluate the accuracy of the regression results.

[0068] Specifically, if the regression result is consistent with the label of the regression task, it is considered that the regression model's prediction is successful, and the corresponding evaluation index performance is better. The prediction success rate of the regression model can be indirectly reflected through these evaluation indicators.

[0069] Different evaluation metrics are selected based on different sub-datasets to evaluate the accuracy of the regression results, including:

[0070] The evaluation indicators are mean absolute error, absolute coefficient and mean square error.

[0071] The mean absolute error is: Among them, MAE is the mean absolute error, n is the number of samples in the batch, Represents the predicted value of the i-th sample, θ * is the parameter of the optimal regression model obtained after training, y i is the true value of the corresponding sample. MAE is an indicator to measure the prediction error of the model. The smaller the value, the closer the prediction result is to the true value and the better the model performance.

[0072] The absolute coefficient is: Among them, R2 is the absolute coefficient, is the mean of the corresponding true values. R 2 It is an indicator for evaluating the goodness of fit of the regression model. The larger the value (the upper limit is 1), the better the model can explain the changes in the target variable.

[0073] The mean square error is: Here, MSE is the mean squared error. A lower MSE indicates that the model has better prediction performance overall, that is, the predicted value is closer to the true value.

[0074] Table 1 is a performance comparison of the UTKFace dataset under semi-supervised settings of different proportions provided by the embodiments of the present application. The top of Table 1 lists different proportions, represented by γ, which are 5%, 10%, and 20% respectively. The results are averaged based on six random runs, and the results are presented in the form of "mean ± standard deviation". The best results are shown in bold. The Method in Table 1 is the method part, which shows the compared semi-supervised regression algorithms. Supervised is the lower bound of the SSR method, which is trained only on labeled data without using any unlabeled data. Fully-Supervised is the upper bound of the SSR method, which is trained on all labeled and unlabeled data, and assumes that the true labels of the unlabeled data are known. Compared with other methods, the algorithm proposed in this application always obtains the best or suboptimal results at all proportions. In particular, when labeled data is scarce (γ = 5%), the method proposed in this application (Ours) is 2.7% lower in MAE than the suboptimal algorithm SimRegMatch, and in R 2 It increased by 2.1%.

[0075] Table 1

[0076]

[0077] The embodiment of the present application also provides a semi-supervised regression device 300 based on the pseudo-label heteroscedasticity hypothesis, such as Figure 3 As shown, the device includes: a construction module 301, a selection module 302, a training module 303, a regression result output module 304 and an evaluation module 305.

[0078] The construction module 301 is used to construct a semi-supervised regression dataset, wherein the semi-supervised regression dataset includes multiple sub-datasets, each sub-dataset corresponds to a different type of regression task, and different semi-supervised settings are performed for the regression task type of each sub-dataset.

[0079] The selection module 302 is used to select an adapted backbone network and a pseudo-label uncertainty estimation network according to different types of regression tasks.

[0080] The training module 303 is used to iteratively execute the model training steps until the stopping condition is met, thereby obtaining a trained regression model. The model training steps include: constructing a two-layer optimization framework to jointly optimize the parameters of the regression model and the parameters of the pseudo-label uncertainty estimation network. The regression model is used to predict the sample pseudo-labels, and the heteroscedasticity modeling is performed on the pseudo-labels of the unlabeled samples. The lower layer loss is obtained, and the parameters of the regression model are updated based on the lower layer loss. The upper layer loss is obtained based on the updated regression model, and the parameters of the pseudo-label uncertainty estimation network are updated based on the upper layer loss.

[0081] The regression result output module 304 is used to input the regression task to be processed into the trained regression model after the model training is completed, and output the corresponding regression result.

[0082] The evaluation module 305 is used to select different evaluation indicators according to different sub-datasets to evaluate the accuracy of the regression results.

[0083] Some modules in the apparatus described herein may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0084] The devices or modules described in the above application embodiments can be implemented by computer chips or physical devices, or by products with certain functions. For ease of description, the above devices are described separately by function in various modules. When implementing the embodiments of this application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0085] The methods, devices, or modules described in this application can be implemented in the form of computer-readable program code. The controller can be implemented in any appropriate manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in the form of pure computer-readable program code, it is entirely possible to implement the same function of the controller in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the means for implementing various functions may be considered to be both a software module for implementing the method and a structure within a hardware component.

[0086] like Figure 4 As shown, an embodiment of the present application also provides a semi-supervised regression server based on the pseudo-label heteroscedasticity assumption, including a memory 401 and a processor 402; the memory 401 is used to store computer-executable instructions; the processor 402 is used to execute computer-executable instructions to implement a semi-supervised regression method based on the pseudo-label heteroscedasticity assumption described above in the embodiment of the present application.

[0087] An embodiment of the present application also provides a computer-readable storage medium, which stores executable instructions. When a computer executes the executable instructions, it can implement the semi-supervised regression method based on the pseudo-label heteroscedasticity assumption described above in the embodiment of the present application.

[0088] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of the present application can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, or it can be embodied in the implementation process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the method described in the embodiment of the present application.

[0089] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to in detail. Each embodiment focuses on the differences from other embodiments. All or part of this application can be used in many general or special computer system environments or configurations.

[0090] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some or all of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. A semi-supervised regression method based on pseudo-label heteroscedasticity assumption, characterized in that: include: Constructing a semi-supervised regression dataset; wherein the semi-supervised regression dataset includes multiple sub-datasets, each sub-dataset corresponds to a different type of regression task, and different semi-supervised settings are performed for the regression task type of each sub-dataset; According to different types of regression tasks, select the appropriate backbone network and pseudo-label uncertainty estimation network; Iterate the model training steps until the stopping condition is met to obtain a trained regression model; The model training step includes: constructing a two-layer optimization framework to jointly optimize the parameters of the regression model and the parameters of the pseudo-label uncertainty estimation network; Use the regression model to predict the sample pseudo-label, perform heteroscedasticity modeling on the pseudo-label of the unlabeled sample, obtain the lower layer loss, and update the parameters of the regression model based on the lower layer loss; Obtain the upper layer loss based on the updated regression model, and update the parameters of the pseudo-label uncertainty estimation network based on the upper layer loss; After the model training is completed, the regression task to be processed is input into the trained regression model and the corresponding regression results are output; Different evaluation metrics are selected according to different sub-datasets to evaluate the accuracy of the regression results.

2. The semi-supervised regression method based on pseudo-label heteroscedasticity hypothesis according to claim 1, characterized in that The pseudo-label uncertainty estimation network adopts a lightweight architecture design, specifically including: The pseudo-label uncertainty estimation network is a multi-layer perceptron consisting of only a single hidden layer. It has the ability to approximate any continuous function and evaluates the uncertainty of the pseudo-label by mapping the pseudo-label of unlabeled samples to uncertainty values.

3. The semi-supervised regression method based on pseudo-label heteroscedasticity hypothesis according to claim 2, characterized in that: The expression for mapping the pseudo label of an unlabeled sample to an uncertainty value is: Among them, z j represents the logarithmic variance of the uncertainty of the j-th unlabeled sample, represents the jth unlabeled sample, g φ represents the pseudo-label uncertainty estimation network, represents the uncertainty of the pseudo label of the jth unlabeled sample.

4. The semi-supervised regression method based on pseudo-label heteroscedasticity hypothesis according to claim 3, characterized in that The expression for modeling heteroscedasticity of pseudo labels of unlabeled samples is: in, is the predicted value of the jth unlabeled sample, is the pseudo label of the jth unlabeled sample, σ j is the uncertainty of the pseudo label of the jth unlabeled sample, is the negative log-likelihood of the jth sample, and p is the probability density of the pseudo-label given the unlabeled sample.

5. The semi-supervised regression method based on pseudo-label heteroscedasticity hypothesis according to claim 4, characterized in that: The lower layer loss includes supervised loss and heteroskedastic unsupervised loss; The lower layer loss is obtained as follows: in, is the lower layer loss, To monitor losses, is the heteroskedastic unsupervised loss that takes into account the uncertainty of unlabeled samples, θ is the parameter of the regression model, and φ is the parameter of the pseudo-label uncertainty estimation network.

6. The semi-supervised regression method based on pseudo-label heteroscedasticity hypothesis according to claim 5, characterized in that: The upper layer loss is obtained as follows: in, is the upper loss, is the data sampled from the labeled dataset, y k is the kth sample Tags, To update the parameters The kth sample obtained under the regression model The predicted value obtained, θ * (φ) indicates that the parameters of the regression model are updated under the influence of the parameters of the pseudo-label uncertainty estimation network.

7. The semi-supervised regression method based on pseudo-label heteroscedasticity hypothesis according to claim 1, characterized in that The two-layer optimization framework is constructed by: By alternately updating the parameters of the regression model and the parameters of the pseudo-label uncertainty estimation network, the uncertainty of the pseudo-labels of unlabeled samples can suppress unreliable pseudo-labels, while ensuring the learning of unlabeled samples whose pseudo-labels are correct but there is an error between the predicted value and the pseudo-label.

8. The semi-supervised regression method based on pseudo-label heteroscedasticity hypothesis according to claim 6, characterized in that: The expression for updating the parameters of the regression model based on the lower layer loss is: Among them, θ t+1 (φ t ) is the parameter of the regression model at the t+1th iteration, θ t is the parameter of the regression model at the tth iteration, φ t is the parameter of the pseudo-label uncertainty estimation network at the tth iteration, α is the learning rate of the regression model, α>0, Loss for the lower layer The gradient of the regression model with respect to its parameters θ.

9. The semi-supervised regression method based on pseudo-label heteroscedasticity hypothesis according to claim 8, characterized in that The expression for updating the parameters of the pseudo-label uncertainty estimation network based on the upper layer loss is: Among them, φ t+1 is the parameter of the pseudo-label uncertainty estimation network at the t+1th iteration, β is the learning rate of the pseudo-label uncertainty estimation network, β>0, Upper layer loss The gradient with respect to the parameters φ of the pseudo-label uncertainty estimation network.

10. The semi-supervised regression method based on pseudo-label heteroscedasticity hypothesis according to claim 9, characterized in that: Different evaluation indicators are selected according to different sub-datasets to evaluate the accuracy of the regression results, including: The evaluation indicators are mean absolute error, absolute coefficient and mean square error; The mean absolute error is: Among them, MAE is the mean absolute error, n is the number of samples in the batch, Represents the predicted value of the i-th sample, θ * is the parameter of the optimal regression model obtained after training, y i is the true value of the corresponding sample; The absolute coefficient is: Among them, R 2 is the absolute coefficient, is the mean of the corresponding true values; The mean square error is: Among them, MSE is the mean square error.

Images