Unbalanced regression problem solving method based on multi-task learning

Abstract

The invention is suitable for the technical field of data driving, and provides an unbalanced regression problem solving method based on multi-task learning. According to the method, through a multi-task learning framework, the knowledge of the auxiliary task is effectively utilized, and the prediction accuracy of the main task under the unbalanced data condition is remarkably improved; by utilizing additional information and constraints provided by the auxiliary task, the model can better learn in a data scarce region, so that challenges brought by unbalanced data are relieved; in combination with self-adaptive gradient scaling and dynamic adjustment of the weight of the loss function, gradient contributions of different tasks in the training process are effectively balanced, and the stability and convergence speed of the training process are improved; after training, the method supports pruning of auxiliary task branches, so that the size of the model is equivalent to that of a single-task learning model. The simplification capability ensures high efficiency and resource friendliness of the multi-task learning method in practical application, and the method is suitable for being deployed in a resource-limited environment.

Description

Technical Field

[0001] The present invention belongs to the field of data-driven technology, and in particular relates to a method for solving imbalanced regression problems based on multi-task learning. Background Art

[0002] In modern industrial production, data-driven decision-making has become an important means to improve efficiency and reduce costs. However, as a key statistical tool, regression analysis often faces the problem of data imbalance, which can lead to a decrease in the prediction accuracy of the model in some important categories, thus affecting the performance of the entire system. For example, in applications such as quality control and equipment failure prediction, samples from minority categories often carry key information, while samples from majority categories may occupy the vast majority of the data set. This imbalance makes it difficult for traditional regression models to effectively learn the features of minority categories, resulting in wrong decisions. Therefore, it is of great practical significance to develop new methods that can effectively deal with the problem of unbalanced regression.

[0003] Regression tasks under imbalanced data face the following unique challenges: 1) Lack of clear category boundaries: Unlike classification tasks, regression tasks deal with continuous target values ​​and have no clear class boundaries, which affects the learning of data patterns.

[0004] 2) Limitations of sampling and enhancement methods: Direct resampling and commonly used data enhancement methods may destroy the relationship between features and target variables in regression tasks, resulting in the model being unable to accurately capture the true data pattern.

[0005] 4) Overfitting problem: Continuous target values ​​make the model prone to overfitting to noise and specific patterns in sparse areas, reducing the generalization ability of the model.

[0006] 5) Computational complexity: Although ensemble methods perform well for regression tasks, their high computational requirements limit their use in industrial applications, so more efficient methods are needed to handle imbalanced data.

[0007] In response to the above challenges, the present invention proposes a solution to the imbalanced regression problem based on multi-task learning. Summary of the invention

[0008] The purpose of the present invention is to provide a solution to the imbalanced regression problem based on multi-task learning, aiming to solve the problems raised in the above background technology.

[0009] To achieve the above object, the present invention provides the following technical solutions: A solution to the imbalanced regression problem based on multi-task learning includes the following steps: Step S1, multi-task learning model design: Under the hard parameter sharing framework, design a multi-task learning network architecture and define a shared basic network, which is used to extract common features of input data. In the case of data scarcity, the auxiliary task introduces gradient information different from the main task; design a task-specific layer, which is used to process specific information related to the main task and the auxiliary task, and capture the personalized features of each task; Step S2, loss function and optimization strategy: define the total loss function of the main task and auxiliary tasks, including the loss of the main task and the losses of all auxiliary tasks, and balance them through their respective weights; Step S3, adaptive loss function: optimize the learning effect of the main task during the training process by adaptive gradient scaling and dynamic adjustment of the loss function weight.

[0010] Furthermore, in the multi-task learning network architecture, the model parameters Divided into shared parameters and task-specific parameters , shared parameters Used to learn general features for all tasks, specific task parameters Used to capture the personalized characteristics of each task; The shared base network receives input x , and processed through shared parameters to generate a shared feature; the definition of the shared basic network is as follows: Formula 1: ; in represents a shared basic network; Represents shared parameters; Represents shared features; The top-level network for each task is built on shared features Above, specifically expressed as: Formula 2: ; in Representation Task i The predicted value of Representation Task t network; Represents a specific task parameter.

[0011] Furthermore, the total loss function of the main task and the auxiliary task is: Formula 3: ; in is the total loss function of the main task and auxiliary tasks; It is the loss of the main task; It is i The loss of auxiliary tasks; are the weights of the main task and auxiliary task respectively; during training, the model optimizes the shared parameters by considering the losses of all tasks and task-specific parameters ; During the training process, the model parameters By using the learning rate Perform gradient descent to update: Formula 4: ; in Indicates t+ Model parameters for 1 iteration; Indicates t The model parameters of the iteration; t Indicates t Iterations; is the total loss Relative to model parameters The gradient is: Formula 5: ; in, Relative to the parameter The gradient of K It is K Auxiliary tasks.

[0012] Furthermore, the specific method of the adaptive gradient scaling is as follows: At each iteration, the loss function of the main task and each auxiliary task is calculated with respect to the shared parameters The gradient of Let the gradient of the main task be: Formula 6: ; in represents the gradient of the main task; Relative to the parameter The gradient of For i Auxiliary tasks, the gradient is: Formula 7: ; in Indicates i The gradient of the auxiliary task; For each task, the L2 norm of the gradient is calculated to measure its magnitude: Formula 8: ; Formula 9: ; in represents the main task gradient; Represents auxiliary tasks iThe gradient of Calculate the scaling factor for each task based on the L2 norm of the gradient of each task : Formula 10: ; in represents the gradient of the first auxiliary task; represents the gradient of the Kth auxiliary task; Representation Task i The gradient of Multiply the gradient of each task by its corresponding scaling factor and the current loss weight : Formula 11: ; in Representation Task i Adjusted gradient; After applying the scaling factors and weights, the model parameters are updated via backpropagation : Formula 12: ; in, is the weighted sum of the gradients of the main task and all auxiliary tasks.

[0013] Furthermore, the specific method of dynamically adjusting the loss function is as follows: At the beginning of training, the weights of all tasks are set to similar values ​​to promote the learning of common features; as the training progresses, the weight of the main task gradually increases and the weight of the auxiliary task gradually decreases; in the last cycle, the weight of the main task reaches 1 and the weight of the auxiliary task drops to 0.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention uses a multi-task learning framework to effectively utilize the knowledge of auxiliary tasks and significantly improve the prediction accuracy of the main task under unbalanced data conditions.

[0015] 2. The present invention utilizes the additional information and constraints provided by auxiliary tasks to enable the model to learn better in areas where data is scarce, thereby alleviating the challenges brought by unbalanced data.

[0016] 3. The present invention combines adaptive gradient scaling with dynamic adjustment of loss function weights to effectively balance the gradient contributions of different tasks during the training process, thereby improving the stability and convergence speed of the training process.

[0017] 4. After training, the present invention can prune auxiliary task branches to make the model size comparable to that of the single-task learning model. This pruning capability ensures the efficiency and resource-friendliness of multi-task learning methods in practical applications, and is suitable for deployment in resource-constrained environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of the present invention.

[0019] Figure 2 Schematic diagram of the significant imbalance in the distribution of NH3 emission data.

[0020] Figure 3 A comparison chart of the model prediction value and the true value in the test set of a specific low-sample area.

[0021] Figure 4 The error curves of the multi-task learning model and the single-task learning model on the test set in a specific low-sample area. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0023] like Figure 1 As shown, it is a flowchart of a method for solving the imbalanced regression problem based on multi-task learning provided by an embodiment of the present invention. The method is implemented by a software system, runs on an Ubuntu 18.04 operating system, and uses the PyTorch 2.1.0 package to build a multi-task learning model.

[0024] First, we collect data related to the industrial imbalanced data regression task and screen out key feature variables that can reflect the target characteristics, such as input variables, output variables, and features required for auxiliary tasks. Then, we combine actual industrial data with experimental data to construct a full-condition dataset suitable for regression tasks.

[0025] Next, in PyCharm software, a multi-task learning model is built. The model design includes two parts: a shared basic network and a task-specific layer. The shared basic network is used to extract the common features of the input data and can effectively capture the correlation between different tasks. All tasks share the shared basic network to effectively capture the correlation between different tasks. In the case of data scarcity, the auxiliary task provides additional information and constraints for the learning of the main task by introducing gradient information different from the main task. These additional information and constraints help the model learn features better. In addition, the loss of the auxiliary task introduces a regularization effect. Due to the existence of the auxiliary task, the model needs to consider the loss of the main task and the auxiliary task at the same time during the optimization process, which helps to balance the parameters of different tasks and prevent the main task from overfitting to the limited training data due to insufficient data. On top of the shared basic network, each task has an independent specific layer that focuses on processing information related to the task. The task-specific layer is used to process specific information related to the main task and the auxiliary task to improve the prediction accuracy of the main task.

[0026] Before training, the optimization of model parameters is based on the dataset of industrial imbalanced data regression tasks, combined with industrial data collected from actual application scenarios. The data are preprocessed, including feature selection and standardization, to ensure data quality. In order to effectively train and validate the model, we divide the dataset into training set, validation set, and test set. Select appropriate optimizers (such as Adam), learning rates, training cycles, batch sizes, and time steps, etc. During training, the training set is used to train model parameters, and the validation set is used to adjust hyperparameters during training. In addition, the learning effect of the main task is optimized during training by adaptive gradient scaling and dynamic adjustment of the loss function weights.

[0027] Finally, the test set is used to verify the actual effect of the model and evaluate the performance of the main task and auxiliary tasks to ensure the effectiveness of the model under imbalanced data conditions.

[0028] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0029] 1. Industrial imbalance regression dataset; 1.1 Real-time estimation of ammonia emissions in commercial vehicle selective catalytic reduction (SCR) systems; The effectiveness and feasibility of the proposed method are verified by an actual industrial case study of real-time estimation of ammonia emissions in a commercial vehicle SCR system. The SCR system is a key technology for reducing nitrogen oxide (NOx) emissions in commercial vehicles, catalyzing NOx into harmless nitrogen and water by injecting urea. However, ammonia (NH3), as a necessary intermediate in this process, poses significant risks to the environment and health if emitted, and may affect the accuracy of NOx sensors, thereby affecting system control. The widespread use of ammonia sensors in practical applications is limited due to their high cost, limited reliability, narrow measurement range, and large errors. Therefore, accurate prediction of NH3 emissions is critical to optimizing SCR system performance, ensuring compliance, and reducing environmental impact.

[0030] Under ideal conditions, NH3 in urea solution will react completely with nitrogen oxides, resulting in no NH3 emissions. However, in actual applications, changes in reaction conditions (such as temperature, urea injection rate, and exhaust flow) may lead to incomplete reactions, causing some NH3 to bypass the reaction with NOx and be emitted directly, a phenomenon known as ammonia slip. Therefore, most data points show zero emissions, while non-zero NH3 emission values ​​are relatively rare and fluctuate significantly, resulting in an unbalanced data set.

[0031] 1.2 Data collection experiment and data processing method; The NH3 emission data collection experiment was conducted on a calibrated engine test bench equipped with a CuZeolite catalyst SCR system using a commercial diesel engine that meets current emission standards. The test bench is equipped with a SHENCKHD460 dynamometer that can accurately control engine load and speed to replicate real driving conditions. The Engine Control Unit (ECU) continuously provides key input parameters to achieve real-time NH3 emission estimation, including exhaust flow, urea injection rate, SCR system temperature, and NOx concentration at the inlet and outlet. NH3 concentration is measured using a high-precision gas analyzer (HORIBA MEXA7100EGR) with an accuracy of ±1 ppm to ensure reliable data. The experiment follows a standard procedure under controlled conditions, covering a variety of operating conditions such as low load, high load and transient scenarios to collect comprehensive NH3 emission data.

[0032] After processing, the entire dataset contains about 36k data points. Figure 2 As shown, the data distribution reveals a significant imbalance in NH3 emissions, with the vast majority of values ​​concentrated in the zero emission range. To ensure the robustness of the evaluation, balanced validation and test sets were manually constructed to reduce the impact of this imbalance. The dataset is divided into 25.2k samples for training and 10.8k samples equally divided into validation and test sets.

[0033] 2. Design and implementation of multi-task learning model; 2.1 Multi-task learning network architecture; In the multi-task learning network architecture under the hard parameter sharing framework, the model parameters Divided into shared parameters and task-specific parameters , shared parameters Used to learn general features for all tasks, specific task parameters Used to capture the personalized characteristics of each task.

[0034] Shared Base Network: The shared base network receives input x , and processed through shared parameters to generate a shared feature, which is the common basis for all tasks and provides input for subsequent task-specific layers. The definition of the shared basic network is: Formula 1: ; in represents a shared basic network; Represents shared parameters; Represents shared features.

[0035] Task-specific layers: The top layer of each task is built on shared features Above, these task-specific layers receive shared features as input and through task-specific parameters Further processing is performed to produce output for each task. Specifically expressed as: Formula 2: ; in Representation Task i The predicted value of Representation Task t network; Represents a specific task parameter.

[0036] Loss function and optimization: Unlike traditional multi-task learning, the auxiliary tasks in this invention are not intended to improve their own performance, but to enhance the learning effect of the main task by sharing knowledge. Auxiliary tasks support the learning of the main task by introducing different gradient information. The total loss function of the main task and auxiliary tasks is the goal of model optimization, which includes the loss of the main task and the loss of all auxiliary tasks, and is balanced by their respective weights.

[0037] The total loss function of the main task and auxiliary task is: Formula 3: ; in is the total loss function of the main task and auxiliary tasks; It is the loss of the main task; It is i The loss of auxiliary tasks; are the weights of the main task and auxiliary task respectively; during training, the model optimizes the shared parameters by considering the losses of all tasks and task-specific parameters ; During the training process, the model parameters By using the learning rate Perform gradient descent to update: Formula 4: ; in Indicates t+ Model parameters for 1 iteration; Indicates t The model parameters of the iteration; t Indicates t iterations (for each mini-batch); is the total loss Relative to model parameters The gradient is: Formula 5: ; in, Relative to the parameter The gradient of K It is K Auxiliary tasks.

[0038] In each iteration, the model parameters are adjusted according to the total loss Relative to model parameters Gradient to update.

[0039] 2.2 Adaptive gradient scaling and dynamic adjustment of loss function weights during model training; In multi-task learning, differences in data distribution and gradient update rates between tasks may lead to an imbalance in gradient size, thus affecting the optimization of shared parameters, especially the performance of the main task. When the gradient of the auxiliary task is significantly larger than the gradient of the main task, the shared parameters of the model will be dominated by the auxiliary task, thereby reducing the attention and performance of the main task; on the contrary, when the gradient of the auxiliary task is too small, their contribution to the optimization of the shared parameters is negligible and cannot effectively support the main task. To this end, the present invention adopts the Adaptive Gradient Scaling (AGS) method to balance the gradient contributions of different tasks.

[0040] At each iteration, the loss function of the main task and each auxiliary task is calculated with respect to the shared parameters The gradient of the main task is: Formula 6: ; in represents the gradient of the main task; Relative to the parameter The gradient of For i Auxiliary tasks, the gradient is: Formula 7: ; in Indicates i The gradient of the auxiliary task; For each task, the L2 norm of the gradient is calculated to measure its magnitude: Formula 8: ; Formula 9: ; in represents the main task gradient; Represents auxiliary tasks i The gradient of To balance the gradient contributions of different tasks, the scaling factor of each task is calculated based on the L2 norm of the gradient of each task. : Formula 10: ; in represents the gradient of the first auxiliary task; represents the gradient of the Kth auxiliary task; Representation Task i The gradient of Multiply the gradient of each task by its corresponding scaling factor and the current loss weight : Formula 11: ; in Representation Task i Adjusted gradient; After applying the scaling factors and weights, the model parameters are updated via backpropagation : Formula 12: ; in, is the weighted sum of the gradients of the main task and all auxiliary tasks.

[0041] AGS ensures that no single task dominates parameter updates, so that the contributions of the main task and auxiliary tasks remain balanced. Once this balance is established, the task weights are adjusted dynamically. The strategy can effectively improve the model performance.

[0042] During the training process, the strategy of dynamically adjusting the weight of the loss function allows the weight of the task to be adjusted as the training stage changes. In the early stage of training, the weights of all tasks are set to similar values ​​to promote the learning of common features; as the training progresses, the weight of the main task gradually increases, while the weight of the auxiliary task gradually decreases. In the last cycle, the weight of the main task reaches 1, and the weight of the auxiliary task drops to 0. This strategy allows the model to make full use of the information of the auxiliary tasks in the early stage, while gradually focusing on the main task in the later stage of training, thereby improving the overall optimization effect.

[0043] 3. Model performance evaluation and result analysis; In the performance evaluation of the multi-task learning model proposed in the present invention, the single-task learning model (STL) is used as the benchmark model and LSTM is used as the backbone network. LSTM is suitable for real-time processing due to its low data and computing requirements, and provides good interpretability and cost-effectiveness, which is supported by extensive engineering experience. To ensure the fairness of performance comparison, the present invention determines the hyperparameter settings of the single-task learning model through a comprehensive hyperparameter search process. All subsequent comparison models (SMOGN, BAGGING), including the multi-task learning model (MTL), use the same hyperparameters as the single-task learning model to ensure the comparability of the evaluation results and accurately reflect the performance of each model.

[0044] In order to further verify the effectiveness of the multi-task learning model of the present invention in processing unbalanced regression tasks, several benchmark models are introduced for comparison. These benchmark models use existing technologies to alleviate the problems caused by data imbalance, and specifically select SMOGN and Bagging models because of their effectiveness in balancing regression task data.

[0045] The performance of SMOGN, BAGGING, STL, and MTL is evaluated using commonly used regression evaluation metrics, including R ² (coefficient of determination), MAE (mean absolute error), and RMSE (root mean square error).

[0046]

[0047] in, n is the sample size, is the true value of the data, is the predicted value of the model. RMSE is the square root of the average of the sum of the squares of the differences between the observed and predicted values, which is used to measure the difference between the predicted and observed values. The smaller its value, the higher the prediction accuracy of the model. R ²A measure of how well the variability predicted by the model matches the actual variability in the data.R The value of ² is usually between 0 and 1. The closer the value is to 1, the better the model explains the variation in the data.

[0048] The collected commercial vehicle SCR system ammonia emission test set is divided into a full test set (FT) and a specific low sample area test set (LT). The specific low sample area test set focuses on non-zero emission data, which exhibits a more complex and difficult to predict distribution. The final multi-task learning model is formed by training the auxiliary task (predicting the cross-sensitivity coefficient) together with the main task, and the performance is compared with the baseline model.

[0049] The trained model is used to infer the test set. The comparison between the predicted value and the true value (Measurement) of the model in the test set of a specific low sample area is shown in the figure below. Figure 3 As shown, the relevant evaluation indicators of the model test set are shown in Table 1:

[0050] from Figure 3 It can be seen from the data in Table 1 that the multi-task learning model of the present invention shows strong performance in both the complete test set and the low sample area, especially in the low sample area, the MSE and MAE are significantly lower than other models. The multi-task learning model benefits from the knowledge transfer of auxiliary tasks and shows high robustness. Figure 4 The error curves of the multi-task learning model and the single-task learning model on a specific low-sample area test set are shown. It can be seen from the figure that the error of the single-task learning model increases significantly in the low-sample area, indicating that the unbalanced distribution hinders its ability to extract meaningful patterns, resulting in poor generalization performance. In contrast, the error of the multi-task learning model of the present invention remains more consistent in high-sample and low-sample areas.

[0051] Therefore, the present invention designs a solution to the imbalanced regression problem based on multi-task learning for industrial imbalanced data regression tasks, which can effectively improve the performance of the model under imbalanced data conditions, thereby achieving more accurate and reliable predictions.

[0052] The above are only preferred embodiments of the present invention. It should be pointed out that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These should also be regarded as the protection scope of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A solution to the imbalanced regression problem based on multi-task learning, characterized in that: The following steps are involved: Step S1, multi-task learning model design: Under the hard parameter sharing framework, design a multi-task learning network architecture and define a shared basic network, which is used to extract common features of input data. In the case of data scarcity, the auxiliary task introduces gradient information different from the main task; design a task-specific layer, which is used to process specific information related to the main task and the auxiliary task, and capture the personalized features of each task; Step S2, loss function and optimization strategy: define the total loss function of the main task and auxiliary tasks, including the loss of the main task and the losses of all auxiliary tasks, and balance them through their respective weights; Step S3, adaptive loss function: optimize the learning effect of the main task during the training process by adaptive gradient scaling and dynamic adjustment of the loss function weight.

2. The method for solving the imbalanced regression problem based on multi-task learning according to claim 1, characterized in that: In the multi-task learning network architecture, the model parameters Shared Parameters and task-specific parameters , shared parameters Used to learn general features for all tasks, specific task parameters Used to capture the personalized characteristics of each task; The shared base network receives input x , and processed through shared parameters to generate a shared feature; the definition of the shared basic network is as follows: Formula 1: ; in represents a shared basic network; Represents shared parameters; Represents shared features; The top-level network for each task is built on shared features Above, specifically expressed as: Formula 2: ; in Indicates the task i The predicted value of Indicates the task t network; Represents a specific task parameter.

3. The method for solving the imbalanced regression problem based on multi-task learning according to claim 1, characterized in that: The total loss function of the main task and auxiliary task is: Formula 3: ; in is the total loss function of the main task and auxiliary tasks; It is the loss of the main task; It is i The loss of auxiliary tasks; are the weights of the main task and auxiliary task respectively; During training, the model optimizes the shared parameters by considering the losses of all tasks. and task-specific parameters ; During the training process, the model parameters By using the learning rate Perform gradient descent to update: Formula 4: ; in Indicates t+ Model parameters for 1 iteration; Indicates t The model parameters of the iteration; t Indicates t Iterations; is the total loss Relative to model parameters The gradient is: Formula 5: ; in, Relative to the parameter The gradient of K It is K Auxiliary tasks.

4. The method for solving the imbalanced regression problem based on multi-task learning according to claim 3, characterized in that: The specific method of the adaptive gradient scaling is as follows: At each iteration, the loss function of the main task and each auxiliary task is calculated with respect to the shared parameters The gradient of Let the gradient of the main task be: Formula 6: ; in represents the gradient of the main task; Relative to the parameter The gradient of For i Auxiliary tasks, the gradient is: Formula 7: ; in Indicates i The gradient of the auxiliary task; For each task, the L2 norm of the gradient is calculated to measure its magnitude: Formula 8: ; Formula 9: ; in represents the main task gradient; Represents auxiliary tasks i The gradient of Calculate the scaling factor for each task based on the L2 norm of the gradient of each task : Formula 10: ; in represents the gradient of the first auxiliary task; represents the gradient of the Kth auxiliary task; Indicates the task i The gradient of Multiply the gradient of each task by its corresponding scaling factor and the current loss weight : Formula 11: ; in Indicates the task i Adjusted gradient; After applying the scaling factors and weights, the model parameters are updated via backpropagation : Formula 12: ; in, is the weighted sum of the gradients of the main task and all auxiliary tasks.

5. The method for solving the imbalanced regression problem based on multi-task learning according to claim 1, characterized in that: The specific method of dynamically adjusting the loss function is as follows: At the beginning of training, the weights of all tasks are set to similar values ​​to promote the learning of common features; as the training progresses, the weight of the main task gradually increases and the weight of the auxiliary task gradually decreases; in the last cycle, the weight of the main task reaches 1 and the weight of the auxiliary task drops to 0.

Images