Method and apparatus for establishing a prediction model

Abstract

The embodiment of the application discloses a method and device for establishing a prediction model, which are applied to the technical field of deep learning. The main technical scheme comprises the following steps: obtaining a training data set, wherein the training data set comprises sample data of C categories and labels corresponding to the sample data; taking the sample data as the input of a first prediction model, taking the labels corresponding to the sample data as the target output of the first prediction model, training the first prediction model, wherein the first prediction model comprises a feature extraction network and a prediction network, the feature extraction network is used to extract feature representation by using the sample data, and the prediction network is used to obtain a prediction result for the sample data by using the feature representation; and in the training, a momentum gradient descent method is used to update parameters of the first prediction model, wherein the weight of the momentum in the parameter update is determined according to the difference degree between the long-tail data distribution and the uniform distribution of the training data set. The application can reduce the possibility of falling into a local optimum and improve the prediction effect.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a method and apparatus for building a prediction model. Background Technology

[0002] Naturally collected samples typically exhibit a long-tail distribution, meaning that a small subset of categories (the "head categories") collect the vast majority of samples, while the majority of categories (the "tail categories") receive only a small number of samples. Taking image classification as an example, for training sets of animal recognition tasks, millions of images of common animals like cats and dogs can be easily collected, but for rare animals like snow leopards, the number of images collected is very limited; the rarer the animal, the more difficult it is to collect images. This results in a severe class imbalance in the collected dataset, causing prediction models trained on deep neural networks to overfit to the head categories, significantly impacting model performance. Therefore, a method for building prediction models is urgently needed to mitigate the overfitting problem caused by long-tail data and improve prediction performance. Summary of the Invention

[0003] In view of this, this application provides a method and apparatus for building a prediction model to mitigate the problem of overfitting to the head category caused by long-tail data and improve prediction performance.

[0004] This application provides the following solution:

[0005] Firstly, a method for establishing a prediction model is provided, the method comprising:

[0006] Obtain a training dataset, which includes sample data of C categories and the labels corresponding to the sample data, where C is a positive integer greater than 1;

[0007] The sample data is used as the input of the first prediction model, and the label corresponding to the sample data is used as the target output of the first prediction model. The first prediction model is trained. The first prediction model includes a feature extraction network and a prediction network. The feature extraction network is used to extract feature representations using the sample data, and the prediction network is used to obtain prediction results for the sample data using the feature representations.

[0008] In the training, the first prediction model is updated with momentum gradient descent, wherein the weight of momentum in the parameter update is determined based on the degree of difference between the long-tail data distribution and the uniform distribution of the training dataset.

[0009] According to one achievable method in an embodiment of this application, updating the parameters of the first prediction model using momentum gradient descent includes:

[0010] In each iteration of the training, the parameters of the first prediction model updated in the previous iteration and the momentum of the current iteration are weighted and processed to update the parameters of the first prediction model. The momentum of the current iteration is determined based on the momentum used in the previous iteration and the gradient of the loss function. The direction of parameter update is opposite to the direction of momentum in the current iteration, while the direction of momentum in the current iteration is the same as the direction of parameter update in the previous iteration and the same as the direction of gradient of the loss function.

[0011] According to one achievable method in an embodiment of this application, the absolute value of the weight of the momentum in the parameter update is greater than or equal to 1, and the degree of difference between the long-tail data distribution and the uniform distribution of the training dataset includes the KL divergence value between the long-tail data distribution and the uniform distribution of the training dataset.

[0012] According to one achievable method in an embodiment of this application, the weight of the momentum in the parameter update is the larger of the value obtained by multiplying the divergence calculation value by log2 C and 1, wherein the divergence calculation value is the KL divergence value of the long-tail data distribution and the uniform distribution of the training dataset.

[0013] According to one achievable method in the embodiments of this application, there are N prediction models, where N is a positive integer greater than 1; the method further includes: resampling the training dataset N times using the bootstrap method to obtain N datasets, which are used to train N first prediction models respectively.

[0014] After training, a second prediction model is obtained by using the N first prediction models obtained from training. The second prediction model is used to integrate the prediction results output by the N prediction networks after inputting the data to be predicted to determine the prediction result of the data to be predicted.

[0015] According to one achievable method in this application embodiment, integrating the prediction results output by N prediction networks to determine the category of the data to be predicted includes:

[0016] The prediction results output by the N prediction networks are voted on, and the category of the data to be predicted is determined based on the voting results.

[0017] According to one achievable method in an embodiment of this application, the method further includes:

[0018] The second prediction model is further trained using the training dataset. The training objective is to minimize the difference between the prediction result output by the second prediction model and the corresponding label. During the training process of the second prediction model, the parameters of each first prediction model are kept unchanged, and the weights corresponding to the N prediction results in the voting are learned.

[0019] According to one possible implementation method in an embodiment of this application, the sample data is an image sample;

[0020] The feature representation extraction using sample data includes: extracting one or any combination of color features, texture features, shape features, spatial relationship features, and frequency features from image samples and encoding them to obtain feature representations;

[0021] The prediction result is the image category, image segmentation region, or target detection result of the image.

[0022] Secondly, an apparatus for establishing a prediction model is provided, the apparatus comprising:

[0023] The sample acquisition unit is configured to acquire a training dataset, which includes sample data of C categories and labels corresponding to the sample data, where C is a positive integer greater than 1;

[0024] The model training unit is configured to take sample data as input to a first prediction model and the label corresponding to the sample data as the target output of the first prediction model to train the first prediction model. The first prediction model includes a feature extraction network and a prediction network. The feature extraction network is used to extract feature representations using the sample data, and the prediction network is used to obtain prediction results for the sample data using the feature representations.

[0025] In the training, the first prediction model is updated with momentum gradient descent, wherein the weight of momentum in the parameter update is determined based on the degree of difference between the long-tail data distribution and the uniform distribution of the training dataset.

[0026] According to a third aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0027] According to a fourth aspect, an electronic device is provided, characterized in that it comprises:

[0028] One or more processors; and

[0029] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any one of the first aspects above.

[0030] According to the specific embodiments provided in this application, this application can achieve the following technical effects:

[0031] 1) This application introduces momentum into the training of the prediction model and uses momentum gradient descent to update the model parameters. It can also adaptively determine the weight of momentum in parameter updates according to different long-tail data distributions, increase the model's attention to tail categories, improve the diversity of feature representations, reduce the possibility of getting trapped in local optima, and thus improve the prediction effect.

[0032] 2) The way momentum is set as a weight parameter in parameter updates in this application can, on the one hand, weaken the influence of long-tail distribution on momentum in previous iterations, balance the gradient, and increase the model's attention to the tail category, so that the learned feature representation is more diverse while maintaining intra-class compactness. On the other hand, it can adaptively adjust the step size, thereby enabling the model to obtain the optimal solution better.

[0033] 3) In this application, Booststrap is used to resample the training dataset N times to obtain N datasets, which are then used to train N first prediction models. A second prediction model is then obtained by ensembling multiple first prediction models. This multi-expert network approach can learn more diverse feature representations and effectively reduce the model's variance, thereby further improving the model's stability and accuracy.

[0034] 4) The prediction results of each first prediction model in this application can be used to determine the prediction results of the second prediction model by voting. This method can further improve the accuracy of model prediction.

[0035] 5) In this application, the training dataset can be used to further train the second prediction model while keeping the parameters of each first prediction model unchanged. The weights of N prediction networks in the voting can be learned. This reweighting method on multiple prediction networks can further improve the accuracy of model prediction.

[0036] 6) The method provided in this application can be widely applied to application scenarios such as image category prediction, segmentation region prediction or object detection. Even if image samples are more likely to exhibit long-tail phenomena in display scenarios, by introducing momentum during training, the feature representations extracted from one or any combination of color features, texture features, shape features, and spatial relationship features obtained from the image can be more diversified, reducing the possibility of getting trapped in local optima, and making the model more accurate in image category prediction, segmentation region prediction or object detection.

[0037] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 An exemplary system architecture diagram is shown that can be applied to embodiments of this application;

[0040] Figure 2 A flowchart illustrating the method for establishing a prediction model provided in this application embodiment;

[0041] Figure 3 A schematic diagram illustrating the principle of training the second prediction model provided in an embodiment of this application;

[0042] Figure 4 A schematic diagram illustrating the principle of the second prediction model provided in the embodiments of this application;

[0043] Figure 5 A schematic block diagram of an apparatus for establishing a prediction model provided in an embodiment of this application;

[0044] Figure 6 A schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0046] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0047] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0048] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0049] Current methods for addressing long-tail distribution mainly include resampling and reweighting. Resampling adjusts the number of training samples from different classes in the training set to ensure a balance in the number of samples per class. Reweighting primarily applies a larger penalty to the training data of the tail classes in the loss function. The core purpose of these methods is to rebalance the dataset. However, they cause the following two problems:

[0050] 1) The classifier tends to correctly classify the tail category samples, that is, the model overfits to the tail category and underfits to the head category, resulting in inaccurate prediction model.

[0051] 2) It significantly promotes the learning of the classifier, but unintentionally impairs the representational power of the learned deep features. Although the learned model can effectively separate the first and last samples, the learned intra-class feature representations are relatively scattered, which also affects the accuracy of the prediction model.

[0052] In view of this, this application adopts a novel approach, incorporating momentum into the training process of the prediction model to learn more diverse feature representations. To facilitate understanding of this application, the system architecture to which this application applies will first be briefly described. Figure 1 An exemplary system architecture that can be applied to embodiments of this application is shown, such as Figure 1 As shown, the system architecture includes a model building device that builds a prediction model offline, and a prediction device that predicts the data to be predicted online.

[0053] The model building device, after acquiring training data, can use the method provided in the embodiments of this application to train the model and obtain a prediction model.

[0054] The prediction device uses an established prediction model to make predictions on the data to be predicted. For example, it predicts the image category from image data.

[0055] The model building and prediction devices can be configured as separate servers, or they can be located on the same server or server cluster, or they can be located on a separate or the same cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product within the cloud computing service system, designed to address the management difficulties and weak service scalability inherent in traditional physical hosts and Virtual Private Servers (VPs). The model training and prediction devices can also be located on computer terminals with strong computing capabilities.

[0056] It should be understood that Figure 1 The number of model training devices, prediction devices, and prediction models shown in the diagram is merely illustrative. Depending on implementation needs, any number of model training devices, prediction devices, and prediction models can be included.

[0057] Figure 2 This is a flowchart of a method for establishing a prediction model provided in an embodiment of this application. This method can be... Figure 1 The model building device in the system shown is executed. For example... Figure 2 As shown, the method may include the following steps:

[0058] Step 202: Obtain the training dataset, which includes sample data of C categories and the corresponding labels of the sample data, where C is a positive integer greater than 1.

[0059] Step 204: Use the sample data as input to the first prediction model and the labels corresponding to the sample data as the target output of the first prediction model to train the first prediction model. The first prediction model includes a feature extraction network and a prediction network. The feature extraction network is used to extract feature representations from the sample data, and the prediction network is used to obtain prediction results for the sample data using the feature representations. During training, the momentum gradient descent method is used to update the parameters of the first prediction model. The weight of momentum in the parameter update is determined according to the degree of difference between the long-tail data distribution and the uniform distribution of the training dataset.

[0060] As can be seen from the above process, this application introduces momentum into the training of the prediction model, uses momentum gradient descent to update the model parameters, and can adaptively determine the weight of momentum in parameter updates according to different long-tail data distributions, thereby improving the diversity of feature representations, reducing the possibility of getting trapped in local optima, and thus improving the prediction effect.

[0061] The training dataset obtained in step 202 above can be an image dataset, a text dataset, etc. Taking an image dataset as an example, based on the actual prediction scenario, C categories of image samples and the labels for each image sample can be obtained. The labels are set according to the specific prediction task. For example, if the prediction task is an image classification task, the corresponding prediction result is the image category, then the label is the category of the image sample. Another example is an image segmentation task, the corresponding prediction result is the image segmentation region, then the label is the label of each region contained in the image sample. Yet another example is an object detection task, the corresponding prediction result is the object type and location in the image, then the label is the object type and location label in the image sample.

[0062] The following detailed description of step 204 in the above process, with reference to an embodiment, focuses on this step. A prediction model can typically be divided into a feature extraction network and a prediction network in terms of its overall functional architecture. The feature extraction network is used to extract feature representations from the input data, and the prediction network is used to obtain prediction results for the input data using these feature representations.

[0063] Feature extraction networks, which can be implemented based on convolutional neural networks, Transformer networks, etc., are used to extract feature representations from the data to be predicted. Taking an image as an example, features such as color, texture, shape, spatial relationships, and frequency can be extracted or any combination thereof and encoded to obtain feature representations. Feature extraction networks can extract image features globally from the entire image or extract features from each image patch obtained after segmenting the image.

[0064] Prediction networks can be classification networks, regression networks, etc., and their function is to map feature representations to prediction results.

[0065] It should be noted that the terms "first" and "second" used in the embodiments of this application do not have limitations in terms of size, order, or quantity, but are only used to distinguish between the two prediction models. For example, "first prediction model" and "second prediction model" are used to distinguish between the two prediction models in terms of name.

[0066] In step 204, momentum is first introduced into the training of the first prediction model. The first prediction model includes a feature extraction network and a prediction network. The feature extraction network extracts feature representations from the sample data, and the prediction network uses these feature representations to obtain prediction results for the sample data. During training, the sample data serves as the input to the first prediction model, and the corresponding label serves as the target output. Momentum gradient descent is used to update the parameters of the first prediction model during training.

[0067] To facilitate understanding, let's first briefly explain the concept of momentum gradient descent. Momentum gradient descent is a gradient descent method where, in each iteration, the direction of parameter updates is opposite to the direction of momentum used in that iteration. The momentum direction of the current iteration is the same as the direction of the previous parameter updates and also the same as the gradient direction of the loss function. Alternatively, we can consider the direction of parameter updates in each iteration as a weighted combination of the opposite direction of the current gradient and the direction of the previous parameter updates. In other words, the direction of parameter updates depends not only on the direction of the current gradient but also on the past parameter update directions. The formula for each iteration can be shown below:

[0068] θ t =θ t-1 -m t (1)

[0069]

[0070] Where, θ t Let θ be the model parameters for the t-th iteration. t-1 Let represent the model parameters for the (t-1)th iteration, η be the learning rate, and μ be the momentum decay ratio. The function for calculating the gradient, m t and m t-1 Let be the momentum of the t-th and t-1-th iterations.

[0071] In the embodiments of this application, in each iteration, the parameters of the first prediction model are updated based on the parameters of the first prediction model after the previous iteration and the momentum of the current iteration. The momentum of the current iteration is determined based on the momentum used in the previous iteration and the gradient of the loss function.

[0072] The learning rate and momentum decay ratio mentioned above can be set empirically or experimentally. One possible approach is to initialize the learning rate η to a single value, such as 0.1, and then reduce it after a certain training phase, for example, reducing it to 0.01 when training reaches 80% and to 0.0001 when training reaches 90%. The momentum decay ratio can be set to a fixed value, such as 0.9. Both the learning rate and momentum decay ratio can be set according to different training datasets.

[0073] The principle behind the momentum gradient update described above is that during gradient update, an additional gradient is applied in a certain direction, enabling the function to escape a local optimum when it reaches a local optimum. For example... When a local optimum is reached, the value is 0, but the momentum m t-1 It is highly likely that θ is not zero, therefore, tIt can still be updated, thus escaping local optima. In long-tailed datasets, the contributions of each class to momentum and the current gradient are unbalanced, so the model will still be heavily biased towards the head class and converge to a local optimum.

[0074] Therefore, in this embodiment, a weight parameter β is introduced to adjust the momentum. Standard data processed by machine learning typically assumes that the number of samples corresponding to each category in the dataset approximately follows a uniform distribution, i.e., class balance. Therefore, the difference between the long-tail distribution and the uniform distribution of the training dataset reflects the imbalance of the training dataset. The more imbalanced the training dataset, the more likely the prediction model is to favor the head category. Therefore, in each iteration of training, the parameters of the first prediction model updated in the previous iteration and the momentum in the current iteration are weighted to update the parameters of the first prediction model. In this embodiment, the weight β of momentum in parameter updates can be determined based on the difference between the long-tail distribution and the uniform distribution of the training dataset. After introducing this weight parameter, the above formulas (1) and (2) are improved as follows:

[0075] θ t =θ t-1 -βm t (3)

[0076]

[0077] Where β > 1. As one possible implementation, it can be determined using the following formula:

[0078] β=max((I KL ·log2C),1) (5)

[0079] Among them, I KL This is the KL (Kullback-Leibler Divergence) divergence between the long-tailed distribution P and the uniform distribution Q of the training dataset. KL divergence measures the "distance" between two probability distribution functions, reflecting the degree of difference between the long-tailed and uniform distributions of the training dataset. Other metrics can also be used to represent this. KL The following formula can be used:

[0080]

[0081] C represents the number of categories, p i Let represent the probability of the i-th class appearing in the training dataset, which can be obtained statistically. Let q represent the probability of a uniform distribution, which can be taken as...

[0082] It should be noted that in formula (4) This is one of the settings; it can also be set to be consistent with... Other related values, such as wait.

[0083] On the one hand, the weight parameter β can weaken the influence of the long-tail distribution on momentum in previous iterations, balance the gradient, and increase the model's attention to the tail classes, making the learned feature representations more diverse while maintaining intra-class compactness, thus reducing the possibility of getting trapped in local optima. On the other hand, the step size can be adaptively adjusted, thereby better enabling the model to obtain the optimal solution.

[0084] Furthermore, the training objective described above is to minimize the difference between the output of the first prediction model and the corresponding labels in the training data. Specifically, a loss function can be constructed based on this training objective. In each iteration, the model parameters of the first prediction model are updated using the value of the loss function and methods such as momentum gradient descent, until a preset training termination condition is met. This termination condition may include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0085] One feasible approach is to use the trained first prediction model to predict the data to be predicted after training. This involves inputting the data to be predicted into the first prediction model and obtaining the prediction result output by the model for that data.

[0086] However, as a better alternative, a multi-expert network approach can be used to construct the prediction model. In this embodiment, a Bagging (Bootstrap aggregating) method can be used to implement the multi-expert network. The Bagging algorithm, also known as the group learning algorithm, is a group learning algorithm in machine learning. Bagging is a technique that reduces generalization error by combining several models. The main idea is to train multiple parallel base models separately, and then integrate the prediction results of the multiple base models to obtain the final result.

[0087] In other words, it can be used Figure 2 The process described herein trains N first prediction models, where N is a positive integer greater than 1. To increase the diversity of the N first prediction models, a bootstrap method can be used to resample the training dataset N times, resulting in N datasets; then these N datasets are used to train the models according to... Figure 2 The method shown is used to train N first prediction models.

[0088] Bootstrap, in essence, involves resampling a given dataset with replacement to create multiple new datasets, each with the same sample size as the original dataset. If the training dataset in this embodiment has n samples, then a new dataset of size n is drawn from this training dataset using sampling with replacement. This sampling process is repeated N times to obtain N new datasets. Sampling with replacement means that after sampling one or more samples from the dataset, the sampled samples are returned to the dataset before subsequent sampling is performed. This ensures that the new datasets obtained are different from each other, resulting in a greater diversity of the N trained first prediction models.

[0089] like Figure 3 As shown, a first prediction model is trained using a new dataset, and N first prediction models are obtained by training on N new datasets. The training of each first prediction model is independent, and momentum gradient descent is used to update parameters during training. For example, the momentum of the first prediction model 1 is represented by M. 1 The gradient is represented as The momentum of the first prediction model 1 is represented by M. 2 The gradient is represented as …;The momentum of the first prediction model N is represented by M. N The gradient is represented as

[0090] After training, the second prediction model can be composed of N first prediction models arranged side by side. The second prediction model is used to integrate the prediction results output by the N prediction networks after inputting the data to be predicted, to determine the final prediction result for the data. For example... Figure 4 As shown, the second prediction model may further include a voting module. When the second prediction model is used to predict the data to be predicted, each of the first prediction models will output a prediction result for the data to be predicted, and the voting module will integrate the prediction results. As one possible implementation, the voting module can vote on the prediction results output by the N prediction networks for the data to be predicted, and determine the prediction result for the data to be predicted based on the voting result.

[0091] Voting is an ensemble learning method that follows the principle of majority rule. By integrating multiple models (the first prediction model in this application), variance is reduced, thereby improving the robustness and generalization ability of the model.

[0092] When conducting a vote, either a hard vote or a soft vote can be used. A hard vote determines the final prediction based on the prediction that occurs most frequently among the predictions of the N first prediction models. This can be achieved by assigning voting weights to each first prediction model and then weighting the frequency of the prediction based on those weights.

[0093] Soft voting, also known as soft voting, involves weighted summation of the probabilities of each prediction result based on the weights of the first prediction models, and finally selecting the prediction result with the highest sum of probabilities. Because soft voting takes into account the additional information of probability, it can yield more accurate results than hard voting. For example, by weighted summation of the probabilities of the data to be predicted in category 1 from the outputs of each first prediction model, the final probability of the data to be predicted in category 1 can be obtained. By performing this process for each category, the final probability of the data to be predicted in each category can be obtained, and the category with the highest probability can then be determined as the prediction result for the data to be predicted.

[0094] As one possible approach, the weights of each first prediction model in the voting process can be used as hyperparameters and set using empirical or experimental values.

[0095] As another feasible approach, a second prediction model can be trained using the training dataset. Similarly, the sample data is used as input, and the labels corresponding to the sample data are used as the target output of the second prediction model; that is, the training objective is to minimize the difference between the prediction result output by the second prediction model and the corresponding label. During this training process, the parameters of the prediction network remain unchanged; only the weights of the N prediction networks in the voting are learned, i.e., only the parameters of the voting module are updated.

[0096] This multi-expert network approach generates richer datasets, and the learned expert networks (i.e., the first-order prediction models) are more diverse and have lower variance. It is represented as follows:

[0097] Var(X i )=σ 2 ,i=1,2,…,N (7)

[0098]

[0099] Among them, X i Let be the prediction result of the i-th first prediction model. Assume the variance of each first prediction model is Var(X). i The values ​​are the same, both being σ. 2 The variance of the final second prediction model For the variance of each first prediction model Clearly, the variance can be significantly reduced, indicating that the model stability has been significantly improved.

[0100] On the one hand, momentum is added during training to escape local optima and learn more diverse feature representations, mitigating the overfitting problem to the head class caused by long-tail data. On the other hand, multi-expert networks can learn more diverse feature representations, and based on these diverse representations, the variance of the model is effectively reduced, thereby further improving the model's stability and accuracy. This momentum-plus-multi-expert-network approach makes the model simple and effective, without requiring complex hyperparameter tuning.

[0101] The methods provided in this application embodiment can be applied to various application scenarios, including but not limited to:

[0102] Application Scenario 1: Applied to image category recognition tasks.

[0103] Taking animal category recognition in images as an example, the training dataset can include image samples of animals in each category, along with their category labels. In real life, images of common animals such as cats, dogs, chickens, and ducks are readily available and belong to the head category. However, images of animals like snow leopards, ligers, pangolins, hobbies, hawk-owls, and baiji dolphins are difficult to obtain, costly to acquire, and few in number, belonging to the tail category. This results in a long-tailed distribution of samples in the training dataset. To mitigate the problem of inaccurate predictions caused by the prediction model (in this application, an image category recognition model) overfitting to the head category, the method described in this embodiment can be used to build the prediction model.

[0104] The training dataset is resampled N times using the Bootstrap method, resulting in N datasets used to train N first prediction models. The training of each first prediction model is independent. Momentum is introduced during the training of each model, and momentum gradient descent is used to update the parameters. The weight of momentum in the parameter update is determined based on the difference between the long-tailed data distribution and the uniform distribution of the training dataset. The training objective is to minimize the difference between the prediction result output by the first prediction model and the label of the image sample.

[0105] After training N first prediction models, the N first prediction models and the voting module are used to construct a second prediction model. The training dataset can be used to train the second prediction model further. The training objective is to minimize the difference between the prediction results output by the second prediction model and the corresponding labels of the image samples. During the training of the second prediction model, the parameters of each first prediction model are kept unchanged, and only the weights of the N prediction results of the N first prediction models in the voting are learned, that is, the parameters of the voting module.

[0106] The final trained second prediction model is used for image category recognition online or offline. The second prediction model predicts the image category of the input image to be recognized. Specifically, each of the first prediction models in the second prediction model obtains its own image category prediction for the image to be recognized. A voting module then uses the prediction results of each first prediction model to vote on the predictions and outputs the image category of the image to be recognized based on the voting result.

[0107] Experiments have shown that this method can significantly improve the accuracy and stability of image category recognition.

[0108] Application Scenario 2: Applied to image segmentation tasks.

[0109] Image segmentation refers to dividing an image into several non-overlapping regions with similar features. Examples include segmenting scenes and objects, segmenting the human body from the background, and segmenting faces and bodies, etc.

[0110] Image segmentation has wide applications in the field of autonomous driving. Taking scene and object segmentation in autonomous driving as an example, the training dataset can include image samples collected in various scenarios by a collection vehicle (which can be a dedicated collection vehicle, a regular vehicle, or an autonomous vehicle, etc.) and labels for scene and object regions in the image samples. In real life, there are many image samples collected in normal scenarios, such as normal intersections, normal obstacles, and normal stop lines under normal weather conditions, which belong to the head category. However, there are very few image samples under extreme weather conditions such as blizzards and heavy rain, which belong to the tail category. This also results in a long-tailed distribution of samples in the training dataset. In order to mitigate the problem of inaccurate predictions caused by the prediction model (in this application, the image segmentation model) overfitting to the head category, the method in the embodiment of this application can be used to build the prediction model.

[0111] The training dataset is resampled N times using the Bootstrap method, resulting in N datasets used to train N first prediction models. The training of each first prediction model is independent. Momentum is introduced during the training of each model, and momentum gradient descent is used to update the parameters. The weight of momentum in the parameter update is determined based on the difference between the long-tailed data distribution and the uniform distribution of the training dataset. The training objective is to minimize the difference between the image segmentation result output by the first prediction model and the label of the image sample.

[0112] After training N first prediction models, the N first prediction models and the voting module are used to construct a second prediction model. The training dataset can be used to train the second prediction model further. The training objective is to minimize the difference between the image segmentation result output by the second prediction model and the corresponding label of the image sample. During the training of the second prediction model, the parameters of each first prediction model are kept unchanged, and only the weights of the N prediction results of the N first prediction models in the voting are learned, that is, the parameters of the voting module.

[0113] The final trained second prediction model is used for online or offline segmentation of scene and object regions in an image. The second prediction model predicts scene and object regions in the input image to be segmented. Specifically, each of the first prediction models in the second prediction model obtains a segmentation result for the image to be segmented. A voting module then uses the segmentation results of each first prediction model to vote on the results and outputs the segmentation results of the scene and object regions in the image to be segmented based on the voting result.

[0114] Experiments have shown that this method can significantly improve the recognition accuracy and stability of image segmentation.

[0115] Application Scenario 3: Applied to the task of object detection in images.

[0116] Object detection is also widely used in the field of autonomous driving, requiring the accurate identification of target types and their location information, such as obstacles and traffic facilities, from images captured by autonomous vehicles. Taking object detection in autonomous driving as an example, the training dataset can include image samples and labels indicating the type and location information of targets in the image samples. Similarly, in real life, there are many image samples of normal traffic facilities and obstacles, such as normal traffic lights, pedestrians, and vehicles, which belong to the head category. However, there are very few image samples of irregular intersections, irregularly shaped vehicles, and pedestrians with special postures, which belong to the tail category. This results in a long-tailed distribution of samples in the training dataset. To mitigate the problem of inaccurate predictions caused by the prediction model (in this application, the object detection model) overfitting to the head category, the method in the embodiments of this application can be used to build the prediction model.

[0117] The training dataset is resampled N times using the Bootstrap method to obtain N datasets, each used to train a first prediction model. The training of each first prediction model is independent. Momentum is introduced during the training of each first prediction model, and momentum gradient descent is used to update the parameters. The weight of momentum in the parameter update is determined based on the difference between the long-tailed data distribution and the uniform distribution of the training dataset. The training objective is to minimize the difference between the prediction result (including target type and location information) output by the first prediction model and the label of the image sample.

[0118] After training N first prediction models, the N first prediction models and the voting module are used to construct a second prediction model. The training dataset can be used to train the second prediction model further. The training objective is to minimize the difference between the prediction results output by the second prediction model and the corresponding labels of the image samples. During the training of the second prediction model, the parameters of each first prediction model are kept unchanged, and only the weights of the N prediction results of the N first prediction models in the voting are learned, that is, the parameters of the voting module.

[0119] The final trained second prediction model is used for object detection in images, either online or offline. The second prediction model predicts the type and location information of objects contained in the input image to be detected. Specifically, the prediction results of each of the first prediction models for the object type and location information of the image to be detected are then voted on by a voting module, and the object detection result of the image to be detected is output based on the voting result.

[0120] Experiments have shown that this method can significantly improve the accuracy and stability of image target detection.

[0121] Besides the application scenarios mentioned above, there may be other application scenarios, which will not be listed here.

[0122] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0123] According to another embodiment, an apparatus for establishing a predictive model is provided. Figure 5 A schematic block diagram of an apparatus for building a predictive model according to one embodiment is shown, the apparatus being disposed in Figure 1The model building apparatus in the illustrated architecture. For example... Figure 5 As shown, the device 500 may include a sample acquisition unit 501 and a model training unit 502.

[0124] The sample acquisition unit 501 is configured to acquire a training dataset, which includes sample data of C categories and the corresponding labels of the sample data, where C is a positive integer greater than 1.

[0125] Model training unit 502 is configured to take sample data as input to the first prediction model and the labels corresponding to the sample data as the target output of the first prediction model to train the first prediction model. The first prediction model includes a feature extraction network and a prediction network. The feature extraction network is used to extract feature representations from the sample data, and the prediction network is used to obtain prediction results for the sample data using the feature representations. During training, momentum gradient descent is used to update the parameters of the first prediction model, where the weight of momentum in the parameter update is determined based on the degree of difference between the long-tail data distribution and the uniform distribution of the training dataset.

[0126] The first prediction model can be a convolutional neural network, a Transformer network, etc., and its function is to extract feature representations from the data to be predicted. Taking an image as an example, it can extract one or any combination of features such as color, texture, shape, and spatial relationship. The prediction network can be a classification network, a regression network, etc., and its function is to map the feature representations to the prediction results.

[0127] As one possible approach, when updating the parameters of the first prediction model using momentum gradient descent, the model training unit 502 can perform the following:

[0128] In each iteration of training, the parameters of the first prediction model updated in the previous iteration and the momentum of the current iteration are weighted and processed to update the parameters of the first prediction model. The momentum of the current iteration is determined based on the momentum used in the previous iteration and the gradient of the loss function. The direction of parameter update is opposite to the direction of momentum in the current iteration, while the direction of momentum in the current iteration is the same as the direction of parameter update in the previous iteration and the same as the direction of gradient of the loss function. For details, please refer to the descriptions of formulas (3) and (4) in the method embodiment.

[0129] As one possible approach, the absolute value of the momentum weight in the parameter update is greater than or equal to 1, and the degree of difference between the long-tail data distribution and the uniform distribution of the training dataset includes the KL divergence value between the long-tail data distribution and the uniform distribution of the training dataset.

[0130] As one preferred approach, the weight of momentum in parameter updates is the larger of the value obtained by multiplying the divergence calculation value by log2 C and 1. The divergence calculation value is the KL divergence between the long-tailed data distribution and the uniform distribution of the training dataset. For specific representations, please refer to formulas (5) and (6) in the method embodiment.

[0131] One feasible approach is to use the trained first prediction model to predict the data to be predicted after training. This involves inputting the data to be predicted into the first prediction model and obtaining the prediction result from the first prediction model for that data.

[0132] However, as a better alternative, a multi-expert network approach can be used to construct the prediction model. That is, there are N first prediction models, where N is a positive integer greater than 1. The method also includes using Bootstrap to resample the training dataset N times, obtaining N datasets, each used to train one of the N first prediction models.

[0133] After training, the N first prediction models obtained from training are used to form a second prediction model. The second prediction model is used to integrate the prediction results output by the N prediction networks after inputting the data to be predicted to determine the prediction result of the data to be predicted.

[0134] For example, the second prediction model may further include a voting module. When the second prediction model is used to predict the data to be predicted, each of the first prediction models will output a prediction result for that data, and the voting module will integrate the prediction results. As one possible implementation, the voting module can vote on the prediction results output by the N prediction networks for the data to be predicted, and determine the prediction result for the data to be predicted based on the voting result.

[0135] When conducting a vote, either a hard vote or a soft vote can be used. A hard vote determines the final prediction based on the prediction that occurs most frequently among the predictions of the N first prediction models. This can be achieved by assigning voting weights to each first prediction model and then weighting the frequency of the prediction based on those weights.

[0136] The so-called soft voting method involves weighted summation of the probabilities of each prediction result based on the weights of each primary prediction model, and finally selecting the prediction result with the highest sum of probabilities. Soft voting takes into account the additional information of probability, thus yielding more accurate results than hard voting.

[0137] As one possible approach, the weights of each first prediction model in the voting process can be used as hyperparameters and set using empirical or experimental values.

[0138] As another possible approach, the model training unit 502 described above can also use the training dataset to further train the second prediction model. The training objective is to minimize the difference between the prediction result output by the second prediction model and the corresponding label. During the training of the second prediction model, the parameters of each prediction network are kept unchanged, and the weights of N prediction networks in the voting are learned.

[0139] It should be noted that the embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0140] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0141] And an electronic device, comprising:

[0142] One or more processors; and

[0143] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0144] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0145] in, Figure 6 An exemplary architecture of an electronic device is shown, which may include a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, and a memory 620. The processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620 can communicate with each other via a communication bus 630.

[0146] The processor 610 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs and implement the technical solution provided in this application.

[0147] The memory 620 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 620 can store the operating system 621 for controlling the operation of the electronic device 600, and the basic input / output system (BIOS) 622 for controlling the low-level operations of the electronic device 600. Additionally, it can store a web browser 623, a data storage management system 624, and a device 625 for building predictive models, etc. The aforementioned device 625 for building predictive models can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when the technical solution provided in this application is implemented through software or firmware, the relevant program code is stored in the memory 620 and is called and executed by the processor 610.

[0148] Input / output interface 613 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0149] Network interface 614 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0150] Bus 630 includes a pathway for transmitting information between various components of the device, such as processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620.

[0151] It should be noted that although the above-described device only shows the processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, memory 620, bus 630, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0152] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0153] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0154] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for establishing a predictive model, characterized in that, The method includes: Obtain a training dataset, which includes sample data of C categories and the corresponding labels of the sample data, where C is a positive integer greater than 1; wherein the sample data are image samples; The first prediction model is trained by using sample data as input and the corresponding labels as output. The first prediction model includes a feature extraction network and a prediction network. The feature extraction network extracts feature representations from the sample data, and the prediction network uses these feature representations to obtain prediction results for the sample data. Extracting feature representations from the sample data includes extracting one or any combination of color features, texture features, shape features, spatial relationship features, and frequency features from the image samples and encoding them to obtain feature representations. The prediction result is an image category, an image segmentation region, or an image target detection result. In the training, momentum gradient descent is used to update the parameters of the first prediction model. The weight of momentum in the parameter update is determined based on the degree of difference between the long-tail data distribution and the uniform distribution of the training dataset. The absolute value of the momentum weight in the parameter update is greater than 1, and the degree of difference between the long-tail data distribution and the uniform distribution of the training dataset includes the KL divergence value between the long-tail data distribution and the uniform distribution. The weight of momentum in the parameter update is calculated as: the divergence value multiplied by... The larger of the obtained value and 1, the divergence calculation value is the KL divergence value between the long-tail data distribution and the uniform distribution of the training dataset; The step of updating the parameters of the first prediction model using momentum gradient descent includes: In each iteration of the training, the parameters of the first prediction model updated in the previous iteration and the momentum of the current iteration are weighted and processed to update the parameters of the first prediction model. The momentum of the current iteration is determined by weighting the momentum used in the previous iteration and the gradient of the loss function. The direction of parameter update is opposite to the direction of momentum in the current iteration, and the direction of momentum in the current iteration is the same as the direction of parameter update in the previous iteration and the same as the direction of gradient of the loss function.

2. The method according to claim 1, characterized in that, The first prediction model consists of N models, where N is a positive integer greater than 1; the method further includes: resampling the training dataset N times using the bootstrap method to obtain N datasets, which are then used to train the N first prediction models. After training, a second prediction model is obtained by using the N first prediction models obtained from training. The second prediction model is used to integrate the prediction results output by the N prediction networks after inputting the data to be predicted to determine the prediction result of the data to be predicted.

3. The method according to claim 2, characterized in that, The method of integrating the prediction results from N prediction networks to determine the category of the data to be predicted includes: The prediction results output by the N prediction networks are voted on, and the category of the data to be predicted is determined based on the voting results.

4. The method according to claim 3, characterized in that, The method further includes: The second prediction model is further trained using the training dataset. The training objective is to minimize the difference between the prediction result output by the second prediction model and the corresponding label. During the training process of the second prediction model, the parameters of each first prediction model are kept unchanged, and the weights corresponding to the N prediction results in the voting are learned.

5. An apparatus for establishing a predictive model, characterized in that, The device includes: The sample acquisition unit is configured to acquire a training dataset, which includes sample data of C categories and labels corresponding to the sample data, where C is a positive integer greater than 1; wherein the sample data are image samples. The model training unit is configured to take sample data as input to a first prediction model and the labels corresponding to the sample data as the target output of the first prediction model to train the first prediction model. The first prediction model includes a feature extraction network and a prediction network. The feature extraction network is used to extract feature representations from the sample data, and the prediction network is used to obtain prediction results for the sample data using the feature representations. The extraction of feature representations from the sample data includes: extracting one or any combination of color features, texture features, shape features, spatial relationship features, and frequency features from image samples and encoding them to obtain feature representations. The prediction result is an image category, an image segmentation region, or an image target detection result. In the training, momentum gradient descent is used to update the parameters of the first prediction model. The weight of momentum in the parameter update is determined based on the degree of difference between the long-tail data distribution and the uniform distribution of the training dataset. The absolute value of the momentum weight in the parameter update is greater than 1, and the degree of difference between the long-tail data distribution and the uniform distribution of the training dataset includes the KL divergence value between the long-tail data distribution and the uniform distribution. The weight of momentum in the parameter update is calculated as: the divergence value multiplied by... The larger of the obtained value and 1, the divergence calculation value is the KL divergence value between the long-tail data distribution and the uniform distribution of the training dataset; The step of updating the parameters of the first prediction model using momentum gradient descent includes: In each iteration of the training, the parameters of the first prediction model updated in the previous iteration and the momentum of the current iteration are weighted and processed to update the parameters of the first prediction model. The momentum of the current iteration is determined by weighting the momentum used in the previous iteration and the gradient of the loss function. The direction of parameter update is opposite to the direction of momentum in the current iteration, and the direction of momentum in the current iteration is the same as the direction of parameter update in the previous iteration and the same as the direction of gradient of the loss function.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 4.

7. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 4.

Images