Multi-task learning method, device and electronic equipment

Abstract

The embodiments of the present application relate to the field of image processing technology, and disclose a multi-task learning method, device and electronic device. The method obtains at least two data sets to obtain a task balance factor and a data set balance factor, wherein the task balance factor is used to adjust the number of labels between different tasks, and the data set balance factor is used to adjust the number of labels of each task in different data sets. The task balance factor and the data set balance factor are used to determine the sampling rate of the corresponding labels of each task in each data set, so as to sample each data set to obtain sample data for training of each task, and then perform multi-task training to obtain multi-task training results. The embodiments of the present application can solve the problem of label imbalance caused by different numbers of labels or imbalanced proportions or different data distributions of different tasks in multiple data sets when performing multi-task single-stage training, thereby balancing the performance of different tasks in multi-task learning.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a multi-task learning method, device and electronic device. Background Art

[0002] In many fields such as natural language processing, speech recognition, and computer vision, multi-task learning (MLT) improves learning efficiency and prediction accuracy by utilizing data from different learning tasks, learning shared information of multiple subtasks at the same time, and predicting multiple targets from the learned shared features, thereby obtaining better robustness and generalization ability, thereby improving the performance of each task and reducing the risk of overfitting.

[0003] General machine learning models are targeted at a single specific task, such as handwritten digit recognition, object detection, etc. Models for different tasks are learned separately on their own data sets. Using a single data set for multiple tasks requires that the data set has labels that correspond to multiple tasks one by one. Such data sets have high labeling costs and are difficult to adapt to all multi-tasks. In addition, such data sets are prone to insufficient data volume, leading to overfitting problems. Therefore, multi-task learning usually uses multiple data sets to supplement the number of labels for different tasks and adapt to different multi-task models.

[0004] In order to improve the performance of multiple tasks, a multi-stage training method is usually used for multi-task learning. However, the training process of this training method is cumbersome, resulting in insufficient training efficiency. Therefore, in order to improve the training efficiency, a single-stage method is usually used for multi-task learning. However, if a single-stage multi-task training is completed on multiple data sets, the following situations may occur:

[0005] (1) The amount of labeled data sets for different tasks varies. For example, some tasks contain abundant labeled image data, such as object detection and segmentation tasks, while some tasks lack labeled image data, such as pedestrian attribute classification tasks.

[0006] (2) The number of labels for different tasks in different datasets is different or the ratio is unbalanced. Some datasets do not contain labels for all tasks. For example, some datasets have a large number of depth maps but no segmentation labels, while some datasets have a small number of one-to-one corresponding depth maps and segmentation labels.

[0007] (3) The data distribution of different data sets is different. For example, the data distribution of indoor data sets is different from that of outdoor data sets. For example, data sets with one-to-one correspondence with depth maps are generally indoors, which makes the number of depth map labels in indoor data sets relatively large, while the number of depth map labels in outdoor data sets is relatively small.

[0008] From the above, it can be seen that if the image data of the data sets of related tasks are directly integrated together for training, it is easy to cause an imbalance in the number of training samples for different tasks in a single-stage multi-task training, resulting in good performance for some tasks and poor performance for others, thus making task learning unbalanced.

[0009] The current technical solutions have at least the following technical problems:

[0010] For single-stage multi-task training, directly inputting image data from multiple datasets may result in different numbers of labels for different tasks in different datasets, imbalanced proportions, or different data distributions, leading to imbalanced labels for different tasks, that is, an imbalance in the number of training samples for different tasks, resulting in good performance for some tasks and poor performance for others, thus making task learning unbalanced. Summary of the invention

[0011] The embodiments of the present application provide a multi-task learning method, device and electronic device to solve the technical problem of label imbalance caused by different numbers of labels or imbalanced proportions or different data distributions between different tasks in different data sets when directly inputting image data of multiple data sets during multi-task single-stage training, so that the labels between different tasks are balanced, thereby balancing the performance of different tasks in multi-task learning.

[0012] In a first aspect, an embodiment of the present application provides a multi-task learning method, the method comprising:

[0013] Obtain at least two data sets, where the at least two data sets are used for training at least two tasks, wherein each task corresponds to a label, and the number of label types contained in each data set is not greater than the number of tasks;

[0014] Based on the label balanced sampling mechanism, at least two data sets are sampled, and all the sampled data are input into the first model as training sample data for multi-task training to obtain multi-task prediction results;

[0015] Among them, the label balanced sampling mechanism includes:

[0016] Obtain the task balance factor corresponding to each task and the dataset balance factor of the label corresponding to each task in each dataset, wherein the task balance factor is used to adjust the number of labels between different tasks, and the dataset balance factor is used to adjust the number of labels of each task in different datasets;

[0017] Determine the sampling rate of the corresponding label of each task in each data set based on the task balance factor and the data set balance factor;

[0018] According to the sampling rate, each of the at least two data sets is sampled to obtain sample data for training of each task.

[0019] In some embodiments, determining the sampling rate of the corresponding label of each task in each data set according to the task balance factor and the data set balance factor includes:

[0020] Assuming there are K tasks and n data sets, the sampling rate of the label corresponding to each task in each data set is: in, is the task balance factor corresponding to the kth task, δ i,k is the dataset balancing factor for the corresponding label of the k-th task in the i-th dataset, i∈{1,2…n}, k∈{1,2…K}.

[0021] In some embodiments, sampling each of the at least two data sets according to a sampling rate to obtain sample data for training each task includes:

[0022] In at least two data sets, for each task in each data set corresponding label, a random number generation algorithm is used to generate a random number;

[0023] If the random number is less than the sampling rate, the tag is added to the first tag set;

[0024] After traversing all labels in all data sets, a second label set corresponding to each task is generated, and data corresponding to all labels in the second label set are determined as sample data for training each task.

[0025] In some embodiments, all sampled data are input as training sample data into the first model for multi-task training to obtain multi-task prediction results, including:

[0026] All the sampled data are input into the first model as training sample data for single-stage multi-task training;

[0027] Adding the prediction results of the sample data used for training corresponding to each task to the prediction result set, so as to simultaneously obtain the prediction result set of each task in at least two tasks;

[0028] The prediction result set of each task is combined, and the combined prediction result set is determined as the multi-task prediction result.

[0029] In some embodiments, the first model includes at least two task branches, each task branch corresponds to a task, and the prediction result of the sample data for training corresponding to each task is added to the prediction result set to simultaneously obtain the prediction result set of each task in at least two tasks, including:

[0030] Each task branch in the first model predicts each sample data of the task corresponding to it to obtain a prediction result corresponding to each sample data, and the prediction result corresponding to each sample data is added to the prediction result set of each task to simultaneously obtain a prediction result set for each task in at least two tasks.

[0031] In some embodiments, at least two task branches share the same backbone network, and the backbone network is used for each task branch to predict each sample to obtain a prediction result corresponding to each sample.

[0032] In some embodiments, the method further comprises:

[0033] According to the task balance factor corresponding to each task, determine the number of tags corresponding to each task, including:

[0034] Assume that there are K tasks and n data sets, where the number of images in the n data sets is {N 1 ,N 2 …N n}, then the number of labels corresponding to each task is:

[0035]

[0036] in, is the task balance factor corresponding to the kth task, N i,k is the number of labels of the k-th task in the i-th dataset, i∈{1,2…n}, k∈{1,2…K}.

[0037] In some embodiments, the method further comprises:

[0038] According to the task balance factor corresponding to each task and the data set balance factor of the label of each task in each data set, the number of labels of each task in each data set is determined, specifically including:

[0039] Assume that there are K tasks and n data sets, where the number of images in the n data sets is {N 1 ,N 2 …N n}, then the number of labels for each task in each dataset is:

[0040]

[0041] in, is the task balance factor corresponding to the kth task, δ i,k is the dataset balancing factor for the label of the kth task in the i-th dataset, N i,kis the number of labels of the k-th task in the i-th dataset, i∈{1,2…n}, k∈{1,2…K}.

[0042] In a second aspect, an embodiment of the present application provides a multi-task learning device, the device comprising:

[0043] A data set acquisition module, used to acquire at least two data sets, the at least two data sets are used for training at least two tasks, wherein each task corresponds to a label, and the number of label types contained in each data set is not greater than the number of tasks;

[0044] A prediction result determination module, used to sample at least two data sets based on a label balanced sampling mechanism, and input all the sampled data as training sample data into the first model for multi-task training to obtain a multi-task prediction result;

[0045] Among them, the prediction result determination module includes:

[0046] A balance factor acquisition unit, used to acquire a task balance factor corresponding to each task and a data set balance factor of a label corresponding to each task in each data set, wherein the task balance factor is used to adjust the number of labels between different tasks, and the data set balance factor is used to adjust the number of labels of each task in different data sets;

[0047] A sampling rate determination unit, used to determine the sampling rate of the corresponding label of each task in each data set according to the task balance factor and the data set balance factor;

[0048] The training sample determination unit is used to sample each of the at least two data sets according to the sampling rate to obtain sample data for training of each task.

[0049] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0050] A memory and one or more processors, the one or more processors are used to execute one or more computer programs stored in the memory, and when the one or more processors execute the one or more computer programs, the electronic device implements the multi-task learning method as in the first aspect.

[0051] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions, which, when executed by a processor, enable the processor to execute the multi-task learning method of the first aspect.

[0052] Beneficial effects of the embodiments of the present application: Different from the prior art, the embodiments of the present application provide a multi-task learning method, including: obtaining at least two data sets, the at least two data sets are used for training at least two tasks, wherein each task corresponds to a label, and the number of label types contained in each data set is not greater than the number of tasks; based on a label balanced sampling mechanism, sampling at least two data sets, and inputting all sampled data as training sample data into a first model for multi-task training to obtain multi-task prediction results; wherein the label balanced sampling mechanism includes: obtaining a task balance factor corresponding to each task and a data set balance factor of the label corresponding to each task in each data set, wherein the task balance factor is used to adjust the number of labels between different tasks, and the data set balance factor is used to adjust the number of labels of each task in different data sets; according to the task balance factor and the data set balance factor, determining the sampling rate of the label corresponding to each task in each data set; according to the sampling rate, sampling each of the at least two data sets to obtain sample data for training of each task.

[0053] By acquiring at least two data sets, obtaining a task balance factor and a data set balance factor, wherein the task balance factor is used to adjust the number of labels between different tasks, and the data set balance factor is used to adjust the number of labels of each task in different data sets, and using the task balance factor and the data set balance factor to determine the sampling rate of the corresponding labels of each task in each data set, so as to sample each data set and obtain sample data for training of each task, and then perform multi-task training to obtain multi-task training results. The embodiment of the present application can solve the technical problem of label imbalance caused by different numbers of labels or imbalanced proportions or different data distributions between different tasks in different data sets when performing multi-task single-stage training, so as to balance the labels between different tasks, thereby balancing the performance of different tasks in multi-task learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0055] Figure 1 It is a schematic diagram of an application environment of a multi-task learning method provided in an embodiment of the present application;

[0056] Figure 2 is a schematic diagram of multi-task learning provided in an embodiment of the present application;

[0057] Figure 3It is a flowchart of a multi-task learning method provided in an embodiment of the present application;

[0058] Figure 4 It is a schematic diagram of a framework of multi-task learning provided in an embodiment of the present application;

[0059] Figure 5 is another flowchart of multi-task learning provided by an embodiment of the present application;

[0060] Figure 6 yes Figure 5 A detailed flowchart of step S504 in FIG.

[0061] Figure 7 yes Figure 5 A detailed flowchart of step S505 in FIG.

[0062] Figure 8 It is a flowchart of a label balanced sampling mechanism for multi-task learning provided in an embodiment of the present application;

[0063] Fig. 9 is a structural diagram of a multi-task learning device provided in an embodiment of the present application;

[0064] Fig.10 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The present application is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements can also be made without departing from the concept of the present application. These all belong to the protection scope of the present application.

[0066] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0067] It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other, all within the scope of protection of the present application. In addition, although the functional module division is performed in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a sequence different from the module division in the device or the flow chart. In addition, the words "first", "second", "third", etc. used herein do not limit the data and execution order, but only distinguish the same items or similar items with basically the same functions and effects.

[0068] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in this specification and in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in this specification includes any and all combinations of one or more of the related listed items.

[0069] In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0070] Before explaining the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0071] (1) Neural network, also referred to as neural network (NNs) or connection model, is an algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. Neural networks rely on the complexity of the system to adjust the interconnected relationships between a large number of internal nodes to achieve the purpose of processing information. Specifically, a neural network can be composed of neural units, which can be specifically understood as a neural network with an input layer, a hidden layer, and an output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. Among them, a neural network with many hidden layers is called a deep neural network (DNN). The work of each layer in a neural network can be described by the mathematical expression y=a(W·x+b). From a physical level, the work of each layer in a neural network can be understood as completing the transformation from input space to output space (i.e., the row space to the column space of a matrix) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimension increase / reduction; 2. Enlargement / reduction; 3. Rotation; 4. Translation; 5. "Bending". The operations of 1, 2, and 3 are completed by "W·x", the operation of 4 is completed by "+b", and the operation of 5 is implemented by "a()". The word "space" is used here because the classified object is not a single thing, but a class of things. Space refers to the collection of all individuals of this class of things. W is the weight matrix of each layer of the neural network. Each value in the matrix represents the weight value of a neuron in this layer. The matrix W determines the spatial transformation from the input space to the output space mentioned above, that is, the W of each layer of the neural network controls how to transform the space. The purpose of training a neural network is to finally obtain the weight matrix of all layers of the trained neural network. Therefore, the training process of a neural network is essentially to learn how to control spatial transformation, and more specifically to learn the weight matrix.

[0072] It should be noted that in the embodiments of the present application, the models used in multi-task learning are essentially neural networks. Common components in neural networks include convolutional layers, pooling layers, normalization layers, and inverse convolutional layers. By assembling these common components in neural networks, a model is designed. When the model parameters (weight matrices of each layer) are determined so that the model error meets the preset conditions or the number of model parameters is adjusted to reach a preset threshold, the model converges.

[0073] The convolution layer is configured with multiple convolution kernels, each of which is set with a corresponding step size to perform convolution operations on the image. The purpose of the convolution operation is to extract different features of the input image. The first convolution layer may only extract some low-level features such as edges, lines, and corners. Deeper convolution layers can iteratively extract more complex features from low-level features.

[0074] The deconvolution layer is used to map a low-dimensional space to a high-dimensional space while maintaining the connection relationship / pattern between them (the connection relationship here refers to the connection relationship during convolution). The deconvolution layer is configured with multiple convolution kernels, each of which is set with a corresponding step size to perform deconvolution operations on the image. Generally, the framework library used to design neural networks (such as the PyTorch library) has a built-in upsumple() function, which can be called to achieve low-dimensional to high-dimensional spatial mapping.

[0075] Pooling layers can reduce the dimension of data or represent images with higher-level features by imitating the human visual system. Common operations of pooling layers include maximum pooling, mean pooling, random pooling, median pooling, and combined pooling. Generally speaking, pooling layers are periodically inserted between convolutional layers of neural networks to achieve dimensionality reduction.

[0076] The normalization layer is used to normalize all neurons in the intermediate layer to prevent gradient explosion and gradient disappearance.

[0077] (2) Loss function refers to a function that maps the value of a random event or its related random variables to a non-negative real number to represent the "risk" or "loss" of the random event. The loss function is a non-negative real function that is used to quantify the difference between the predicted label and the true label predicted by the model. In applications, the loss function is usually associated with the optimization problem as a learning criterion, that is, the model is solved and evaluated by minimizing the loss function. For example, it is used for model parameter estimation in statistics and machine learning. In the process of training a neural network, because we hope that the output of the neural network is as close as possible to the value we really want to predict, we can compare the current network's predicted value with the desired target value, and then update the weight matrix of each layer of the neural network according to the difference between the two (however, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the neural network). For example, if the network's predicted value is too high, adjust the weight matrix to make it predict a lower value, and continue to adjust until the neural network can predict the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which is an important equation used to measure the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing the output value (loss) as much as possible.

[0078] The technical solution of the present application is described in detail below in conjunction with the accompanying drawings.

[0079] See also Figure 1 , Figure 1 It is a schematic diagram of an application environment of a multi-task learning method provided in an embodiment of the present application;

[0080] like Figure 1 As shown, the application environment 100 includes: an electronic device 101 and a server 102, and the electronic device 101 and the server 102 communicate via wired or wireless communication.

[0081] The electronic device 101 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The electronic device 101 may be provided with a client, which may be a video client, a browser client, an online shopping client, an instant messaging client, etc., and this application does not limit the type of the client.

[0082] The electronic device 101 and the server 102 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application. The electronic device 101 may obtain at least two data sets and predict the image data in the data sets to obtain a multi-task prediction result, wherein the image data in the data sets may be images stored in the memory of the electronic device 101 or images received from other devices, such as images sent by the server 102.

[0083] Alternatively, the electronic device 101 may receive at least two data sets sent by the server 102, store at least two data sets, and perform multi-task training based on the at least two data sets to obtain a multi-task prediction result. The user may browse the images stored in the electronic device, send the images to the server 102, and the server 102 may perform multi-task training on at least two data sets to obtain a multi-task prediction result, and send the multi-task prediction result 102 to the electronic device 102. It is understandable that the electronic device may acquire image data through an image acquisition device, wherein the image acquisition device may be built into the electronic device 101 or may be externally connected to the electronic device 101, and this application does not limit this.

[0084] It is understandable that the electronic device 101 may generally refer to one of a plurality of electronic devices, and the embodiment of the present application is only illustrated by the electronic device 101. Those skilled in the art may know that the number of the above electronic devices may be more or less. For example, the above electronic device may be only one, or the above electronic device may be dozens or hundreds, or more. The embodiment of the present application does not limit the number and device type of the electronic devices.

[0085] Among them, server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.

[0086] The server 102 and the electronic device 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application. The server 102 may maintain an image database for storing multiple data sets, each data set including multiple images. The server 102 may receive images and / or multi-task training instructions sent by the electronic device 101, and generate at least two data sets based on the images sent by the electronic device 102, perform multi-task training on at least two data sets according to the multi-task training instructions to obtain multi-task prediction results, and send the multi-task prediction results to the electronic device 101.

[0087] It is understandable that the number of the above servers 102 may be more or less, and the embodiment of the present application does not limit this. Of course, the server 102 may also include other functional servers to provide more comprehensive and diversified services.

[0088] See also Figure 2 , Figure 2 is a schematic diagram of multi-task learning provided in an embodiment of the present application;

[0089] like Figure 2 As shown in the figure, in multi-task learning (MTL), in a multi-task learning network, features of multiple inputs are used together to obtain shared parameters, where shared parameters can be used for the training of each task, and task-specific parameters corresponding to each task are obtained through multiple shared parameters. For example, Task 1, Task 2, Task 3, and Task 4 use shared parameters for task learning respectively, and each task corresponds to specific task-specific parameters for training the corresponding task. It can be understood that multiple shared parameters are back-propagated in parallel to the four outputs. Since the four outputs share the bottom hidden layer, the feature representations used for a task in these hidden layers can also be used by other tasks, prompting multiple tasks to learn together. The core idea of ​​multi-task learning is to train multiple tasks in parallel and share the feature representations learned by different tasks.

[0090] It is understandable that multi-task learning is an inductive transfer method that makes full use of domain-specific information implicit in the training signals of multiple related tasks. During the backpropagation process, multi-task learning allows the features dedicated to a certain task in the shared hidden layer to be used by other tasks; multi-task learning will be able to learn features that are applicable to several different tasks, which are often not easy to learn in a single-task learning network. The goal of inductive transfer is to use additional sources of information to improve the learning performance of the current task, including improving generalization accuracy, learning speed, and the understandability of the learned model. Providing a stronger inductive bias is a way to transfer and improve generalization ability, which can produce better generalization ability on a fixed training set, or reduce the number of training samples required to achieve the same level of performance. Inductive bias causes an inductive learner to prefer some hypotheses. Multi-task learning uses the information implicit in the training signals of related tasks as an inductive bias to improve generalization ability.

[0091] The embodiment of the present application provides a multi-task learning method to balance the number of labels of each task, thereby balancing the performance of different tasks in multi-task learning.

[0092] For details, please refer to Figure 3 , Figure 3 It is a flowchart of a multi-task learning method provided in an embodiment of the present application;

[0093] The multi-task learning method is applied to the above-mentioned electronic device. Specifically, the executor of the multi-task learning method is one or more processors of the electronic device.

[0094] like Figure 3 As shown, the multi-task learning method includes:

[0095] Step S301: obtaining at least two data sets, where the at least two data sets are used for training at least two tasks, wherein each task corresponds to a label, and the number of label types contained in each data set is not greater than the number of tasks;

[0096] Specifically, the electronic device obtains at least two data sets, each of which includes multiple image data, i.e., target images, and at least two data sets are used for training at least two tasks, wherein each task corresponds to a label, and the number of label types contained in each data set is not greater than N, where N is the number of tasks. For example, if the number of tasks N is 10, i.e., there are N different tasks that need to be trained, then the number of label types in each data set is not greater than 10, for example, one data set includes 10 labels, and another data set includes 9 labels. In order to avoid interference caused by other labels that are not used for training, the embodiment of the present application assumes that the labels in each data set are used for task training.

[0097] It can be understood that a set of samples is called a data set. The label refers to the value that needs to be predicted in the image. The label can be a discrete value or a continuous value, for example, the label is age, whether it is human, etc.

[0098] Please refer to Figure 4 , Figure 4 It is a schematic diagram of a framework of multi-task learning provided in an embodiment of the present application;

[0099] like Figure 4 As shown, the framework of the multi-task learning includes: input, model, and output, wherein the input is at least two data sets, for example: data set 1 and data set 2, each data set is an image data set, including multiple image data, the model is a deep learning model, and the output is the prediction result of the image data, for example: detection result and / or classification result.

[0100] Specifically, at least two data sets are input into a model, namely a first model, the first model includes a backbone network, the first model is used to train at least two tasks, corresponding to at least two task branches, for example: the first model is used to train a detection task and an attribute classification task, the detection task corresponds to a detection branch, the detection branch is used to output a detection result of the image data, the detection result includes a detection frame of a target object in the image data and a score of the target object; the attribute classification task corresponds to an attribute classification branch, the attribute classification branch is used to output a classification result of the image data, and a classification score of an attribute category of the target object in the classification result; wherein, The label corresponding to the detection result output by the detection branch is: detection label, that is, the coordinate of the upper left corner of the detection box and the width and height of the detection box. The detection label is a 4-dimensional variable, for example: the 4-dimensional variable is expressed as (x, y, w, h), x is the horizontal coordinate of the upper left corner of the detection box, y is the vertical coordinate of the upper left corner, w is the width of the detection box, and h is the height of the detection box. The label corresponding to the classification result output by the attribute classification branch is the category label, that is, the category ID. For example: [infant, child, adult, unknown] The IDs corresponding to the four categories are [0, 1, 2, 3] respectively. If the detection box in the image data is a baby, the category label is 0 at this time.

[0101] The attribute classification branch predicts the classification score for each pixel in the image data, and the classification score output by the attribute classification branch is the classification score corresponding to the central pixel of the detection frame of the target object output by the detection branch. Specifically, the detection branch outputs the coordinates of the upper left corner and the lower right corner, and the center of the detection frame of the target object can be obtained through geometric operations. It can be understood that the detection frame of the target object is a rectangular frame, and the center of the detection frame of the target object is the center of the rectangular frame.

[0102] Step S302: based on the label balanced sampling mechanism, sampling is performed on at least two data sets, and all the sampled data are input into the first model as training sample data for multi-task training to obtain multi-task prediction results.

[0103] Specifically, the label balanced sampling mechanism includes:

[0104] Obtain a task balance factor corresponding to each task and a dataset balance factor for the label corresponding to each task in each dataset, wherein the task balance factor is used to adjust the number of labels between different tasks, and the dataset balance factor is used to adjust the number of labels of each task in different datasets; determine a sampling rate of the label corresponding to each task in each dataset based on the task balance factor and the dataset balance factor; sample each dataset in at least two datasets based on the sampling rate to obtain sample data for training of each task.

[0105] It can be understood that the label balanced sampling mechanism is used to sample at least two data sets acquired by the electronic device so that the labels of each task in the image data input into the first model are balanced. To further explain the working principle of the label balanced sampling mechanism, please refer to Figure 5 , Figure 5 is another flowchart of multi-task learning provided by an embodiment of the present application;

[0106] like Figure 5 As shown in Figure 1, the multi-task learning process includes:

[0107] Step S501: obtaining at least two data sets, where the at least two data sets are used for training at least two tasks, wherein each task corresponds to a label, and the number of label types contained in each data set is not greater than the number of tasks;

[0108] Step S502: Obtain the task balance factor corresponding to each task and the data set balance factor of the label corresponding to each task in each data set;

[0109] Specifically, the task balance factor is used to adjust the number of labels between different tasks. For example, among the labels of all data sets, the number of labels corresponding to the first task is different from the number of labels corresponding to the second task. Therefore, the task balance factor is used to balance the number of labels between the first task and the second task. For example, in the entire data set, the number of labels corresponding to the first task is 10,000, while the number of labels corresponding to the second task is 5,000. At this time, the task balance factor corresponding to the first task is set to 0.1, and the task balance factor corresponding to the second task is set to 0.2, so that the number of labels corresponding to the first task and the second task are both 1,000 during the sampling process.

[0110] Specifically, the dataset balance factor is used to adjust the number of labels for each task in different datasets. For example, the number of labels corresponding to the first task in the sampling process is determined to be 1000 through the task balance factor. At this time, the dataset balance factor is used to distribute 1000 labels to each dataset. It can be understood that the number of labels corresponding to each dataset is different, so the number of labels required to be sampled for each dataset is also different. Assuming that each dataset does not contain noise data or the proportion of noise data is the same, then the dataset balance factor corresponding to each dataset is the same, that is, the dataset balance factor corresponding to each dataset is 1, where noise data refers to image data with label errors (NoisyLabels) in the training data.

[0111] In an embodiment of the present application, if the proportion of noise data contained in different data sets is different, the data set balance factors corresponding to different data sets are different. For example, if the proportion of noise data corresponding to the first data set is X, then the data set balance factor corresponding to the first data set is: 1-X, where X∈[0,1]. It can be understood that if the data set balance factor corresponding to each data set is 1-X, then the number of labels sampled for each task is less than the number of labels determined by the task balance factor. At this time, in order to fill the remaining number of labels, sampling can be performed from several data sets with the smallest proportion of noise data, for example, the remaining number of labels can be sampled from a data set with the smallest proportion of noise data, or at least two data sets are sorted in order of the proportion of noise data from small to large, and the top three data sets with the proportion of noise data in the order of small to large are obtained, and the remaining number of labels is determined. Samples are performed from the top three data sets in equal or unequal proportions to determine the number of labels corresponding to each data set, wherein the smaller the proportion of noise data is, the larger the sampling ratio is. After determining the number of labels corresponding to each data set, adjust the data set balance factor corresponding to each data set according to the number of labels corresponding to each data set. For example, if the number of labels corresponding to a task in all data sets is 1000, the task balance factor corresponding to the task is 0.1, thereby determining that the number of labels corresponding to the task in the sampling process is 1000. If the number of labels corresponding to the label in a data set is 1000, and the number of labels sampled by the task in the data set is 120, then the data set balance factor is 1.2, that is, the number of labels sampled by a task in a data set = the number of labels corresponding to a task in a data set * task balance factor * data set balance factor, for example: 120 = 1000*0.1*1.2.

[0112] Step S503: Determine the sampling rate of the corresponding label of each task in each data set according to the task balance factor and the data set balance factor;

[0113] Specifically, according to the task balance factor and the data set balance factor, the sampling rate of the corresponding label of each task in each data set is determined, including:

[0114] Assuming there are K tasks and n data sets, the sampling rate of the label corresponding to each task in each data set is: in, is the task balance factor corresponding to the kth task, δ i,k is the dataset balancing factor for the corresponding label of the k-th task in the i-th dataset, i∈{1,2…n}, k∈{1,2…K}.

[0115] For example: The task balance factor corresponding to the first task is 0.1, and the dataset balance factor of the first task corresponding to the label in the first dataset is δ 1,1 =1, then the sampling rate of the corresponding label of the first task in the first dataset is 0.1*1=0.1.

[0116] Step S504: sampling each of the at least two data sets according to the sampling rate to obtain sample data for training of each task;

[0117] For details, please refer to Figure 6 , Figure 6 yes Figure 5 A detailed flowchart of step S504 in FIG.

[0118] like Figure 6 As shown, step S504: sampling each of the at least two data sets according to the sampling rate to obtain sample data for training of each task, including:

[0119] Step S5041: In at least two data sets, for the corresponding label of each task in each data set, a random number generation algorithm is used to generate a random number;

[0120] Specifically, in at least two data sets, for each label of each task in each data set, a random number is generated using a random number generation algorithm, wherein the value range of the random number is [0, 1]. In an embodiment of the present application, the random number generation method is completed by a random number generation algorithm, wherein the random number generation algorithm includes an algorithm for generating a random number uniformly distributed between [0, 1], such as a linear congruence generator (LCG), a mixed congruence method, and the like.

[0121] Step S5042: If the random number is less than the sampling rate, the tag is added to the first tag set;

[0122] Specifically, the value range of the random number is [0,1], and the value range of the sampling rate is [0,1]. When the random number is less than the sampling rate, the tag is added to the first tag set. It can be understood that when the number of tags is greater, the number of sampled tags will be closer to the number corresponding to the sampling rate, that is, the number of sampled tags ≈ the total number of tags * sampling rate.

[0123] Step S5043: After traversing all labels in all data sets, a second label set corresponding to each task is generated, and the data corresponding to all labels in the second label set are determined as sample data for training each task.

[0124] Specifically, after traversing all the labels in each data set, and then traversing all the data sets, thereby traversing all the labels in all data sets, the first label set finally obtained is used as the second label set, and the image data corresponding to all the labels in the second label set is used as the sample data for training each task.

[0125] By traversing all labels in all data sets to determine sample data for training each task, the embodiment of the present application can better determine the samples corresponding to each task, which is beneficial to multi-task training.

[0126] Step S505: After sampling all data sets, all sampled data are input into the first model as training sample data to perform multi-task training to obtain multi-task prediction results.

[0127] For details, please refer to Figure 7 , Figure 7 yes Figure 5 A detailed flowchart of step S505 in FIG.

[0128] like Figure 7 As shown, step S505: inputting all the sampled data as training sample data into the first model for multi-task training to obtain multi-task prediction results, including:

[0129] Step S5051: input all the sampled data as training sample data into the first model to perform single-stage multi-task training;

[0130] In an embodiment of the present application, the first model includes at least two task branches, each task branch corresponds to a task one by one, wherein at least two task branches share a backbone network.

[0131] Please refer to Figure 4 ,like Figure 4 Take the multi-task learning framework shown in FIG. 1 as an example. The framework includes two task branches, namely the detection branch and the attribute classification branch.

[0132] In multi-task training, training the detection branch and the attribute classification branch simultaneously requires one-to-one corresponding detection boxes and attribute classification labels. However, the amount of training data for different tasks is different, resulting in unbalanced multi-task training. For example: for the detection branch, there are a large number of detection boxes classified as people, but these data sets lack attribute labels, so multi-task training cannot be performed; for the attribute classification branch, there are data sets that contain both detection boxes and attribute category labels, but the amount of data is small, which can easily lead to overfitting. Therefore, the embodiment of the present application proposes to adjust the label ratios of different data sets and different tasks through a label balanced sampling mechanism, thereby balancing the training of multiple tasks.

[0133] like Figure 4 As shown, taking age stage classification as an example, it includes two task branches, namely detection branch and age stage classification branch, wherein the detection branch corresponds to the detection task, and the age stage classification branch corresponds to the age stage classification task. For example, the present application embodiment includes 4 age stages, namely infants, children, adults, and unknown age. Assume that data set 1 only contains data sets with detection box labels, and contains N 1 Dataset 2 contains both detection box labels and age classification labels, a total of N 2 pictures.

[0134] In the mixed data set composed of data set 1 and data set 2, there are a total of N 1 +N 2 images are used to train the detection branch, N 2 pictures to train the age classification branch. 1 >>N 2 , then the number of labels in the age classification branch will be much smaller than that in the detection branch, resulting in overfitting of the age classification branch. Therefore, the present application uses a task balance factor to represent the task balance factor between different tasks. The task balance factor is also a sampling factor for the detection branch and the age classification branch. The value range of the task balance factor is [0,1].

[0135] Moreover, in the mixed dataset composed of dataset 1 and dataset 2, the data of the same branch comes from different datasets. Because the sources of different datasets are different, in order to improve the generalization ability of the model, it is necessary to adjust the label ratio between datasets for different branches. For example, for the detection branch, it is hoped that as many human boxes as possible are detected, and it is not limited to human data such as infants and children. This application uses the dataset balancing factor β 1 and β 2 Represents the ratio of dataset 1 and dataset 2 in the detection label, through the dataset balance factor δ 1 and δ 2Represents the ratio of dataset A and dataset B in age attribute labels. In order to achieve the balance of each label, preferably, the dataset balance factor β is set 1 , β 2 , δ 1 and δ 2 Both are 1.

[0136] like Figure 4 As shown in the figure, the sample data sampled in dataset 1 and dataset 2 are mixed together and input into the model. Through forward propagation, the detection branch of the model outputs the detection box and the score of the corresponding category, and the attribute classification branch outputs the score of the attribute category. In each iteration, the label balance sampling mechanism determines which labels in the batch data of this iteration can calculate the loss function through sampling, and its corresponding loss function can be reversely transmitted. The loss function is calculated with the sampled labels and the parameters are updated through reverse transmission. For example, the score of the corresponding category output by the detection branch of the model includes the softmax score of whether it is a human, and the score of the attribute category includes the softmax score of each age stage. It can be understood that directional conduction has nothing to do with the number of samples, that is, assuming that a batch of data has 32 samples before sampling and 24 samples after sampling, the loss function of the samples that need to be reversely transmitted is finally summed and the gradient is calculated, so the number of samples will not affect the reverse conduction.

[0137] The training samples for each task are determined by sampling, so that the number of samples for each task is different. Assuming that the number of samples of the prediction result set and the label set before sampling is 32, the number of samples of the prediction result set and the label set after sampling is less than or equal to 32, for example: 24 samples.

[0138] It is understandable that each task corresponds to a loss function, and each task is iteratively trained based on the loss function to update the parameters of each task. The joint objective function of multi-task learning is the linear weighting of all task loss functions. Among them, the weights can be assigned according to the importance of different tasks, or according to the difficulty of the tasks. Preferably, all tasks in the embodiment of the present application are set with the same weight.

[0139] In the actual training process, through iterative training, the gradient is calculated and the parameters of each task are updated, for example, the first number threshold is set to determine whether the number of iterations is greater than the first number threshold. Specifically, the embodiment of the present application adopts the stochastic gradient descent (SGD) algorithm to update the parameters of each task, wherein the number of iterations is set to 24 times, the initial learning rate is set to 0.001, the learning rate update strategy is the cosine annealing method, and the minimum learning rate is e -5 .

[0140] It is understandable that cosine annealing can reduce the learning rate through the cosine function, in which the cosine value first decreases slowly as x increases, then decreases rapidly, and then decreases slowly again. In the cosine annealing mode, the decreasing mode can be coordinated with the learning rate to produce a better update effect.

[0141] It is understandable that the number of task branches in the embodiments of the present application is not limited and may include two or more task branches.

[0142] Preferably, all task branches in the embodiment of the present application share the same backbone network, and the backbone network is used for each task branch to predict each sample to obtain the prediction result corresponding to each sample, so as to realize single-stage multi-task training and improve training efficiency. In the embodiment of the present application, the backbone network is a feature extractor for extracting features, wherein the backbone network includes a neural network, such as ResNet, MobileNet, VGG and other neural networks.

[0143] It can be understood that multiple tasks share the same backbone network, and the features output by the backbone network are shared features. The features output by the shared network corresponding to the shared parameters are used to predict the output results of different tasks. In other words, multiple tasks share the same feature extractor and output the results of different tasks at the same time, which is conducive to multi-task training.

[0144] Step S5052: adding the prediction results of the sample data for training corresponding to each task to the prediction result set, so as to simultaneously obtain the prediction result set of each task in at least two tasks;

[0145] Specifically, the prediction results of the sample data for training corresponding to each task are added to the prediction result set to simultaneously obtain the prediction result set of each task in at least two tasks, including:

[0146] Each task branch in the first model predicts each sample data of the task corresponding to it to obtain a prediction result corresponding to each sample data, and the prediction result corresponding to each sample data is added to the prediction result set of each task to simultaneously obtain a prediction result set for each task in at least two tasks.

[0147] Since each task corresponds to a set of sample data for training, and each sample data corresponds to a prediction result, after obtaining the prediction result corresponding to each sample data, the prediction result is added to the prediction result set corresponding to the task. After traversing all the sample data corresponding to the task, the final prediction result set is obtained. The final prediction result set is the prediction result set corresponding to the task. Similarly, the next task is processed to obtain the prediction result set of each of at least two tasks.

[0148] Step S5053: combining the prediction result set of each task, and determining the combined prediction result set as the multi-task prediction result.

[0149] Specifically, after traversing all sample data, a prediction result set corresponding to each task is obtained, the prediction result set corresponding to each task is combined, and the combined prediction result set is determined as a multi-task prediction result.

[0150] Compared with the method in which a model trained on a single task infers the data missing labels on the task, and uses the generated results as pseudo labels to increase the amount of training data for the task, and finally fine-tunes the multi-task model pre-trained on the pseudo labels on the data corresponding to each task label, the embodiment of the present application performs a single-stage multi-task training based on the training samples corresponding to each task. Since a single-stage multi-task training is adopted, the model training can be completed in one stage without going through the process of rough training and then fine-tuning. Therefore, the present application can improve the efficiency of multi-task training.

[0151] Please refer to Figure 8 , Figure 8 It is a flowchart of a label balanced sampling mechanism for multi-task learning provided in an embodiment of the present application;

[0152] Among them, the label balanced sampling mechanism extracts the labels of the training data by sampling, and adjusts the label quantities of different data sets and different tasks with different sampling rates.

[0153] like Figure 8 As shown in Figure 1, the process of the label balanced sampling mechanism for multi-task learning includes:

[0154] Step S801: Obtain the corresponding label of a task in a data set;

[0155] Specifically, assume that there are K tasks and n data sets, where the number of images in the n data sets is {N 1 ,N 2 …N n}, is the task balance factor corresponding to the kth task, Ni,k is the number of labels of the k-th task in the i-th dataset, i∈{1,2…n}, k∈{1,2…K}.

[0156] Step S802: Generate a random number;

[0157] Specifically, assume that the random number is c, where the value range of c is [0,1].

[0158] Step S803: whether the random number is less than the task balance factor * the data set balance factor;

[0159] Specifically, determine whether the random number is less than the task balance factor * data set balance factor, that is, whether c is less than If yes, proceed to step S809; if no, proceed to step S804;

[0160] Step S804: whether to traverse all data sets;

[0161] Specifically, it is determined whether all data sets have been traversed, that is, whether the current data set is the last data set. If so, the process proceeds to step S805; if not, the process proceeds to step S807;

[0162] Step S805: whether to traverse all tasks;

[0163] Specifically, it is determined whether all tasks have been traversed, that is, whether the current task is the last task. If so, the process proceeds to step S806; if not, the process proceeds to step S808;

[0164] Step S806: Generate a final label set and a final prediction result set;

[0165] Specifically, if it is determined to traverse all data sets and all tasks, the current label set is used as the final label set, and the current prediction result set is used as the final prediction result set.

[0166] Step S807: enter the next data set;

[0167] Step S808: Enter the next task;

[0168] Step S809: adding the tag to the corresponding tag set, and adding the prediction result of the image data corresponding to the tag to the prediction result set;

[0169] It is understandable that sampling by sampling rate may cause sampling error, resulting in an error between the number of sample data and the expected number of data. Therefore, in the embodiment of the present application, the sampling error is reduced by determining the number of labels corresponding to each task. Specifically, the number of labels corresponding to each task is determined according to the task balance factor corresponding to each task, which specifically includes:

[0170] Assume that there are K tasks and n data sets, where the number of images in the n data sets is {N 1 ,N 2 …N n}, then the number of labels corresponding to each task is:

[0171]

[0172] in, is the task balance factor corresponding to the kth task, N i,k is the number of labels of the k-th task in the i-th dataset, i∈{1,2…n}, k∈{1,2…K}.

[0173] For example: suppose there are 2 tasks and 10 data sets, among which the number of labels related to the first task in each data set is 1000, then the number of labels related to the first task in the 10 data sets is 10000, and the number of labels related to the second task in each data set is 500, then the number of labels related to the second task is 5000. At this time, in order to balance the number of labels between the first task and the second task, the task balance factor corresponding to the first task is set to 0.1, and the task balance factor corresponding to the second task is set to 0.2. At this time, it can be determined that the number of labels corresponding to the first task is 10000*0.1=1000, and the number of labels corresponding to the second task is determined to be 5000*0.2=1000.

[0174] By determining the number of labels corresponding to each task, the number of sample data corresponding to each task can be determined based on the determined number of labels, so that the number of sample data input into the first model for training of each task is equal. Since the number of sample data of different tasks is better balanced, the performance of different tasks in multi-task learning is better balanced.

[0175] It can be understood that the labels corresponding to each task are distributed in different data sets. In order to better achieve label balance, the multi-task learning method further determines the number of labels for each task in each data set. Specifically, according to the task balance factor corresponding to each task and the data set balance factor of the label of each task in each data set, the number of labels for each task in each data set is determined, which specifically includes:

[0176] Assume that there are K tasks and n data sets, where the number of images in the n data sets is {N 1 ,N 2 …N n}, then the number of labels for each task in each dataset is:

[0177]

[0178] in, is the task balance factor corresponding to the kth task, δ i,k is the dataset balancing factor for the label of the kth task in the i-th dataset, N i,k is the number of labels of the k-th task in the i-th dataset, i∈{1,2…n}, k∈{1,2…K}.

[0179] For example: suppose there are 2 tasks and 10 data sets, among which the number of labels related to the first task in each data set is 1000, then the number of labels related to the first task in the 10 data sets is 10000, and the number of labels related to the second task in each data set is 500, then the number of labels related to the second task is 5000. At this time, in order to balance the number of labels between the first task and the second task, the task balance factor corresponding to the first task is set to 0.1, and the task balance factor corresponding to the second task is set to 0.2. At this time, it can be determined that the number of labels corresponding to the first task is 10000*0.1=1000, and the number of labels corresponding to the second task is determined to be 5000*0.2=1000, so that the number of labels corresponding to the two tasks is equal.

[0180] At this time, the 1000 labels corresponding to the first task need to be distributed to each data set. For example, the data set balance factor of the labels of the first task in the first data set is 1. At this time, the sampling rate = task balance factor * data set balance factor = 0.1*1 = 0.1, then the number of labels of the first task in the first data set is 1000*0.1 = 100. Similarly, the data set balance factor of the labels of the first task in the second data set is 1, and the number of labels in the second data set is also 1000*0.1 = 100, ..., and the number of labels in the tenth data set is also 1000*0.1 = 100, so that the sample data for training of the first task can be obtained. The determination of the number of labels of the data set corresponding to the second task is similar to that of the first task, and will not be repeated here.

[0181] By determining the number of labels for each task in each data set, the sampling error in the sampling process can be reduced. In addition, the labels corresponding to each task can be better mapped to each data set to achieve a better label distribution, which is conducive to better achieving label balance in the multi-task learning process, thereby balancing the performance of different tasks in multi-task learning.

[0182] It should be noted that the embodiments of the present application may include two or more tasks, which are not limited here and all fall within the protection scope of the embodiments of the present application.

[0183] In an embodiment of the present application, a multi-task learning method is provided, including: obtaining at least two data sets, the at least two data sets are used for training at least two tasks, wherein each task corresponds to a label, and the number of label types contained in each data set is not greater than the number of tasks; based on a label balanced sampling mechanism, sampling the at least two data sets, and inputting all the sampled data as training sample data into a first model for multi-task training to obtain a multi-task prediction result; wherein the label balanced sampling mechanism includes: obtaining a task balance factor corresponding to each task and a data set balance factor of the label corresponding to each task in each data set, wherein the task balance factor is used to adjust the number of labels between different tasks, and the data set balance factor is used to adjust the number of labels of each task in different data sets; according to the task balance factor and the data set balance factor, determining the sampling rate of the label corresponding to each task in each data set; according to the sampling rate, sampling each of the at least two data sets to obtain sample data for training of each task.

[0184] By acquiring at least two data sets, obtaining a task balance factor and a data set balance factor, wherein the task balance factor is used to adjust the number of labels between different tasks, and the data set balance factor is used to adjust the number of labels of each task in different data sets, and using the task balance factor and the data set balance factor to determine the sampling rate of the corresponding labels of each task in each data set, so as to sample each data set and obtain sample data for training of each task, and then perform multi-task training to obtain multi-task training results. The embodiment of the present application can solve the technical problem of label imbalance caused by different numbers of labels or imbalanced proportions or different data distributions between different tasks in different data sets when performing multi-task single-stage training, so as to balance the labels between different tasks, thereby balancing the performance of different tasks in multi-task learning.

[0185] See also Fig. 9 , Fig. 9 is a structural diagram of a multi-task learning device provided in an embodiment of the present application;

[0186] The multi-task learning device is applied to an electronic device. Specifically, the multi-task learning device is applied to one or more processors of the electronic device.

[0187] like Fig. 9 As shown, the multi-task learning device 90 includes:

[0188] The data set acquisition module 901 is used to acquire at least two data sets, where the at least two data sets are used for training at least two tasks, wherein each task corresponds to a label, and the number of label types contained in each data set is not greater than the number of tasks;

[0189] A prediction result determination module 902 is used to sample at least two data sets based on a label balanced sampling mechanism, and input all the sampled data as training sample data into the first model for multi-task training to obtain a multi-task prediction result;

[0190] The prediction result determination module 902 includes:

[0191] The balance factor acquisition unit 9021 is used to acquire the task balance factor corresponding to each task and the data set balance factor of the label corresponding to each task in each data set, wherein the task balance factor is used to adjust the number of labels between different tasks, and the data set balance factor is used to adjust the number of labels of each task in different data sets;

[0192] The sampling rate determination unit 9022 is used to determine the sampling rate of the corresponding label of each task in each data set according to the task balance factor and the data set balance factor;

[0193] The training sample determination unit 9023 is used to sample each of the at least two data sets according to the sampling rate to obtain sample data for training of each task.

[0194] In the embodiments of the present application, the multi-task learning device can also be constructed by hardware devices. For example, the multi-task learning device can be constructed by one or more chips, and each chip can work in coordination with each other to complete the multi-task learning method described in the above embodiments. For another example, the multi-task learning device can also be constructed by various logic devices, such as a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components.

[0195] The multi-task learning device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0196] The multi-task learning device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0197] The multi-task learning device provided in the embodiment of the present application can achieve Figure 3 To avoid repetition, the various implementation processes will not be described here.

[0198] It should be noted that the above-mentioned multi-task learning device can execute the multi-task learning method provided in the above-mentioned embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not fully described in the embodiment of the multi-task learning device, please refer to the multi-task learning method provided in the embodiment of the present application.

[0199] In an embodiment of the present application, a multi-task learning device is provided, including: a data set acquisition module, used to acquire at least two data sets, the at least two data sets are used for training at least two tasks, wherein each task corresponds to a label, and the number of label types contained in each data set is not greater than the number of tasks; a prediction result determination module, used to sample at least two data sets based on a label balance sampling mechanism, and input all sampled data as training sample data into a first model for multi-task training to obtain a multi-task prediction result; wherein the prediction result determination module includes: a balance factor acquisition unit, used to acquire a task balance factor corresponding to each task and a data set balance factor of a label corresponding to each task in each data set, wherein the task balance factor is used to adjust the number of labels between different tasks, and the data set balance factor is used to adjust the number of labels of each task in different data sets; a sampling rate determination unit, used to determine the sampling rate of the label corresponding to each task in each data set according to the task balance factor and the data set balance factor; a training sample determination unit, used to sample each of the at least two data sets according to the sampling rate, to obtain sample data for training of each task.

[0200] By acquiring at least two data sets, obtaining a task balance factor and a data set balance factor, wherein the task balance factor is used to adjust the number of labels between different tasks, and the data set balance factor is used to adjust the number of labels of each task in different data sets, and using the task balance factor and the data set balance factor to determine the sampling rate of the corresponding labels of each task in each data set, so as to sample each data set and obtain sample data for training of each task, and then perform multi-task training to obtain multi-task training results. The embodiment of the present application can solve the technical problem of label imbalance caused by different numbers of labels or imbalanced proportions or different data distributions between different tasks in different data sets when performing multi-task single-stage training, so as to balance the labels between different tasks, thereby balancing the performance of different tasks in multi-task learning.

[0201] The present application also provides an electronic device. Fig.10 , Fig.10 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0202] like Fig.10 As shown, the electronic device 10 includes at least one processor 11 and a memory 12 ( Fig.10 (a bus connection and a processor are used as an example).

[0203] The processor 11 is used to provide computing and control capabilities to control the electronic device 10 to perform corresponding tasks, for example, to control the electronic device 10 to perform the multi-task learning method in any of the above method embodiments, including: obtaining at least two data sets, the at least two data sets are used for training at least two tasks, wherein each task corresponds to a label, and the number of label types contained in each data set is not greater than the number of tasks; based on the label balanced sampling mechanism, sampling the at least two data sets, and inputting all the sampled data as training sample data into the first model for multi-task training to obtain multi-task prediction results; wherein the label balanced sampling mechanism includes: obtaining a task balance factor corresponding to each task and a data set balance factor of each task corresponding to the label in each data set, wherein the task balance factor is used to adjust the number of labels between different tasks, and the data set balance factor is used to adjust the number of labels of each task in different data sets; according to the task balance factor and the data set balance factor, determining the sampling rate of the label corresponding to each task in each data set; according to the sampling rate, sampling each of the at least two data sets to obtain sample data for training of each task.

[0204] By acquiring at least two data sets, obtaining a task balance factor and a data set balance factor, wherein the task balance factor is used to adjust the number of labels between different tasks, and the data set balance factor is used to adjust the number of labels of each task in different data sets, and using the task balance factor and the data set balance factor to determine the sampling rate of the corresponding labels of each task in each data set, so as to sample each data set and obtain sample data for training of each task, and then perform multi-task training to obtain multi-task training results. The embodiment of the present application can solve the technical problem of label imbalance caused by different numbers of labels or imbalanced proportions or different data distributions between different tasks in different data sets when performing multi-task single-stage training, so as to balance the labels between different tasks, thereby balancing the performance of different tasks in multi-task learning.

[0205] The processor 11 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a hardware chip or any combination thereof; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above-mentioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0206] The memory 12, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the multi-task learning method in the embodiment of the present application. The processor 11 can implement the multi-task learning method in any of the following method embodiments by running the non-transitory software programs, instructions and modules stored in the memory 12. Specifically, the memory 12 may include a volatile memory (VM), such as a random access memory (RAM); the memory 12 may also include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD) or other non-transitory solid-state storage device; the memory 12 may also include a combination of the above types of memories.

[0207] In the embodiment of the present application, the memory 12 may also include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0208] In the embodiment of the present application, the electronic device 10 may also have components such as a wired or wireless network interface, a keyboard, and an input / output interface for input and output. The electronic device 10 may also include other components for realizing device functions, which will not be described in detail here.

[0209] The present application also provides a computer-readable storage medium, such as a memory including a program code, which can be executed by a processor to complete the multi-task learning method in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CDROM), a magnetic tape, a floppy disk, and an optical data storage device.

[0210] The embodiment of the present application also provides a computer program product, which includes one or more program codes, and the program code is stored in a computer-readable storage medium. The processor of the electronic device reads the program code from the computer-readable storage medium, and the processor executes the program code to complete the method steps of the multi-task learning method provided in the above embodiment.

[0211] A person skilled in the art will appreciate that all or part of the steps for implementing the above embodiments may be accomplished by hardware or by hardware associated with a program code, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0212] Through the description of the above implementation methods, ordinary technicians in this field can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Ordinary technicians in this field can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0213] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Under the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present application as mentioned above, which are not provided in detail for the sake of simplicity. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-task learning method, It is characterized in that The method comprises: Obtain at least two data sets, where the at least two data sets are used for training at least two tasks, wherein each task corresponds to a label, and the number of label types contained in each data set is not greater than the number of tasks; Based on a label balanced sampling mechanism, sampling is performed on at least two of the data sets, and all the sampled data are input as training sample data into the first model for multi-task training to obtain a multi-task prediction result; The label balanced sampling mechanism includes: Obtain a task balance factor corresponding to each task and a data set balance factor of a label corresponding to each task in each data set, wherein the task balance factor is used to adjust the number of labels between different tasks, and the data set balance factor is used to adjust the number of labels of each task in different data sets; Determine, according to the task balance factor and the data set balance factor, a sampling rate of a label corresponding to each task in each data set; According to the sampling rate, each of the at least two data sets is sampled to obtain sample data for training of each task.

2. The method according to claim 1, It is characterized in that The step of determining the sampling rate of the label corresponding to each task in each data set according to the task balance factor and the data set balance factor includes: Assuming there are K tasks and n data sets, the sampling rate of the label corresponding to each task in each data set is: in, is the task balance factor corresponding to the kth task, δ i,k is the dataset balancing factor for the corresponding label of the k-th task in the i-th dataset, i∈{1,2…n}, k∈{1,2…K}.

3. The method according to claim 1 or 2, It is characterized in that The step of sampling each of the at least two data sets according to the sampling rate to obtain sample data for training of each task includes: In at least two of the data sets, for a corresponding label of each task in each data set, a random number generation algorithm is used to generate a random number; If the random number is less than the sampling rate, adding the tag to the first tag set; After traversing all labels in all data sets, a second label set corresponding to each task is generated, and data corresponding to all labels in the second label set are determined as sample data for training each task.

4. The method according to claim 1, It is characterized in that The step of inputting all the sampled data as training sample data into the first model for multi-task training to obtain multi-task prediction results includes: All the sampled data are input into the first model as training sample data for single-stage multi-task training; Adding the prediction results of the sample data used for training corresponding to each task to the prediction result set, so as to simultaneously obtain the prediction result set of each task in at least two tasks; The prediction result set of each task is combined, and the combined prediction result set is determined as the multi-task prediction result.

5. The method according to claim 4, It is characterized in that The first model includes at least two task branches, each task branch corresponds to a task, and the prediction result of the sample data for training corresponding to each task is added to the prediction result set to simultaneously obtain the prediction result set of each task in the at least two tasks, including: Each task branch in the first model predicts each sample data of the task corresponding to it to obtain a prediction result corresponding to each sample data, and the prediction result corresponding to each sample data is added to the prediction result set of each task to simultaneously obtain a prediction result set for each task in at least two tasks.

6. The method according to claim 5, It is characterized in that At least two task branches share the same backbone network, and the backbone network is used for each task branch to predict each sample to obtain a prediction result corresponding to each sample.

7. The method according to claim 1, It is characterized in that The method further comprises: According to the task balance factor corresponding to each task, determine the number of tags corresponding to each task, including: Assume that there are K tasks and n data sets, where the number of images in the n data sets is {N 1 ,N 2 …N n }, then the number of labels corresponding to each task is: in, is the task balance factor corresponding to the kth task, N i,k is the number of labels of the k-th task in the i-th dataset, i∈{1,2…n}, k∈{1,2…K}.

8. The method according to claim 1, It is characterized in that The method further comprises: According to the task balance factor corresponding to each task and the data set balance factor of the label of each task in each data set, the number of labels of each task in each data set is determined, specifically including: Assume that there are K tasks and n data sets, where the number of images in the n data sets is {N 1 ,N 2 …N n }, then the number of labels for each task in each dataset is: in, is the task balance factor corresponding to the kth task, δ i,k is the dataset balancing factor for the label of the kth task in the i-th dataset, N i,k is the number of labels of the k-th task in the i-th dataset, i∈{1,2…n}, k∈{1,2…K}.

9. A multi-task learning device, It is characterized in that The device comprises: A data set acquisition module, used to acquire at least two data sets, wherein the at least two data sets are used for training at least two tasks, wherein each task corresponds to a label, and the number of label types contained in each data set is not greater than the number of tasks; A prediction result determination module, used to sample at least two of the data sets based on a label balanced sampling mechanism, and input all the sampled data as training sample data into the first model for multi-task training to obtain a multi-task prediction result; Wherein, the prediction result determination module includes: A balance factor acquisition unit, used to acquire a task balance factor corresponding to each task and a data set balance factor of a label corresponding to each task in each data set, wherein the task balance factor is used to adjust the number of labels between different tasks, and the data set balance factor is used to adjust the number of labels of each task in different data sets; A sampling rate determination unit, used to determine a sampling rate of a label corresponding to each task in each data set according to the task balance factor and the data set balance factor; The training sample determination unit is used to sample each of the at least two data sets according to the sampling rate to obtain sample data for training of each task.

10. An electronic device, It is characterized in that include: A memory and one or more processors, wherein the one or more processors are used to execute one or more computer programs stored in the memory, and when the one or more processors execute the one or more computer programs, the electronic device implements the method as described in any one of claims 1 to 8.

Images