Multi-task model training method, detection method, device, terminal equipment and medium
By combining task uncertainty and priority in multi-task learning, the joint training process is improved, the detection accuracy of the multi-task perception model is enhanced, and the problem of low perception accuracy in existing technologies is solved.
Patent Information
- Application Number
- CN202210845122.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-18
- Publication Date
- 2025-10-28
- Estimated Expiration
- 2042-07-18
AI Technical Summary
Existing joint models have low perceptual accuracy in multi-task learning and lack consideration for task priority.
By acquiring multi-task sample images, feature extraction and loss calculation are performed. The multi-task model is jointly trained by combining task uncertainty and priority. Backpropagation is performed using a loss adapter that weights the uncertainty and task priority loss function.
It improves the detection accuracy of the multi-task joint model for multiple perception tasks, improves the joint training process, and combines task uncertainty and priority.
Smart Images
Figure CN115130604B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and in particular to a multi-task model training method, detection method, apparatus, terminal equipment, and medium. Background Technology
[0002] In the overall autonomous driving technology solution, perception, as the most fundamental link, often needs to output multiple types of information to downstream processes (planning and control). For example, in parking scenarios, it needs to detect parking spaces, walls, pillars, drivable space, and obstacles (dynamic and static obstacles). Therefore, after receiving perception data (such as perception data from cameras, LiDAR, and millimeter-wave radar), the perception model needs to output multiple pieces of information simultaneously.
[0003] Regarding the choice of approach, one approach is to have multiple perception models output multiple perception information. While this approach offers high accuracy, its real-time performance is poor. In practical applications, another approach is generally chosen: a single perception model outputs multiple perception information. This type of perception model is usually called a joint model, and the training of the joint model is called joint training. This falls within the same category as multi-task learning research in academia.
[0004] Existing joint models typically consider optimizing all tasks during joint training, but this approach does not achieve high perceptual accuracy for multiple perceptual tasks. Summary of the Invention
[0005] The main objective of this invention is to provide a multi-task model training method, detection method, device, terminal equipment, and medium, aiming to improve the detection accuracy of the multi-task perception model for multiple perception tasks.
[0006] To achieve the above objectives, embodiments of the present invention provide a multi-task model training method, the method comprising:
[0007] Obtain multi-task sample images;
[0008] Feature extraction is performed on the multi-task sample images to obtain feature maps;
[0009] Based on the feature map, the loss value for each task is calculated;
[0010] Based on the loss values of each task, the multi-task model is jointly trained by combining task attribute parameters and task priorities to obtain a trained multi-task perception model.
[0011] Optionally, the task attribute parameters include task uncertainty, and the step of jointly training the multi-task model based on the loss values of each task, combined with the task attribute parameters and task priorities, to obtain the trained multi-task perception model includes:
[0012] The loss values of each task are input into the uncertainty-based loss function integrator and the task priority-based loss function integrator, respectively, to obtain the uncertainty-based loss function and the task priority-based loss function. The uncertainty-based loss function is determined based on the loss value of each task and the corresponding uncertainty, and the task priority-based loss function is determined based on the loss value of each task and the corresponding weight.
[0013] By weighting the uncertainty-based loss function and the task-priority-based loss function with a loss adapter, multi-task learning training is performed to obtain the loss function value and then backpropagation is performed to obtain the multi-task perception model.
[0014] Optionally, the step of weighting the uncertainty-based loss function and the task-priority-based loss function with a loss adapter, performing multi-task learning training to obtain the loss function value, and then performing backpropagation to obtain the multi-task perception model includes:
[0015] Learning weights are assigned to the uncertainty-based loss function and the task-priority-based loss function using a loss adapter;
[0016] Based on the assigned learning weights, the loss adapter weights the uncertainty-based loss function and the task-priority-based loss function to perform multi-task learning training, obtain the loss function value, and perform backpropagation to obtain a multi-task perception model.
[0017] Optionally, the step of assigning learning weights to the uncertainty-based loss function and the task-priority-based loss function via a loss adapter includes:
[0018] The loss adapter employs a preset linear or incremental function and assigns learning weights to the uncertainty-based loss function and the task-priority-based loss function based on the number of training steps.
[0019] Optionally, the task attribute parameters include task loss convergence, and the step of jointly training the multi-task model based on the loss values of each task, combined with the task attribute parameters and task priorities, to obtain the trained multi-task perception model includes:
[0020] The loss values of each task are input into the loss function integrator based on task loss convergence and the loss function integrator based on task priority, respectively, to obtain the loss function based on task loss convergence and the loss function based on task priority. The loss function based on task loss convergence is determined based on the loss value of each task and the corresponding loss convergence, and the loss function based on task priority is determined based on the loss value of each task and the corresponding weight.
[0021] By weighting the loss function based on task loss convergence and the loss function based on task priority using a loss adapter, multi-task learning training is performed to obtain the loss function value, which is then backpropagated to obtain a multi-task perception model.
[0022] Optionally, the step of weighting the loss function based on task loss convergence and the loss function based on task priority using a loss adapter, performing multi-task learning training, obtaining loss function values, and performing backpropagation to obtain a multi-task perception model includes:
[0023] Learning weights are assigned to the task-based loss function and the task-priority-based loss function using a loss adapter.
[0024] Based on the assigned learning weights, the loss function based on task loss convergence and the loss function based on task priority are weighted by the loss adapter to perform multi-task learning training, obtain the loss function value, and perform backpropagation to obtain a multi-task perception model.
[0025] Optionally, the step of assigning learning weights to the loss function based on task loss convergence and the loss function based on task priority through the loss adapter includes:
[0026] The loss adapter employs a preset linear or incremental function and assigns learning weights to the task-based loss function and the task-priority-based loss function based on the number of training steps.
[0027] Optionally, the step of extracting features from the multi-task sample images to obtain feature maps includes:
[0028] The multi-task sample images are input into the backbone network of the multi-task perception model for feature extraction to obtain feature maps.
[0029] Optionally, the step of calculating the loss value for each task based on the feature map includes:
[0030] The feature map is input into the detection head of each task for detection to obtain the detection value of each task.
[0031] The loss value for each task is calculated based on the detection values of each task.
[0032] Optionally, the step of obtaining multi-task sample images may further include:
[0033] The multi-task sample image is processed to obtain a multi-task sample image of the first preset size.
[0034] Optionally, the step of inputting the multi-task sample image into the backbone network of the multi-task perception model for feature extraction to obtain a feature map includes:
[0035] The first preset-size multi-task sample image is input into the backbone network of the multi-task perception model for feature extraction to obtain a feature map of the second preset size.
[0036] Furthermore, this invention also proposes a multi-task detection method, the method comprising:
[0037] Acquire multi-task awareness data;
[0038] The multi-task perception data is input into a pre-trained multi-task perception model for processing. The multi-task perception model is obtained by joint training of task attribute parameters and task priority. The task attribute parameters include at least task uncertainty or task loss convergence.
[0039] The multi-task perception model outputs multi-task perception information.
[0040] Furthermore, embodiments of the present invention also propose a multi-task model training device, the device comprising:
[0041] The sample image acquisition module is used to acquire sample images for multiple tasks;
[0042] The feature extraction module is used to extract features from the multi-task sample images to obtain feature maps;
[0043] The calculation module is used to calculate the loss value for each task based on the feature map;
[0044] The training module is used to jointly train the multi-task model based on the loss values of each task, combined with task attribute parameters and task priorities, to obtain a trained multi-task perception model.
[0045] Furthermore, embodiments of the present invention also propose a multi-task detection device, the device comprising:
[0046] The perception data acquisition module is used to acquire multi-task perception data;
[0047] The processing module is used to input the multi-task perception data into a pre-trained multi-task perception model for processing. The multi-task perception model is obtained by joint training of task attribute parameters and task priority. The task attribute parameters include at least task uncertainty or task loss convergence.
[0048] The output module is used to output multi-task perception information through the multi-task perception model.
[0049] Furthermore, this embodiment of the invention also proposes a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of multi-task model training as described above, or the steps of multi-task detection method as described above.
[0050] Furthermore, embodiments of the present invention also propose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the multi-task model training described above, or the steps of the multi-task detection method described above.
[0051] This invention proposes a multi-task model training method, detection method, apparatus, terminal device, and medium. The method involves acquiring multi-task sample images; extracting features from the multi-task sample images to obtain feature maps; calculating the loss value for each task based on the feature maps; and jointly training the multi-task model based on the loss values of each task, combined with task uncertainty and task priority, to obtain a trained multi-task perception model. Therefore, by improving the joint training process and combining task uncertainty (determined by the attributes of the task itself) and priority (determined by actual business needs), the detection accuracy of the multi-task joint model for multiple perception tasks is improved. Attached Figure Description
[0052] Figure 1 This is a schematic diagram of the functional modules of the terminal device to which the multi-task model training device of the present invention belongs;
[0053] Figure 2 This is a flowchart illustrating an embodiment of the multi-task model training method of the present invention;
[0054] Figure 3 This is a detailed flowchart of an embodiment of the multi-task model training method of the present invention;
[0055] Figure 4 This is a flowchart illustrating an embodiment of the multi-task detection method of the present invention;
[0056] Figure 5 This is a schematic diagram of the functional modules of an embodiment of the multi-task model training device of the present invention;
[0057] Figure 6 This is a schematic diagram of the functional modules of an embodiment of the multi-task detection device of the present invention.
[0058] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0059] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0060] The main solution of this invention is as follows: First, acquire multi-task sample images; second, extract features from the multi-task sample images to obtain feature maps; third, calculate the loss value for each task based on the feature maps; and fourth, jointly train the multi-task model based on the loss values of each task, combined with task uncertainty and task priority, to obtain a trained multi-task perception model. Thus, by improving the joint training process and combining task uncertainty (determined by the attributes of the task itself) and priority (determined by actual business needs), the detection accuracy of the multi-task joint model for multiple perception tasks is improved.
[0061] Technical terms involved in the embodiments of this invention:
[0062] Joint-training: Multi-task training (or joint training)
[0063] Model backbone: The main network of the model, such as ResNet34;
[0064] Task_i head: The detection header of the i-th task;
[0065] Loss_i: The loss function for the i-th task;
[0066] Task-priority Loss Combiner: A loss function integrator based on task priority;
[0067] Uncertainty Loss Combiner: An integrator of loss functions based on uncertainty;
[0068] Loss Apdator: An adapter for linearly weighted loss functions based on task priority and loss functions based on uncertainty.
[0069] Adapted Loss: The loss function value obtained by linearly weighting the loss function of task priority and the loss function of uncertainty using the Loss Apdator.
[0070] Back propagation: the process of propagating backwards.
[0071] The embodiments of the present invention take into account that existing multi-task learning generally considers optimizing all tasks, such as adding task uncertainty parameters, but lacks consideration of task priority.
[0072] Therefore, this invention proposes a solution that improves the joint training process and combines task uncertainty and priority to enhance the detection accuracy of the multi-task joint model for multiple perception tasks.
[0073] Specifically, refer to Figure 1 , Figure 1 This is a functional module diagram of the terminal device to which the multi-task model training device of the present invention belongs. The multi-task model training device can be a data processing device independent of the terminal device, and it can be carried on the terminal device in hardware or software form. The terminal device can be a smart mobile terminal with data processing capabilities, such as a mobile phone or tablet computer, or it can be a fixed terminal device or server with data processing capabilities.
[0074] In this embodiment, the terminal device to which the multi-task model training device belongs includes at least an output module 110, a processor 120, a memory 130, and a communication module 140.
[0075] The memory 130 stores the operating system and the multi-task model training program; the output module 110 may be a display screen, etc. The communication module 140 may include a WIFI module, a mobile communication module, and a Bluetooth module, etc., and communicates with external devices or servers through the communication module 140.
[0076] When the multi-task model training program in memory 130 is executed by the processor, it performs the following steps:
[0077] Obtain multi-task sample images;
[0078] Feature extraction is performed on the multi-task sample images to obtain feature maps;
[0079] Based on the feature map, the loss value for each task is calculated;
[0080] Based on the loss values of each task, the multi-task model is jointly trained by combining task attribute parameters and task priorities to obtain a trained multi-task perception model.
[0081] Furthermore, when the multi-task model training program in memory 130 is executed by the processor, it also performs the following steps:
[0082] The loss values of each task are input into the uncertainty-based loss function integrator and the task priority-based loss function integrator, respectively, to obtain the uncertainty-based loss function and the task priority-based loss function. The uncertainty-based loss function is determined based on the loss value of each task and the corresponding uncertainty, and the task priority-based loss function is determined based on the loss value of each task and the corresponding weight.
[0083] By weighting the uncertainty-based loss function and the task-priority-based loss function with a loss adapter, multi-task learning training is performed to obtain the loss function value and then backpropagation is performed to obtain the multi-task perception model.
[0084] Furthermore, when the multi-task model training program in memory 130 is executed by the processor, it also performs the following steps:
[0085] Learning weights are assigned to the uncertainty-based loss function and the task-priority-based loss function using a loss adapter;
[0086] Based on the assigned learning weights, the loss adapter weights the uncertainty-based loss function and the task-priority-based loss function to perform multi-task learning training, obtain the loss function value, and perform backpropagation to obtain a multi-task perception model.
[0087] Furthermore, when the multi-task model training program in memory 130 is executed by the processor, it also performs the following steps:
[0088] The loss adapter employs a preset linear or incremental function and assigns learning weights to the uncertainty-based loss function and the task-priority-based loss function based on the number of training steps.
[0089] Furthermore, when the multi-task model training program in memory 130 is executed by the processor, it also performs the following steps:
[0090] The loss values of each task are input into the loss function integrator based on task loss convergence and the loss function integrator based on task priority, respectively, to obtain the loss function based on task loss convergence and the loss function based on task priority. The loss function based on task loss convergence is determined based on the loss value of each task and the corresponding loss convergence, and the loss function based on task priority is determined based on the loss value of each task and the corresponding weight.
[0091] By weighting the loss function based on task loss convergence and the loss function based on task priority using a loss adapter, multi-task learning training is performed to obtain the loss function value, which is then backpropagated to obtain a multi-task perception model.
[0092] Furthermore, when the multi-task model training program in memory 130 is executed by the processor, it also performs the following steps:
[0093] Learning weights are assigned to the task-based loss function and the task-priority-based loss function using a loss adapter.
[0094] Based on the assigned learning weights, the loss function based on task loss convergence and the loss function based on task priority are weighted by the loss adapter to perform multi-task learning training, obtain the loss function value, and perform backpropagation to obtain a multi-task perception model.
[0095] Furthermore, when the multi-task model training program in memory 130 is executed by the processor, it also performs the following steps:
[0096] The loss adapter employs a preset linear or incremental function and assigns learning weights to the task-based loss function and the task-priority-based loss function based on the number of training steps.
[0097] Furthermore, when the multi-task model training program in memory 130 is executed by the processor, it also performs the following steps:
[0098] The multi-task sample images are input into the backbone network of the multi-task perception model for feature extraction to obtain feature maps.
[0099] Furthermore, when the multi-task model training program in memory 130 is executed by the processor, it also performs the following steps:
[0100] The feature map is input into the detection head of each task for detection to obtain the detection value of each task.
[0101] The loss value for each task is calculated based on the detection values of each task.
[0102] Furthermore, when the multi-task model training program in memory 130 is executed by the processor, it also performs the following steps:
[0103] The multi-task sample image is processed to obtain a multi-task sample image of the first preset size.
[0104] Furthermore, when the multi-task model training program in memory 130 is executed by the processor, it also performs the following steps:
[0105] The first preset-size multi-task sample image is input into the backbone network of the multi-task perception model for feature extraction to obtain a feature map of the second preset size.
[0106] This embodiment, through the above-described scheme, specifically involves acquiring multi-task sample images; extracting features from the multi-task sample images to obtain feature maps; calculating the loss value for each task based on the feature maps; and jointly training the multi-task model based on the loss values of each task, combined with task uncertainty and task priority, to obtain a trained multi-task perception model. Thus, by improving the joint training process and incorporating task uncertainty and priority, the detection accuracy of the multi-task joint model for multiple perception tasks is improved.
[0107] Based on, but not limited to, the terminal device architecture described above, embodiments of the method of the present invention are proposed.
[0108] The execution subject of the method in this embodiment can be a multi-task model training device, a multi-task model training terminal device, or a server. This embodiment takes a multi-task model training device as an example. The multi-task model training device can be integrated into terminal devices such as smartphones and tablets with data processing functions, or it can be integrated into a fixed terminal or server.
[0109] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the multi-task model training method of the present invention. The multi-task model training method includes:
[0110] Step S101: Obtain multi-task sample images;
[0111] The main purpose of this invention is to train a multi-task perception model, thereby obtaining the multi-task perception model through training.
[0112] The multi-task perception model in this embodiment can be implemented based on a neural network.
[0113] like Figure 3 As shown, the entire neural network consists of two parts: the Joint-training Model backbone (the main network of the multi-task training or joint training model, such as ResNet34) and the Task_i head. The former shares parameters with all tasks, while the latter means that each task has its own Task_i head (the detection head of the i-th task), which does not share parameters with other tasks. For example, there is a parking space detection head fsd_head, a drivable space detection head ds_head, a dynamic obstacle detection head mod_head, etc.
[0114] The sources of multi-task sample images can be varied. For example, perception data for various tasks can be obtained from actual autonomous parking scenarios, such as perception data from cameras, LiDAR, and millimeter-wave radar, which can then be used to obtain sample images of various tasks, such as parking spaces, drivable spaces, and obstacles.
[0115] Taking the front-side model of an autonomous parking scenario as an example, front-side images of the autonomous parking scenario can be collected.
[0116] In one implementation, the size of the collected sample images can be set according to actual needs or obtained through processing. For example, the multi-task sample images can be processed to obtain a multi-task sample image of a first preset size, such as a side-front image of 640*480*3.
[0117] Step S102: Extract features from the multi-task sample images to obtain feature maps;
[0118] In one implementation method, the multi-task sample images are input into the backbone network of the multi-task perception model (e.g., Figure 3 Feature extraction is performed on the Joint-training Model backbone shown to obtain the feature map.
[0119] The backbone network of a multi-task training or joint training model can be, for example, ResNet34.
[0120] In one implementation, after acquiring the multi-task sample image, the multi-task sample image can be processed to obtain a processed multi-task sample image of a first preset size.
[0121] Then, the multi-task sample image of the first preset size is input into the backbone network of the multi-task perception model for feature extraction to obtain a feature map of the second preset size.
[0122] For example, taking the front-side model in an autonomous parking scenario as an example, after inputting a front-side image of size 640*480*3 into the Joint-training Model Backbone, a feature map of size 20*15*512 can be obtained (downsampled by 32 times).
[0123] Step S103: Calculate the loss value for each task based on the feature map;
[0124] The feature map is input into the detection head of each task for detection to obtain the detection value of each task; the loss value of each task is calculated based on the detection value of each task.
[0125] Step S104: Based on the loss values of each task, and in combination with task attribute parameters and task priorities, perform joint training of the multi-task model to obtain the trained multi-task perception model.
[0126] Among them, the task attribute parameters are determined by the attributes of the task itself, and the task priority is determined by the actual business requirements.
[0127] In this embodiment, task attribute parameters may include: task uncertainty, task loss convergence, etc.
[0128] The following section elaborates on the specific scheme for jointly training a multi-task perception model based on the loss values of each task, combined with the task attribute parameters and task priority, using task uncertainty and task loss convergence as the task attribute parameters respectively:
[0129] In one implementation, the task attribute parameters include task uncertainty. The step of jointly training a multi-task model based on the loss values of each task, combined with the task attribute parameters and task priorities, to obtain a trained multi-task perception model may include:
[0130] First, the loss values of each task are input into the uncertainty-based loss function integrator and the task priority-based loss function integrator, respectively, to obtain the uncertainty-based loss function and the task priority-based loss function. The uncertainty-based loss function is determined based on the loss value of each task and the corresponding uncertainty, and the task priority-based loss function is determined based on the loss value of each task and the corresponding weight.
[0131] The uncertainty of each task is a learnable parameter with an initial value of 1.
[0132] The loss weights for each task can be preset, such as by setting them manually.
[0133] Then, by weighting the uncertainty-based loss function and the task-priority-based loss function with a loss adapter, multi-task learning training is performed to obtain the loss function value and backpropagation to obtain the multi-task perception model.
[0134] Specifically, learning weights are assigned to the uncertainty-based loss function and the task-priority-based loss function through a loss adapter;
[0135] Specifically, the loss adapter can employ a preset linear function or an incremental function, and assign learning weights to the uncertainty-based loss function and the task-priority-based loss function based on the number of training steps.
[0136] Then, based on the assigned learning weights, the loss function based on uncertainty and the loss function based on task priority are weighted by the loss adapter to perform multi-task learning training, obtain the loss function value, and perform backpropagation to obtain the multi-task perception model.
[0137] As another implementation, the task attribute parameters include task loss convergence. The step of jointly training the multi-task model based on the loss values of each task, combined with the task attribute parameters and task priorities, to obtain the trained multi-task perception model may include:
[0138] First, the loss values of each task are input into the loss function integrator based on task loss convergence and the loss function integrator based on task priority, respectively, to obtain the loss function based on task loss convergence and the loss function based on task priority. The loss function based on task loss convergence is determined based on the loss value of each task and the corresponding loss convergence, and the loss function based on task priority is determined based on the loss value of each task and the corresponding weight.
[0139] The convergence of the loss for each task is a learnable parameter with an initial value of 1.
[0140] The loss weights for each task can be preset, such as by setting them manually.
[0141] Then, by weighting the loss function based on task loss convergence and the loss function based on task priority through a loss adapter, multi-task learning training is performed to obtain the loss function value and backpropagation to obtain the multi-task perception model.
[0142] Specifically, a loss adapter is used to assign learning weights to the loss function based on task loss convergence and the loss function based on task priority; wherein, the loss adapter can use a preset linear function or an incremental function, and assign learning weights to the loss function based on task loss convergence and the loss function based on task priority based on the number of training steps.
[0143] Then, based on the assigned learning weights, the loss function based on task loss convergence and the loss function based on task priority are weighted by the loss adapter to perform multi-task learning training, obtain the loss function value, and perform backpropagation to obtain the multi-task perception model.
[0144] Therefore, by acquiring multi-task sample images; extracting features from the multi-task sample images to obtain feature maps; calculating the loss value for each task based on the feature maps; and jointly training the multi-task model based on the loss values of each task, combined with task uncertainty and task priority, a trained multi-task perception model is obtained. Thus, by improving the joint training process and incorporating task uncertainty and priority, the detection accuracy of the multi-task joint model for multiple perception tasks is improved.
[0145] The following is combined with Figure 3 The following is a detailed explanation of the scheme in this embodiment for jointly training a multi-task perception model based on the loss values of each task, combined with task uncertainty and task priority, to obtain a trained multi-task perception model:
[0146] Figure 3 The parameters are explained below:
[0147] Joint-training refers to multi-task training or joint training; Model backbone refers to the main network of the model, such as ResNet34; Task_i head refers to the detection head of the i-th task; Loss_i refers to the loss function of the i-th task; Task-priority Loss Combiner refers to the task-priority-based loss function combiner; Uncertainty Loss Combiner refers to the uncertainty-based loss function combiner; Loss Apdator refers to the adapter between the linearly weighted task-priority loss function and the uncertainty loss function.
[0148] The Adapted Loss is the loss function value obtained by linearly weighting the task priority loss function and the uncertainty loss function using the Loss Apdator. Back propagation refers to back propagation.
[0149] Specifically, as one implementation method, such as Figure 3 As shown, the entire neural network consists of two parts: the Joint-training Model backbone and the Task_i head. The former shares parameters with all tasks, while the latter indicates that each task has its own task_i head and does not share parameters with other tasks.
[0150] First, the multi-task sample images are input into the backbone network of the multi-task perception model (e.g., Figure 3 Feature extraction is performed on the Joint-training Model backbone shown to obtain the feature map.
[0151] The backbone network of a multi-task training or joint training model can be, for example, ResNet34.
[0152] For example, taking the front-side model in an autonomous parking scenario as an example, after inputting a front-side image of size 640*480*3 into the Joint-training Model Backbone, a feature map of size 20*15*512 can be obtained (downsampled by 32 times).
[0153] Then, as Figure 3 As shown, the feature maps of each task are passed to each task_i head (e.g., parking space detection head fsd_head, drivable space detection head ds_head, dynamic obstacle detection head mod_head, etc.) to obtain the predicted value output_i (the detection value referred to in this embodiment) of each task_i.
[0154] Then, the loss_i of each task (e.g., loss_fsd, loss_ds, loss_mod, etc.) is calculated using output_i and the corresponding truth value gt_i.
[0155] Then, the loss value loss_i (also known as the loss function of each task) of each task is passed to Task-priority Loss Combiner and Uncertainty Loss Combiner respectively.
[0156] Specifically, after collecting the loss for each task, the task-priority-based loss function Loss_priority is calculated using the Task-priority Loss Combiner. The specific calculation formula is as follows:
[0157]
[0158] Among them, Loss i Let ω be the loss value of the i-th task (also called the loss function of the i-th task). i The weights of the loss function for the i-th task can be set manually, for example, 3.0 for dynamic obstacle detection, 2.0 for wall and pillar detection, and 1.5 for lane line detection.
[0159] Using Equation (1) above, the task priority-based loss function integrator can calculate the loss value of all tasks through preset weights, and obtain a set of loss values based on task priority.
[0160] Meanwhile, after collecting the losses for each task, the Loss_uncert, a loss function based on task uncertainty, is calculated using the Uncertainty Loss Combiner. The specific calculation formula is as follows:
[0161]
[0162] Among them, Loss i Let σ be the loss value of the i-th task (also called the loss function of the i-th task). i : represents the uncertainty of the i-th task, which is a learnable parameter that can be initially set to 1.0.
[0163] Using Equation (2) above, the task uncertainty-based loss function integrator can calculate the loss value of all tasks through learnable weights, thus obtaining a set of loss values based on task uncertainty.
[0164] Finally, a multi-task learning model is obtained by weighting the uncertainty-based loss function and the task-priority-based loss function using a Loss Adapter, yielding the loss function value, and then performing backpropagation. The specific formula for calculating the loss function value is as follows:
[0165]
[0166] Where T is the total number of training steps, t is the current number of training steps; Loss_uncert is the loss function based on task uncertainty, and Loss_priority is the loss function based on task priority.
[0167] The Loss Adaptor assigns weights based on the current training step t and the total training steps T. When t is 0, i.e., at the beginning of training, Loss_uncert takes the lead, allowing the network to better learn multi-tasks based on the uncertainty of the task itself. When t > 0.5T, Loss_priority takes the lead, allowing the network to allocate learning weights for each task according to task priority.
[0168] Therefore, the above solution, based on a multi-task learning method for actual business applications, combines loss_uncert and loss_priority to improve model accuracy. At the same time, it can adaptively allocate the weights of loss_uncert and loss_priority, thereby improving the flexibility of model training and further enhancing the model's accuracy.
[0169] This embodiment combines task uncertainty (determined by the attributes of the task itself) and task priority (determined by actual business) to improve the accuracy of the joint model for multiple perception tasks.
[0170] It's important to note that the purpose of task uncertainty is to enable the network to better perform multi-task learning based on the inherent uncertainty of each task. This can be replaced by other overall optimization schemes for multi-task learning used in academia. For example, replacing `loss_uncertainty` (or `loss_uncert`) with `loss_convergence`, or, as in the alternative implementation mentioned above, determining the learning weights of each task based on the convergence of its own loss. Furthermore, while the Loss Adaptor mentioned in the above embodiments is linear, other non-linear increasing functions can also be used to allocate the weights for `loss_uncertainty` and `loss_priority`.
[0171] Reference Figure 4 , Figure 4This is a flowchart illustrating an embodiment of the multi-task detection method of the present invention. The multi-task detection method includes:
[0172] Step S201: Obtain multi-task awareness data;
[0173] Step S202: Input the multi-task perception data into a pre-trained multi-task perception model for processing. The multi-task perception model is obtained by joint training of task attribute parameters and task priority. The task attribute parameters include at least task uncertainty or task loss convergence.
[0174] Step S203: Output multi-task perception information through the multi-task perception model.
[0175] Furthermore, embodiments of the present invention can also perform sensory data inspection based on the multi-task perception model trained in the above embodiments, and output multi-task perception information.
[0176] For example, after receiving sensing data (such as sensing data from cameras, LiDAR, and millimeter-wave radar), a multi-task perception model can simultaneously output multiple sensing and detection information.
[0177] The multi-task perception model is obtained by jointly training task attribute parameters and task priority. The task attribute parameters include at least task uncertainty or task loss convergence.
[0178] The specific training process of the multi-task perception model in this embodiment can be referred to the above embodiments, and will not be repeated here.
[0179] Compared with existing technologies, this embodiment improves the joint training process by combining task uncertainty (determined by the attributes of the task itself) and priority (determined by actual business), thereby enhancing the detection accuracy of the multi-task joint model for multiple perception tasks.
[0180] In addition, such as Figure 5 As shown in the figure, this embodiment of the invention also proposes a multi-task model training device, the device comprising:
[0181] The sample image acquisition module is used to acquire sample images for multiple tasks;
[0182] The feature extraction module is used to extract features from the multi-task sample images to obtain feature maps;
[0183] The calculation module is used to calculate the loss value for each task based on the feature map;
[0184] The training module is used to jointly train the multi-task model based on the loss values of each task, combined with task attribute parameters and task priorities, to obtain a trained multi-task perception model.
[0185] The principle and implementation process of multi-task model training in this embodiment are explained in the above embodiments and will not be repeated here.
[0186] In addition, such as Figure 6 As shown in the figure, this embodiment of the invention also proposes a multi-task detection device, the device comprising:
[0187] The perception data acquisition module is used to acquire multi-task perception data;
[0188] The processing module is used to input the multi-task perception data into a pre-trained multi-task perception model for processing. The multi-task perception model is obtained by joint training of task attribute parameters and task priority. The task attribute parameters include at least task uncertainty or task loss convergence.
[0189] The output module is used to output multi-task perception information through the multi-task perception model.
[0190] The principle and implementation process of multi-task detection in this embodiment are explained in the above embodiments and will not be repeated here.
[0191] This invention also proposes a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the above-described functionality. Figure 2 , Figure 3 The steps of multi-task model training in the illustrated embodiment, or as described above. Figure 4 The steps of the multi-task detection method in the illustrated embodiment.
[0192] This invention also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described functionality. Figure 2 , Figure 3 The steps of multi-task model training in the illustrated embodiment, or as described above. Figure 4 The steps of the multi-task detection method in the illustrated embodiment.
[0193] Since this multi-task model training program or multi-task detection program adopts all the technical solutions of all the foregoing embodiments when executed by the processor, it has at least all the beneficial effects brought about by all the technical solutions of all the foregoing embodiments, which will not be repeated here.
[0194] This invention proposes a multi-task model training method, detection method, apparatus, terminal device, and medium. The method involves acquiring multi-task sample images; extracting features from the multi-task sample images to obtain feature maps; calculating the loss value for each task based on the feature maps; and jointly training the multi-task model based on the loss values of each task, combined with task uncertainty and task priority, to obtain a trained multi-task perception model. Therefore, by improving the joint training process and combining task uncertainty (determined by the attributes of the task itself) and priority (determined by actual business needs), the detection accuracy of the multi-task joint model for multiple perception tasks is improved.
[0195] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or approach that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or approach. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or approach that includes that element.
[0196] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0197] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the methods of each embodiment of the present invention.
[0198] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. A multi-task model training method, characterized in that, The method includes: Obtain multi-task sample images; Feature extraction is performed on the multi-task sample images to obtain feature maps; Based on the feature map, the loss value for each task is calculated; Based on the loss values of each task, the multi-task model is jointly trained by combining task attribute parameters and task priority to obtain a trained multi-task perception model. The task attribute parameters include either task uncertainty or task loss convergence. The steps for jointly training the multi-task model based on the loss values of each task, combined with task attribute parameters and task priorities, to obtain the trained multi-task perception model include: The loss values of each task are input into the loss function integrator based on task attribute parameters and the loss function integrator based on task priority, respectively, to obtain the loss function based on task attribute parameters and the loss function based on task priority. By weighting the loss function based on task attribute parameters and the loss function based on task priority using a loss adapter, multi-task learning training is performed to obtain the loss function value, which is then backpropagated to obtain a multi-task perception model.
2. The multi-task model training method according to claim 1, characterized in that, The task attribute parameter-based loss function integrator includes an uncertainty-based loss function integrator, and the task attribute parameter-based loss function includes an uncertainty-based loss function. The step of inputting the loss values of each task into the task attribute parameter-based loss function integrator and the task priority-based loss function integrator to obtain the task attribute parameter-based loss function and the task priority-based loss function includes: The loss values of each task are input into the uncertainty-based loss function integrator and the task priority-based loss function integrator, respectively, to obtain the uncertainty-based loss function and the task priority-based loss function. The uncertainty-based loss function is determined based on the loss value of each task and the corresponding uncertainty, and the task priority-based loss function is determined based on the loss value of each task and the corresponding weight. The steps of weighting the loss function based on task attribute parameters and the loss function based on task priority using a loss adapter, performing multi-task learning training, obtaining the loss function value, and performing backpropagation to obtain the multi-task perception model include: By weighting the uncertainty-based loss function and the task-priority-based loss function with a loss adapter, multi-task learning training is performed to obtain the loss function value and then backpropagation is performed to obtain the multi-task perception model.
3. The multi-task model training method according to claim 2, characterized in that, The steps of weighting the uncertainty-based loss function and the task-priority-based loss function with a loss adapter, performing multi-task learning training to obtain the loss function value, and then performing backpropagation to obtain the multi-task perception model include: Learning weights are assigned to the uncertainty-based loss function and the task-priority-based loss function using a loss adapter; Based on the assigned learning weights, the loss adapter weights the uncertainty-based loss function and the task-priority-based loss function to perform multi-task learning training, obtain the loss function value, and perform backpropagation to obtain a multi-task perception model.
4. The multi-task model training method according to claim 3, characterized in that, The step of assigning learning weights to the uncertainty-based loss function and the task-priority-based loss function through the loss adapter includes: The loss adapter employs a preset linear or incremental function and assigns learning weights to the uncertainty-based loss function and the task-priority-based loss function based on the number of training steps.
5. The multi-task model training method according to claim 1, characterized in that, The task attribute parameter-based loss function integrator includes a task loss convergence-based loss function integrator, and the task attribute parameter-based loss function includes a task loss convergence-based loss function. The step of inputting the loss values of each task into the task attribute parameter-based loss function integrator and the task priority-based loss function integrator respectively to obtain the task attribute parameter-based loss function and the task priority-based loss function includes: The loss values of each task are input into the loss function integrator based on task loss convergence and the loss function integrator based on task priority, respectively, to obtain the loss function based on task loss convergence and the loss function based on task priority. The loss function based on task loss convergence is determined based on the loss value of each task and the corresponding loss convergence, and the loss function based on task priority is determined based on the loss value of each task and the corresponding weight. The steps of weighting the loss function based on task attribute parameters and the loss function based on task priority using a loss adapter, performing multi-task learning training, obtaining the loss function value, and performing backpropagation to obtain the multi-task perception model include: By weighting the loss function based on task loss convergence and the loss function based on task priority using a loss adapter, multi-task learning training is performed to obtain the loss function value, which is then backpropagated to obtain a multi-task perception model.
6. The multi-task model training method according to claim 5, characterized in that, The steps of weighting the loss function based on task loss convergence and the loss function based on task priority using a loss adapter, performing multi-task learning training, obtaining loss function values, and performing backpropagation to obtain a multi-task awareness model include: Learning weights are assigned to the task-based loss function and the task-priority-based loss function using a loss adapter. Based on the assigned learning weights, the loss function based on task loss convergence and the loss function based on task priority are weighted by the loss adapter to perform multi-task learning training, obtain the loss function value, and perform backpropagation to obtain a multi-task perception model.
7. The multi-task model training method according to claim 6, characterized in that, The step of assigning learning weights to the loss function based on task loss convergence and the loss function based on task priority through the loss adapter includes: The loss adapter employs a preset linear or incremental function and assigns learning weights to the task-based loss function and the task-priority-based loss function based on the number of training steps.
8. The multi-task model training method according to any one of claims 1-7, characterized in that, The step of extracting features from the multi-task sample images to obtain feature maps includes: The multi-task sample images are input into the backbone network of the multi-task perception model for feature extraction to obtain feature maps.
9. The multi-task model training method according to any one of claims 1-7, characterized in that, The step of calculating the loss value for each task based on the feature map includes: The feature map is input into the detection head of each task for detection to obtain the detection value of each task. The loss value for each task is calculated based on the detection values of each task.
10. The multi-task model training method according to claim 8, characterized in that, Following the step of acquiring multi-task sample images, the following also includes: The multi-task sample image is processed to obtain a multi-task sample image of the first preset size.
11. The multi-task model training method according to claim 10, characterized in that, The step of inputting the multi-task sample images into the backbone network of the multi-task perception model for feature extraction to obtain feature maps includes: The first preset-size multi-task sample image is input into the backbone network of the multi-task perception model for feature extraction to obtain a feature map of the second preset size.
12. A multi-task detection method, characterized in that, The method includes: Acquire multi-task awareness data; The multi-task perception data is input into a pre-trained multi-task perception model for processing. The multi-task perception model is obtained by joint training of task attribute parameters and task priority. The task attribute parameters include at least task uncertainty or task loss convergence. The multi-task perception model outputs multi-task perception information. The multi-task awareness model is obtained through joint training by combining task attribute parameters and task priorities, and the steps include: The loss values of each task are input into the loss function integrator based on task attribute parameters and the loss function integrator based on task priority, respectively, to obtain the loss function based on task attribute parameters and the loss function based on task priority. By weighting the loss function based on task attribute parameters and the loss function based on task priority using a loss adapter, multi-task learning training is performed to obtain the loss function value, which is then backpropagated to obtain a multi-task perception model.
13. A multi-task model training device, characterized in that, The device includes: The sample image acquisition module is used to acquire sample images for multiple tasks; The feature extraction module is used to extract features from the multi-task sample images to obtain feature maps; The calculation module is used to calculate the loss value for each task based on the feature map; The training module is used to jointly train the multi-task model based on the loss values of each task, combined with task attribute parameters and task priority, to obtain a trained multi-task perception model. The task attribute parameters include either task uncertainty or task loss convergence. The training module is also used for: The loss values of each task are input into the loss function integrator based on task attribute parameters and the loss function integrator based on task priority, respectively, to obtain the loss function based on task attribute parameters and the loss function based on task priority. By weighting the loss function based on task attribute parameters and the loss function based on task priority using a loss adapter, multi-task learning training is performed to obtain the loss function value, which is then backpropagated to obtain a multi-task perception model.
14. A multi-task detection device, characterized in that, The device includes: The perception data acquisition module is used to acquire multi-task perception data; The processing module is used to input the multi-task perception data into a pre-trained multi-task perception model for processing. The multi-task perception model is obtained by joint training of task attribute parameters and task priority. The task attribute parameters include at least task uncertainty or task loss convergence. The output module is used to output multi-task perception information through the multi-task perception model; The processing module is further configured to: The loss values of each task are input into the loss function integrator based on task attribute parameters and the loss function integrator based on task priority, respectively, to obtain the loss function based on task attribute parameters and the loss function based on task priority. By weighting the loss function based on task attribute parameters and the loss function based on task priority using a loss adapter, multi-task learning training is performed to obtain the loss function value, which is then backpropagated to obtain a multi-task perception model.
15. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of multi-task model training as described in any one of claims 1-11, or the steps of the multi-task detection method as described in claim 12.
16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of multi-task model training as described in any one of claims 1-11, or the steps of the multi-task detection method as described in claim 12.