A training method and device for a multi-task model

By updating the multi-task model with parameters of the single-task model, the problem of iteration time-consuming in the training process of multi-task model is solved, and a more efficient training process is achieved.

CN114821538BActive Publication Date: 2024-11-19BEIJING HORIZON ROBOTICS TECH RES & DEV CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202210563203.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-19
Publication Date
2024-11-19
Estimated Expiration
2042-05-19

AI Technical Summary

Technical Problem

In the existing perception algorithm, the iteration of multi-task models during the training process takes a long time and slow training speed, which affects the training efficiency, especially when the model parameters change, it needs to be retrained, resulting in an increase in the number of iterations.

Method used

By using the pre-established single-task model to train the training data set, the parameters of each single-task model are obtained, and the initial multi-task model is updated based on these parameters, reducing the number of iterations and improving training efficiency.

Benefits of technology

It effectively reduces the number of iterations in the multi-task model training process, improves training speed, improves training efficiency, and avoids the need for retraining caused by training exceptions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114821538B_ABST
    Figure CN114821538B_ABST
Patent Text Reader

Abstract

Disclosed are a method and device for training a multi-task model, which are used to train an initial multi-task model. The initial multi-task model may include at least one first abnormal sub-task layer. The method provided by the present disclosure can use a training data set to train at least one pre-established first single-task model to obtain first model parameters corresponding to each first single-task model, wherein the first single-task model includes a second abnormal sub-task layer, the second abnormal sub-task layer is obtained based on the first abnormal sub-task layer, and the initial multi-task model is updated based on the first model parameters to obtain a trained multi-task model, so as to realize the training of the initial multi-task model. This solves the problem in the prior art that when training a multi-task model, if an abnormality occurs during the training process, the training data needs to be reused to train the multi-task model, thereby reducing iteration time and improving training efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of intelligent driving technology, and in particular to a training method and device for a multi-task model. Background Art

[0002] With the technological breakthroughs in the field of artificial intelligence, advanced driver assistance systems and unmanned driving technologies have developed rapidly. As an important part of advanced driver assistance systems and unmanned driving technologies, perception algorithms are a prerequisite for vehicles to achieve safe driving. Vehicles can obtain information about the vehicle's current environment through on-board sensors such as on-board cameras. The on-board processing system then processes and refines the environmental information through perception algorithms, providing the vehicle control system with the ability to make corresponding decisions.

[0003] Most existing perception algorithms rely on multi-task models to implement calculations. However, during the training process of the multi-task model, once a model parameter changes, the multi-task model needs to be retrained, which will result in long iteration time and slow training speed, affecting training efficiency. Summary of the invention

[0004] The embodiments of the present disclosure provide a method and device for training a multi-task model to solve the problem in the prior art that the training of the multi-task model is time-consuming and slow in training speed, which affects the training efficiency. Specifically, the embodiments of the present disclosure provide the following technical solutions:

[0005] According to a first aspect of the present disclosure, a multi-task model training method is provided, comprising:

[0006] Get the training dataset;

[0007] Using the training data set to train at least one pre-established first single-task model, to obtain first model parameters corresponding to each of the first single-task models, wherein each of the first single-task models includes a second abnormal subtask layer, and each of the second abnormal subtask layers is obtained based on each of the first abnormal subtask layers;

[0008] The initial multi-task model is updated based on the first model parameters to obtain a trained multi-task model.

[0009] According to a second aspect of the present disclosure, a multi-task model training device is provided, which is used to train an initial multi-task model, wherein the initial multi-task model includes at least one first abnormal subtask layer, and the device includes:

[0010] Training data acquisition module: used to obtain training data sets;

[0011] A training module, which trains at least one pre-established first single-task model using the training data set obtained by the training data acquisition module to obtain first model parameters corresponding to each of the first single-task models, wherein each of the first single-task models includes a second abnormal subtask layer, and each of the second abnormal subtask layers is obtained based on each of the first abnormal subtask layers;

[0012] A parameter updating module is used to update the initial multi-task model according to the first model parameters obtained by the training module to obtain a trained multi-task model.

[0013] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the above-mentioned multi-task model training method.

[0014] According to a fourth aspect of the present disclosure, an electronic device is provided, the electronic device comprising

[0015] processor;

[0016] a memory for storing instructions executable by the processor;

[0017] The processor is used to read the executable instructions from the memory and execute the instructions to implement the above-mentioned multi-task model training method.

[0018] The training method of the multi-task model disclosed in the present invention is used to train the initial multi-task model. The initial multi-task model may include at least one first abnormal sub-task layer. The method provided by the present invention can use the training data set to train at least one pre-established first single-task model to obtain the first model parameters corresponding to each first single-task model, wherein the first single-task model includes a second abnormal sub-task layer, the second abnormal sub-task layer is obtained based on the first abnormal sub-task layer, and the initial multi-task model is updated based on the first model parameters to obtain the trained multi-task model, so as to realize the training of the initial multi-task model. This solves the problem in the prior art that when training a multi-task model, if an abnormality occurs during the training process, the training data needs to be reused to train the multi-task model, thereby reducing iteration time and improving training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0020] Figure 1 Schematic diagram of the structure of an initial multi-task model provided by an exemplary embodiment of the present disclosure.

[0021] Figure 2 It is a structural diagram of a multi-task model training system provided by an exemplary embodiment of the present disclosure.

[0022] Figure 3 It is a flowchart of a training method for a multi-task model provided by an exemplary embodiment of the present disclosure.

[0023] Figure 4 It is a flowchart of updating an initial multi-task model provided by an exemplary embodiment of the present disclosure.

[0024] Figure 5 It is a flowchart of training a single-task model provided by an exemplary embodiment of the present disclosure.

[0025] Figure 6 It is a flowchart of obtaining target parameters provided by an exemplary embodiment of the present disclosure.

[0026] Figure 7 It is a flowchart of obtaining target shared layer parameters provided by an exemplary embodiment of the present disclosure.

[0027] Figure 8 It is a flowchart of updating an initial multi-task model provided by an exemplary embodiment of the present disclosure.

[0028] Fig. 9 It is a flowchart of a training method for a multi-task model provided by an exemplary embodiment of the present disclosure.

[0029] Fig.10 It is a flowchart of a training method for a multi-task model provided by an exemplary embodiment of the present disclosure.

[0030] Fig.11 It is a structural diagram of a multi-task model training device provided by an exemplary embodiment of the present disclosure.

[0031] Fig.12 It is a structural diagram of a parameter updating module provided by an exemplary embodiment of the present disclosure.

[0032] Fig.13 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0034] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0035] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0036] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0037] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0038] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0039] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0040] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0041] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0042] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0043] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0044] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0045] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0046] Application Overview

[0047] In application scenarios such as assisted driving or autonomous driving, the vehicle needs to be able to perceive the surrounding environment. In order to meet this requirement, it is necessary to collect videos including the vehicle's surrounding environment, and use perception algorithms to calculate the feature data corresponding to each video frame in the video frame by frame to obtain the environmental information contained in each video frame (such as traffic light information, lane line information, obstacle location information, etc.). Most existing perception algorithms rely on multi-task models to implement calculations. If the environmental information contained in the video frame is regarded as multiple subtasks, the multi-task model can process these subtasks simultaneously. Compared with using multiple single-task models to process these subtasks separately, the multi-task model undoubtedly reduces the amount of calculation and resource usage.

[0048] The multi-task model can be obtained by training the initial multi-task model using a training data set, wherein the training data set may include training data annotated with feature labels corresponding to each piece of environmental information. When the initial multi-task model is trained using the training data set, each training data in the training data set needs to be used as the input of the initial multi-task model, and the feature label corresponding to each training data needs to be used as the expected output of the initial multi-task model, and the loss function needs to be calculated to update the model parameters of the multi-task model according to the loss function.

[0049] However, when the initial multi-task model is trained in the above manner, if an abnormality occurs during the training process, for example, the training result cannot reach the preset convergence condition due to an abnormal setting of the learning rate of the initial multi-task model, then it is necessary to adjust the learning rate of the multi-task model and retrain the initial multi-task model using the training data until the convergence condition is reached. Only then can the training be terminated to obtain a multi-task model for simultaneously processing each subtask. This results in long iteration time and slow training speed, affecting the training efficiency.

[0050] Exemplary Systems

[0051] Figure 1 is a schematic diagram of the structure of an initial multi-task model provided by an exemplary embodiment of the present disclosure. Figure 1 As shown, it includes an input module 110 for obtaining data to be processed, a calculation module 120 for calculating the data to be processed, and an output module 130 for outputting the calculation result of the calculation module 120. Among them, the data to be processed can be training data or task data of the task to be performed. When the initial multi-task model is trained, the input module 110 is used to obtain training samples. When the initial multi-task model is used for prediction, the input module 110 is used to obtain the task data of the task to be performed. The present disclosure does not limit the format of the data to be processed input to the input module 110, and it can be flexibly selected according to the application field or task type of the initial multi-task model. For example, for image recognition tasks, the data to be processed can be image data, and for natural language understanding tasks, the data to be processed can be text data.

[0052] The calculation module 120 includes a first calculation unit 121 and a second calculation unit 122. Specifically, the first calculation unit 121 may be a neural network with a multi-branch structure for processing the unique features of each multi-task, and the second calculation unit 122 may be a neural network with a single-branch structure for processing the common features of each multi-task. The initial multi-task model is trained using the training data to obtain a multi-task model that can efficiently and accurately process each sub-task.

[0053] The output module 130 may perform code conversion on the calculation result output by the calculation module 120 , so as to convert the calculation result input to the output module 130 from unreadable digital data into readable audio / video / text data.

[0054] Figure 2 Schematic diagram of the structure of a multi-task model training system provided by an exemplary embodiment of the present disclosure. Figure 2As shown, the system includes a data storage device 210, a model training platform 220 and a parameter storage device 230. The data storage device 210 may be a data storage server, which may be used to store training data sets, and the storage capacity of the data storage server is much larger than the storage capacity of the computing server in the model training platform 220. The training data set may be a data set including training data such as language data, image data and video data, and the training data set may be composed of multiple sub-training data sets, and each sub-training data set includes multiple training data of the same type and / or the true value (feature label) corresponding to each training data.

[0055] The model training platform 220 includes a computing server for iterative calculation, which can also be called a node, which can be a common computer, a mobile terminal, a workstation or a general server, a dedicated server, etc., and a switch for data communication between computing servers. The computing server has a local storage, and its capacity is smaller than that of the data storage server. When training the model, each computing server reads certain training data from the data storage server to a local storage device by sampling, so as to train the initial multi-task model according to the read training data. Specifically, the model training platform 220 is used to obtain the task category of each subtask that can be processed by the initial multi-task model, and match the corresponding single-task model based on the task category of each subtask. For example, in application scenarios such as assisted driving or automatic driving, the task category of each subtask can include a signal light recognition task, an obstacle recognition task, and a lane line recognition task, etc. The single-task model corresponding to the task category of each subtask can be a single-task model for processing a signal light recognition task, a single-task model for processing an obstacle recognition task, and a single-task model for processing a lane line recognition task. The model training platform 220 can also use the training data to train each single-task model separately, obtain the model parameters corresponding to each single-task model after training, and sort the model parameters according to the preset numerical relationship to obtain the target parameters.

[0056] The parameter storage device 230 is used to store the target parameters obtained through training. When the model training platform 220 completes training, the final target parameters can be input into the parameter storage device 230 for storage, so as to update the model parameters corresponding to the computing module 120 of the initial multi-task model according to the target parameters, thereby realizing the training of the initial multi-task model.

[0057] Exemplary Methods

[0058] Figure 3 is a flowchart of a training method for a multi-task model provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 3 As shown, the method comprises the following steps:

[0059] S301: Obtain a training data set.

[0060] The training data set may include a plurality of training data corresponding to each subtask in the multi-task processable by the initial multi-task model and a true value corresponding to each training data. The training data may be image features corresponding to each sample image obtained by encoding and converting each sample image collected in advance, and the training data may also be each sample image collected in advance.

[0061] In application scenarios such as assisted driving or autonomous driving, the initial multi-task model can simultaneously process subtasks corresponding to six task categories, namely, traffic light recognition task, pedestrian detection task, obstacle detection task, lane line detection task, vehicle distance detection task and sign recognition task. In order to ensure the training accuracy of the initial multi-task model and enable the trained multi-task model to efficiently and accurately process the above six subtasks in parallel, it is necessary to obtain the training data corresponding to the above six subtasks and the true value corresponding to each training data from the training data set, wherein the number of collected training data corresponding to each subtask may be different.

[0062] Each training data corresponds to at least one true value. For example, the true value corresponding to the training data corresponding to the traffic light recognition task can be the lighting condition of the traffic light (red light, yellow light, green light, red light and yellow light at the same time, or green light and yellow light at the same time). Each condition is a true value. Therefore, the types of true values ​​corresponding to the training data corresponding to the traffic light recognition task can be 6. The true value corresponding to the training data corresponding to the lane line detection task can be the type of lane line in the training data, the distance from the lane line to the driving vehicle, etc. The true value corresponding to the training data corresponding to the obstacle detection task can be the type of obstacle in the training data, the distance from the obstacle to the driving vehicle, the area value of the obstacle, etc. The present disclosure does not specifically limit the types and quantities of the true values ​​corresponding to each training data.

[0063] S302: Using the training data set to train at least one pre-established first single-task model, to obtain first model parameters corresponding to each first single-task model.

[0064] In order to improve the task processing accuracy and efficiency of the initial multi-task model, it is necessary to iterate the initial multi-task model to update the parameters of the initial multi-task model and obtain the trained multi-task model. The initial multi-task model includes at least one first abnormal sub-task layer, each first abnormal sub-task layer is used to learn or process a sub-task in the multi-tasks respectively. For example, when the initial multi-task model is trained, each first abnormal sub-task layer is used to learn sub-tasks of different task categories respectively. When the initial multi-task model is used for prediction, each first abnormal sub-task layer is used to process sub-tasks of different task categories respectively.

[0065] In some embodiments, the pre-collected test image may be processed using an initial multi-task model, and the first abnormal sub-task layer may be determined based on the processing result of the initial multi-task model. For example, the test image includes traffic light image features (red light on), pedestrian image features (pedestrians exist), obstacle image features (obstacles exist), lane line image features (zebra crossing exists), vehicle distance image features (vehicles are 5 meters ahead), and sign image features (left turn prohibited). By using the initial multi-task model to process the pre-collected test image, the prediction features corresponding to each image feature predicted by the initial multi-task model can be obtained. Each prediction feature can include traffic light prediction features (red light on), pedestrian prediction features (no pedestrians exist), obstacle prediction features (no obstacles exist), lane line prediction features (zebra crossing exists), vehicle distance prediction features (vehicles are 2 meters ahead), and sign prediction features (left turn prohibited). It can be seen that the pedestrian prediction features, obstacle prediction features, and vehicle distance prediction features obtained according to the initial multi-task model are different from the pedestrian image features, obstacle image features, and vehicle distance image features corresponding to the test image. Therefore, it can be determined that there are three first abnormal sub-task layers in the initial multi-task model, which are network layers for processing pedestrian detection tasks, obstacle detection tasks, and vehicle distance detection tasks, respectively.

[0066] According to the task categories of the subtasks that can be learned or processed by each first abnormal subtask layer, the first single-task models equal to the number of the first abnormal subtask layers can be pre-established, and each first single-task model is used to learn or process a subtask that can be learned or processed by the first abnormal subtask layer. For example, based on the aforementioned initial multi-task model that can have three first abnormal subtask layers, the pre-established first single-task models can be: a first single-task model for processing a pedestrian detection task, a first single-task model for processing an obstacle detection task, and a first single-task model for processing a vehicle distance detection task.

[0067] In some embodiments, each first single-task model includes a second abnormal subtask layer, and each second abnormal subtask layer is obtained based on each first abnormal subtask layer, wherein the network structure of each second abnormal subtask layer is the same as that of each first abnormal subtask layer. Therefore, the dimension of the model parameter corresponding to each second abnormal subtask layer is the same as the dimension of the model parameter corresponding to each first abnormal subtask layer. For example, the model parameter corresponding to one of the second abnormal subtask layers is The model parameters corresponding to one of the first abnormal subtask layers are Then the dimension of the model parameters corresponding to the second abnormal subtask layer and the dimension of the model parameters corresponding to the first abnormal subtask layer are both 2×2.

[0068] It should be noted that the model parameters corresponding to the second abnormal subtask layer and the model parameters corresponding to the first abnormal subtask layer are merely illustrative examples of the dimensional relationship between the two. The dimensions of the model parameters corresponding to the second abnormal subtask layer and the dimensions of the model parameters corresponding to the first abnormal subtask layer can be in the form of any numerical combination (for example, n×m×r, where n, m and r are all positive integers), as long as the dimensions of the model parameters corresponding to the second abnormal subtask layer and the dimensions of the model parameters corresponding to the first abnormal subtask layer are equal.

[0069] In addition, although the network structure of each second abnormal subtask layer is the same as that of each first abnormal subtask layer, and the dimension of the model parameters corresponding to each second abnormal subtask layer is also the same as the dimension of the model parameters corresponding to each first abnormal subtask layer, the model parameters corresponding to each second abnormal subtask layer may not be equal to the model parameters corresponding to each first abnormal subtask layer.

[0070] In some embodiments, the first single-task model also includes a coding conversion layer. When the first single-task model is trained using a training data set, the coding conversion layer will identify whether the training data in the training data set is coded data. The coded data is used to characterize the image features corresponding to each sample image. The coded data can be recognized by each network layer (for example, the second abnormal sub-task layer) included in the first single-task model. If the coding conversion layer recognizes that the training data is coded data, the training data can directly participate in the training of the first single-task model. If the coding conversion layer recognizes that the training data is not coded data, for example, the training data is pre-collected sample images, the coding conversion layer can convert the training data into coded data, and the training data that has undergone coding conversion can participate in the training of the first single-task model.

[0071] When the first single-task model is trained using the training data set, the training data corresponding to each subtask in the training data set and the true value corresponding to each training data can be used to train each first single-task model (such as using the training data corresponding to the obstacle recognition task and the true value corresponding to the training data to train the first single-task model for processing the obstacle recognition task, etc., which will not be described in detail in this disclosure). When the preset training conditions are met, the training is stopped, and the first model parameters corresponding to each first single-task model are obtained.

[0072] S303: Update the initial multi-task model based on the first model parameters to obtain a trained multi-task model.

[0073] Each time each first single-task model is trained, a corresponding first model parameter is obtained. The model parameters of the initial multi-task model can be updated based on the first model parameters to iterate the initial multi-task model.

[0074] Among them, the number of iterations of the model parameters of the initial multi-task model is the same as the number of trainings of each first single-task model. When the training of each first single-task model reaches the preset training condition, the training is stopped, and the model parameters of the multi-task model are updated for the last time based on the obtained first model parameters to obtain a trained multi-task model.

[0075] The training method of the multi-task model disclosed in the present invention is used to train the initial multi-task model. The initial multi-task model may include at least one first abnormal sub-task layer. The method provided by the present invention can use the training data set to train at least one pre-established first single-task model to obtain the first model parameters corresponding to each first single-task model, wherein the first single-task model includes a second abnormal sub-task layer, the second abnormal sub-task layer is obtained based on the first abnormal sub-task layer, and the initial multi-task model is updated based on the first model parameters to obtain the trained multi-task model, so as to realize the training of the initial multi-task model. This solves the problem in the prior art that when training a multi-task model, if an abnormality occurs during the training process, the training data needs to be reused to train the multi-task model, thereby reducing iteration time and improving training efficiency.

[0076] Figure 4 It is a flowchart of updating an initial multi-task model based on first model parameters provided by an exemplary embodiment of the present disclosure.

[0077] like Figure 4 As shown in the above Figure 3 Based on the embodiment shown, in an exemplary embodiment of the present disclosure, the step of updating the initial multi-task model based on the first model parameter shown in step S303 may specifically include the following steps:

[0078] S401: Obtain second model parameters corresponding to at least one pre-established second single-task model.

[0079] While the initial multi-task model is used to process the pre-collected test image and the first abnormal sub-task layer is determined based on the processing result of the initial multi-task model, the first sub-task layer can also be determined. Taking the method of determining the first abnormal sub-task layer as an example, since the pedestrian prediction features, obstacle prediction features and vehicle distance prediction features obtained by the initial multi-task model are different from the pedestrian image features, obstacle image features and vehicle distance image features corresponding to the test image, it can be determined that there are three first abnormal sub-task layers in the initial multi-task model, namely, the network layers for processing the pedestrian detection task, the obstacle detection task and the vehicle distance detection task respectively. Correspondingly, since the traffic light prediction features, the lane line prediction features and the sign prediction features are consistent with the traffic light image features, the lane line image features and the sign image features corresponding to the test image (or the difference is less than the threshold), it can be determined that there are three first sub-task layers in the initial multi-task model, namely, the network layers for processing the traffic light detection task, the lane line detection task and the sign detection task respectively.

[0080] According to the task categories of the subtasks that can be learned or processed by each first subtask layer, second single-task models equal in number to the first subtask layers can be pre-established, and each second single-task model is used to learn or process a subtask that can be learned or processed by the first subtask layer. For example, based on the aforementioned initial multi-task model with three first subtask layers, the pre-established second single-task models can be: a second single-task model for processing a traffic light detection task, a second single-task model for processing a lane line detection task, and a second single-task model for processing a sign detection task.

[0081] In some embodiments, each second single-task model includes a second subtask layer, and each second subtask layer is obtained based on each first subtask layer. The network structure of each second subtask layer is the same as that of each first subtask layer. Therefore, the dimension of the model parameters corresponding to each second subtask layer is the same as that of the model parameters corresponding to each first subtask layer. For example, the model parameters corresponding to one of the second subtask layers are The model parameters corresponding to one of the first subtask layers are Then the dimensions of the model parameters corresponding to the second subtask layer and the dimensions of the model parameters corresponding to the first subtask layer are both 2×2.

[0082] It should be noted that the model parameters corresponding to the second subtask layer and the model parameters corresponding to the first subtask layer are merely illustrative examples of the dimensional relationship between the two. The dimensions of the model parameters corresponding to the second subtask layer and the dimensions of the model parameters corresponding to the first subtask layer can be in the form of any numerical combination (for example, n×m×r, where n, m and r are all positive integers), as long as the dimensions of the model parameters corresponding to the second subtask layer and the dimensions of the model parameters corresponding to the first subtask layer are equal.

[0083] In addition, although the network structure of each second subtask layer is the same as that of each first subtask layer, and the dimensions of the model parameters corresponding to each second subtask layer are also the same as the dimensions of the model parameters corresponding to each first subtask layer, the model parameters corresponding to each second subtask layer may not be equal to the model parameters corresponding to each first subtask layer.

[0084] In some embodiments, the second single-task model may also include a transcoding layer. The structure and purpose of the transcoding layer of the second single-task model may refer to the above description of the transcoding layer of the first single-task model, which will not be elaborated here.

[0085] S402: Obtain target parameters based on the first model parameters and the second model parameters.

[0086] Among them, the first model parameters refer to the parameters describing the first single task model composed of multiple constraint parameters, and the second model parameters refer to the parameters describing the second single task model composed of multiple constraint parameters. Based on the fusion of each first model parameter and each second model parameter with the preset numerical relationship, the target parameter can be determined.

[0087] S403: Update the initial multi-task model based on the target parameters.

[0088] The target parameters are used to update each network layer in the initial multi-task model to indirectly realize the training of the initial multi-task model and obtain a trained multi-task model. The trained multi-task model can realize parallel processing of multiple tasks efficiently and accurately.

[0089] like Figure 5 As shown in the above Figure 4 Based on the embodiment shown, in an exemplary embodiment of the present disclosure, the step of training at least one pre-established first single-task model using a training data set shown in step S302 may specifically include the following steps:

[0090] S501: Based on the first shared layer included in the initial multi-task model, determine the second abnormal shared layer included in each first single-task model.

[0091] Among them, the initial multi-task model includes a first shared layer, and the first shared layer is used to simultaneously learn or process each subtask in the multi-task according to the training data corresponding to each subtask. The training data used to train the first shared layer is the training data that appears repeatedly in the training data set. Taking the subtasks being the obstacle detection task and the pedestrian detection task as an example, the training data set may include a sub-training data set corresponding to the obstacle detection task and a sub-training data set corresponding to the pedestrian detection task. The sub-training data set corresponding to the obstacle detection task may contain training data on the position information of pedestrian A, and the sub-training data set corresponding to the pedestrian detection task may also contain training data on the position information of pedestrian A. Then the training data on the position information of pedestrian A is the training data that appears repeatedly in the training data set. For another example, the sub-training data set corresponding to the obstacle detection task may contain training data on the area occupied by pedestrian A, and the sub-training data set corresponding to the pedestrian detection task may contain training data on the area occupied by pedestrian A. Then the training data on the area occupied by pedestrian A is the training data that appears repeatedly in the training data set, and so on.

[0092] Each first single-task model includes a second abnormal shared layer, each second abnormal shared layer is obtained based on the first shared layer, wherein the network structure of each second abnormal shared layer is the same as that of the first shared layer, therefore, the dimension of the model parameters corresponding to each second abnormal shared layer is the same as that of the model parameters corresponding to the first shared layer, for example, the model parameters corresponding to one of the second abnormal shared layers are The model parameters corresponding to the first shared layer are Then the dimension of the model parameters corresponding to the second abnormal shared layer and the dimension of the model parameters corresponding to the first shared layer are both 2×2.

[0093] It should be noted that the model parameters corresponding to the second abnormal shared layer and the model parameters corresponding to the first shared layer are merely illustrative examples of the dimensional relationship between the two. The dimensions of the model parameters corresponding to the second abnormal shared layer and the dimensions of the model parameters corresponding to the first shared layer can be in the form of any numerical combination (for example, n×m×r, where n, m and r are all positive integers), as long as the dimensions of the model parameters corresponding to the second abnormal shared layer and the dimensions of the model parameters corresponding to the first shared layer are equal.

[0094] In addition, although the network structure of each second abnormal shared layer is the same as that of the first shared layer, and the dimensions of the model parameters corresponding to each second abnormal shared layer are also the same as the dimensions of the model parameters corresponding to each first shared layer, the model parameters corresponding to each second abnormal shared layer may not be equal to the model parameters corresponding to each first shared layer.

[0095] Each second abnormal sharing layer is used to learn or process each subtask in the multi-task according to the training data corresponding to each subtask. For example, the training data set includes training data about the position information of pedestrian A and training data about the area occupied by pedestrian A. When the first single-task model for processing the obstacle detection task and the first single-task model for processing the pedestrian detection task are trained using the training data set, the training data about the position information of pedestrian A and the training data about the area occupied by pedestrian A are both used to train the second abnormal sharing layer of the first single-task model for processing the obstacle detection task and the second abnormal sharing layer of the first single-task model for processing the pedestrian detection task. However, the model parameters corresponding to the second abnormal sharing layer of the first single-task model for processing the obstacle detection task and the second abnormal sharing layer of the first single-task model for processing the pedestrian detection task obtained after training may not be equal.

[0096] In some embodiments, each first single-task model corresponds to its own first model parameter, and the first model parameters include first subtask layer parameters and first shared layer parameters, wherein the model parameters corresponding to the above-mentioned second abnormal subtask layer can be determined as the first subtask layer parameters, and the model parameters corresponding to the above-mentioned second abnormal shared layer can be determined as the first shared layer parameters.

[0097] S502: Train each first single-task model using the training data set to obtain first subtask layer parameters corresponding to each second abnormal subtask layer, and obtain first shared layer parameters corresponding to each second abnormal shared layer.

[0098] In some embodiments, before using the training data set to train each first single-task model, each training data in the training data set can be classified and corresponding category labels can be added based on the task category corresponding to each first single-task model. Based on each category label, multiple sub-training data sets are generated, and each sub-training data set includes training data with the same category label added. According to the category label of the training data in each sub-training data set, a corresponding category label is added to each sub-training data set, and each sub-training data set with a category label is used to train the first single-task model of the corresponding task category.

[0099] Each first single-task model can be trained using each sub-training data set with added category labels to obtain the first sub-task layer parameters corresponding to the second abnormal sub-task layer and the first shared layer parameters corresponding to the second abnormal shared layer in each first single-task model. Wherein, when training the first single-task model, the first single-task model will sequentially generate corresponding prediction values ​​based on the training data in the input sub-training data set, and calculate the loss value between the prediction value and the true value of the training data based on a specific loss function, so as to update the first sub-task layer parameters and the first shared layer parameters corresponding to the first single-task model according to the loss value, until the preset training conditions are met, and the trained first single-task models are obtained, and the first sub-task layer parameters and the first shared layer parameters corresponding to each first single-task model are obtained, wherein the specific loss function can be an absolute value loss function, a square loss function, an intersection-over-union loss function, etc., which is not limited by the present disclosure.

[0100] In some embodiments, there are association relationships and non-association relationships between the task categories. Based on the association relationships between the task categories, the training data in each sub-training data set is classified to obtain an associated data set and a non-association data set.

[0101] Among them, there is at least one completely identical training data in each sub-training data set corresponding to the task category with an associated relationship, and the training data in the sub-training data sets corresponding to the task category with an unassociated relationship are all different. The training data that appear repeatedly in each sub-training data set can be determined as associated data, and the training data that appears only once in each sub-training data set can be determined as unassociated data.

[0102] For example, the sub-training data set corresponding to the pedestrian detection task may include training data on the position information of pedestrian A and pedestrian B, and the sub-training data set corresponding to the obstacle detection task may include training data on the position information of pedestrian A and vehicle A. Then, there is an association relationship between the pedestrian detection task and the obstacle detection task. The training data on the position information of pedestrian A is the associated data corresponding to the pedestrian detection task and the obstacle detection task, and the training data on the position information of pedestrian B and the training data on the position information of vehicle A are the non-associated data corresponding to the pedestrian detection task and the obstacle detection task. For another example, the training data on the position information of vehicle A is the non-associated data corresponding to the pedestrian detection task and the obstacle detection task. The sub-training data set corresponding to the lane line detection task may include training data on the lane line information on both sides of the vehicle and the lane line information ahead and behind the vehicle. The sub-training data set corresponding to the vehicle distance detection task may include training data on the lane line information on both sides of the vehicle and the distance between the vehicle and other vehicles. Then, there is a correlation between the lane line detection task and the vehicle distance detection task. The training data on the lane line information on both sides of the vehicle is the correlated data corresponding to the lane line detection task and the vehicle distance detection task, while the training data on the lane line information ahead and behind the vehicle and the training data on the distance between the vehicle and other vehicles are the non-correlated data corresponding to the lane line detection task and the vehicle distance detection task.

[0103] Based on the associated data set, each second abnormal shared layer can be trained to obtain the first shared layer parameters. Based on the non-associated data set, each second abnormal sub-task layer can be trained to obtain the first sub-task layer parameters.

[0104] Among them, when the second abnormal shared layer of each first single-task model is trained based on each associated data set, the second abnormal shared layer will generate corresponding prediction values ​​based on the training data in the associated data set in turn, and calculate the loss value between the prediction value and the true value of the training data based on a specific loss function, so as to update the first shared layer parameters corresponding to the first single-task model according to the loss value until the preset training conditions are met, and the trained first single-task models are obtained, and the first shared layer parameters corresponding to each first single-task model are obtained.

[0105] Similarly, when the second abnormal subtask layer of each first single task model is trained based on each non-correlated data set, the second abnormal subtask layer will generate corresponding prediction values ​​based on the training data in the non-correlated data set in turn, and calculate the loss value between the prediction value and the true value of the training data based on a specific loss function, so as to update the first subtask layer parameters corresponding to the first single task model according to the loss value until the preset training conditions are met, thereby obtaining the trained first single task models, and obtaining the first subtask layer parameters corresponding to each first single task model.

[0106] like Figure 6 As shown in the above Figure 5On the basis of the embodiment shown, in an exemplary embodiment of the present disclosure, the target parameters include target subtask layer parameters and target shared layer parameters. The step of obtaining the target parameters based on the first model parameters and the second model parameters shown in step S402 may specifically include the following steps:

[0107] S601: Based on the first shared layer, determine the second shared layer corresponding to each second single-task model.

[0108] Among them, the network structure of each second shared layer is the same as that of the first shared layer. Therefore, the dimension of the model parameters corresponding to each second shared layer is the same as the dimension of the model parameters corresponding to the first shared layer. The relationship between each second shared layer and the first shared layer can refer to the explanation of the relationship between the above-mentioned second abnormal shared layer and the first shared layer, which will not be repeated here.

[0109] In some embodiments, each second single-task model corresponds to its own second model parameters, and the second model parameters include second subtask layer parameters and second shared layer parameters, wherein the model parameters corresponding to the above-mentioned second subtask layer can be determined as the second subtask layer parameters, and the model parameters corresponding to the above-mentioned second shared layer can be determined as the second shared layer parameters.

[0110] S602: Obtain target shared layer parameters based on the fusion of each first shared layer parameter and each second shared layer parameter.

[0111] Exemplarily, the number of first single-task models is 3, and the first shared layer parameters of each first single-task model can be N1, N2, and N3, respectively; the number of second single-task models is 3, and the second shared layer parameters of each second single-task model can be N4, N5, and N6, respectively. Based on the fusion of N1, N2, N3, N4, N5, and N6 with the preset numerical relationship, the target shared layer parameters can be obtained.

[0112] S603: Obtain target subtask layer parameters based on a set of first subtask layer parameters.

[0113] Exemplarily, if the number of first single-task models is 3, the first subtask layer parameters of each first single-task model are M1, M2, and M3 respectively, and based on the set of each first subtask layer parameter, the target subtask layer parameters can be obtained as {M1, M2, M3}.

[0114] like Figure 7 As shown in the above Figure 6 On the basis of the embodiment shown, in an exemplary embodiment of the present disclosure, the step of obtaining the target shared layer parameters based on the fusion of each first shared layer parameter and each second shared layer parameter shown in step S602 may specifically include the following steps:

[0115] S701: In response to the first shared layer parameters and the second shared layer parameters being the same, determining any one of the first shared layer parameters and the second shared layer parameters as a target shared layer parameter.

[0116] Exemplarily, if the first shared layer parameters and the second shared layer parameters are the same, that is, N1=N2=N3=N4=N5=N6, any shared layer parameter can be extracted, for example, N5 can be extracted, and N5 is determined as the target shared layer parameter.

[0117] S702: In response to the first shared layer parameters and the second shared layer parameters being different, an average value of the first shared layer parameters and the second shared layer parameters is determined as a target shared layer parameter.

[0118] Exemplarily, if the first shared layer parameters and the second shared layer parameters are different, the average value of the first shared layer parameters and the second shared layer parameters is calculated, that is, And the average value is determined as the target shared layer parameter.

[0119] S703: In response to the first shared layer parameters and the second shared layer parameters being different, determine a median value of the first shared layer parameters and the second shared layer parameters as a target shared layer parameter.

[0120] In some embodiments, if the first shared layer parameters and the second shared layer parameters are different, the first shared layer parameters and the second shared layer parameters are sorted in order from small to large to generate a shared layer parameter set, and it is detected whether the number of items in the shared layer parameter set is an even number. If it is an even number, the average of the two middle items in the shared layer parameter set is determined as the target shared layer parameter. For example, the shared layer parameter set is {N4, N2, N6, N1, N5, N3}, where N4>N2>N6>N1>N5>N3. Since the number of items in the shared layer parameter set is 6, which is an even number, the two middle items N6 and N1 in the shared layer parameter set are obtained, and the average of N6 and N1 is calculated. Determine as the target shared layer parameter; if the number of items in the shared layer parameter set is an odd number, obtain the median in the shared layer parameter set, and determine the median in the shared layer parameter set as the target shared layer parameter. For example, the shared layer parameter set is {N4, N2, N6, N1, N5, N3, N7}, where N4>N2>N6>N1>N5>N3>N7. Since the number of items in the shared layer parameter set is 7, which is an odd number, obtain the median N1 in the shared layer parameter set, and determine the median N1 as the target shared layer parameter.

[0121] like Figure 8 As shown in the above Figure 7Based on the embodiment shown, in an exemplary embodiment of the present disclosure, the step of updating the initial multi-task model based on the target parameters shown in step S403 may specifically include the following steps:

[0122] S801: Based on the target subtask layer parameters, update each first abnormal subtask layer.

[0123] In order to save training time and improve training efficiency, only the first abnormal subtask layers in the initial multi-task model can be updated based on the target subtask layer parameters. Exemplarily, the target subtask layer parameters can be {M1, M2, M3}. The initial multi-task model includes 3 first abnormal subtask layers. These 3 first abnormal subtask layers can be updated based on M1, M2, and M3 respectively. After the update, if the first abnormal subtask layer can perform normal learning and processing based on the input data, the first abnormal subtask layer will become the first subtask layer after the update.

[0124] S802: In response to an abnormality in the first shared layer, update the first shared layer based on the target shared layer parameters.

[0125] Exemplarily, the target shared layer parameters may be It can be based on The first shared layer of the initial multi-task model is updated, and the first shared layer with abnormal conditions can perform normal learning and processing based on the input data after being updated.

[0126] The training method of the multi-task model disclosed in the present invention can test the initial multi-task model in advance through a test image, confirm the network layer with abnormal conditions in the multi-task model, namely, the first abnormal sub-task layer and the first shared layer, and obtain the first model parameters (first sub-task layer parameters and first shared layer parameters) corresponding to the first single-task model by training the first single-task model which is preset with the same task category as that processed by each first abnormal sub-task layer. At the same time, obtain the network layer without abnormal conditions in the multi-task model, namely, the first sub-task layer, and obtain the second model parameters (second sub-task layer parameters and second shared layer parameters) corresponding to the second single-task model which is preset with the same task category as that processed by each first sub-task layer, update each first abnormal sub-task layer using the first sub-task layer parameters, and update the first shared layer with abnormal conditions based on the target shared layer parameters obtained by fusing the first shared layer parameters and the second shared layer parameters, so as to obtain the trained multi-task model. This method of only updating the network layers with abnormal situations in the initial multi-task model can shorten the training time. In addition, if an abnormal training situation occurs during the training of a first single-task model, only the first single-task model with the abnormal training situation needs to be retrained, without the need to retrain every first single-task model, which can reduce iteration time and improve training efficiency.

[0127] In some embodiments, if the initial multi-task model is not tested in advance using a test image, the present disclosure also provides a method for training a multi-task model, such as Fig. 9 As shown, the following steps are included:

[0128] S901: Based on the task category of each subtask in the multi-task, a single-task model corresponding to the task category of each subtask is obtained.

[0129] Among them, the acquired single-task models can be the first single-task model or the second single-task model. For example, the acquired single-task models can all be first single-task models, or the acquired single-task models can all be second single-task models, or a part of the acquired single-task models can be the first single-task model and the other part can be the second single-task model. The present disclosure does not make specific restrictions, and the type of each single-task model may not be acquired in advance through testing.

[0130] S902: Use the training data set to train each single-task model respectively, and obtain model parameters corresponding to each trained single-task model respectively.

[0131] In some embodiments, there is no need to pre-test the type of each single-task model, and each single-task model can be indiscriminately trained using the training data set, that is, the first single-task model and the second single-task model are indiscriminately trained to obtain the first model parameters corresponding to each first single-task model and the second model parameters corresponding to each second single-task model after training.

[0132] Among them, the first model parameters may include first shared layer parameters corresponding to the second abnormal shared layer of the first single task model and first subtask layer parameters corresponding to the second abnormal subtask layer; the second model parameters may include second shared layer parameters corresponding to the second shared layer of the second single task model and second subtask layer parameters corresponding to the second subtask layer.

[0133] S903: Obtain target parameters based on the model parameters corresponding to each single-task model.

[0134] In some embodiments, the target parameters include target shared layer parameters and target subtask layer parameters. The target subtask layer parameters can be obtained based on a set of first subtask layer parameters and second subtask layer parameters; the target shared layer parameters can be obtained based on a fusion of first shared layer parameters, second shared layer parameters and preset numerical relationships.

[0135] Among them, when the first shared layer parameters and the second shared layer parameters are the same, any one of the first shared layer parameters and the second shared layer parameters can be determined as the target shared layer parameter.

[0136] Alternatively, when the first shared layer parameters and the second shared layer parameters are different, the average value of the first shared layer parameters and the second shared layer parameters can be determined as the target shared layer parameter.

[0137] Alternatively, when the first shared layer parameters and the second shared layer parameters are different, the median value of the first shared layer parameters and the second shared layer parameters may be determined as the target shared layer parameter.

[0138] S904: Update the initial multi-task model based on the target parameters.

[0139] Based on the target shared layer parameters and target subtask layer parameters obtained in step S903, each network layer in the initial multi-task model is updated, which can indirectly realize the training of the initial multi-task model. The trained multi-task model can efficiently and accurately realize parallel processing of multiple tasks.

[0140] The training method of the multi-task model provided in this embodiment obtains single-task models corresponding to the task categories of each subtask in the multi-tasks that can be processed by the initial multi-task model, and uses the training data set to indiscriminately train each single-task model to obtain model parameters corresponding to each single-task model, and obtains target parameters based on the model parameters corresponding to each single-task model, so as to update the initial multi-task model through the target parameters to obtain the trained multi-task model. The method of not testing the initial multi-task model with test images in advance can make the training process simpler and save training steps. In addition, indiscriminately training each single-task model (the first single-task model and the second single-task model) can improve the training accuracy.

[0141] In some embodiments, if the initial multi-task model is not tested in advance using a test image, the present disclosure also provides a method for training a multi-task model, such as Fig.10 As shown, the following steps are included

[0142] S101: Based on the task category of each subtask in the multi-task, a single-task model corresponding to the task category of each subtask is obtained.

[0143] Among them, the acquired single-task models can be the first single-task model or the second single-task model. For example, the acquired single-task models can all be the first single-task models, or the acquired single-task models can all be the second single-task models, or a part of the acquired single-task models can be the first single-task model and the other part can be the second single-task model. The present disclosure does not make any specific restrictions.

[0144] S102: Test each single-task model using a test image to obtain a first single-task model and a second single-task model.

[0145] Among them, each single-task model is used to process the test image respectively, and based on the processing results of each single-task model, it is determined whether each single-task model has an abnormal network layer, the single-task model with the abnormal network layer is determined as the first single-task model, and the single-task model of the network layer without abnormal conditions is determined as the second single-task model.

[0146] S103: Train each first single-task model using the training data set to obtain first model parameters corresponding to each first single-task model, and obtain second model parameters corresponding to each second single-task model.

[0147] Among them, the first single-task model includes a second abnormal subtask layer and a second abnormal shared layer, the first model parameters include first subtask layer parameters and first shared layer parameters, and each first single-task model is trained using a training data set to obtain the first subtask layer parameters corresponding to the trained first abnormal subtask layer, and the first shared layer parameters corresponding to the trained second abnormal shared layer can be obtained.

[0148] S104: Obtain target parameters based on the model parameters corresponding to each single-task model.

[0149] In some embodiments, the target parameters include target shared layer parameters and target subtask layer parameters. The target subtask layer parameters can be obtained based on a set of first subtask layer parameters and second subtask layer parameters; the target shared layer parameters can be obtained based on a fusion of first shared layer parameters, second shared layer parameters and preset numerical relationships.

[0150] Among them, when the first shared layer parameters and the second shared layer parameters are the same, any one of the first shared layer parameters and the second shared layer parameters can be determined as the target shared layer parameter.

[0151] Alternatively, when the first shared layer parameters and the second shared layer parameters are different, the average value of the first shared layer parameters and the second shared layer parameters can be determined as the target shared layer parameter.

[0152] Alternatively, when the first shared layer parameters and the second shared layer parameters are different, the median value of the first shared layer parameters and the second shared layer parameters may be determined as the target shared layer parameter.

[0153] S105: Update the initial multi-task model based on the target parameters.

[0154] Based on the target shared layer parameters and target subtask layer parameters obtained in step S104, each network layer in the initial multi-task model is updated, which can indirectly realize the training of the initial multi-task model and obtain a trained multi-task model. The trained multi-task model can efficiently and accurately realize parallel processing of multiple tasks.

[0155] The training method of the multi-task model provided in this embodiment obtains a single-task model corresponding to the task category of each sub-task in the multi-task, tests each single-task model with a test image, obtains a first single-task model and a second single-task model, trains each first single-task model with a training data set, obtains first model parameters corresponding to each first single-task model, and obtains second model parameters corresponding to each second single-task model, and obtains target parameters based on the model parameters corresponding to each single-task model, so as to update the initial multi-task model according to the target parameters. By adopting this method of not using a test image to test the initial multi-task model in advance, but using a test image to test whether each single-task model has an abnormality, the training time for training the single-task model can be shortened, the training amount can be reduced, and the training efficiency can be improved.

[0156] Exemplary Devices

[0157] See also Fig.11 , is a schematic diagram of the structure of a multi-task model training device provided in an embodiment of the present disclosure, and the device is used to implement all or part of the functions of the aforementioned method embodiment. Specifically, the multi-task model training device includes a training data acquisition module 810, a training module 820, and a parameter update module 830. In addition, the device may also include an image processing module, a receiving module, and a sending module, etc., which are not limited in this embodiment.

[0158] Specifically, the training data acquisition module 810 in the embodiment of the present disclosure is used to acquire a training data set;

[0159] The training module 820 in the embodiment of the present disclosure trains at least one pre-established first single-task model using the training data set acquired by the training data acquisition module to obtain first model parameters corresponding to each first single-task model.

[0160] Wherein, each first single-task model includes a second abnormal subtask layer, and each second abnormal subtask layer is obtained based on each first abnormal subtask layer.

[0161] The parameter updating module 830 in the embodiment of the present disclosure is used to update the initial multi-task model according to the first model parameters obtained by the training module to obtain a trained multi-task model.

[0162] Optionally, in another implementation of this embodiment, the initial multi-task model further includes a plurality of first sub-task layers, and the parameter updating module 830 further includes a parameter acquiring unit 910, a parameter analyzing unit 920 and a parameter updating unit 930, such as Fig.12 shown.

[0163] The parameter acquisition unit 910 in the embodiment of the present disclosure is used to obtain the second model parameters corresponding to at least one pre-established second single-task model. Each second single-task model includes a second subtask layer, and each second subtask layer is obtained based on each first subtask layer. The number analysis unit 920 is used to obtain the target parameter based on each first model parameter and each second model parameter. The parameter updating unit 930 is used to update the initial multi-task model based on the target parameter.

[0164] Optionally, in another implementation of this embodiment, obtaining a second model parameter corresponding to at least one pre-established second single-task model further includes:

[0165] The at least one pre-established second single-task model is trained using the training data set to obtain second model parameters corresponding to each second single-task model.

[0166] Optionally, in another implementation of this embodiment, the initial multi-task model further includes a first shared layer, each first single-task model further includes a second abnormal shared layer, each second abnormal shared layer is obtained based on the first shared layer, and the first model parameters include first subtask layer parameters and first shared layer parameters;

[0167] Using the training data set to train at least one pre-established first single-task model, obtaining first model parameters corresponding to each first single-task model includes:

[0168] The first single-task models are trained using the training data set to obtain first subtask layer parameters corresponding to each second abnormal subtask layer, and first shared layer parameters corresponding to each second abnormal shared layer.

[0169] Optionally, in another implementation of this embodiment, the second single-task model further includes a second shared layer, the second shared layer is obtained based on the first shared layer, the second model parameters include second subtask layer parameters and second shared layer parameters, and the target parameters include target subtask layer parameters and target shared layer parameters; obtaining the second model parameters corresponding to at least one second single-task model includes:

[0170] Obtain the second shared layer parameters corresponding to each second shared layer respectively; based on each of the first model parameters and each of the second model parameters, obtain the target parameters, including: obtaining the target shared layer parameters based on the fusion of each of the first shared layer parameters and each of the second shared layer parameters; obtaining the target subtask layer parameters based on the set of each of the first subtask layer parameters.

[0171] Optionally, in another implementation of this embodiment, obtaining the target shared layer parameters based on the fusion of each first shared layer parameter and each second shared layer parameter includes:

[0172] Based on the fusion of each first shared layer parameter and each second shared layer parameter with a preset numerical relationship, the target shared layer parameter is determined.

[0173] Optionally, in another implementation of this embodiment, the step of determining the target shared layer parameter based on fusing each first shared layer parameter and each second shared layer parameter with a preset numerical relationship includes:

[0174] In response to the first shared layer parameters and the second shared layer parameters being the same, any one of the first shared layer parameters and the second shared layer parameters is determined as the target shared layer parameter; or, in response to the first shared layer parameters and the second shared layer parameters being different, the average value of the first shared layer parameters and the second shared layer parameters is determined as the target shared layer parameter, or the median value of the first shared layer parameters and the second shared layer parameters is determined as the target shared layer parameter.

[0175] Optionally, in another implementation of this embodiment, updating the initial multi-task model based on the target parameter includes:

[0176] Based on the target subtask layer parameters, each first abnormal subtask layer is updated; in response to the abnormality of the first shared layer, the first shared layer is updated based on the target shared layer parameters.

[0177] In addition, in the embodiment of the device, the functions of the various modules shown in FIG. 11 are the same as those in the above-mentioned Figure 3 The method embodiment shown corresponds to, for example, the acquisition module 810 is used to execute the aforementioned method step 301, the training module 820 is used to execute the aforementioned method step 302, the parameter updating module 830 is used to execute the aforementioned method step 303, etc.

[0178] Exemplary Electronic Devices

[0179] Below, reference Fig.13 The electronic device 10 may be one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.

[0180] Fig.13 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0181] like Fig.13 As shown, the electronic device 10 includes one or more processors 11 and a memory 12 .

[0182] The processor 11 may be a central processing unit 910 (CPU) or other forms of processing units 910 having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0183] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may run the program instructions to implement the steps in the multi-task model training method of each embodiment of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0184] In one example, the electronic device 10 may further include: an input device 13 and an output device 14, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0185] For example, when the electronic device is the first device or the second device, the input device 13 may be the above-mentioned vehicle-mounted camera 200, which is used to capture input signals of the environment image. When the electronic device is a stand-alone device, the input device 13 may be a communication network connector, which is used to receive the collected input signals from the first device and the second device.

[0186] In addition, the input device 13 may also include, for example, a keyboard, a mouse, and the like.

[0187] The output device 14 can output various information to the outside, including determined distance information, direction information, etc. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0188] Of course, to simplify, Fig.13Only some of the components related to the present disclosure in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 10 may also include any other appropriate components.

[0189] Exemplary computer program products and computer-readable storage media

[0190] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the multi-task model training method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0191] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0192] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, cause the processor to execute the steps of the multi-task model training method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0193] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0194] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limiting. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0195] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0196] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0197] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0198] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A training method for a multi-task model, comprising an initial multi-task model, wherein the initial multi-task model comprises at least one first abnormal subtask layer, a plurality of first subtask layers and a first shared layer, the method comprising: Get the training dataset; Using the training data set to train at least one pre-established first single-task model, first model parameters corresponding to each of the first single-task models are obtained, wherein each of the first single-task models includes a second abnormal subtask layer and a second abnormal shared layer, each of the second abnormal subtask layers is obtained based on each of the first abnormal subtask layers, and each of the second abnormal shared layers is obtained based on the first shared layer; the first model parameters include the first subtask layer parameters corresponding to the second abnormal subtask layer and the first shared layer parameters corresponding to the second abnormal shared layer; The initial multi-task model is updated based on the first model parameters to obtain a trained multi-task model.

2. The method according to claim 1, wherein: The updating of the initial multi-task model based on the first model parameters comprises: Acquire a second model parameter corresponding to at least one pre-established second single-task model, wherein each of the second single-task models includes a second subtask layer, and each of the second subtask layers is obtained based on each of the first subtask layers; Obtaining a target parameter based on each of the first model parameters and each of the second model parameters; The initial multi-task model is updated based on the target parameters.

3. The method according to claim 2, wherein: Acquiring a second model parameter corresponding to at least one pre-established second single-task model further includes: The training data set is used to train at least one pre-established second single-task model to obtain second model parameters corresponding to each of the second single-task models.

4. The method according to claim 2, wherein: The second single-task model further includes a second shared layer, the second shared layer is obtained based on the first shared layer, the second model parameters include second subtask layer parameters and second shared layer parameters, and the target parameters include target subtask layer parameters and target shared layer parameters; The obtaining of second model parameters respectively corresponding to at least one second single-task model comprises: Obtain second shared layer parameters respectively corresponding to each of the second shared layers; The obtaining of the target parameter based on each of the first model parameters and each of the second model parameters comprises: Obtaining the target shared layer parameter based on the fusion of each of the first shared layer parameters and each of the second shared layer parameters; Based on a set of the first subtask layer parameters, the target subtask layer parameters are obtained.

5. The method according to claim 4, wherein: The obtaining the target shared layer parameter based on the fusion of each of the first shared layer parameters and each of the second shared layer parameters includes: The target shared layer parameter is determined based on the fusion of each of the first shared layer parameters and each of the second shared layer parameters with a preset numerical relationship.

6. The method according to claim 5, wherein: The determining the target shared layer parameter based on fusing each of the first shared layer parameters and each of the second shared layer parameters with a preset numerical relationship includes: In response to the first shared layer parameters and the second shared layer parameters being the same, determining any one of the first shared layer parameters and the second shared layer parameters as the target shared layer parameter; or, In response to the fact that the first shared layer parameters and the second shared layer parameters are different, the average value of the first shared layer parameters and the second shared layer parameters is determined as the target shared layer parameter, or the median value of the first shared layer parameters and the second shared layer parameters is determined as the target shared layer parameter.

7. The method according to claim 6, wherein: The updating of the initial multi-task model based on the target parameter comprises: Based on the target subtask layer parameter, updating each of the first abnormal subtask layers; In response to an abnormality in the first shared layer, the first shared layer is updated based on the target shared layer parameters.

8. A multi-task model training device, used for training an initial multi-task model, wherein the initial multi-task model comprises at least one first abnormal subtask layer, a plurality of first subtask layers and a first shared layer, the device comprising: Training data acquisition module: used to obtain training data sets; A training module, using the training data set obtained by the training data acquisition module to train at least one pre-established first single-task model, to obtain first model parameters corresponding to each of the first single-task models, wherein each of the first single-task models includes a second abnormal subtask layer and a second abnormal shared layer, each of the second abnormal subtask layers is obtained based on each of the first abnormal subtask layers, and each of the second abnormal shared layers is obtained based on the first shared layer; the first model parameters include the first subtask layer parameters corresponding to the second abnormal subtask layer and the first shared layer parameters corresponding to the second abnormal shared layer; A parameter updating module is used to update the initial multi-task model according to the first model parameters obtained by the training module to obtain a trained multi-task model.

9. A computer-readable storage medium storing a computer program for executing the multi-task model training method according to any one of claims 1 to 7.

10. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the multi-task model training method described in any one of claims 1-7 above.

Citation Information

Patent Citations

  • Multi-task learning convolutional neural network-based face attribute analysis method

    CN106529402A

  • Gradient normalization systems and methods for adaptive loss balancing in deep multitask networks

    CN111373419A

  • Object classification method and device, readable medium and electronic equipment

    CN113435528A

  • Model training method and device, storage medium and electronic equipment

    CN113516239A

  • Picture attribute prediction method and device, electronic equipment and storage medium

    CN114170484A