A training method and device for a multi-task model

Through the hierarchical training method, a multi-task model training data set of category labels, feature labels and decoding labels is used to form a classification network, feature extraction network and decoding network, which solves the problem of low training efficiency of multi-task model and realizes scalability and Improved efficiency.

CN114943291BActive Publication Date: 2024-11-26BEIJING HORIZON ROBOTICS TECH RES & DEV CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210583424.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-25
Publication Date
2024-11-26
Estimated Expiration
2042-05-25

AI Technical Summary

Technical Problem

The training efficiency of existing multitasking models is inefficient, resulting in the problem of increasing training time when improving scalability.

Method used

Through a hierarchical training method, a training data set including category tags, feature tags and decoding tags is used to train the multi-task model for classification networks, feature extraction networks and decoding networks to form a multi-task model for classification networks, feature extraction networks and decoding networks. Task model.

Benefits of technology

While improving the scalability of the multi-task model, it significantly improves the training efficiency of the multi-task model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114943291B_ABST
    Figure CN114943291B_ABST
Patent Text Reader

Abstract

The present disclosure provides a training method and device for a multi-task model. The multi-task model to be trained is trained based on a first training data set including category labels. The multi-task model trained using the first training data set is trained based on a second training data set including feature labels. The multi-task model trained using the second training data set is trained based on a third training data set including decoding labels to obtain a multi-task model including a classification network, a decoding network and a feature extraction network, wherein the feature extraction network is a subnetwork of the classification network, and the decoding network is a subnetwork of the feature extraction network. The training method provided by the present disclosure improves the scalability of the multi-task model while improving the training efficiency of the multi-task model by performing hierarchical training on the multi-task model to be trained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of intelligent driving technology, and in particular to a training method and device for a multi-task model. Background Art

[0002] With the technological breakthroughs in the field of artificial intelligence, advanced driver assistance systems and unmanned driving technologies have developed rapidly. As an important part of advanced driver assistance systems and unmanned driving technologies, perception algorithms are a prerequisite for vehicles to achieve safe driving. Vehicles can obtain information about the vehicle's current environment through on-board sensors such as on-board cameras. The on-board processing system then processes and refines the environmental information through perception algorithms, providing the vehicle control system with the ability to make corresponding decisions.

[0003] Most perception algorithms rely on multi-task models to implement calculations. The more subtasks a multi-task model can process in parallel, the better the scalability of the multi-task model. However, the better the scalability of the multi-task model, the longer it takes to train the multi-task model. Therefore, how to improve the scalability of the multi-task model while improving the training efficiency of the multi-task model is an urgent problem that needs to be solved. Summary of the invention

[0004] The embodiments of the present disclosure provide a method and device for training a multi-task model to solve the problem of low training efficiency of the multi-task model in the prior art. Specifically, the embodiments of the present disclosure provide the following technical solutions:

[0005] According to a first aspect of the present disclosure, a multi-task model training method is provided, comprising:

[0006] Based on a first training data set including category labels, training the multi-task model to be trained to obtain a multi-task model including a classification network, wherein the classification network is used to classify the preset tasks;

[0007] Based on a second training data set including feature labels, the multi-task model trained using the first training data set is trained to obtain a multi-task model including a feature extraction network, wherein the feature extraction network is used to identify feature labels of the training data in a preset task category, and the feature extraction network is a subnetwork of the classification network;

[0008] Based on a third training data set including decoding labels, the multi-task model trained using the second training data set is trained to obtain a multi-task model including a decoding network, wherein the decoding network is used to decode feature labels corresponding to the training data, and the decoding network is a subnetwork of the feature extraction network.

[0009] According to a second aspect of the present disclosure, a multi-task model training device is provided, comprising:

[0010] A first training module: based on a first training data set including category labels, training the multi-task model to be trained to obtain a multi-task model including a classification network, wherein the classification network is used to classify the preset tasks;

[0011] A second training module: based on a second training data set including feature labels, the multi-task model trained by the first training module is trained to obtain a multi-task model including a feature extraction network, wherein the feature extraction network is used to identify feature labels of training data in a preset task category, and the feature extraction network is a subnetwork of the classification network;

[0012] The third training module: Based on a third training data set including decoding labels, the multi-task model trained by the second training module is trained to obtain a multi-task model including a decoding network, wherein the decoding network is used to decode feature labels corresponding to the training data, and the decoding network is a subnetwork of the feature extraction network.

[0013] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the above-mentioned multi-task model training method.

[0014] According to a fourth aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0015] processor;

[0016] a memory for storing instructions executable by the processor;

[0017] The processor is used to read the executable instructions from the memory and execute the instructions to implement the above-mentioned multi-task model training method.

[0018] The training method of the multi-task model disclosed in the present invention trains the multi-task model to be trained based on a first training data set including category labels, trains the multi-task model trained using the first training data set based on a second training data set including feature labels, and trains the multi-task model trained using the second training data set based on a third training data set including decoding labels, so as to obtain a multi-task model including a classification network, a feature extraction network and a decoding network, wherein the feature extraction network is a subnetwork of the classification network, and the decoding network is a subnetwork of the feature extraction network. The training method provided in the present invention performs hierarchical training on the multi-task model to be trained, so as to improve the scalability of the multi-task model while improving the training efficiency of the multi-task model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0020] Figure 1 Schematic diagram of the structure of a multi-task model to be trained provided by an exemplary embodiment of the present disclosure.

[0021] Figure 2 It is a structural diagram of a multi-task model training system provided by an exemplary embodiment of the present disclosure.

[0022] Figure 3 It is a flowchart of a training method for a multi-task model provided by an exemplary embodiment of the present disclosure.

[0023] Figure 4 It is a flowchart of a training method for a multi-task model provided by an exemplary embodiment of the present disclosure.

[0024] Figure 5 It is a flowchart of a training method for a multi-task model provided by an exemplary embodiment of the present disclosure.

[0025] Figure 6 It is a flowchart of a training method for a multi-task model provided by an exemplary embodiment of the present disclosure.

[0026] Figure 7 It is a structural diagram of a multi-task model training device provided by an exemplary embodiment of the present disclosure.

[0027] Figure 8 It is a structural diagram of a first training module provided by an exemplary embodiment of the present disclosure.

[0028] Fig. 9 It is a structural diagram of a second training module provided by an exemplary embodiment of the present disclosure.

[0029] Fig.10 It is a structural diagram of a third training module provided by an exemplary embodiment of the present disclosure.

[0030] Fig.11 It is an information interaction diagram of a multi-task model training device provided by an exemplary embodiment of the present disclosure.

[0031] Fig.12 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0033] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0034] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0035] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0036] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0037] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0038] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0039] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0040] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0041] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0042] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0043] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0044] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0045] Application Overview

[0046] In application scenarios such as assisted driving or autonomous driving, the vehicle needs to be able to perceive the surrounding environment. In order to meet this requirement, it is necessary to collect videos including the vehicle's surrounding environment, and use perception algorithms to calculate the feature data corresponding to each video frame in the video frame by frame to obtain the environmental information contained in each video frame (such as traffic light information, lane line information, obstacle location information, etc.). Most existing perception algorithms rely on multi-task models to implement calculations. If the environmental information contained in the video frame is regarded as multiple subtasks, the multi-task model can process these subtasks simultaneously. Compared with using multiple single-task models to process these subtasks separately, the multi-task model undoubtedly reduces the amount of calculation and resource usage.

[0047] The number of subtasks that a multi-task model can process in parallel is an important indicator for measuring the scalability of the multi-task model. The more subtasks a multi-task model can process in parallel, the better the scalability of the multi-task model. However, the better the scalability of the multi-task model, the longer it takes to train the multi-task model. Therefore, how to improve the training efficiency of the multi-task model while improving the scalability of the multi-task model is an urgent problem to be solved.

[0048] Exemplary Systems

[0049] Figure 1is a schematic diagram of the structure of a multi-task model to be trained provided by an exemplary embodiment of the present disclosure. Figure 1 As shown, it includes a data acquisition module 110 for acquiring data to be processed and a calculation module 120 for calculating the data to be processed. The data to be processed may be training data or training data of the task to be performed. When the multi-task model to be trained is trained, the data acquisition module 110 is used to acquire training samples. When the multi-task model to be trained is used for prediction, the data acquisition module 110 is used to acquire the training data of the task to be performed. The present disclosure does not limit the format of the data to be processed acquired by the data acquisition module 110, and it can be flexibly selected according to the application field or task type of the multi-task model to be trained. For example, for an image recognition task, the data to be processed may be image data, and for a natural language understanding task, the data to be processed may be text data.

[0050] The calculation module 120 includes a first calculation unit 121, a second calculation unit 122, and a third calculation unit 123. Specifically, the first calculation unit 121 can be a neural network with a multi-branch structure, and each branch of the first calculation unit 121 is connected to the second calculation unit 122. The second calculation unit 122 can also be a neural network with a multi-branch structure, and each branch of the second calculation unit 122 is connected to the third calculation unit 123. The third calculation unit 123 can be a neural network with a single branch structure. The first calculation unit 121, the second calculation unit 122, and the third calculation unit 123 are all used to calculate the data to be processed to obtain different types of features of the data to be processed.

[0051] Figure 2 Schematic diagram of the structure of a multi-task model training system provided by an exemplary embodiment of the present disclosure. Figure 2 As shown, the system includes a data storage device 210, a model training platform 220 and a parameter storage device 230. The data storage device 210 may be a data storage server, which may be used to store training data sets, and the storage capacity of the data storage server is much larger than the storage capacity of the computing server in the model training platform 220. The training data set may be a data set including training data such as language data, image data and video data, and the training data set may be composed of multiple sub-training data sets, and each sub-training data set includes multiple training data of the same type and / or labels (true values) corresponding to each training data.

[0052] The model training platform 220 includes a computing server for iterative calculation, which can also be called a node, which can be a common computer, a mobile terminal, a workstation or a general server, a dedicated server, etc., and a switch for data communication between computing servers. The computing server has a local storage, and its capacity is smaller than that of the data storage server. When training the model, each computing server reads certain training data from the data storage server to a local storage device by sampling, so as to train the multi-task model to be trained according to the read training data. Specifically, the model training platform 220 is used to train the multi-task model to be trained using the training data read from the data storage server, so as to iterate the model parameters of the multi-task model to be trained, thereby obtaining a multi-task model including a classification network, a decoding network and a feature extraction network.

[0053] The parameter storage device 230 is used to store the model parameters of the multi-task model trained by the model training platform 220 for subsequent calls.

[0054] Exemplary Methods

[0055] Figure 3 is a flowchart of a training method for a multi-task model provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 3 As shown, the method comprises the following steps:

[0056] S301: Based on a first training data set including category labels, a multi-task model to be trained is trained to obtain a multi-task model including a classification network.

[0057] Multi-tasks are tasks that the trained multi-task model is expected to learn or calculate in parallel. The first training data set includes at least one category label, and the category label is used to characterize the task category of each subtask in the multi-task. For example, in the application scenarios of assisted driving or autonomous driving, the subtasks in the multi-task can be preset as signal light recognition tasks, pedestrian detection tasks, obstacle detection tasks, vehicle distance detection tasks, sign recognition tasks, etc. The first training data set may include a signal light label for characterizing the signal light task category, a pedestrian position label for characterizing the pedestrian position task category, a pedestrian number label for characterizing the pedestrian number task category, a vehicle position label for characterizing the vehicle position task category, and a sign label for characterizing the sign content task category, etc.

[0058] It should be noted that the above-mentioned traffic light recognition task, pedestrian detection task, obstacle detection task, vehicle distance detection task, and signboard recognition task are only the five subtasks listed as examples in this disclosure. This disclosure does not impose specific restrictions on the types and numbers of subtasks. The category labels included in the above-mentioned first training data set are only the category labels listed as examples in this disclosure. This disclosure does not impose specific restrictions on the number and names of the category labels included in the first training data set.

[0059] The multi-task model to be trained may include a first branch network such as ResNet, MobileNet, etc. for classifying data. Based on the first training data set, the multi-task model to be trained is trained, and the first branch network can be trained as a specific classification network, so that the trained multi-task model can classify the multi-task data to be processed.

[0060] Among them, the task categories after the trained multi-task model classifies the multi-task data to be processed are related to the category labels included in the first training data set for training the multi-task model. For example, if the first training data set includes traffic light labels, pedestrian position labels, pedestrian quantity labels, vehicle position labels and signboard labels, after training the multi-task model using the first training data set, the obtained multi-task model can classify the task data to be processed into any one or several of the traffic light task categories, pedestrian position task categories, pedestrian quantity task categories, vehicle position task categories and signboard content task categories.

[0061] S302: Based on a second training data set including feature labels, the multi-task model trained using the first training data set is trained to obtain a multi-task model including a feature extraction network.

[0062] The second training data set includes at least one feature label, which is used to characterize the feature information of each subtask in the multi-task in the corresponding task category. For example, in the traffic light task category, the feature label can be used to characterize the lighting status of the traffic light (red light, yellow light, green light). In the sign recognition task category, the feature label can be used to characterize the content of the detected sign (no left turn, no honking, slow down, etc.). The present disclosure does not impose specific restrictions on the number and names of the feature labels included in the second training data set.

[0063] The multi-task model to be trained may include a second branch network for extracting feature information from data, the number of the second branch network is the same as the number of branches of the first branch network, and each branch of the first branch network is connected to a second branch network. Based on the second training data set, the multi-task model trained using the first training data set continues to be trained, and the second branch network for extracting feature information from data can be trained as a specific feature extraction network, which is a subnetwork of the classification network, and the feature extraction network is used to identify the feature labels of the task data to be processed in the corresponding task category.

[0064] Among them, the feature information after the trained multi-task model extracts features from the multi-task data to be processed is related to the feature labels included in the second training data set for training the multi-task model. For example, if the second training data set includes feature labels in the traffic light task category (red light label, yellow light label, green light label), feature labels in the sign recognition task category (no left turn label, no honking label, slow down label, etc.), after the multi-task model is trained using the second training data set, the obtained multi-task model will classify the task data to be processed, and will further extract features from the task data of each task category. For example, the multi-task model can extract the feature label (red light label) corresponding to the task data in the traffic light task category, thereby determining the characteristics of the task data (the task data is used to characterize the red light).

[0065] S303: Based on a third training data set including decoding labels, the multi-task model trained using the second training data set is trained to obtain a multi-task model including a decoding network.

[0066] The third training data set is a coding language that can be recognized by the multi-task model, and the third training data set includes at least one decoding label, and the decoding label is used to represent the decoding information corresponding to the feature information of each sub-task in the multi-task. For example, in the traffic light task category, the feature information is the coding information used to represent that the traffic light is in the red light state, which can be represented by the coding language {01, 1}, and the corresponding decoding label can be represented as {red, light}. The present disclosure does not make specific restrictions on the content and form of the decoding label included in the third training data set.

[0067] The multi-task model to be trained may include a single-branch network for decoding data, the number of single-branch networks is the same as the number of branches of the second branch network, and each branch of the second branch network is connected to a single-branch network. Based on the third training data set, the multi-task model trained using the second training data set continues to be trained, and the single-branch network for decoding data can be trained as a specific decoding network, which is a subnetwork of the feature extraction network, and the decoding network is used to decode the feature labels corresponding to the training data.

[0068] Among them, the decoding information obtained after the trained multi-task model decodes the multi-task data to be processed is related to the decoding label included in the third training data set for training the multi-task model. For example, if the third training data set includes the decoding label {red, light} in the traffic light task category, after the multi-task model is trained using the third training data set, the obtained multi-task model will classify the task data to be processed and extract features to obtain feature information in a specific task category, and will further decode the obtained feature information to convert the encoded information {01, 1} used to characterize the traffic light being in the red light state into decoding information {red, light}.

[0069] In this embodiment, the multi-task model to be trained is trained based on a first training data set including category labels, the multi-task model trained using the first training data set is trained based on a second training data set including feature labels, and the multi-task model trained using the second training data set is trained based on a third training data set including decoding labels, to obtain a multi-task model including a classification network, a feature extraction network, and a decoding network, thereby improving the scalability of the multi-task model and the training efficiency of the multi-task model.

[0070] Figure 4 It shows that Figure 3 A schematic diagram of a process of training a multi-task model to be trained in the embodiment shown.

[0071] like Figure 4 As shown in the above Figure 3 Based on the illustrated embodiment, in an exemplary embodiment of the present disclosure, the first training data set further includes first training data, and the training of the multi-task model to be trained shown in step S301 may specifically include the following steps:

[0072] S401: Use the multi-task model to be trained to predict first training data to obtain a prediction task category corresponding to the first training data.

[0073] Among them, the first training data corresponds to at least one category label, and the category label is used to characterize the task category of the first training data. For example, in the application scenario of assisted driving or automatic driving, each subtask in the multi-task can be preset as a signal light recognition task, a pedestrian detection task, an obstacle detection task, a vehicle distance detection task, a sign recognition task, etc. The first training data set may include first training data for characterizing the lighting of the signal light, first training data for characterizing the position of the pedestrian, first training data for characterizing the number of pedestrians, first training data for characterizing the position of the surrounding vehicles, and first training data for characterizing the content of the sign, etc. Each first training data corresponds to at least one category label. For example, the category label corresponding to the first training data for characterizing the lighting of the signal light may be a signal light category label; the category label corresponding to the first training data for characterizing the position of the pedestrian may be a pedestrian category label, an obstacle category label, etc.; the category label corresponding to the first training data for characterizing the position of the surrounding vehicles may be an obstacle category label, a vehicle distance category label; and the category label corresponding to the training data for characterizing the content of the sign may be a sign category label. The category labels corresponding to the above first training data are only the category labels exemplarily listed in the present disclosure, and the present disclosure does not specifically limit the number and type of category labels corresponding to the first training data.

[0074] When the multi-task model to be trained is trained using the first training data set, the multi-task model to be trained will predict the first training data in the first training data set to obtain the prediction task category corresponding to the first training data. For example, the multi-task model to be trained will predict the first training data used to characterize the lighting condition of the traffic light, and the obtained prediction task category may be a traffic light recognition task category, or may be any task category such as a pedestrian detection task category, an obstacle detection task category, a vehicle distance detection task category, and a sign recognition task category.

[0075] S402: Determine a first loss value according to the prediction task category and the category label.

[0076] After the multi-task model to be trained predicts the first training data and obtains the predicted task category corresponding to the first training data, the first loss value between the predicted task category and the category label corresponding to the first training data can be calculated based on a specific loss function. For example, the category label corresponding to the first training data used to characterize the lighting condition of a traffic light is a traffic light category label. If the multi-task model to be trained predicts the first training data used to characterize the lighting condition of a traffic light and obtains the predicted task category as an obstacle detection task category, the first loss value between the traffic light category label and the obstacle detection task category can be calculated based on the specific loss function.

[0077] Among them, the specific loss function can be an absolute value loss function, a square loss function, an intersection-over-union loss function, etc., which is not limited in this disclosure.

[0078] S403: Adjust a first set of parameters corresponding to the classification network according to the first loss value.

[0079] The first set of parameters corresponding to the classification network is adjusted with the goal of minimizing the first loss value, and the first set of parameters corresponding to the classification network is iterated to obtain a trained multi-task model including the classification network.

[0080] In some embodiments, before adjusting the first set of parameters corresponding to the classification network, a first threshold value may be preset, and the first threshold value is a loss value between a category label corresponding to the first training data and a desired prediction task category calculated based on a specific loss function. After obtaining the first loss value, the numerical relationship between the first loss value and the first threshold value may be detected. If it is detected that the first loss value is greater than the preset first threshold value, the training effect of the multi-task model to be trained does not meet expectations, and it is necessary to adjust the first set of parameters corresponding to the classification network based on the first loss value. If it is detected that the first loss value is less than or equal to the preset first threshold value, the training effect of the multi-task model to be trained meets expectations, and a multi-task model including a classification network is obtained.

[0081] Figure 5 Shown as Figure 4 A schematic diagram of a process for training a multi-task model in the embodiment shown.

[0082] like Figure 5 As shown in the above Figure 4 Based on the illustrated embodiment, in an exemplary embodiment of the present disclosure, the second training data set includes second training data, and the training of the multi-task model trained using the first training data set shown in step S302 may specifically include the following steps:

[0083] S501: Use the multi-task model trained with the first training data set to predict the second training data to obtain prediction feature information corresponding to the second training data.

[0084] Among them, the second training data corresponds to at least one feature label, and the feature label is used to characterize the feature information of the second training data in the corresponding task category. For example, in the application scenario of assisted driving or automatic driving, the task category corresponding to the second training data may be a signal light recognition task category, a pedestrian detection task category, an obstacle detection task category, a vehicle distance detection task category, a sign recognition task category, etc. The second training data set may include second training data for characterizing the lighting status of signal lights, second training data for characterizing the position of pedestrians, second training data for characterizing the number of pedestrians, second training data for characterizing the position of surrounding vehicles, and second training data for characterizing the content of signboards, etc. Each Each of the second training data corresponds to at least one feature label. For example, the feature label corresponding to the second training data used to characterize the lighting of the traffic light may be a red light feature label, a red light off feature label, a green light feature label, a yellow light feature label, etc.; the feature label corresponding to the second training data used to characterize the position of the pedestrian may be a pedestrian position coordinate feature label, etc.; the category label corresponding to the first training data used to characterize the position of the surrounding vehicles may be a vehicle position coordinate feature label, a vehicle driving status feature label, etc.; the feature label corresponding to the training data used to characterize the content of the sign may be a no left turn feature label, a no honking feature label, a slow down feature label, a speed limit of 80 km / h feature label, etc. The feature labels corresponding to the above second training data are only exemplary feature labels listed in the present disclosure, and the present disclosure does not specifically limit the number and type of feature labels corresponding to the second training data.

[0085] When the multi-task model trained with the first training data set is trained with the second training data set, the multi-task model predicts the second training data in the second training data set to obtain the predicted feature information corresponding to the second training data. For example, the multi-task model predicts the second training data used to characterize the lighting condition of the traffic light, and the obtained predicted feature information may be any one or more of the red light feature information, the green light feature information, and the yellow light feature information.

[0086] S502: Determine a second loss value according to the predicted feature information and the feature label.

[0087] The multi-task model trained with the first training data set predicts the second training data. After obtaining the predicted feature information corresponding to the second training data, the second loss value between the predicted feature information and the feature label corresponding to the second training data can be calculated based on a specific loss function. For example, the feature label corresponding to the second training data used to characterize the lighting of the traffic light is a red light feature label. If the predicted feature information obtained by the multi-task model from predicting the second training data used to characterize the lighting of the traffic light is red light feature information, the second loss value between the red light feature label and the red light feature information can be calculated based on the specific loss function.

[0088] Among them, the specific loss function can be set with reference to the aforementioned embodiment and will not be described in detail here.

[0089] S503: Adjust a second set of parameters corresponding to the feature extraction network according to the second loss value.

[0090] The second set of parameters corresponding to the feature network is adjusted with the goal of minimizing the second loss value, and the second set of parameters corresponding to the feature network is iterated to obtain a trained multi-task model including a classification network and a feature network.

[0091] In some embodiments, before adjusting the second set of parameters corresponding to the feature network, a second threshold value may be preset, and the second threshold value is a loss value between a feature label corresponding to the second training data and the expected predicted feature information calculated based on a specific loss function. After obtaining the second loss value, the numerical relationship between the second loss value and the second threshold value may be detected. If it is detected that the second loss value is greater than the preset second threshold value, the training effect of training the multi-task model does not meet expectations, and the second set of parameters corresponding to the feature network needs to be adjusted based on the second loss value. If it is detected that the second loss value is less than or equal to the preset second threshold value, the training effect of training the multi-task model meets expectations, and a multi-task model including a classification network and a feature network is obtained.

[0092] Figure 6 Shown as Figure 5 A schematic diagram of a process for training a multi-task model in the embodiment shown.

[0093] like Figure 6 As shown in the above Figure 5 Based on the illustrated embodiment, in an exemplary embodiment of the present disclosure, the third training data set includes third training data, and the training of the multi-task model trained using the second training data set shown in step S303 may specifically include the following steps:

[0094] S601: predicting third training data using the multi-task model trained using the second training data set to obtain predicted decoding information corresponding to the third training data.

[0095] Among them, the third training data corresponds to at least one decoding label, and the decoding label is used to characterize the decoding information corresponding to the feature information of the third training data. For example, in the application scenario of assisted driving or automatic driving, the feature information corresponding to the third training data in the signal light recognition task category can be red light on feature information, green light off feature information, yellow light off feature information, etc. The third training data set may include third training data for characterizing the feature information of the red light on, third training data for characterizing the feature information of the green light off, and third training data for characterizing the feature information of the yellow light off. Each of the third training data corresponds to at least one decoding label. For example, the third training data for characterizing the feature information of the red light on can be The third training data used to characterize the green light off feature information can be represented by {01, 1} in a coding language, and the third training data used to characterize the yellow light off feature information can be represented by {03, 0} in a coding language, wherein 01, 02, and 03 can be used to represent red light, green light, and yellow light, and 0 and 1 can be used to represent the light status. Then, the decoding label corresponding to the third training data used to characterize the red light on feature information can be a red light on decoding label, represented by {red, light}, the decoding label corresponding to the third training data used to characterize the green light off feature information can be a green light off decoding label, represented by {green, light out}, and the decoding label corresponding to the third training data used to characterize the yellow light off feature information can be a yellow light off decoding label, represented by {yellow, light out}. The decoding labels corresponding to the above third training data are only exemplary decoding labels listed in the present disclosure, and the present disclosure does not specifically limit the number and type of decoding labels corresponding to the third training data.

[0096] When the multi-task model trained with the second training data set is trained with the third training data set, the multi-task model predicts the third training data in the third training data set to obtain the predicted decoding information corresponding to the third training data. For example, the multi-task model predicts the third training data {01, 1} used to characterize the red light feature information, and the obtained predicted decoding information can be any one or more of the red light decoding information {red, light}, the red light off decoding information {red, lightout}, the yellow light decoding information {yellow, light}, and the green light off decoding information {green, light out}.

[0097] S602: Determine a third loss value according to the predicted decoding information and the decoding label.

[0098] The multi-task model trained with the second training data set predicts the third training data. After obtaining the predicted decoding information corresponding to the third training data, the third loss value between the predicted decoding information and the decoding label corresponding to the third training data can be calculated based on a specific loss function, wherein the specific loss function can be set with reference to the aforementioned embodiment and will not be elaborated here.

[0099] The third loss value is used to characterize the difference between the predicted decoding information and the decoding label. The training effect of the multi-task model trained on the second training data set can be determined based on the third loss value. The smaller the third loss value, the better the training effect of the multi-task model. For example, the decoding label corresponding to the third training data used to characterize the red light feature information can be a red light decoding label. If the predicted decoding information obtained by the multi-task model from the third training data used to characterize the red light feature information is a red light off decoding information, the third loss value between the red light off decoding label and the red light on decoding information can be calculated based on a specific loss function.

[0100] Among them, the specific loss function can be set with reference to the aforementioned embodiment and will not be described in detail here.

[0101] S603: Adjust a third set of parameters corresponding to the decoding network according to the third loss value.

[0102] Among them, the third set of parameters corresponding to the decoding network is adjusted with the goal of minimizing the third loss value, and the third set of parameters corresponding to the decoding network is iterated to obtain a trained multi-task model including the decoding network.

[0103] In some embodiments, before adjusting the third set of parameters corresponding to the decoding network, a third threshold value may be preset, and the third threshold value is a loss value between a decoding label corresponding to the third training data and the expected predicted feature information calculated based on a specific loss function. After obtaining the third loss value, the numerical relationship between the third loss value and the third threshold value may be detected. If it is detected that the third loss value is greater than the preset third threshold value, the training effect of training the multi-task model does not meet expectations, and the third set of parameters corresponding to the decoding network needs to be adjusted based on the third loss value. If it is detected that the third loss value is less than or equal to the preset third threshold value, the training effect of training the multi-task model meets expectations, and a multi-task model including a classification network, a feature network, and a decoding network is obtained.

[0104] Exemplary Devices

[0105] See also Figure 7, is a schematic diagram of the structure of a multi-task model training device provided in an embodiment of the present disclosure, and the device is used to implement all or part of the functions of the aforementioned method embodiment. Specifically, the multi-task model training device includes a first training module 710, a second training module 720, and a third training module 730. In addition, the device may also include an image processing module, a receiving module, and a sending module, etc., which are not limited in this embodiment.

[0106] Specifically, the first training module 710 in the embodiment of the present disclosure is used to train the multi-task model to be trained based on a first training data set including category labels to obtain a multi-task model including a classification network.

[0107] Among them, the classification network is used to classify the preset tasks.

[0108] The second training module 720 in the disclosed embodiment is used to train the multi-task model trained by the first training module based on a second training data set including feature labels to obtain a multi-task model including a feature extraction network.

[0109] Among them, the feature extraction network is used to identify the feature labels of the training data in the preset task category, and the feature extraction network is a subnetwork of the classification network.

[0110] The third training module 730 in the embodiment of the present disclosure is used to train the multi-task model trained by the second training module based on a third training data set including decoding labels, to obtain a multi-task model including a decoding network.

[0111] Among them, the decoding network is used to decode the feature labels corresponding to the training data, and the decoding network is a subnetwork of the feature extraction network.

[0112] Optional, such as Figure 8 As shown, in another implementation of this embodiment, the first training data set includes first training data and category labels corresponding to the first training data, and the first training module 710 also includes a first prediction unit 711, a first calculation unit 712 and a first adjustment unit 713.

[0113] The first prediction unit 711 is used to predict the first training data using the multi-task model to be trained to obtain the prediction task category corresponding to the first training data. The first calculation unit 712 is used to determine the first loss value according to the category label and the prediction task category predicted by the first prediction unit. The first adjustment unit 713 is used to adjust the first set of parameters corresponding to the classification network according to the first loss value determined by the first calculation unit.

[0114] Optional, such as Fig. 9As shown, in another implementation of this embodiment, the second training data set includes second training data and feature labels corresponding to the second training data, and the second training module 720 also includes a second prediction unit 721, a second calculation unit 722 and a second adjustment unit 723.

[0115] The second prediction unit 721 is used to predict the second training data using the multi-task model trained by the first training module 710 to obtain the predicted feature information corresponding to the second training data. The second calculation unit 722 is used to determine the second loss value according to the feature label and the predicted feature information predicted by the second prediction unit 721. The second adjustment unit 723 is used to adjust the second set of parameters corresponding to the feature extraction network according to the second loss value determined by the second calculation unit 722.

[0116] Optional, such as Fig.10 As shown, in another implementation of this embodiment, the third training data set includes third training data and decoding labels corresponding to the third training data, and the third training module 730 also includes a third prediction unit 731, a third calculation unit 732 and a third adjustment unit 733.

[0117] Among them, the third prediction unit 731 is used to predict the third training data using the multi-task model trained by the second training module 720 to obtain the predicted decoding information corresponding to the third training data. The third calculation unit 732 is used to determine the third loss value according to the decoding label and the predicted decoding information predicted by the third prediction unit 731. The third adjustment unit 733 is used to adjust the third set of parameters corresponding to the decoding network according to the third loss value determined by the third calculation unit 731.

[0118] Optional, such as Fig.11 As shown, in another implementation of this embodiment, the multi-task model training device also includes a control module 740, which is used to control the first adjustment unit 713 to adjust the first group of parameters corresponding to the classification network based on the first loss value when it is determined that the first loss value calculated by the first calculation unit 712 is greater than the preset first threshold; and, when it is determined that the second loss value calculated by the second calculation unit 722 is greater than the preset second threshold, control the second adjustment unit 723 to adjust the second group of parameters corresponding to the feature extraction network based on the second loss value; and, when it is determined that the third loss value calculated by the third calculation unit 732 is greater than the preset third threshold, control the third adjustment unit 733 to adjust the third group of parameters corresponding to the decoding network based on the third loss value.

[0119] In addition, in the embodiment of the present device, Figure 7 The functions of the modules shown are similar to those described above. Figure 3The method embodiment shown corresponds to, for example, the first training module 710 is used to execute the aforementioned method step 301, the second training module 720 is used to execute the aforementioned method step 302, the third training module 730 is used to execute the aforementioned method step 303, and so on.

[0120] Exemplary Electronic Devices

[0121] Below, reference Fig.12 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.

[0122] Fig.12 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.

[0123] like Fig.12 As shown, the electronic device 11 includes one or more processors 111 and a memory 112 .

[0124] The processor 111 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 11 to perform desired functions.

[0125] The memory 112 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 111 may run the program instructions to implement the training method of the multi-task model of each embodiment of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0126] In one example, the electronic device 11 may further include: an input device 113 and an output device 114 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0127] For example, when the electronic device is the first device or the second device, the input device 113 may be the microphone or microphone array described above, for capturing input signals from a sound source. When the electronic device is a stand-alone device, the input device 113 may be a communication network connector, for receiving collected input signals from the first device and the second device.

[0128] In addition, the input device 113 may also include, for example, a keyboard, a mouse, and the like.

[0129] The output device 114 can output various information to the outside, including determined distance information, direction information, etc. The output device 114 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0130] Of course, to simplify, Fig.12 Only some of the components related to the present disclosure in the electronic device 11 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 11 may also include any other appropriate components.

[0131] Exemplary computer program products and computer-readable storage media

[0132] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the training method of the multi-task model according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0133] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0134] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, cause the processor to execute the steps in the training method of a multi-task model according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0135] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0136] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0137] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0138] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0139] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0140] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A multi-task model training method, applied to a multi-task model training system, include: Based on a first training data set including category labels stored in a data storage device in the multi-task model training system, a multi-task model to be trained is trained through a model training platform in the multi-task model training system to obtain a multi-task model including a classification network, wherein the classification network is used to classify preset tasks; Based on a second training data set including feature labels stored in the data storage device, the multi-task model trained using the first training data set is trained through the model training platform to obtain a multi-task model including a feature extraction network, wherein the feature extraction network is used to identify feature labels of the training data in a preset task category, the feature extraction network is a subnetwork of the classification network, and the feature labels are used to characterize feature information of each subtask in the multi-task in the corresponding task category; Based on a third training data set including decoding labels stored in the data storage device, the multi-task model trained using the second training data set is trained through the model training platform to obtain a multi-task model including a decoding network, wherein the decoding network is used to decode feature labels corresponding to the training data, the decoding network is a subnetwork of the feature extraction network, and the decoding labels are used to represent decoding information corresponding to feature information of each subtask in the multi-task; Among them, the trained multi-task model is used to perform parallel processing on each of the preset tasks in the multi-task; the training data in the first training data set, the second training data set and the third training data set all include at least one of language data, image data and video data.

2. The method according to claim 1, in, The first training data set includes first training data and the category label corresponding to the first training data; Based on the first training data set including the category labels, the multi-task model to be trained is trained, including: Using the multi-task model to be trained to predict the first training data, to obtain a prediction task category corresponding to the first training data; Determining a first loss value according to the prediction task category and the category label; According to the first loss value, a first set of parameters corresponding to the classification network is adjusted.

3. The method according to claim 2, in, The second training data set includes second training data and feature labels corresponding to the second training data; Training the multi-task model trained using the first training dataset based on a second training dataset including feature labels, comprising: Using the multi-task model trained with the first training data set to predict the second training data, to obtain prediction feature information corresponding to the second training data; Determining a second loss value according to the predicted feature information and the feature label; According to the second loss value, a second set of parameters corresponding to the feature extraction network is adjusted.

4. The method according to claim 3, in, The third training data set includes third training data and decoding labels corresponding to the third training data; Training the multi-task model trained using the second training dataset based on a third training dataset including decoded labels, comprising: Using the multi-task model trained with the second training data set to predict the third training data, to obtain prediction decoding information corresponding to the third training data; Determining a third loss value according to the predicted decoding information and the decoding label; According to the third loss value, a third set of parameters corresponding to the decoding network is adjusted.

5. The method according to claim 4, further comprising: include: Determine that the first loss value is greater than a preset first threshold, and adjust a first set of parameters corresponding to the classification network based on the first loss value; Determine that the second loss value is greater than a preset second threshold, and adjust a second set of parameters corresponding to the feature extraction network based on the second loss value; Determine that the third loss value is greater than a preset third threshold, and adjust a third set of parameters corresponding to the decoding network based on the third loss value.

6. A training device for a multi-task model, include: A first training module: based on a first training data set including category labels stored in a data storage device in a multi-task model training system, the multi-task model to be trained is trained through a model training platform in the multi-task model training system to obtain a multi-task model including a classification network, wherein the classification network is used to classify preset tasks; A second training module: based on a second training data set including feature labels stored in the data storage device, the multi-task model trained by the first training module is trained through the model training platform to obtain a multi-task model including a feature extraction network, wherein the feature extraction network is used to identify feature labels of training data in a preset task category, the feature extraction network is a subnetwork of the classification network, and the feature labels are used to characterize feature information of each subtask in the multi-task in the corresponding task category; A third training module: based on a third training data set including decoding labels stored in the data storage device, the multi-task model trained by the second training module is trained through the model training platform to obtain a multi-task model including a decoding network, wherein the decoding network is used to decode feature labels corresponding to the training data, the decoding network is a subnetwork of the feature extraction network, and the decoding labels are used to represent decoding information corresponding to feature information of each subtask in the multi-task; Among them, the trained multi-task model is used to perform parallel processing on each of the preset tasks in the multi-task; the training data in the first training data set, the second training data set and the third training data set all include at least one of language data, image data and video data.

7. The device according to claim 6, in, The first training data set includes first training data and the category label corresponding to the first training data; The first training module includes: A first prediction unit predicts first training data using the multi-task model to be trained to obtain a prediction task category corresponding to the first training data; A first calculation unit: determining a first loss value according to the category label and the prediction task category predicted by the first prediction unit; A first adjustment unit: adjusts a first set of parameters corresponding to the classification network according to the first loss value determined by the first calculation unit.

8. The device according to claim 7, in, The second training data set includes second training data and feature labels corresponding to the second training data; The second training module includes: A second prediction unit: predicting the second training data using the multi-task model trained by the first training module to obtain prediction feature information corresponding to the second training data; A second calculation unit: determining a second loss value according to the feature label and the predicted feature information predicted by the second prediction unit; A second adjustment unit: adjusting a second set of parameters corresponding to the feature extraction network according to the second loss value determined by the second calculation unit.

9. The device according to claim 8, in, The third training data set includes third training data and decoding labels corresponding to the third training data; The third training module includes: A third prediction unit: predicting the third training data using the multi-task model trained by the second training module to obtain prediction decoding information corresponding to the third training data; A third calculation unit: determining a third loss value according to the decoding label and the predicted decoding information predicted by the third prediction unit; A third adjustment unit: adjusting a third set of parameters corresponding to the decoding network according to the third loss value determined by the third calculation unit.

10. A computer-readable storage medium storing a computer program for executing the multi-task model training method according to any one of claims 1 to 5.

11. An electronic device, the electronic device include: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the training method of the multi-task model described in any one of claims 1-5 above.