Multi-task model training method, perception method, device, medium and product

Abstract

The invention discloses a multi-task model training method. The method comprises the steps of obtaining a training data set of a plurality of sub-tasks corresponding to a multi-task model; inputting a first sample image in the image data set of the sub-tasks into a multi-task model, and performing multi-scale feature extraction on the first sample image by adopting a first sub-network of the multi-task model to obtain first feature maps of different scales; performing feature fusion processing on the first feature maps of different scales by adopting a second sub-network shared by each sub-task network in an associated task group to which sub-tasks belong in the multi-task model to obtain a second feature map; processing the second feature map by adopting a sub-task network corresponding to sub-tasks in a multi-task model to obtain first sensing images of the sub-tasks, so as to obtain the first sensing images of the sub-tasks; and iteratively updating network parameters of the multi-task model according to the first sensing image and the first sample image corresponding to each sub-task to obtain a trained multi-task model.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a training method, perception method, equipment, medium and product for a multi-task model. Background Art

[0002] At present, with the continuous development of artificial intelligence, the requirements for intelligence are getting higher and higher. In specific application scenarios, a landing artificial intelligence service often contains a variety of different types of processing tasks, so that the service content can be completed more accurately and efficiently through the cooperation of various processing tasks. For example, in the scenario of autonomous driving, accurate perception of the surrounding environment is a crucial part of realizing autonomous driving. It is often achieved by detecting, identifying or segmenting various types of elements on the road: pedestrians, vehicles, lane lines, traffic signs, road surface markings and drivable areas, etc. The detection, identification or segmentation of each element can be used as a perception task. Through the multi-task model, each task shares a backbone network and feature fusion network, and shares the feature maps obtained from the backbone network and feature fusion network. Based on these feature maps, different tasks are processed separately; however, since the task differences between different tasks may be large, when all tasks share the same feature fusion network, the perception results output by the multi-task model in actual applications are poorly accurate. Summary of the invention

[0003] One of the purposes of the present application is to provide a training method, perception method, device, medium and product of a multi-task model to solve the problem in the related art that the task differences between different tasks may be large, and when all tasks share the same feature fusion network, the perception results output by the multi-task model in actual applications are poorly accurate.

[0004] In order to achieve the above purpose, the technical solution of the embodiment of the present application is implemented as follows:

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, the present application provides a method for training a multi-task model, the method comprising:

[0007] Acquire a training data set of multiple subtasks corresponding to the multi-task model, wherein the training data set at least includes a sample image;

[0008] For a training data set of any one of the multiple subtasks, input a first sample image in the image data set of the subtask into the multi-task model, and use a first sub-network of the multi-task model to perform multi-scale feature extraction on the first sample image to obtain first feature maps of different scales;

[0009] Using a second sub-network shared by each sub-task network in the associated task group to which the sub-task belongs in the multi-task model, feature fusion processing is performed on the first feature maps of different scales to obtain a second feature map; wherein the multi-task model includes at least two associated task groups, and each associated task group includes at least one sub-task;

[0010] Processing the second feature map using a subtask network corresponding to the subtask in the multitask model to obtain a first perceived image of the subtask output by the multitask model, thereby obtaining a first perceived image of each subtask;

[0011] According to the first perception image and the first sample image corresponding to each of the subtasks, the network parameters of the multi-task model are iteratively updated to obtain a trained multi-task model.

[0012] According to the above technical means, the first sub-network performs multi-scale feature extraction on the first sample image, which can capture the details and structural information of the image at different scales. Multi-scale feature extraction can enhance the model's ability to represent images and improve the model's performance in various tasks; according to the correlation between different sub-tasks, it is divided into multiple associated task groups, and different associated task groups share different second sub-networks. The second sub-network performs feature fusion processing on the first feature maps of different scales, which can integrate multi-scale features and make full use of information at different scales; each sub-task has a corresponding sub-task network, which can perform special processing for specific tasks. According to the first perception image and the first sample image corresponding to each sub-task, the network parameters of the multi-task model are iteratively updated, which can continuously optimize the performance of the model, improve the generalization ability and robustness of the model, and thus improve the accuracy of the perception results output by the multi-task model in practical applications.

[0013] Furthermore, the first sub-network includes multiple feature extraction sub-modules for extracting features of different scales, the multiple feature extraction sub-modules are connected in series, the feature extraction sub-module includes a first number of first convolutional network units and a second number of second convolutional network units, the second number of second convolutional network units are cascaded with the first number of second convolutional network units, and the first convolutional network unit and the second convolutional network unit are set for different step sizes.

[0014] Furthermore, the second number is 1, and the second convolutional network unit is located at the first position of the feature extraction submodule.

[0015] According to the above technical means, through the second convolutional network unit, each feature level needs to downsample the input feature map by two times so that deeper feature levels can have a larger receptive field, thereby detecting large targets in the image.

[0016] Further, the first convolutional network unit includes a first convolutional branch, a first directly connected branch and a first fusion branch; wherein, the first convolutional branch and the first directly connected branch are connected in parallel and then cascaded with the first fusion branch, the first convolutional branch is composed of multiple convolutional modules, multiple batch normalization modules and activation functions, and the first fusion branch includes an adder; the second convolutional network unit includes at least two second convolutional branches, a third convolutional branch, a second fusion branch and a third fusion branch; wherein, the second convolutional branch of the first part is connected in parallel and then cascaded with the second fusion branch and the third convolutional branch, the second convolutional branch of the second part is connected in parallel with the second convolutional branch of the first part, the second fusion branch and the third convolutional branch, the second convolutional branch and the third convolutional branch are both composed of a convolutional module and a batch normalization module, the second fusion branch includes an adder and an activation function, and the third fusion branch includes an adder.

[0017] Furthermore, the method also includes: modifying a convolution module with a convolution kernel of a first size in the first sub-network into a repeatable convolution module with a multi-branch structure having convolution kernels of different sizes.

[0018] According to the above technical means, the multi-branch structure of the repeatable convolution module adopts different convolution kernels to obtain different receptive fields, and adds the information obtained from different receptive fields to enhance the extraction of feature information and improve model performance.

[0019] Furthermore, the repeatable convolution module includes a fourth convolution branch, a fifth convolution branch, a second directly connected branch and a fourth fusion branch; wherein, the fourth convolution branch, the fifth convolution branch and the second directly connected branch are connected in parallel and then cascaded with the fourth fusion branch; the fourth convolution branch is composed of a convolution module with a convolution kernel of the first size and a batch normalization module, and the fifth convolution branch is composed of a convolution module with a convolution kernel of the second size, and the first size is different from the second size.

[0020] According to the above technical means, when training a multi-task model, training is performed according to three branches, so that the model training is more comprehensive, the extraction of feature information is strengthened, and the model performance is improved.

[0021] Furthermore, the method also includes: after the multi-task model training is completed, according to the additivity of convolution, the repeatable convolution modules are equivalently merged into the convolution modules of the first size, thereby obtaining the trained multi-task model.

[0022] According to the above technical means, the model structure during training has a repeatable convolution module. According to the additivity of convolution, during model reasoning, the three branches are equivalently merged into a single-branch 3×3 convolution, that is, the repeatable convolution module is equivalently merged into a 3×3 ordinary convolution. In this way, the reasoning speed of the model is greatly improved without losing the model performance.

[0023] Furthermore, the iterative updating of the network parameters of the multi-task model according to the first perceived image and the first sample image corresponding to each of the subtasks includes: for any one of the multiple subtasks, determining the target loss of the first perceived image of the subtask according to the first perceived image and the first sample image of the subtask; determining the first gradient data, the second gradient data and the third gradient data corresponding to the subtask according to the target loss of the subtask, thereby obtaining the first gradient data, the second gradient data and the third gradient data corresponding to each subtask; wherein the first gradient data includes the gradient value of the network parameter of the first subnetwork, the second gradient data includes the gradient value of the network parameter of the second subnetwork shared by each subtask network in the associated task group to which the subtask belongs, and the third gradient data includes the gradient value of the network parameter of the subtask network corresponding to the subtask; the network parameters of the multi-task model are iteratively updated according to the first gradient data, the second gradient data and the third gradient data corresponding to each subtask.

[0024] According to the above technical means, by determining the target loss of the subtask, the performance of the subtask under the current model state can be accurately measured. The first gradient data, second gradient data and third gradient data of the network parameters of the subtask corresponding to different networks are calculated according to the target loss, so that the model can perform efficient parameter updates under the framework of multi-task learning. Different subtasks can be independently optimized according to their own target losses and corresponding gradient data, while at the same time exchanging information and collaboratively optimizing through the shared network parts (the first subnetwork and the second subnetwork) to jointly improve the overall performance of the model.

[0025] Further, determining the target loss of the first perceived image of the subtask based on the first perceived image of the subtask and the first sample image includes: determining the image loss of the first perceived image of the subtask based on the first perceived image of the subtask and the first sample image; obtaining convolution kernels of the same size in the first subnetwork; based on an orthogonal loss function, processing any two convolution kernels of the same size to obtain the orthogonal loss corresponding to the subtask; and determining the target loss based on the image loss and the orthogonal loss.

[0026] According to the above technical means, by determining the image loss based on the first perceived image and the first sample image of the subtask, the prediction accuracy of the model on the subtask can be measured. By processing any two convolution kernels of the same size to obtain the orthogonal loss, excessive similarity between the convolution kernels can be avoided, so that the model can better utilize different convolution kernels to extract different features, thereby enriching the feature space of the model, improving the expressiveness and accuracy of the model, and avoiding repeated feature extraction and calculation, reducing the waste of computing resources.

[0027] Furthermore, the network parameters of the multi-task model are adjusted according to the first gradient data, the second gradient data and the third gradient data corresponding to each subtask, including: adjusting the network parameters of the first subnetwork according to the first gradient data corresponding to each subtask; adjusting the network parameters of the second subnetwork shared by each subtask network in the associated task group to which the subtask belongs according to the second gradient data corresponding to each subtask; and adjusting the network parameters of the subtask network corresponding to the subtask according to the third gradient data corresponding to each subtask.

[0028] Furthermore, adjusting the network parameters of the first sub-network according to the first gradient data corresponding to each sub-task includes: obtaining a first weight corresponding to each sub-task; based on the first weight, performing weighted averaging processing on the first gradient data of the network parameters of the first sub-network to obtain first target gradient data of the network parameters of the first sub-network, and inputting the first target gradient data into the first sub-network to complete the adjustment of the network parameters of the first sub-network.

[0029] According to the above technical means, by obtaining the first weight corresponding to each subtask and performing weighted averaging on the first gradient data of the network parameters of the first subnetwork, the needs of each subtask can be comprehensively considered, the balance of multi-task performance can be achieved, and the overall multi-task processing capability can be improved. Inputting the first target gradient data into the first subnetwork for parameter adjustment can make the first subnetwork better adapt to the multi-task environment and improve the adaptability and flexibility of the model. When the task requirements change or new subtasks are added, the first weight can be adjusted to quickly adapt to the new situation without retraining the entire model, saving time and computing resources.

[0030] Furthermore, according to the second gradient data corresponding to each subtask, the network parameters of the second subnetwork shared by each subtask network in the associated task group to which the subtask belongs are adjusted, including: for each subtask, obtaining the second weight of each subtask in the associated task group to which the subtask belongs; based on the second weight, performing weighted averaging processing on the second gradient data of the network parameters of the second subnetwork shared by each subtask network in the associated task group to obtain the second target gradient data of the network parameters of the second subnetwork shared by each subtask network in the associated task group, and inputting the second target gradient data into the shared second subnetwork to complete the adjustment of the network parameters of the second subnetwork.

[0031] According to the above technical means, for each subtask, by obtaining the second weight of each subtask in the associated task group to which it belongs, the correlation between different subtasks can be fully taken into account, the correlation between these tasks can be better utilized, and the learning efficiency of the model can be improved. The second gradient data of the network parameters of the second sub-network shared by each subtask network in the associated task group is weighted averaged, which can balance the optimization requirements of different subtasks with high correlation for the shared network. Inputting the second target gradient data into the shared second sub-network for parameter adjustment can make the second sub-network better adapt to the combination and changes of different subtasks. When a new subtask is added to the associated task group, it is only necessary to recalculate the second weight and perform weighted average processing to quickly adjust the shared network to adapt to the new task requirements.

[0032] In a second aspect, the present application provides a perception method of a multi-task model, the method comprising:

[0033] acquiring a first image;

[0034] Using a multi-task model to perform perception processing on the first image to obtain a task perception image corresponding to the first image;

[0035] Wherein, the multi-task model is obtained based on the training method of the multi-task model described in any one of the first aspects.

[0036] In a third aspect, the present application provides a training device for a multi-task model, the device comprising:

[0037] A first acquisition module is used to acquire a training data set of multiple subtasks corresponding to the multi-task model, wherein the training data set at least includes a sample image;

[0038] A multi-task model processing module, for a training data set of any one of the multiple subtasks, inputting a first sample image in the image data set of the subtask into the multi-task model, and performing multi-scale feature extraction on the first sample image using a first sub-network of the multi-task model to obtain first feature maps of different scales;

[0039] The multi-task model processing module is further used to use a second sub-network shared by each sub-task network in the associated task group to which the sub-task in the multi-task model belongs to perform feature fusion processing on the first feature maps of different scales to obtain a second feature map; wherein the multi-task model includes at least two associated task groups, and each associated task group includes at least one sub-task;

[0040] The multi-task model processing module is further used to process the second feature map using a sub-task network corresponding to the sub-task in the multi-task model to obtain a first perceived image of the sub-task output by the multi-task model, thereby obtaining a first perceived image of each sub-task;

[0041] The multi-task model processing module is also used to iteratively update the network parameters of the multi-task model according to the first perception image and the first sample image corresponding to each of the subtasks to obtain a trained multi-task model.

[0042] In a fourth aspect, the present application provides a perception device of a multi-task model, the device comprising:

[0043] A second acquisition module, used for acquiring a first image;

[0044] a multi-task model processing module, configured to perform perception processing on the first image using a multi-task model to obtain a task perception image corresponding to the first image;

[0045] Wherein, the multi-task model is obtained based on the training method of the multi-task model described in any one of the first aspects.

[0046] In a fifth aspect, the present application provides a computer device, the computer device comprising: a memory and a processor,

[0047] The memory stores a computer program executable on the processor;

[0048] When the processor executes the computer program, it implements part or all of the steps in the training method of the multi-task model as described in any one of the first aspects, or part or all of the steps in the perception method of the multi-task model as described in the second aspect.

[0049] In a sixth aspect, the present application provides a computer-readable storage medium, which stores one or more computer programs, and the one or more computer programs can be executed by one or more processors to implement part or all of the steps in the training method of the multi-task model as described in any one of the first aspect, or part or all of the steps in the perception method of the multi-task model as described in the second aspect.

[0050] In the seventh aspect, the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements part or all of the steps in the training method of the multi-task model as described in any one of the first aspect, or part or all of the steps in the perception method of the multi-task model as described in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0052] Figure 1 A schematic diagram of an implementation flow of an optional multi-task model training method provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of an optional multi-task model provided in an embodiment of the present application Figure 1 ;

[0054] Figure 3 A schematic diagram of an optional multi-task model provided in an embodiment of the present application Figure 2 ;

[0055] Figure 4 A schematic diagram of the structure of an optional first convolutional network unit provided in an embodiment of the present application;

[0056] Figure 5 A schematic diagram of the structure of an optional second convolutional network unit provided in an embodiment of the present application;

[0057] Figure 6 A schematic diagram of the structure of an optional repeatable convolution module provided in an embodiment of the present application;

[0058] Figure 7 A schematic diagram of the structure of an optional first repeatable convolutional network unit and a second repeatable convolutional network unit provided in an embodiment of the present application;

[0059] Figure 8 A schematic diagram of an implementation flow of an optional multi-task model training method provided in an embodiment of the present application;

[0060] Fig. 9 A schematic diagram of an implementation flow of an optional multi-task model training method provided in an embodiment of the present application;

[0061] Fig.10 A schematic diagram of an implementation flow of an optional multi-task model training method provided in an embodiment of the present application;

[0062] Fig.11 A schematic diagram of the structure of an optional basic network module BaseBlock_A provided in an embodiment of the present application;

[0063] Fig.12 A schematic diagram of the structure of an optional basic network module BaseBlock_B provided in an embodiment of the present application;

[0064] Fig.13 A schematic diagram of the structure of an optional re-parameterization module RepBlock provided in an embodiment of the present application;

[0065] Fig.14 A schematic diagram of the structure of an optional basic network module BaseBlock_A and BaseBlock_B fusion re-parameterization module provided in an embodiment of the present application;

[0066] Fig.15 A schematic diagram of the structure of an optional multi-task model training device provided in an embodiment of the present application;

[0067] Fig.16 A schematic diagram of the structure of a sensing device of an optional multi-task model provided in an embodiment of the present application;

[0068] Fig.17 A schematic diagram of the structure of an optional computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0070] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific order for the objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application.

[0072] At present, with the continuous development of artificial intelligence, the requirements for intelligence are getting higher and higher. In specific application scenarios, a landing artificial intelligence service often contains a variety of different types of processing tasks, so that the various processing tasks can work together to complete the service content more accurately and efficiently. The multi-task model is a relatively new and practical direction in the field of deep learning. The multi-task model refers to a neural network obtained by integrating multiple visual tasks using similar information into the same neural network through unified network training. The multi-task model has important applications in many scenarios. For example, in the scenario of autonomous driving, the core of autonomous driving technology, as a frontier field of the modern automobile industry, is to realize the autonomous navigation and decision-making of vehicles. In this process, the machine vision system plays a vital role. By simulating human visual perception ability, it accurately perceives elements in the surrounding environment, including pedestrians, vehicles, lane lines, traffic signs, road markings, and drivable areas. Among them, the detection, recognition or segmentation of each element can be used as a perception task. Through the multi-task model, each task shares a backbone network and a feature fusion network, and shares the feature maps obtained from the backbone network and the feature fusion network. Based on these feature maps, different tasks are processed separately. However, since the task differences between different tasks may be large, when all tasks share the same feature fusion network, the perception results output by the multi-task model in actual applications are less accurate.

[0073] Based on the problems mentioned in the above-mentioned related technologies, the embodiments of the present application provide a multi-task model training method, a multi-task model perception method, an apparatus, a computer device, a storage medium and a computer product, aiming to take into account multi-task performance at the same time, ensure that each task has a good effect, and realize network integration of a larger number of tasks.

[0074] In some embodiments, the present application provides a multi-task model training method, which can be applied to a computer device, and the computer device executes one or more method steps in the following embodiments.

[0075] In the implementation manner of the present application, the computer device may be any type of device with computing capabilities, such as a handheld mobile terminal, a desktop terminal device, a vehicle-mounted terminal device, a wearable device, a server, and a cloud platform, etc., and the present application does not impose any restrictions on this.

[0076] In the implementation manner of the present application, the multi-task model refers to a neural network model that can realize multiple image perception tasks such as detection tasks, segmentation tasks and recognition tasks, for example, pedestrian detection, vehicle detection, lane line segmentation, vehicle recognition, pedestrian recognition, etc., integrating these perception tasks into the same neural network, that is, the multi-task model described in the present application.

[0077] Figure 1 A schematic diagram of an implementation flow of an optional multi-task model training method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method may include the following steps:

[0078] Step 101: Obtain training data sets for multiple subtasks corresponding to a multi-task model, wherein the training data sets at least include sample images.

[0079] In an embodiment of the present application, the training data set is a sample image data set used to train a multi-task model. Different subtasks can have different training data sets, that is, a corresponding training data set is set for each subtask. The training data set includes multiple sample images and label data of the task corresponding to each sample image.

[0080] It should be noted that each subtask can be trained in parallel or sequentially using its corresponding training data set to improve subtask performance and training efficiency.

[0081] In some embodiments, based on the correlation between each subtask in all subtasks, it can be determined whether the sources of the training data sets of different subtasks are the same or different; here, if the correlation between each subtask is greater than or equal to the correlation threshold, the subtasks with correlation greater than or equal to the correlation threshold can select homologous data for data annotation as the training data set, and if the correlation between each subtask is less than the correlation threshold, the subtasks with correlation less than the correlation threshold can select heterogeneous data for data annotation as the training data set. Exemplarily, the multiple subtasks include pedestrian detection tasks and lane segmentation tasks. The pedestrian detection task can use the coco dataset, and the lane segmentation can use the CULane open source dataset, thereby supporting heterogeneous data for model training.

[0082] In some embodiments, for the training data set corresponding to each subtask, a single batch polling method can be used for training. Exemplarily, the multi-task model corresponds to m subtasks, each subtask has a corresponding training data set and training parameters, and the training parameters include the training rounds and training batches of the subtask training. In one iteration, the training data set is divided according to the training batches, and the training data set and training parameters corresponding to each subtask can be sequentially input into the multi-task model to obtain the network parameters of the multi-task model corresponding to the task. Further, the training data set and training parameters corresponding to the next task are used to train the multi-task model corresponding to the next task, until the training data sets of all tasks train the multi-task model, indicating that this iteration is completed.

[0083] Step 102: for a training data set of any one of the multiple subtasks, input a first sample image in the image data set of the subtask into a multi-task model, and use a first sub-network of the multi-task model to perform multi-scale feature extraction on the first sample image to obtain first feature maps of different scales.

[0084] In an embodiment of the present application, a multi-task model is used to perform different tasks such as detection, recognition or segmentation on different objects in an image. The multi-task model can learn multiple related sub-tasks at the same time and fully explore the potential shared information between different sub-tasks.

[0085] In an embodiment of the present application, the multi-task model includes a first sub-network, which is a backbone network (Backbone) shared by all sub-tasks. The first sub-network is used to perform in-depth feature extraction of different scales on the image to be processed to obtain rich semantic information and spatial feature information corresponding to the image to be processed at different scales.

[0086] In the embodiment of the present application, the first feature map is used to characterize the rich semantic information and spatial feature information corresponding to each object in the image screen of the image to be processed at different scales. Here, features of different scales can reflect different aspects of the image. For example, small-scale features can capture local details, and large-scale features can reflect the overall structure. In the target detection task, multi-scale features can help the model better detect targets of different sizes. Among them, the spatial feature information includes but is not limited to the precise location information features, size information features, and shape information features of the object.

[0087] In one possible implementation, referring to Figure 2As shown, an embodiment of the present application provides a structural schematic diagram of a multi-task model. After the computer device obtains the training data sets of multiple subtasks corresponding to the multi-task model, for the training data set of any one of the multiple subtasks, the first sample image in the image data set of each subtask is sequentially input into the multi-task model. Here, for each subtask, the first sub-network 21 of the multi-task model 200 is used to perform multi-scale feature extraction on the first sample image to obtain a first feature map with rich semantic information and spatial feature information at different scales. In this way, the first sub-network of the multi-task model performs multi-scale feature extraction on the first sample image, which can comprehensively capture the information of the image at different levels and scales. Features of different scales can reflect various characteristics of the image, such as details, local structure, and overall contour, and provide rich feature representation for subsequent processing.

[0088] Step 103: Use a second sub-network shared by each sub-task network in an associated task group to which the sub-task belongs in the multi-task model to perform feature fusion processing on the first feature maps of different scales to obtain a second feature map; wherein the multi-task model includes at least two associated task groups, and each associated task group includes at least one sub-task.

[0089] In an embodiment of the present application, the second subnetwork is an intermediate layer network (Neck), which is used to perform feature fusion on the first feature map output by the first subnetwork, thereby upsampling the first feature map of a smaller image scale to obtain a second feature map with larger scale feature information, thereby providing the feature information required for tasks that rely on a large-scale receptive field.

[0090] In the embodiment of the present application, the associated task group refers to the association between multiple subtasks, and the association can be understood as the positive correlation effect of mutual promotion between subtasks.

[0091] In an embodiment of the present application, the multi-task model can process multiple different sub-tasks, and divide all sub-tasks into multiple associated task groups according to their relevance. The sub-tasks in each associated task group share a second sub-network, but not all sub-tasks share a second sub-network, thereby obtaining multiple second sub-networks, and the multiple second sub-networks are decoupled sub-networks to adapt to the task types corresponding to different associated task groups, such as detection task types and segmentation task types.

[0092] For example, according to the relevance of the perception tasks, all subtasks can be divided into detection-related task groups, segmentation-related task groups, and recognition-related task groups; for example, the detection-related task group is used to detect subtasks of different objects, including pedestrian detection subtasks, lane detection subtasks, obstacle detection subtasks, etc., and the segmentation-related task group is used to segment subtasks of different objects, including lane line segmentation subtasks, road edge segmentation subtasks, etc. Of course, according to the task relevance of the perception objects, all subtasks can be divided into pedestrian detection-related task groups and lane line segmentation task association groups. Of course, all subtasks can also be divided according to other relevances, and this application does not make specific restrictions.

[0093] It can be understood that in the implementation manner of the present application, the number of second sub-networks depends on the number of associated task groups divided by multiple sub-tasks. After determining the associated task groups of multiple sub-tasks, it is only necessary to copy the same number of second sub-networks. The specific number can be selected according to different application scenarios, and the present application does not impose any restrictions on this.

[0094] In an embodiment of the present application, during the feature fusion process, the second sub-network upsamples the first feature map of a smaller image scale to obtain a third feature map with larger-scale feature information, and then performs feature fusion corresponding to the task type with the first feature map of the same scale in the first sub-network, thereby providing the feature information required for tasks that rely on a large-scale receptive field, so as to obtain the second feature map output by the second sub-network.

[0095] In one possible implementation, see Figure 2 and Figure 3 As shown, the second sub-network 22 can adopt a cubic convolution and upsampling module. The upsampling module can adopt bilinear interpolation. The upsampling module is used to upsample the first feature map output by the first sub-network, that is, the first feature map is gradually decoded to obtain a second feature map with a larger scale and richer semantic information, so that the subsequent sub-task network can obtain more features and information of the required scale, so as to better complete the corresponding sub-task processing.

[0096] In some embodiments, different associated task groups correspond to different task types, and different task types have different feature fusion methods. Exemplarily, the task types include detection task types (such as pedestrian detection task types) and segmentation task types (such as lane segmentation task types), wherein the second subnetwork for the detection task type upsamples the feature map of a smaller image scale to obtain a feature map with larger-scale feature information, and then uses an element-by-element addition method to perform feature fusion with the feature map of the same scale in the first subnetwork to obtain higher-level semantic information. The second subnetwork for the segmentation task type upsamples the feature map of a smaller image scale to obtain a feature map with larger-scale feature information, and then uses a feature concatenation method to perform feature fusion with the feature map of the same scale in the first subnetwork to obtain more original feature information. Exemplarily, referring to Figure 3 shown.

[0097] In some embodiments, different associated task groups correspond to different task types, and the second sub-networks corresponding to different task types have different numbers of detection heads involved in feature fusion. Figure 3 As shown in the figure, different task types include detection task types (such as pedestrian detection task types) and segmentation task types (such as lane segmentation task types). The first subnetwork includes five image levels (stages). When the second subnetwork upsamples the first feature map, the second subnetwork corresponding to the detection task type performs feature fusion. Since pedestrian detection requires effective detection of both small targets at a long distance and large targets at a close distance, the required receptive fields are of different sizes, and the required semantic feature information and spatial feature information have different levels. Therefore, the second subnetwork can be connected to a detection head (subtask network) at stages 2, 3, 4, and 5 respectively. In this way, after obtaining the second feature maps with different receptive fields and different levels of semantic feature information and spatial feature information, the detection head is used to perceive the second feature map to obtain the pedestrian perception result. When the second subnetwork upsamples the first feature map, the second subnetwork corresponding to the segmentation task type is performing feature fusion. Since the lane line segmentation task requires more refined pixel classification and rich semantic feature information and spatial feature information, the second subnetwork can be connected to a detection head (subtask network) at stage 2. In this way, after obtaining the second feature map with rich semantic feature information and spatial feature information, the detection head is used to perceive the second feature map to obtain the lane line segmentation perception result.

[0098] In one possible implementation, continue to refer to Figure 2, after obtaining the first sub-network 21 using the multi-task model, performing multi-scale feature extraction on the first sample image, and obtaining the first feature maps of different scales, the computer device uses the second sub-network 22 shared by each sub-task network in the associated task group to which the sub-task belongs in the multi-task model to perform feature fusion processing on the first feature maps of different scales to obtain the second feature map. In this way, by sharing the second sub-network by each sub-task network in the associated task group to which the sub-task belongs in the multi-task model, resource sharing and information interaction are realized. The shared second sub-network can learn the common features between multiple sub-tasks, thereby enhancing the generalization ability of the model. The shared second sub-network avoids constructing a feature fusion module for each sub-task separately, thereby greatly reducing the number of parameters and computational complexity of the model. This can significantly reduce the consumption of computing resources and improve the training and reasoning efficiency of the model when processing large-scale data and complex tasks. The second sub-network performs feature fusion processing on the first feature maps of different scales, effectively integrating multi-scale information. This fusion can give full play to the advantages of features of different scales, make up for the shortcomings of features of a single scale, and make the final second feature map have a more powerful representation ability.

[0099] Step 104: Use the subtask network corresponding to the subtask in the multi-task model to process the second feature map to obtain the first perceived image of the subtask output by the multi-task model, thereby obtaining the first perceived image of each subtask.

[0100] In the embodiment of the present application, the subtask network is a head network (Head) private to each subtask, whose input is the second feature map output by the corresponding second subnetwork, and whose output is the perception result corresponding to the subtask, such as the perception image. Exemplarily, the subtask network can adopt head networks such as PointPillars based on anchor points, head networks such as CenterPoint based on center heat map prediction, and DETR3D and Transfusion head networks based on Transformer; the segmentation task head network adopts a multi-layer convolutional network with the number of output channels as the number of categories. Of course, the head network can also be implemented using a multi-layer perceptron (Multilayer Perceptron, MLP), and this application does not make specific restrictions. For example, in one example, each subtask network includes three convolutional layers, wherein the first layer is a convolutional layer with a convolution kernel size of 1, which is used to reduce the dimension of the second feature map, the second layer is a convolutional layer with a convolution kernel of 3, which is used to add nonlinear transformation to each task, and the third layer is an output layer, which is used to output the output result of the subtask.

[0101] In some embodiments, continue to refer to Figure 2 As shown in the figure, the multi-task model integrates subtasks 1 to m, a total of m subtasks, and each subtask corresponds to its own private subtask network, that is, Figure 2 Subtask network 1 to subtask network m shown in .

[0102] In the implementation mode of the present application, when constructing a multi-task model, it is necessary to group multiple subtasks in advance according to the correlation between each subtask in all subtasks to obtain an associated task group. Figure 2 As shown, the m subtasks are divided into n associated task groups, wherein associated task group 1 includes subtasks 1 to p1, associated task group 2 includes subtasks p1+1 to p2, ... associated task group n includes subtasks pz+1 to m.

[0103] After the associated task groups are obtained, a second sub-network is set for each associated task group, so that for associated task groups 1 to n, the following are set: Figure 2 The second sub-networks 1 to n are shown, and there are n second sub-networks in total.

[0104] It can be understood that in the implementation manner of the present application, the number of second sub-networks depends on the number of associated task groups divided by multiple sub-tasks. After determining the associated task groups of multiple sub-tasks, it is only necessary to copy the same number of second sub-networks. The specific number can be selected according to different application scenarios, and the present application does not impose any restrictions on this.

[0105] After obtaining n second sub-networks, for each second sub-network, the private sub-task network of each sub-task can be connected to the back of the second sub-network according to the sub-tasks included in the corresponding associated task group. Taking the second sub-network 1 as an example, the associated task group 1 corresponding to the second sub-network 1 includes a total of p1 sub-tasks, namely, sub-tasks 1 to p1, so the sub-task networks of the p1 tasks can be connected to the back of the second sub-network 1.

[0106] In one possible implementation, continue to refer to Figure 2 The computer device uses the second sub-network 22 shared by each sub-task network in the associated task group to which the sub-task belongs in the multi-task model to perform feature fusion processing on the first feature maps of different scales. After obtaining the second feature map, the sub-task network 23 corresponding to the sub-task in the multi-task model is used to process the second feature map to obtain the first perceived image of the sub-task output by the multi-task model, thereby obtaining the first perceived image of each sub-task.

[0107] Step 105: Iteratively update the network parameters of the multi-task model according to the first perception image and the first sample image corresponding to each subtask to obtain a trained multi-task model.

[0108] In an embodiment of the present application, the target loss corresponding to each subtask is determined based on the first perception image and the first sample image corresponding to each subtask. When it is determined that the training does not converge based on the target losses corresponding to multiple subtasks, the network parameters of the multi-task model are iteratively updated. When it is determined that the training converges based on the target losses corresponding to multiple subtasks, the training is completed.

[0109] In the embodiment of the present application, the first perceived image represents the prediction result of the multi-task model for the subtask, and the first sample image in the training data set represents the true value (Groundtruth). The loss between the first perceived image and the sample image can be calculated based on the pre-set loss function. The loss represents the difference between the network prediction result and the true result. The larger the loss, the worse the network performance. Conversely, the smaller the loss, the better the network performance. The goal of network training is to make the loss value of multiple iterations converge to a better range.

[0110] Still taking the iterative process in the above example as an example, after obtaining the difference between the first perceived image and the first sample image of each subtask, that is, the target loss, the network parameters of the multi-task model can be optimized and adjusted using back propagation according to the target loss.

[0111] The first sub-network structure of the multi-task model is further explained below.

[0112] In some embodiments, the first sub-network includes multiple feature extraction sub-modules for extracting features at different scales, the multiple feature extraction sub-modules are connected in series, the feature extraction sub-modules include a first number of first convolutional network units and a second number of second convolutional network units, the second number of second convolutional network units are cascaded with the first number of second convolutional network units, and the first convolutional network unit and the second convolutional network unit are set for different step sizes.

[0113] In the embodiment of the present application, the first sub-network includes 5 feature levels (stages), each feature level is used to extract features of different scales, each feature level includes a feature extraction sub-block, and multiple feature extraction sub-blocks are connected in series. Here, the feature extraction sub-module of each feature level includes a first convolutional network unit and a second convolutional network unit, the first convolutional neural network and the second convolutional neural network are cascaded, and the second convolutional network unit is located before the first convolutional network unit, the first convolutional network unit is used to extract rich image features, and the second convolutional network unit is used to perform feature downsampling.

[0114] In the embodiment of the present application, the first convolutional network unit and the second convolutional network unit are set for different strides. For example, the first convolutional network unit can be set for the case of stride=1, that is, the size of the feature map input to the first convolutional network unit is the same as the size of the feature map output after processing by the first convolutional network unit. The second convolutional network unit can be set for the case of stride=2, and the second convolutional network unit is used for feature downsampling, that is, the size of the feature map output after processing by the second convolutional network unit is 0.5 times the size of the feature map input to the second convolutional network unit.

[0115] In an embodiment of the present application, the first number of first convolutional network units and the second number of second convolutional network units included in the feature extraction submodule of each feature level are the same or different, and the second number of second convolutional network units are cascaded with the first number of second convolutional network units; further, according to the importance and type of the task, the first number of first convolutional network units and the second number of second convolutional network units set by the feature extraction submodule corresponding to each feature level can be shown in Table 1.

[0116] Stage The first number of first convolutional network units The second number of first convolutional network units 1 0 1 2 1 1 3 5 1 4 2 1 5 1 1

[0117] In some embodiments of Table 1, the second number is 1, and the second convolutional network unit is located at the first position of the feature extraction submodule.

[0118] Here, the first convolutional network unit in the feature extraction submodule corresponding to each feature level is set to the second convolutional network unit. In this way, each feature level stage needs to downsample the input feature map by two times so that deeper feature levels can have a larger receptive field and thus detect large objects in the image.

[0119] In some embodiments, the first convolutional network unit includes a first convolutional branch, a first directly connected branch and a first fusion branch; wherein the first convolutional branch and the first directly connected branch are connected in parallel and then cascaded with the first fusion branch, the first convolutional branch is composed of a plurality of convolutional modules, a plurality of batch normalization modules and an activation function, and the first fusion branch includes an adder;

[0120] The second convolutional network unit includes at least two second convolutional branches, a third convolutional branch, a second fusion branch and a third fusion branch; wherein, the second convolutional branch of the first part is connected in parallel and then cascaded with the second fusion branch and the third convolutional branch, the second convolutional branch of the second part is connected in parallel with the second convolutional branch, the second fusion branch and the third convolutional branch of the first part, the second convolutional branch and the third convolutional branch are both composed of a convolution module and a batch normalization module, and the second fusion branch and the third fusion branch both include adders.

[0121] For the first convolutional network unit, illustratively, refer to Figure 4 As shown, the first convolutional network unit 4 includes a first convolution branch 41, a first directly connected branch 42 and a first fusion branch 43. After the first convolution branch 41 and the first directly connected branch 42 are connected in parallel, they are cascaded with the first fusion branch 43.

[0122] Among them, the first convolution branch 41 includes two convolution modules (Conv), and a batch normalization module (Batch Normalization, BN) corresponding to each convolution module, and an activation function such as a ReLu activation function. It should be noted that the size of the convolution kernel in the convolution module can be a first size, such as a 3×3 convolution kernel. Here, the first convolution module is used to increase the number of feature channels, which is equivalent to the expansion layer in the network, so as to extract rich image features. The degree of channel expansion can be dynamically set by the α hyperparameter to obtain multi-task models with different parameter amounts and complexities to adapt to perception tasks of different difficulties. The second convolution module is used to reduce the number of feature channels to match the output of the residual shortcut connection.

[0123] Among them, the first fusion branch 43 includes an adder, which is used to fuse the feature map input to the first convolutional network unit 4 through the first directly connected branch 42 with the feature map output after the feature map is processed by the first convolutional branch 41, so as to obtain the feature map processed by the first convolutional network unit 4.

[0124] For the second convolutional network unit, exemplarily, refer to Figure 5 As shown, the second convolutional network unit 5 includes three second convolution branches 51, a third convolution branch 52, a second fusion branch 53 and a third fusion branch 54, wherein two second convolution branches 51 are connected in parallel and then cascaded with the second fusion branch 53 and the third convolution branch 52, and one second convolution branch 51 is connected in parallel with the above-mentioned two second convolution branches 51, the second fusion branch 53 and the third convolution branch 52.

[0125] Among them, the three second convolution branches 51 and the third convolution branch 52 are each composed of a convolution module and a batch normalization module, the second fusion branch 53 includes an adder and an activation function, and the third fusion branch 54 includes an adder.

[0126] Among them, the convolution module in the first second convolution branch 51 can be used to extract edge features of the image, the convolution module in the second second convolution branch 51 can be used to extract detail features in the image, and the convolution module in the third second convolution branch 51 can be used to perform residual connection to adapt the output. It should be noted that the size of the convolution kernel in the convolution module can be a first size, such as a 3×3 convolution kernel, and the step size can be set to stride=2.

[0127] Among them, the second fusion branch 53 includes an adder and an activation function. The adder is used to fuse the two feature maps obtained after the feature map input to the second convolutional network unit is processed by the first second convolution branch 51 and the second second convolution branch 51 respectively, and activate the fused features through an activation function such as ReLu.

[0128] The convolution module in the third convolution branch 52 can be used to extract edge features and detail features of the image. It should be noted that the size of the convolution kernel in the convolution module can be a first size, such as a 3×3 convolution kernel, and the stride can be set to stride=1.

[0129] Among them, the third fusion branch 54 includes an adder, which is used to perform feature fusion on the feature map input to the second convolutional network unit, the feature map output after processing by the third second convolutional branch, and the feature map output by the third convolutional branch 52 to obtain a feature map processed by the second convolutional network unit.

[0130] In some embodiments, the first sub-network may further include: modifying a convolution module having a convolution kernel of a first size in the first sub-network into a repeatable convolution module having a multi-branch structure with convolution kernels of different sizes.

[0131] In an embodiment of the present application, the first size may be 3×3, and the convolution kernel in the first sub-network has the same input size and output size as the convolution module of the first size.

[0132] In an embodiment of the present application, the repeatable convolution module is a re-parameterized module, and the convolution module with a convolution kernel of a first size in the first sub-network is modified into a repeatable convolution module with a multi-branch structure having convolution kernels of different sizes. The multi-branch structure of the repeatable convolution module uses different convolution kernels to obtain different receptive fields, and adds the information obtained from different receptive fields to enhance the extraction of feature information and improve model performance. Therefore, when performing target detection, small targets at a long distance and large targets at a close distance can still effectively extract features.

[0133] In some embodiments, the repeatable convolution module includes a fourth convolution branch, a fifth convolution branch, a second direct connection branch, and a fourth fusion branch;

[0134] Among them, the fourth convolution branch, the fifth convolution branch and the second direct connection branch are connected in parallel and then cascaded with the fourth fusion branch; the fourth convolution branch is composed of a convolution module with a convolution kernel of a first size and a batch normalization module, and the fifth convolution branch is composed of a convolution module with a convolution kernel of a second size, and the first size and the second size are different.

[0135] For the repeatable convolution module, illustratively, refer to Figure 6 As shown, the repeatable convolution module 6 includes a fourth convolution branch 61, a fifth convolution branch 62, a second directly connected branch 63 and a fourth fusion branch 64; wherein, the fourth convolution branch 61, the fifth convolution branch 62 and the second directly connected branch 63 are connected in parallel and then cascaded with the fourth fusion branch 64.

[0136] Among them, the fourth convolution branch 61 is composed of a convolution module and a batch normalization module, and the size of the convolution kernel of the convolution module is a first size, which can be 3×3.

[0137] Among them, the fifth convolution branch 62 is composed of a convolution module, and the size of the convolution kernel of the convolution module is the second size, and the second size can be 1×1.

[0138] Among them, the fourth fusion branch 64 includes an adder, which is used to input the feature map of the repeatable convolution module 6 through the fourth convolution branch 61, the fifth convolution branch 62 and the second direct branch 63 respectively, and the feature map processed by the fourth convolution branch 61, the feature map processed by the fifth convolution branch 62, and the feature map processed by the second direct branch 63 are feature fused through the adder 641 to obtain the feature map processed by the repeatable convolution module 6.

[0139] From the above, we can see that when training a multi-task model, training is performed according to three branches, which makes the model training more comprehensive, strengthens the extraction of feature information, and improves model performance.

[0140] In a feasible scenario, continue to refer to Figure 4 , Figure 5 and Figure 6 , the repeatable convolution module 6 replaces the convolution kernels in the first convolution network unit 4 and the second convolution network unit 5 with a convolution module of a first size such as 3×3, to obtain a first repeatable convolution network unit 7, such as Figure 7 The left figure in , and the second reproducible convolutional network unit 8, as shown Figure 7 The right picture in the figure. When the multi-task model is trained, there are many branches in the entire network model, which makes the model training more complete. When the multi-task model is deployed for reasoning, the branches of the repeatable convolution module 6 are equivalently merged to obtain a 3×3 ordinary convolution, that is, the first repeatable convolutional network unit 7 and the second repeatable convolutional network unit 8 in the trained multi-task model are equivalently merged into the first convolutional network unit 4 and the second convolutional network unit 5. In this way, without losing the model performance, the model reasoning speed is greatly improved and the model reasoning time is reduced.

[0141] In some embodiments, after the model training is completed, the method also includes: after the multi-task model training is completed, according to the additivity of the convolution, the repeatable convolution modules are equivalently merged into a convolution module of the first size, thereby obtaining a trained multi-task model.

[0142] In the embodiment of the present application, after obtaining a trained multi-task model with a repeatable convolution module, before the model is actually deployed and used, it is necessary to merge the parameters of the repeatable convolution module to reduce the time consumption of model reasoning. Figure 6 As shown in the figure, the model structure during training contains a repeatable convolution module. According to the additivity of convolution, during model reasoning, the three branches are equivalently merged into a single-branch 3×3 convolution, that is, the repeatable convolution module is equivalently merged into a 3×3 ordinary convolution. In this way, the reasoning speed of the model is greatly improved without losing the model performance.

[0143] In some embodiments, step 105 combines the process of iteratively updating the network parameters of the multi-task model according to the first perception image and the first sample image corresponding to each subtask Figure 8 Provide explanation.

[0144] Step 801: for any one of the multiple subtasks, determine a target loss of the first perceived image of the subtask according to the first perceived image and the first sample image of the subtask;

[0145] In some embodiments, the method further includes: determining the image loss of the first perceived image of the subtask based on the first perceived image and the first sample image of the subtask; obtaining convolution kernels of the same size in the first sub-network; based on the orthogonal loss function, processing any two convolution kernels of the same size to obtain the orthogonal loss corresponding to the subtask; and determining the target loss based on the image loss and the orthogonal loss.

[0146] Here, the target loss includes image loss and orthogonal loss. The image loss is the loss between the first perception image and the sample image. The orthogonal loss is the orthogonal value between any two convolution kernels of the same size. The larger the orthogonal value, the more independent any two convolution kernels of the same size are. The orthogonal loss can be implemented by the following formula (1).

[0147] L ortho =Σ([W i T W j -I]) (1)

[0148] Among them, L ortho is the orthogonal loss, W i is the i-th convolution kernel in the network, W j For W iThe j-th convolution kernel has the same size, and I is the identity matrix.

[0149] It should be noted that by calculating the orthogonal loss, the parameters in the model can be forced to be orthogonal, thereby reducing the redundancy of the parameters in the model. According to the differences between the subtasks in multi-task learning, each convolution kernel can extract different features, thereby improving the overall performance of the model without increasing any inference time.

[0150] In an embodiment of the present application, by determining the image loss based on the first perceived image and the first sample image of the subtask, the prediction accuracy of the model on the subtask can be measured, and any two convolution kernels of the same size are processed to obtain the orthogonal loss, which can avoid excessive similarity between the convolution kernels, so that the model can better utilize different convolution kernels to extract different features, thereby enriching the feature space of the model, improving the expressiveness and accuracy of the model, and avoiding repeated feature extraction and calculation, reducing the waste of computing resources.

[0151] Step 802: determine the first gradient data, second gradient data, and third gradient data corresponding to the subtask according to the target loss of the subtask, thereby obtaining the first gradient data, second gradient data, and third gradient data corresponding to each subtask; wherein the first gradient data includes the gradient value of the network parameter of the first subnetwork, the second gradient data includes the gradient value of the network parameter of the second subnetwork shared by each subtask network in the associated task group to which the subtask belongs, and the third gradient data includes the gradient value of the network parameter of the subtask network corresponding to the subtask.

[0152] In the embodiment of the present application, the first gradient data is the network parameter of the subtask for the first subnetwork, which can accurately adjust the part responsible for multi-scale feature extraction to better adapt to the needs of the subtask and extract more targeted features.

[0153] In the embodiment of the present application, the second gradient data is the network parameters of the second sub-network shared by the sub-task in the associated task group, which helps to find a balanced optimization direction among multiple sub-tasks, ensuring the performance of each sub-task and making full use of the correlation between tasks for effective parameter updates.

[0154] In the embodiment of the present application, the third gradient data is the network parameters of the subtask network corresponding to the subtask, which can be fine-tuned specifically for the specific needs of the subtask to improve the accuracy and performance of the subtask.

[0155] In the embodiment of the present application, the computer device obtains the first gradient value of the network parameter of the first sub-network corresponding to the sub-task, the second gradient value of the network parameter of the second sub-network shared by each sub-task network in the associated task group to which the sub-task corresponding to the sub-task belongs, and the third gradient value of the network parameter of the sub-task network corresponding to the sub-task according to the target loss of a single sub-task through back propagation. In this way, by determining the target loss of the sub-task, the performance of the sub-task in the current model state can be accurately measured. The first gradient data, the second gradient data and the third gradient data of the network parameters of the sub-task corresponding to different networks are calculated according to the target loss, so that the model can be efficiently updated under the framework of multi-task learning. Different sub-tasks can be optimized independently according to their own target loss and corresponding gradient data, and at the same time, information interaction and collaborative optimization are carried out through the shared network part (the first sub-network and the second sub-network) to jointly improve the overall performance of the model.

[0156] Step 803: Iteratively update the network parameters of the multi-task model according to the first gradient data, the second gradient data, and the third gradient data corresponding to each subtask.

[0157] In some embodiments, the method further comprises:

[0158] Step 831: Adjust the network parameters of the first sub-network according to the first gradient data corresponding to each sub-task.

[0159] In an embodiment of the present application, after obtaining the first gradient data corresponding to each subtask, the following process can also be performed to obtain the first weight corresponding to each subtask; based on the first weight, the first gradient data of the network parameters of the first subnetwork are weighted averaged to obtain the first target gradient data of the network parameters of the first subnetwork, and the first target gradient data is input into the first subnetwork to complete the adjustment of the network parameters of the first subnetwork.

[0160] In the embodiment of the present application, the first weight is set based on the importance of the subtask. Each subtask may have different requirements and priorities in practical applications. The introduction of the first weight enables the model to make personalized adjustments to the first subnetwork according to the specific circumstances. For example, for certain key subtasks, a larger weight can be assigned to ensure that the shared subnetwork can extract features more accurately when processing these tasks to meet the high requirements of the tasks. For relatively minor subtasks, the weight can be appropriately reduced to avoid over-optimization on these tasks and affect the performance of other important tasks.

[0161] In the embodiment of the present application, the target gradient data of the network parameters of the calculation sub-network can be obtained by the following formula (2).

[0162]

[0163] Among them, grad is the target gradient data, m is the total number of subtasks, k is the kth subtask, grad is the gradient data of the network parameters, and weight is the weight of the subtask.

[0164] In the embodiment of the present application, after obtaining the first gradient data corresponding to each subtask and the first weight corresponding to each subtask, based on the first gradient data corresponding to each subtask and the first weight corresponding to each subtask, the first gradient data of the network parameters of the first subnetwork are weighted averaged using the above formula (2) to obtain the first target gradient data of the network parameters of the first subnetwork, and the first target gradient data are input into the first subnetwork to complete the adjustment of the network parameters of the first subnetwork. In this way, by obtaining the first weight corresponding to each subtask and performing weighted average processing on the first gradient data of the network parameters of the first subnetwork, the requirements of each subtask can be comprehensively considered, the balance of multi-task performance can be achieved, and the overall multi-task processing capability can be improved. Inputting the first target gradient data into the first subnetwork for parameter adjustment can make the first subnetwork better adapt to the multi-task environment and improve the adaptability and flexibility of the model. When the task requirements change or a new subtask is added, the first weight can be adjusted to quickly adapt to the new situation without retraining the entire model, saving time and computing resources.

[0165] Step 832: According to the second gradient data corresponding to each subtask, adjust the network parameters of the second sub-network shared by each sub-task network in the associated task group to which the subtask belongs.

[0166] In an embodiment of the present application, after obtaining the second gradient data corresponding to each subtask, the following process can also be performed: for each subtask, obtain the second weight of each subtask in the associated task group to which the subtask belongs; based on the second weight, perform weighted averaging on the second gradient data of the network parameters of the second subnetwork shared by each subtask network in the associated task group to obtain the second target gradient data of the network parameters of the second subnetwork shared by each subtask network in the associated task group, and input the second target gradient data into the shared second subnetwork to complete the adjustment of the network parameters of the second subnetwork.

[0167] In the embodiment of the present application, the second weight is the weight of each subtask in the associated task group to which the single subtask belongs.

[0168] In the embodiment of the present application, after obtaining the first gradient data corresponding to each subtask and the second gradient data corresponding to each subtask, based on the second weight of each subtask in the associated task group to which each subtask belongs, and the second gradient data corresponding to each subtask, the above formula (2) is used to perform weighted average processing on the second gradient data of the network parameters of the second subnetwork shared by each subtask network in the associated task group to which the subtask belongs, obtain the second target gradient data of the network parameters of the second subnetwork shared by each subtask network in the associated task group, and input the second target gradient data into the shared second subnetwork to complete the adjustment of the network parameters of the second subnetwork. In this way, for each subtask, by obtaining the second weight of each subtask in the associated task group to which it belongs, the correlation between different subtasks can be fully considered, the correlation between these tasks can be better utilized, and the learning efficiency of the model can be improved. The second gradient data of the network parameters of the second subnetwork shared by each subtask network in the associated task group is weighted averaged, which can balance the optimization requirements of different subtasks with high correlation on the shared network. Inputting the second target gradient data into the shared second subnetwork for parameter adjustment can make the second subnetwork better adapt to the combination and change of different subtasks. When a new subtask is added to the associated task group, it is only necessary to recalculate the second weight and perform weighted average processing to quickly adjust the shared network to adapt to the new task requirements.

[0169] Step 833: According to the third gradient data corresponding to each subtask, adjust the network parameters of the subtask network corresponding to the subtask.

[0170] In the embodiment of the present application, each subtask has its own unique task requirements and data characteristics. By adjusting the subtask network specifically through the third gradient data, accurate optimization of each subtask can be achieved. This enables the subtask network to better adapt to specific subtasks, extract features that better meet the requirements of the task, and improve the performance of the subtask.

[0171] Based on the above-mentioned multi-task model training method, a trained multi-task model is obtained. The embodiment of the present application also provides a multi-task model perception method. Figure 8 Schematic diagram of the process of the perception method of the multi-task model in the embodiment of the present application, such as Fig. 9 As shown, the method includes:

[0172] Step 901: Acquire a first image;

[0173] Step 902: Use the multi-task model to perform perception processing on the first image to obtain a task perception image corresponding to the first image.

[0174] In the embodiment of the present application, the first image is an image to be processed.

[0175] Among them, the multi-task model is any multi-task model obtained based on the training method of the multi-task model in the embodiment of the present application.

[0176] In an embodiment of the present application, a multi-task model is used to perform different tasks such as detection, recognition or segmentation on different objects in an image. The multi-task model can learn multiple related sub-tasks at the same time and fully explore the potential shared information between different sub-tasks.

[0177] In an embodiment of the present application, the multi-task model includes a first subnetwork, multiple second subnetworks and multiple subtask networks. Among them, the first subnetwork is a backbone network (Backbone) shared by all subtasks, and the first subnetwork is used to perform deep feature extraction of different scales on the image to be processed, and obtain rich semantic information and spatial feature information corresponding to the image to be processed at different scales. The second subnetwork is an intermediate layer network (Neck), which is used to perform feature fusion on the first feature map output by the first subnetwork, thereby upsampling the first feature map of the smaller image scale to obtain a second feature map with larger scale feature information, thereby providing the feature information required for tasks that rely on large-scale receptive fields. The subtask network is a head network (Head) private to each subtask, and its input is the second feature map output by the corresponding second subnetwork, and its output is the perception result corresponding to the subtask, such as the perceived image.

[0178] In an embodiment of the present application, a first image is acquired, and a multi-task model is used to perform perception processing on the first image to obtain a task perception image corresponding to the first image. The multi-task model can analyze the first image from multiple task types at the same time. Different tasks usually focus on different aspects of the image. This comprehensive information extraction can provide richer data support for subsequent applications. Whether it is image recognition, target detection, image segmentation or other image processing tasks, more accurate and detailed image features can be obtained from the perception processing of the multi-task model.

[0179] The following is an exemplary description of the training method of the multi-task module through a specific embodiment.

[0180] In related technologies, multi-task model training often relies on relatively complex data labeling, that is, all targets need to be labeled on a picture, which greatly increases the difficulty of labeling and brings huge labeling costs.

[0181] The application number is CN111882040A. The network is trained first, and the scale factor γ of the BN layer obtained by training is used as the importance index of the channel. The corresponding channels with importance indexes below the threshold are discarded, thereby achieving the purpose of compressing the model. This method only uses the scale factor γ of BN as the importance index. After each training, the scale factor of each BN will change, and the size of the cropped model will also be different. The threshold selection is also manually set based on experience, and it cannot guarantee that the effect after each cropping is effective.

[0182] The application number is CN113052189A. It uses grouped convolution and improved channel shuffling to generate a small number of main features, and then uses efficient operations to generate shadow features to ensure the richness and redundancy of features. The bottleneck structure in MobileNetV3 is replaced with this structure to obtain a model with low parameter and computational complexity. This method still uses a large number of depthwise separable convolutions. Although the number of parameters and computational complexity are reduced, memory access will increase, resulting in a relatively high latency in actual vehicle-side applications.

[0183] Application No. CN115860090A, in response to the high cost of multi-task data annotation, proposes a method of obtaining homologous data through model pre-brush. However, this method requires multiple steps for model training, which is cumbersome. If there are too many shared parameters and too few independent parameters, the effect of fine training of single-task parameters will be greatly reduced. If there are too few shared parameters and too many independent parameters, the time consumption of the model will also increase accordingly.

[0184] This application aims to solve the problems of limited computing resources on the vehicle side, high cost of labeling multi-task homologous data, and difficulty in obtaining them. A network model structure based on re-parameterization and a training method for a multi-task model are proposed. Fig.10 As shown, the method comprises the following steps:

[0185] Step 1001: Construct a network basic module.

[0186] It should be noted that, thanks to the powerful capabilities of convolutional neural networks in image recognition and processing, and the efficient performance of residual networks, this application is also constructed based on convolutional and residual modules.

[0187] Here, the basic modules BaseBlock_A (corresponding to the first convolutional network unit mentioned above) and BaseBlock_B (corresponding to the second convolutional network unit mentioned above) for the entire network are constructed. Both modules contain a re-parameterized module RepBlock (corresponding to the repeatable convolutional module mentioned above).

[0188] Among them, the basic module BaseBlock_A is for the case of stride=1. The input and output sizes of the feature map are consistent. It contains two convolutional layers and the corresponding BN layer, and only adds the ReLu activation function after the first convolutional layer, such as Fig.11 As shown in the figure, the first convolution layer is used to increase the number of feature channels, which is equivalent to the expansion layer in the network and is used to extract rich image features. The degree of channel expansion can be dynamically set by the α hyperparameter to obtain models with different parameter quantities and complexities to adapt to perception tasks of different difficulties. The second convolution layer is used to reduce the number of feature channels to match the output of the residual shortcut connection. It should be noted that the size of the convolution kernel in the convolution layer is 3×3.

[0189] Among them, the basic module BaseBlock_B is used for feature downsampling when the step length stride=2, that is, the output size of the feature map is 0.5 times the input size. Fig.12 As shown, the basic module BaseBlock_B contains a total of 4 convolutional layers and corresponding BN layers, as well as a Relu activation layer. The feature map is input into this module and is processed in three branches. Branches one and two extract features through a common convolution with stride=2, and then perform element-wise addition (elementwise add) to achieve feature fusion. After that, they are activated by a ReLu, and then through a common convolution with stride=1. After that, the output of branch three is added element-wise to achieve feature fusion. It should be noted that BaseBlock_B extracts relatively richer features than BaseBlock_A, but consumes more computing power than the latter, so BaseBlock_B is only used for the first module of each level stage. It should be noted that the size of the convolution kernel in the convolution layer is 3×3.

[0190] Thanks to the reparameterization technique that can merge convolutions of multiple branches into one convolution, a convolutional neural network is designed. Fig.13The re-parameterized module RepBlock shown is suitable for situations where the input and output sizes (height and width) are the same. It contains a 3×3 ordinary convolution, a 1×1 point convolution and an identity branch. When the model is trained, it is trained according to the three branches. When the model is exported after the training is completed, the three branches will be equivalently merged into a single-branch 3×3 convolution. Based on this, RepBlock replaces the 3×3 convolution in the above BaseBlock_A and BaseBlock_B to form BaseBlock_With_RepBlock_A (corresponding to the above-mentioned first repeatable convolutional network module) and BaseBlock_With_RepBlock_B (corresponding to the above-mentioned second repeatable convolutional network module), as shown Fig.14 As shown in the figure, when the model is trained, there are many branches in the entire network model, which makes the model training more complete. When the model is deployed for reasoning, the re-parameterized branches are equivalently merged, which greatly improves the reasoning speed of the model without losing model performance.

[0191] Step 1002: Build a multi-task model.

[0192] Here, based on the submodules designed in step 1, a network structure similar to Unet is designed, such as Fig.15 As shown, the multi-task model includes a backbone network (corresponding to the first sub-network mentioned above), a feature fusion network (neck) (corresponding to the second sub-network mentioned above) and a task head network (head) (corresponding to the sub-task network mentioned above), including a pedestrian detection task head and a lane line segmentation task head.

[0193] The backbone network is divided into 5 stages. The sub-blocks in each stage are composed of multiple BaseBlock_With_RepBlock_A and BaseBlock_With_RepBlock_B. The number of sub-blocks in each stage is set according to the importance and type of the task, as shown in Table 1. It should be noted that the first sub-block in each stage is set to BaseBlock_With_RepBlock_B because each stage needs to downsample the input feature map by two times so that the deeper stage can have a larger receptive field and thus detect large objects in the image.

[0194] Here, the feature fusion network is designed as two decoupled networks. Figure 3, which is used to adapt to the two different task types of pedestrian detection and lane segmentation in this solution, such as detection tasks and segmentation tasks. Among them, the neck network used for detection tasks uses element-by-element addition to perform feature fusion to integrate higher-level semantic information; the neck network used for segmentation tasks uses feature concatenation (concat) to perform feature fusion so that the segmentation task pays more attention to the original feature information.

[0195] Here, this application designs two task heads, continue to refer to Figure 3 , multiple pedestrian detection task heads and one lane segmentation task head. Pedestrian detection requires effective detection of both small targets at long distances and large targets at close range, so a detection head is connected to each of stages 3, 4, 5, and 6. The lane segmentation task requires more refined pixel classification, so only one segmentation head is connected to stage 2.

[0196] Step 1003: multi-task model training.

[0197] Here, two heterogeneous open source data sets are used as training data sets. For pedestrian detection, the coco data set is used; for lane segmentation, the CULane open source data set is used.

[0198] For heterogeneous data with different annotations, this application adopts a polling method for training, and is illustrated with a single batch of data. Exemplarily, a batch of data in the coco dataset of the pedestrian detection task is used for forward reasoning, the loss is calculated and back-propagated, and the gradient values ​​of the shared parameters of the backbone network corresponding to the pedestrian detection task, the gradient values ​​of the shared parameters of the feature fusion network used for the pedestrian detection task, and the gradient values ​​of the exclusive parameters of the pedestrian detection task are obtained and cached. Furthermore, a batch of data in the CULane open source dataset of the lane line segmentation task is used for forward reasoning, the loss is calculated and back-propagated, and the gradient values ​​of the shared parameters of the backbone network corresponding to the lane line segmentation task, the gradient values ​​of the shared parameters of the feature fusion network used for the lane line segmentation task, and the gradient values ​​of the exclusive parameters of the lane line segmentation task are obtained and cached. At this time, two cached gradient values ​​will be obtained. Then, according to the weights of the two tasks, the above formula (2) is used to perform weighted average on the gradient values ​​of the shared parameters of the backbone network and the gradient values ​​of the shared parameters of the feature fusion network, so as to calculate the final gradient value that needs to be updated for this model to complete this iterative update until the multi-task model training is completed.

[0199] It should be noted that, since each network basic module in step 1001 uses ordinary convolution, in order to reduce the redundancy of the parameters in the model, an orthogonal loss is introduced, which can be calculated by the above formula (1). Through this loss, the parameters in the model can be forced to be orthogonal; in this way, for the differences of each task in multi-task learning, each convolution kernel can extract different features, thereby improving the overall performance of the model without increasing any reasoning time.

[0200] Step 1004: Merge parameters to export the model.

[0201] Here, after the model training in step 1003, a trained multi-task model with a re-parameterized module is obtained. Before the multi-task model is actually deployed and used, the parameters of the re-parameterized module need to be merged to reduce the time consumption of model inference. Fig.16 As shown in the figure, the model structure during training contains a re-parameterized module. According to the additivity of convolution, the re-parameterized module is equivalently merged into a 3×3 ordinary convolution, that is, the BaseBlock_With_RepBlock_A and BaseBlock_With_RepBlock_B in the trained multi-task model are equivalently merged into BaseBlock_A and BaseBlock_B, thereby improving the model speed.

[0202] The present application embodiment provides a training device for a multi-task model, referring to Fig.15 As shown, Fig.15 A schematic diagram of the structure of a multi-task model training device provided in an embodiment of the present application, wherein the multi-task model training device 15 comprises:

[0203] A first acquisition module 1501 is used to acquire a training data set of multiple subtasks corresponding to the multi-task model, wherein the training data set at least includes a sample image;

[0204] The multi-task model processing module 1502 is used for inputting a first sample image in the image data set of any one of the multiple sub-tasks into the multi-task model, and using a first sub-network of the multi-task model to perform multi-scale feature extraction on the first sample image to obtain first feature maps of different scales;

[0205] The multi-task model processing module 1502 is further used to use a second sub-network shared by each sub-task network in the associated task group to which the sub-task belongs in the multi-task model to perform feature fusion processing on the first feature maps of different scales to obtain a second feature map; wherein the multi-task model includes at least two associated task groups, each associated task group includes at least one sub-task;

[0206] The multi-task model processing module 1502 is further used to process the second feature map using a sub-task network corresponding to a sub-task in the multi-task model to obtain a first perceived image of the sub-task output by the multi-task model, thereby obtaining a first perceived image of each sub-task;

[0207] The multi-task model processing module 1502 is also used to iteratively update the network parameters of the multi-task model according to the first perception image and the first sample image corresponding to each subtask, so as to obtain a trained multi-task model.

[0208] The present application embodiment provides a multi-task model perception device, referring to Fig.16 As shown, Fig.16 A schematic diagram of the structure of a perception device of a multi-task model provided in an embodiment of the present application, wherein the perception device 16 of the multi-task model includes:

[0209] The second acquisition module 1601 is used to acquire a first image;

[0210] The multi-task model processing module 1602 is used to use the multi-task model to perform perception processing on the first image to obtain a task perception image corresponding to the first image; wherein the multi-task model is obtained by the training method of the multi-task model of the above embodiment.

[0211] An embodiment of the present application provides a computer device, which may be a vehicle or a terminal. The computer device includes: a memory and a processor.

[0212] The memory stores a computer program executable on the processor;

[0213] When the processor executes the computer program, it implements part or all of the steps in the training method of the multi-task model or the perception method of the multi-task model as described in the above embodiments.

[0214] The embodiment of the present application provides a computer-readable storage medium, which stores one or more computer programs, and the one or more computer programs can be executed by one or more processors to implement some or all of the steps in the above method. The storage medium can be transient or non-transient.

[0215] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0216] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0217] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0218] A hardware entity diagram of a computer device provided in an embodiment of the present application, the computer device may be a vehicle or a terminal, such as Fig.17 As shown, the hardware entity of the computer device 17 includes: a processor 1701 and a memory 1702, wherein the memory 1702 stores a computer program that can be run on the processor 1701, and the processor 1701 executes part or all of the steps in the above-mentioned training method of the multi-task model or the perception method of the multi-task model.

[0219] Among them, the memory 1702 stores a computer program that can be run on the processor. The memory 1702 is configured to store instructions and applications executable by the processor 1701. It can also cache data to be processed or processed by the processor 1701 and various modules in the computer device 17 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0220] The processor 1701 executes the program to implement any of the steps of the multi-task model training method or the multi-task model perception method described above. The processor 1701 generally controls the overall operation of the computer device 17 .

[0221] An embodiment of the present application provides a storage medium, which stores one or more computer programs. The one or more computer programs can be executed by one or more processors to implement the steps of the multi-task model training method or the multi-task model perception method of any of the above embodiments.

[0222] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0223] The processor may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device that implements the functions of the processor may also be other, and the embodiments of the present application are not specifically limited.

[0224] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and the like; it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0225] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0226] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0227] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0228] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple convolutional network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0229] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0230] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0231] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a vehicle-mounted terminal (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0232] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A training method for a multi-task model, characterized in that: The method comprises: Acquire a training data set of multiple subtasks corresponding to the multi-task model, wherein the training data set at least includes a sample image; For a training data set of any one of the multiple subtasks, input a first sample image in the image data set of the subtask into the multi-task model, and use a first sub-network of the multi-task model to perform multi-scale feature extraction on the first sample image to obtain first feature maps of different scales; Using a second sub-network shared by each sub-task network in the associated task group to which the sub-task belongs in the multi-task model, feature fusion processing is performed on the first feature maps of different scales to obtain a second feature map; wherein the multi-task model includes at least two associated task groups, and each associated task group includes at least one sub-task; Processing the second feature map using a subtask network corresponding to the subtask in the multitask model to obtain a first perceived image of the subtask output by the multitask model, thereby obtaining a first perceived image of each subtask; According to the first perception image and the first sample image corresponding to each of the subtasks, the network parameters of the multi-task model are iteratively updated to obtain a trained multi-task model.

2. The multi-task model training method according to claim 1, characterized in that: The first sub-network includes multiple feature extraction sub-modules for extracting features of different scales. The multiple feature extraction sub-modules are connected in series. The feature extraction sub-module includes a first number of first convolutional network units and a second number of second convolutional network units. The second number of second convolutional network units is cascaded with the first number of second convolutional network units. The first convolutional network unit and the second convolutional network unit are set for different step sizes.

3. The training method of the multi-task model according to claim 2, characterized in that: The second number is 1, and the second convolutional network unit is located at the first position of the feature extraction submodule.

4. The multi-task model training method according to claim 2, characterized in that: The first convolutional network unit includes a first convolution branch, a first directly connected branch and a first fusion branch; wherein the first convolution branch and the first directly connected branch are connected in parallel and then cascaded with the first fusion branch, the first convolution branch is composed of a plurality of convolution modules, a plurality of batch normalization modules and an activation function, and the first fusion branch includes an adder; The second convolutional network unit includes at least two second convolutional branches, a third convolutional branch, a second fusion branch and a third fusion branch; wherein the second convolutional branch of the first part is connected in parallel and then cascaded with the second fusion branch and the third convolutional branch, the second convolutional branch of the second part is connected in parallel with the second convolutional branch, the second fusion branch and the third convolutional branch of the first part, the second convolutional branch and the third convolutional branch are both composed of a convolution module and a batch normalization module, the second fusion branch includes an adder and an activation function, and the third fusion branch includes an adder.

5. The multi-task model training method according to claim 4, characterized in that: The method further comprises: The convolution module with a convolution kernel of a first size in the first sub-network is modified into a repeatable convolution module with a multi-branch structure having convolution kernels of different sizes.

6. The multi-task model training method according to claim 5, characterized in that: The repeatable convolution module includes a fourth convolution branch, a fifth convolution branch, a second direct connection branch and a fourth fusion branch; Among them, the fourth convolution branch, the fifth convolution branch and the second directly connected branch are connected in parallel and then cascaded with the fourth fusion branch; the fourth convolution branch is composed of a convolution module with a convolution kernel of the first size and a batch normalization module, and the fifth convolution branch is composed of a convolution module with a convolution kernel of the second size, and the first size is different from the second size.

7. The multi-task model training method according to claim 5, characterized in that: The method further comprises: After the multi-task model is trained, the repeatable convolution modules are equivalently merged into a convolution module of the first size according to the additivity of convolution, thereby obtaining the trained multi-task model.

8. The multi-task model training method according to any one of claims 1 to 7, characterized in that: The iterative updating of the network parameters of the multi-task model according to the first perception image and the first sample image corresponding to each of the subtasks includes: For any one of the plurality of subtasks, determining a target loss of the first perceived image of the subtask according to the first perceived image of the subtask and the first sample image; According to the target loss of the subtask, the first gradient data, the second gradient data and the third gradient data corresponding to the subtask are determined, so as to obtain the first gradient data, the second gradient data and the third gradient data corresponding to each subtask; wherein the first gradient data includes the gradient value of the network parameter of the first sub-network, the second gradient data includes the gradient value of the network parameter of the second sub-network shared by each sub-task network in the associated task group to which the subtask belongs, and the third gradient data includes the gradient value of the network parameter of the sub-task network corresponding to the subtask; The network parameters of the multi-task model are iteratively updated according to the first gradient data, the second gradient data and the third gradient data corresponding to each subtask.

9. The multi-task model training method according to claim 8, characterized in that: The determining, according to the first perceived image of the subtask and the first sample image, a target loss of the first perceived image of the subtask comprises: determining an image loss of the first perceived image of the subtask according to the first perceived image of the subtask and the first sample image; Obtain convolution kernels with the same size in the first sub-network; Based on the orthogonal loss function, any two convolution kernels of the same size are processed to obtain the orthogonal loss corresponding to the subtask; The target loss is determined based on the image loss and the orthogonal loss.

10. The multi-task model training method according to claim 8, characterized in that: The adjusting the network parameters of the multi-task model according to the first gradient data, the second gradient data and the third gradient data corresponding to each subtask includes: Adjusting the network parameters of the first sub-network according to the first gradient data corresponding to each sub-task; According to the second gradient data corresponding to each subtask, adjusting the network parameters of the second sub-network shared by each sub-task network in the associated task group to which the subtask belongs; According to the third gradient data corresponding to each subtask, the network parameters of the subtask network corresponding to the subtask are adjusted.

11. The multi-task model training method according to claim 10, characterized in that: The adjusting the network parameters of the first sub-network according to the first gradient data corresponding to each sub-task includes: Get the first weight corresponding to each subtask; Based on the first weight, weighted averaging is performed on the first gradient data of the network parameters of the first sub-network to obtain first target gradient data of the network parameters of the first sub-network, and the first target gradient data is input into the first sub-network to complete the adjustment of the network parameters of the first sub-network.

12. The multi-task model training method according to claim 10, characterized in that: The adjusting, according to the second gradient data corresponding to each subtask, the network parameters of the second sub-network shared by each sub-task network in the associated task group to which the subtask belongs includes: For each subtask, obtaining a second weight of each subtask in the associated task group to which the subtask belongs; Based on the second weight, weighted averaging is performed on the second gradient data of the network parameters of the second subnetwork shared by each subtask network in the associated task group to obtain the second target gradient data of the network parameters of the second subnetwork shared by each subtask network in the associated task group, and the second target gradient data is input into the shared second subnetwork to complete the adjustment of the network parameters of the second subnetwork.

13. A perception method of a multi-task model, characterized in that: The method comprises: acquiring a first image; Using a multi-task model to perform perception processing on the first image to obtain a task perception image corresponding to the first image; Wherein, the multi-task model is obtained based on the training method of the multi-task model described in any one of claims 1 to 12.

14. A computer device, characterized in that: The computer device comprises a memory and a processor, The memory stores a computer program executable on the processor; When the processor executes the computer program, the training method of the multi-task model as described in any one of claims 1 to 12, or the perception method as described in claim 13 is implemented.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more computer programs, and the one or more computer programs can be executed by one or more processors to implement the training method of the multi-task model as described in any one of claims 1 to 12, or the perception method as described in claim 13.

16. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the training method of the multi-task model as described in any one of claims 1 to 12, or the perception method as described in claim 13 is implemented.

Images