Method for training a multi-task object detection model, method for multi-task object detection

Abstract

The application is suitable for the field of artificial intelligence, and particularly relates to a method for training a multi-task target detection model and a multi-task target detection method. The training method fuses special and general data, uses multi-task joint learning and domain alignment technology, and cooperatively optimizes helmet, head-shoulder, pedestrian and vehicle detection tasks in a unified framework. By minimizing the total loss containing the task prediction difference and the domain distribution difference, the model learns domain-invariant features, thereby reducing computational redundancy, significantly improving cross-scene detection accuracy and generalization ability, improving the training efficiency of the model and the accuracy of the prediction results output when the model is used to perform a prediction task. The detection method inputs a traffic scene image into the trained multi-task target detection model to obtain the detection results of multiple tasks at one time. The detection accuracy, generalization and real-time performance are considered.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and particularly relates to a method for training a multi-task target detection model, a multi-task target detection method, and an electronic device. BACKGROUND

[0002] In complex road scene scenarios such as expressways and urban expressways, a deep learning model is used for target detection, aiming to realize accurate and real-time perception of traffic participants (such as pedestrians, cyclists, vehicles, etc.), thereby providing key data support for intelligent traffic management, event warning, and automatic driving and other upper-layer applications.

[0003] In related technologies, a dedicated deep learning model is designed and trained for each specific detection task (such as helmet wearing detection, pedestrian detection, vehicle detection, head and shoulder detection, etc.), and these models are simultaneously deployed in actual applications.

[0004] However, training multiple models independently for different tasks will result in a large amount of repeated calculations, especially in basic links such as feature extraction, causing waste of computing resources and reduction of system efficiency. Secondly, in complex road conditions, independent learning of each task model cannot capture the inherent and rich context relationships between different traffic participants. For example, the helmet wearing detection task and the head and shoulder detection task are closely coupled in semantics and share local features of the head region; and pedestrians, cyclists, and vehicles commonly appear in real traffic flow, and their spatial positions and interaction relationships constitute important scene semantic information. Due to the inability to effectively utilize these cross-task context associations, the detection capability of a single model will be greatly reduced when the target is occluded, the lighting condition is poor, or the scene is complex, resulting in low accuracy and robustness of the overall detection, and it is difficult to meet the needs of actual applications. SUMMARY

[0005] The embodiments of the present application provide a method for training a multi-task target detection model, a method for multi-task target detection, and an electronic device, which can simultaneously train a model for performing multiple target detection tasks, and train in combination with multi-source data (special-purpose data and general-purpose data), so that the model can combine scene context information for prediction, improve the training efficiency of the model, and improve the accuracy of the prediction results output when the model is used to perform a prediction task.

[0006] In a first aspect, the embodiments of the present application provide a method for training a multi-task target detection model, which comprises:

[0007] The training data set is obtained, and the training data set includes special data and general data. The special data is image data collected for a specified traffic road scene, and the general data is image data collected for multiple traffic road scenes. The specified traffic road scene and the multiple traffic road scenes have different environmental characteristics. The special data is labeled with a first label and a second label, and the general data is labeled with a third label and a fourth label. The first label corresponds to a first task, the second label corresponds to a second task, the third label corresponds to a third task, and the fourth label corresponds to a fourth task.

[0008] In each round of training, the training data set is input into the multi-task object detection model, and the multi-task object detection model performs the following steps: performing feature extraction and fusion processing on the training data set to obtain a first feature data set corresponding to the special data and a second feature data set corresponding to the general data; mapping the first feature data set and the second feature data set to the same feature dimension space to obtain a third feature data set, and determining a distribution difference between the feature distribution of the special data and the feature distribution of the general data according to the third feature data set; performing object detection on each data in the third feature data set to obtain a first prediction result corresponding to the first task, a second prediction result corresponding to the second task, a third prediction result corresponding to the third task, and a fourth prediction result corresponding to the fourth task.

[0009] A total loss value is obtained according to a first difference between the first prediction result and the first label, a second difference between the second prediction result and the second label, a third difference between the third prediction result and the third label, a fourth difference between the fourth prediction result and the fourth label, and the distribution difference, and the parameters of the multi-task object detection model are updated to minimize the total loss value to obtain the trained multi-task object detection model.

[0010] In the technical scheme, by obtaining the training data set containing special data and general data, and processing labels corresponding to different tasks in the same model, the intrinsic mechanism realizes feature sharing, that is, multiple tasks share the low-level features of the same backbone network, thereby reducing the parameter amount and improving the training efficiency. At the same time, by using the differences between the prediction results of different tasks and the corresponding labels, and the distribution difference between the special data and the general data to update the model, the model can have the ability to process multiple tasks at the same time, and the complementary relationship between the tasks is used to improve the accuracy of the detection result. In the training phase, the model learns the domain-invariant features between the general data and the special data by minimizing the distribution difference, thereby establishing a semantic bridge between the two. This makes the model pay more attention to the essential features in the data rather than the data source when performing a specific task, ultimately improving the robustness of the model, and realizing the effect of deploying only one model in the traffic monitoring system to output multiple detection results.

[0011] With reference to the first aspect, in a possible implementation manner of the first aspect, the multi-task target detection model comprises a backbone feature extraction layer, a multi-scale feature fusion layer, a domain feature alignment layer, a first detection head, a second detection head, a third detection head, and a fourth detection head. The backbone feature extraction layer comprises a local feature backbone, a global semantic backbone, and an attention module. The local feature backbone is configured to extract local texture features of each training data in the training data set. The global semantic backbone is configured to extract global context information of each training data in the training data set. The attention module is configured to perform attention fusion on the local texture features and the global context information. The multi-scale feature fusion layer is configured to generate feature maps at different scales based on an initial feature data set output by the backbone feature extraction layer, to obtain a first feature data set and a second feature data set. The domain feature alignment layer comprises a feature projection layer, a domain classifier, and a gradient inversion layer. The feature projection layer is configured to map the first feature data set and the second feature data set to the same feature dimension space to obtain a third feature data set. The domain classifier is configured to determine whether each feature data in the third feature data set belongs to special data or general data. The gradient inversion layer is configured to determine a distribution difference between a feature distribution of the special data and a feature distribution of the general data based on a determination result output by the domain classifier. The first detection head is configured to perform target detection on the third feature data set to obtain a first prediction result. The second detection head is configured to perform target detection on the third feature data set to obtain a second prediction result. The third detection head is configured to perform target detection on the third feature data set to obtain a third prediction result. The fourth detection head is configured to perform target detection on the third feature data set to obtain a fourth prediction result.

[0012] In this implementation manner, a specific model architecture is introduced. By setting a double-backbone feature fusion structure comprising a local feature backbone and a global semantic backbone, the effect of considering both local texture and global context information is achieved, thereby significantly improving the small target recognition rate in complex scenes. By setting a domain feature alignment layer and using the domain classifier and gradient inversion layer therein to determine the distribution difference, domain adaptive alignment is achieved, solving the problem of distribution difference between private and general data, so that the model maintains stable detection performance under different light, angle, and environmental conditions. By setting multiple independent detection heads, a structural basis is provided for task complementation and deployment unification.

[0013] With reference to the first aspect, in a possible implementation manner of the first aspect, the feature extraction and fusion processing on the training data set to obtain the first feature data set corresponding to the special data and the second feature data set corresponding to the general data comprises:

[0014] The local feature backbone is configured to extract local texture features of each training data in the training data set. The global semantic backbone is configured to extract global context information of each training data in the training data set. The attention module is configured to perform attention fusion on the local texture features and the global context information. The multi-scale feature fusion layer is configured to generate feature maps at different scales based on an initial feature data set output by the backbone feature extraction layer, to obtain a first feature data set and a second feature data set. The domain feature alignment layer comprises a feature projection layer, a domain classifier, and a gradient inversion layer. The feature projection layer is configured to map the first feature data set and the second feature data set to the same feature dimension space to obtain a third feature data set. The domain classifier is configured to determine whether each feature data in the third feature data set belongs to special data or general data. The gradient inversion layer is configured to determine a distribution difference between a feature distribution of the special data and a feature distribution of the general data based on a determination result output by the domain classifier. The first detection head is configured to perform target detection on the third feature data set to obtain a first prediction result. The second detection head is configured to perform target detection on the third feature data set to obtain a second prediction result. The third detection head is configured to perform target detection on the third feature data set to obtain a third prediction result. The fourth detection head is configured to perform target detection on the third feature data set to obtain a fourth prediction result.

[0015] extracting context features from each training data in the training data set respectively through the global semantic stem to obtain a global feature data set corresponding to the training data set, the global feature data set including global context information of each training data;

[0016] For each training data, the following steps are performed to obtain an initial feature data set corresponding to the training data set: splicing the local texture features and the global context information corresponding to the training data to obtain a spliced feature map; adjusting the attention weight of the spliced feature map through an attention module to obtain initial feature data corresponding to the training data; the initial feature data set includes initial feature data corresponding to each training data;

[0017] processing the initial feature data set through a multi-scale feature fusion layer to generate feature maps at different scales, to obtain a first feature data set and a second feature data set.

[0018] In this implementation, the specific steps of feature extraction and fusion are refined, and how to efficiently generate high-quality feature representations is clarified. By extracting local texture features and global context information in parallel and using an attention module for fusion, the model can balance details and macro information, which is crucial for identifying small targets in complex scenes. This structured feature generation method significantly improves the expressiveness of features compared to a single stem, providing better inputs for subsequent multi-task detection and domain alignment, and is a key step to realize the advantages of dual-stem feature fusion.

[0019] In combination with the first aspect, in a possible implementation of the first aspect, the first feature data set and the second feature data set are mapped to the same feature dimension space to obtain a third feature data set, and a distribution difference between a feature distribution of the special-purpose data and a feature distribution of the general-purpose data is determined according to the third feature data set, including:

[0020] mapping the first feature data set and the second feature data set to the same feature dimension space through a feature projection layer to obtain a third feature data set;

[0021] judging the type of each feature data in the third feature data set through a domain classifier to obtain a judgment result, the judgment result being used to guide the division of the third feature data set into a first subset and a second subset, the first subset being composed of feature data belonging to the special-purpose data, and the second subset being composed of feature data belonging to the general-purpose data;

[0022] analyzing the feature distribution of the first subset and the second subset according to the judgment result output by the domain classifier through a gradient inversion layer to determine the distribution difference.

[0023] In this implementation, the domain alignment process is described in detail, the dimensions are unified through the feature projection layer, the domain classifier discriminates, and the gradient inversion layer generates a distribution difference value for optimization according to the discrimination result, forming a closed-loop, adaptive domain alignment mechanism. This design enables the model to actively learn how to generate "domain-independent" features, effectively overcoming the generalization bottleneck caused by data distribution differences, and is the core guarantee for realizing domain adaptive alignment and improving the robustness of the model in unknown environments.

[0024] With reference to the first aspect, in a possible implementation of the first aspect, the first task is a helmet wearing detection task, the second task is a head-shoulder detection task, the third task is a pedestrian detection task, and the fourth task is a vehicle detection task.

[0025] The target detection is performed on each data in the third feature data set respectively to obtain a first prediction result corresponding to the first task, a second prediction result corresponding to the second task, a third prediction result corresponding to the third task, and a fourth prediction result corresponding to the fourth task, including:

[0026] The target detection is performed on the third feature data set by the first detection head to obtain the first prediction result; the first prediction result includes a first bounding box, a first classification result, and a first confidence, the first classification result is used to indicate a target type at the first bounding box, and the first confidence is used to indicate whether a target exists at the first position;

[0027] The target detection is performed on the third feature data set by the second detection head to obtain the second prediction result; the second prediction result includes a second bounding box, a second classification result, and a second confidence, the second classification result is used to indicate a target type at the second bounding box, and the second confidence is used to indicate whether a target exists at the second position;

[0028] The target detection is performed on the third feature data set by the third detection head to obtain the third prediction result; the third prediction result includes a third bounding box, a third classification result, and a third confidence, the third classification result is used to indicate a target type at the third bounding box, and the third confidence is used to indicate whether a target exists at the third position;

[0029] The target detection is performed on the third feature data set by the fourth detection head to obtain the fourth prediction result; the fourth prediction result includes a fourth bounding box, a fourth classification result, and a fourth confidence, the fourth classification result is used to indicate a target type at the fourth bounding box, and the fourth confidence is used to indicate whether a target exists at the fourth position.

[0030] In this implementation, the abstract task is combined with the specific traffic scene detection task (helmet wearing, head and shoulder, pedestrian, vehicle), which clearly defines the actual application scenario and functional boundary of the model. By defining the specific output of each detection head (bounding box, classification result, confidence), it clearly shows how the model converts complex visual information into structured data that can be directly used by subsequent applications.

[0031] In combination with the first aspect, in a possible implementation of the first aspect, the first difference value is composed of a first classification difference value, a first bounding box difference value, and a first confidence difference value; the second difference value is composed of a second classification difference value, a second bounding box difference value, and a second confidence difference value; the third difference value is composed of a third classification difference value, a third bounding box difference value, and a third confidence difference value; the fourth difference value is composed of a fourth classification difference value, a fourth bounding box difference value, and a fourth confidence difference value.

[0032] The total loss value is obtained according to the first difference value between the first prediction result and the first label, the second difference value between the second prediction result and the second label, the third difference value between the third prediction result and the third label, the fourth difference value between the fourth prediction result and the fourth label, and the distribution difference value, and the parameters of the multi-task target detection model are updated to minimize the total loss value to obtain the trained multi-task target detection model, including:

[0033] The first weight difference value is obtained according to the weighted sum of the first classification difference value, the first bounding box difference value, and the first confidence difference value.

[0034] The second weight difference value is obtained according to the weighted sum of the second classification difference value, the second bounding box difference value, and the second confidence difference value.

[0035] The third weight difference value is obtained according to the weighted sum of the third classification difference value, the third bounding box difference value, and the third confidence difference value.

[0036] The fourth weight difference value is obtained according to the weighted sum of the fourth classification difference value, the fourth bounding box difference value, and the fourth confidence difference value.

[0037] The total loss value is obtained according to the weighted sum of the first weight difference value, the second weight difference value, the third weight difference value, the fourth weight difference value, and the distribution difference value, and the parameters of the multi-task target detection model are updated to minimize the total loss value to obtain the trained multi-task target detection model.

[0038] In this implementation, the design of the loss function is introduced, providing an optimized and convergent training target. By decomposing the difference value of each task into three sub-differences of classification, bounding box and confidence, and performing weighted summation, and then combining with the distribution difference to form the total loss, the learning process of the model is finely controlled. This multi-task, multi-objective joint optimization strategy ensures that the model improves the detection accuracy of each task while also considering the domain alignment effect, ultimately achieving the optimal overall performance, and achieving the effect of collaborative optimization of multi-source task joint detection structure.

[0039] In combination with the first aspect, in a possible implementation of the first aspect, the multi-scale feature fusion layer adopts a bidirectional feature pyramid structure, and each initial feature data in the initial feature data set includes original output features output by the backbone feature extraction layer at different scales;

[0040] The initial feature data set is processed by the multi-scale feature fusion layer to generate feature maps at different scales, obtaining the first feature data set and the second feature data set, including:

[0041] For each initial feature data, the fusion output features of the initial feature data at different scales are obtained as the feature maps of the initial feature data at different scales; wherein when obtaining the fusion output features at the i-th scale, the following steps are performed: performing down-sampling processing on the fusion output features at the (i-1)-th scale to obtain a down-sampling result, i is a positive integer; performing up-sampling processing on the fusion output features at the (i+1)-th scale to obtain an up-sampling result; and performing weighted fusion on the down-sampling result, the up-sampling result and the original output features at the i-th scale to obtain the fusion output features of the initial feature data at the i-th scale as the feature map at the i-th scale;

[0042] The first feature data set and the second feature data set are obtained according to the fusion output features of each initial feature data in the initial feature data set at different scales.

[0043] In this implementation, the specific implementation of the multi-scale feature fusion layer is introduced, providing an efficient and intelligent feature fusion mechanism. A bidirectional feature pyramid structure is adopted, and up-sampling, down-sampling and weighted fusion are performed on features at different scales, so that the model can adaptively integrate information from different receptive fields. This design greatly improves the detection capability of the model for cross-scale targets, especially small targets, so that the model learns data features at different scales and improves the small target detection effect.

[0044] In the second aspect, the embodiments of the present application provide a multi-task target detection method, which is executed by the multi-task target detection model trained in the first aspect, and the method includes:

[0045] obtaining a to-be-detected image, the to-be-detected image being an image collected from a traffic road scene;

[0046] inputting the to-be-detected image into the multi-task target detection model to obtain a first detection result, a second detection result, a third detection result and a fourth detection result, the first detection result being a result obtained by performing a helmet wearing detection task on the to-be-detected image, the second detection result being a result obtained by performing a head-shoulder detection task on the to-be-detected image, the third detection result being a result obtained by performing a pedestrian detection task on the to-be-detected image, and the fourth detection result being a result obtained by performing a vehicle detection task on the to-be-detected image.

[0047] In the technical solution, the inference application method of the model is introduced, the trained model is directly associated with the actual application scenario, and a complete closed loop from training to deployment is formed. The role of the model is defined, that is, the corresponding detection result is output for different tasks, and the specific meanings of the results in the traffic scene are emphasized.

[0048] In combination with the second aspect, in a possible implementation manner of the second aspect, at least one detection result is selected from the first detection result, the second detection result, the third detection result and the fourth detection result according to a requirement of a detection business.

[0049] In this implementation manner, an optional post-processing step is added, which embodies the flexibility and practicality of the model in actual application. The model is not a fixed system, and its output can be dynamically selected and combined according to specific business requirements (such as only caring about vehicles or only caring about pedestrians). The capability of on-demand output enables the model to serve as a general front-end perception module, which can be flexibly adapted to various intelligent traffic application systems, greatly widening the applicable range and market prospect of the model.

[0050] In a third aspect, an embodiment of the present application provides a device for training a multi-task target detection model, comprising:

[0051] a data acquisition module configured to acquire a training data set, the training data set comprising special data and general data, the special data being image data collected from a specified traffic road scene, the general data being image data collected from multiple types of traffic road scenes, the specified traffic road scene and the multiple types of traffic road scenes having different environmental characteristics, the special data being labeled with a first label and a second label, and the general data being labeled with a third label and a fourth label; the first label corresponding to a first task, the second label corresponding to a second task, the third label corresponding to a third task, and the fourth label corresponding to a fourth task;

[0052] The training module is configured to input the training data set into the multi-task object detection model in each round of training, and perform the following steps by the multi-task object detection model: performing feature extraction and fusion processing on the training data set to obtain a first feature data set corresponding to the special data and a second feature data set corresponding to the general data; mapping the first feature data set and the second feature data set to the same feature dimension space to obtain a third feature data set, and determining a distribution difference between a feature distribution of the special data and a feature distribution of the general data according to the third feature data set; and performing object detection on each data in the third feature data set respectively to obtain a first prediction result corresponding to the first task, a second prediction result corresponding to the second task, a third prediction result corresponding to the third task, and a fourth prediction result corresponding to the fourth task.

[0053] The training module is further configured to obtain a total loss value according to a first difference between the first prediction result and the first label, a second difference between the second prediction result and the second label, a third difference between the third prediction result and the third label, a fourth difference between the fourth prediction result and the fourth label, and the distribution difference, and update parameters of the multi-task object detection model to obtain the trained multi-task object detection model.

[0054] In a fourth aspect, an embodiment of the present application provides a multi-task object detection device, which comprises:

[0055] The data acquisition module is configured to acquire a to-be-detected image, the to-be-detected image being an image obtained by collecting a traffic road scene;

[0056] The detection module is configured to input the to-be-detected image into the multi-task object detection model to obtain a first detection result, a second detection result, a third detection result, and a fourth detection result, the first detection result being a result obtained by performing a helmet wearing detection task on the to-be-detected image, the second detection result being a result obtained by performing a head-shoulder detection task on the to-be-detected image, the third detection result being a result obtained by performing a pedestrian detection task on the to-be-detected image, and the fourth detection result being a result obtained by performing a vehicle detection task on the to-be-detected image, the multi-task object detection model being trained by using the training method of the first aspect.

[0057] In a fifth aspect, the present application further provides an electronic device. The electronic device comprises a memory, one or more processors, and a computer program stored in the memory and executable on the processors. The electronic device implements the method of any one of the implementation manners of the first aspect when executing the computer program, or the electronic device implements the method of any one of the implementation manners of the second aspect when executing the computer program.

[0058] In a fifth aspect, the present application provides a training device. The training device comprises a memory, one or more processors, and a computer program stored in the memory and executable on the processors. The training device implements the method of any of the implementations of the first aspect when executing the computer program.

[0059] In a seventh aspect, the present application provides an inference device. The inference device comprises a memory, one or more processors, and a computer program stored in the memory and executable on the processors. The inference device implements the method of any of the implementations of the second aspect when executing the computer program.

[0060] In an eighth aspect, the present application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the method of any of the implementations of the first aspect.

[0061] In a ninth aspect, the present application provides a computer program product, which, when executed on an electronic device, causes the electronic device to perform the method of any of the implementations of the first aspect.

[0062] It can be understood that the beneficial effects of the third to seventh aspects described above can be referred to the related description in the first and second aspects, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0064] Figure 1 is an architecture diagram of a multi-task target detection model provided by an embodiment of the present application;

[0065] Figure 2 is a process schematic diagram of multi-task target detection using a multi-task target detection model provided by an embodiment of the present application;

[0066] Figure 3 is a flowchart of a method for training a multi-task target detection model provided by an embodiment of the present application;

[0067] Figure 4 is a flowchart of a method for multi-task target detection provided by an embodiment of the present application;

[0068] Figure 5 is a structural block diagram of an apparatus for training a multi-task target detection model provided by an embodiment of the present application;

[0069] Figure 6 is a structural block diagram of a multi-task target detection device provided by an embodiment of the present application;

[0070] Figure 7 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0071] In a complex road condition scene such as a highway and an urban expressway, a deep learning model is used for target detection, aiming to realize accurate and real-time perception of traffic participants (such as pedestrians, cyclists, vehicles, etc.), thereby providing key data support for intelligent traffic management, event warning, and automatic driving and other upper-layer applications.

[0072] In the related art, a dedicated deep learning model is designed and trained for each specific detection task, and these models are deployed simultaneously in actual application. However, this paradigm of training a model for each detection task has significant defects.

[0073] Firstly, there is serious feature redundancy among different tasks, and the model needs to repeatedly learn edge, texture and other low-level visual features, resulting in waste of computing resources and low training efficiency. Secondly, a single-task model lacks understanding of the overall traffic scene, and is difficult to capture the spatial and semantic relationships between different traffic participants. For example, in a traffic road scene, the relative position relationship between pedestrians and vehicles, and the coexistence relationship between cyclists and helmets will affect the model's accurate judgment of the target state, thereby limiting the detection accuracy and robustness in complex scenes.

[0074] For example, when a target (such as a helmet) is partially occluded or too small in size, a single-task model lacks context information and can only rely on local features to make an accurate judgment, which is likely to result in missed detection. For example, a single-task model cannot understand the complex scene semantics containing spatial relationships such as "a pedestrian is about to walk out from in front of a bus", it can only report "there is a pedestrian" and "there is a bus" in isolation, and cannot associate the two to form a prediction of "dangerous event". This lack of understanding of the overall scene makes the model's detection performance drop sharply and lack robustness when facing complex situations such as sudden changes in light, target density, and frequent interaction.

[0075] Moreover, in practical applications, it is often necessary to fuse special data and general data. Special data is usually collected for specific business needs, with the characteristics of single scene and fine annotation. While general data is rich in scene and wide in coverage, but lacks fine-grained annotation for specific tasks. These two types of data have significant distribution differences in terms of lighting, viewing angle, and environment, i.e., domain shift problem. If they are directly mixed for training, the model will learn non-essential features related to the data source, resulting in good performance on specific datasets, but poor generalization ability when facing new and unseen environments, which cannot meet the requirements of stability and reliability in actual deployment.

[0076] Therefore, how to build a visual perception framework that can uniformly process multiple types of target detection tasks and effectively fuse multi-source heterogeneous data and overcome domain shift problems has become a key bottleneck to improve the performance of intelligent transportation systems. To solve the above problems, a method for training a multi-task target detection model and a multi-task target detection method are proposed. The method designs a unified model architecture containing a double-main feature fusion, a learnable multi-scale feature fusion, and a domain self-adaptive alignment module. Multiple source data and multiple loss functions are used to train the model, and finally a model that can detect multiple types of targets with high precision, high robustness, and real-time performance in complex road conditions is obtained, effectively solving the core pain points of waste of computing resources, lack of cross-task context, and poor cross-domain generalization ability in related technologies.

[0077] For example, as shown in Figure 1 , the Figure 1 is an architecture diagram of a multi-task target detection model provided by an embodiment of the present application.

[0078] The multi-task target detection model includes a shared feature layer 100 and a multi-task detection head 140.

[0079] The shared feature layer 100 includes a backbone feature extraction layer 110, a multi-scale feature fusion layer (FPN++) 120, and a domain feature alignment layer (Domain Adapter) 130.

[0080] The backbone feature extraction layer 110 includes a local feature backbone 111, a global semantic backbone 112, and an attention module 113. The domain feature alignment layer 130 includes a feature projection layer 131, a domain classifier 132, and a gradient inversion layer 133.

[0081] The multi-task detection head 140 includes a first detection head 141, a second detection head 142, a third detection head 143, and a fourth detection head 144, wherein the first detection head 141 is used to perform a helmet wearing detection task, the second detection head 142 is used to perform a head-shoulder detection task, the third detection head 143 is used to perform a pedestrian detection task, and the fourth detection head 144 is used to perform a vehicle detection task.

[0082] Wherein, the local feature backbone 111 adopts a CSPDarkNet53 architecture (Cross Stage Partial Network DarkNet53, a backbone network for extracting local details (such as edges and textures) of an image), the global semantic backbone 112 adopts a Swin Transformer-L architecture (Shifted-window Transformer-Large, a Transformer-based visual backbone network capable of effectively capturing the global context relationship of an image), and the attention module 113 refers to a CA (Channel Attention, a channel attention mechanism that automatically learns the importance weight of each channel and then enhances or suppresses the feature according to the weight; for the current task, the weight of the more important channel will be adjusted higher; for the unimportant channel, the weight will be adjusted lower).

[0083] When the training data set is input to the multi-task object detection model, the local feature backbone 111 and the global semantic backbone 112 will extract features in parallel, the local feature backbone 111 is used to extract the local texture features of the training data, and the global semantic backbone 112 is used to extract the global context information of the training data. After the feature maps output by the two are spliced, they are fused through the attention module 113, and the obtained result is used as input to the multi-scale feature fusion layer 120.

[0084] The multi-scale feature fusion layer 120 adopts an improved BiFPN structure (Bidirectional Feature Pyramid Network, a structure that allows bidirectional information flow and introduces a learnable weighting mechanism to adaptively adjust the contribution of different layer features), which can generate feature maps at different scales according to the backbone feature extraction layer 110, as input to the domain feature alignment layer 130.

[0085] The feature projection layer 131 of the domain feature alignment layer 130 is configured to map the feature maps of different scales generated by the backbone feature extraction layer 110 into a fixed-dimensional space. The domain classifier 132 is configured to determine whether the mapped feature data is from a special data or general data. The gradient reversal layer 133 is configured to determine the distribution difference between the feature distribution of the special data and the feature distribution of the general data according to the determination result of the domain classifier 132, and realize data domain alignment through back propagation, so that the multi-task target detection model pays more attention to domain-invariant features rather than domain features of training data, that is, the multi-task target detection model avoids focusing on the source of the input data when extracting features of the data, which leads to good performance on a specific data set but poor generalization ability when facing new data sources.

[0086] The multi-task detection head 140 receives the features mapped into the unified dimensional space output by the domain feature alignment layer 130, and the four task detection heads therein perform corresponding analysis tasks in parallel according to the features. The multi-task detection head 140 uses a dynamic task head structure (Dynamic Task Head), each task has independent detection head parameters but shares feature input, and each detection head includes a classification sub-layer (Class Head), a regression sub-layer (Box Head), and an IoU estimation sub-layer (Objectness Head) capable of outputting prediction results respectively. The complete output results of the multi-task target detection model are obtained by integrating the results output by each task detection head.

[0087] Illustratively, as shown in FIG. 1, the multi-task target detection model includes a backbone feature extraction layer 110, a domain feature alignment layer 130, and a multi-task detection head 140. Figure 2 , Figure 2 is a process diagram of multi-task target detection using a multi-task target detection model according to an embodiment of the present application.

[0088] The to-be-detected image a is obtained by image acquisition on a certain traffic driving scene, and includes the following targets: a first pedestrian 231, a second pedestrian 232, a first vehicle 221, a second vehicle 222, a driver 233, and a helmet 211 worn by the driver 233, which presents the pedestrians, vehicles, and helmet wearing situation in the road scene. The to-be-detected image a is input into the trained multi-task target detection model, and the multi-task target detection model outputs a detection result b after performing multi-task target detection.

[0089] The multi-task target detection model calls each task detection head to perform target detection on the to-be-detected image a respectively, and outputs the results of each detection head respectively. The detection result b is obtained by integrating the output contents of all detection heads. Illustratively, the detection result b includes the following contents:

[0090] 1. Vehicle detection result 1 corresponding to the first vehicle 221, containing a bounding box [300, 50, 450, 180], a classification result "99% probability of a small car", and a confidence of 0.98; indicating that the target (the first vehicle 221) existing at the bounding box is a small car, and the probability of the target existing at the bounding box is 0.98;

[0091] 2. Vehicle detection result 2 corresponding to the second vehicle 222, containing a bounding box [200, 250, 280, 350], a classification result "95% probability of a motorcycle", and a confidence of 0.96; indicating that the target (the second vehicle 222) existing at the bounding box is a motorcycle, and the probability of the target existing at the bounding box is 0.96;

[0092] 3. Helmet wearing detection result 3 corresponding to the helmet 211, containing a bounding box [250, 120, 300, 180], a classification result "99% probability of a helmet", and a confidence of 0.92; indicating that a helmet wearing event exists at the bounding box, and the probability of the target existing at the bounding box is 0.92;

[0093] 4. Pedestrian detection result 4 corresponding to the first pedestrian 231, containing a bounding box [100, 200, 150, 300], a classification result "99% probability of a pedestrian", and a confidence of 0.95; indicating that the target (the first pedestrian 231) existing at the bounding box is a pedestrian, and the probability of the target existing at the bounding box is 0.98;

[0094] 5. Pedestrian detection result 5 corresponding to the second pedestrian 232, containing a bounding box [400, 280, 450, 380], a classification result "95% probability of a pedestrian", and a confidence of 0.93; indicating that the target (the second pedestrian 232) existing at the bounding box is a pedestrian, and the probability of the target existing at the bounding box is 0.93;

[0095] 6. Head and shoulder detection result 6 corresponding to the driver 233, containing a bounding box [220, 180, 280, 280], a classification result "96% probability of a head and shoulder (human head and shoulder region)", and a confidence of 0.94; indicating that a head and shoulder exists at the bounding box, and the probability of the head and shoulder existing at the bounding box is 0.98;

[0096] 7. First head and shoulder detection result 7 corresponding to the first pedestrian 231, containing a bounding box [120, 100, 170, 160], a classification result "94% probability of a head and shoulder", and a confidence of 0.88; indicating that a head and shoulder exists at the bounding box, and the probability of the head and shoulder existing at the bounding box is 0.88;

[0097] 8. The second head and shoulder detection result 8 corresponding to the second pedestrian 232 includes a bounding box [380,260,430,360], a classification result of "93% probability head and shoulder (human head and shoulder region)", and a confidence level of 0.91; indicating that there is a head and shoulder at the bounding box, and the probability of the head and shoulder at the bounding box is 0.91.

[0098] The above results fall into four categories: helmet wearing detection, head and shoulder detection, pedestrian detection, and vehicle detection. Each result consists of a bounding box, classification result, and confidence score, achieving accurate identification and attribute determination of targets in the scene.

[0099] It is worth noting that the data dimensions and specific attribute types included in the detection results for each type of task are configurable. In addition to the common data items / attributes (bounding box, category, confidence score) exemplified above, other attributes can be configured individually based on the characteristics of each task. For example, for the head and shoulder detection task, an "orientation" attribute can be additionally configured to indicate the direction of the pedestrian's head and shoulders; or, for the vehicle detection task, attributes such as "color" and "license plate" can be additionally configured to indicate the color of the vehicle, whether the vehicle has a license plate, etc. The representation of each attribute can be arbitrary. For example, the output form of the "orientation" attribute can be a vector, and the output form of the "category" attribute in the helmet wearing detection task is a binary attribute "yes" or "no". This application does not impose any limitations on this.

[0100] The method for training a multi-task object detection model provided in this application will be explained based on the above description, such as... Figure 3 As shown, Figure 3 This is a flowchart of a method for training a multi-task object detection model according to an embodiment of this application. The method includes the following steps.

[0101] S310, Obtain the training dataset.

[0102] The training dataset includes specialized data and general data. The specialized data consists of image data collected from a specific traffic road scene, while the general data consists of image data collected from multiple types of traffic road scenes. The specific traffic road scene and the multiple types of traffic road scenes have different environmental characteristics.

[0103] Specialized data is labeled with a first label and a second label, while general data is labeled with a third label and a fourth label; the first label corresponds to the first task, the second label corresponds to the second task, the third label corresponds to the third task, and the fourth label corresponds to the fourth task.

[0104] Among them, the first, second, third and fourth tasks are target detection tasks related to traffic and road scenarios, and each task is different from the others.

[0105] For example, each task can be any one of the following tasks:

[0106] (1) A helmet-wearing detection task: identifying a rider (e.g., a motorcycle or bicycle driver) in an image and further determining whether the rider is wearing a safety helmet;

[0107] (2) A head-shoulder detection task: detecting the head and shoulder regions of a pedestrian in an image;

[0108] (3) A pedestrian detection task: locating and identifying a pedestrian from an image or video;

[0109] (4) A vehicle detection task: detecting various types of vehicles in an image, such as cars, buses, trucks, motorcycles, etc.;

[0110] (5) A traffic sign detection task: identifying and locating various types of traffic signs on the road, such as speed limit signs, parking signs, turn signs, etc.;

[0111] (6) A traffic light detection task: detecting a traffic light and identifying its current state, i.e., red, yellow, or green;

[0112] (7) A drivable area detection task: identifying the road surface area where a vehicle can safely drive.

[0113] Optionally, the first task is the helmet-wearing detection task, the second task is the head-shoulder detection task, the third task is the pedestrian detection task, and the fourth task is the vehicle detection task.

[0114] In some embodiments, the specialized data refers to data collected and labeled specifically according to the task requirements, and the general data refers to data publicly available on the network, which can be downloaded and obtained according to requirements.

[0115] Specialized data is usually collected to address specific business requirements, and its scene is relatively fixed and the annotation is fine. For example, for the helmet-wearing detection task, in order to achieve "one helmet and one wearing" safety supervision, cameras are deployed at the entrance of a highway or a specific section of a city, and images of passing motorcycles and riders are captured, resulting in a set of image data as specialized data. These image data may have similar lighting conditions, shooting angles, and backgrounds, and have been finely annotated by humans, clearly marking the bounding box of each rider and whether the rider is wearing a helmet.

[0116] In this application, the specialized data annotation corresponds to the labels of the first task (helmet-wearing detection) and the second task (head-shoulder detection).

[0117] For example, for a special data image containing a cyclist, its first label can include: (1) a classification result indicating that the target is a "cyclist" or a "helmeted head"; (2) a bounding box demarcating the precise coordinates of the target in the image; and (3) a confidence indicating the certainty of the existence of the target.

[0118] For example, for the same special data image containing a cyclist, its second label can include: (1) a classification result indicating that the target is a "head-shoulder"; (2) a bounding box demarcating the precise coordinates of the head-shoulder region in the image; and (3) a confidence indicating the certainty of the existence of the head-shoulder region.

[0119] The general data is usually derived from large-scale and diversified public data sets, aiming to cover as rich real-world scenarios as possible. For example, one data set contains driving scene images from different cities, different weathers (sunny, rainy, snowy), and different times (daytime, nighttime), in which a large number of vehicles, pedestrians, traffic signs, and other targets are labeled. Another data set focuses on pedestrian detection in urban street scenes and contains a large number of pedestrians with different postures and different degrees of occlusion. The above-mentioned example data sets are general data.

[0120] In this application, the labels in the general data correspond to the third task (pedestrian detection) and the fourth task (vehicle detection).

[0121] For example, for a general data image containing a pedestrian, its third label can include: (1) a classification result indicating that the target is a "pedestrian"; (2) a bounding box demarcating the position of the pedestrian; and (3) a confidence indicating the certainty of the existence of the pedestrian.

[0122] For example, for a general data image containing a vehicle, its fourth label can include: (1) a classification result indicating that the target is a "vehicle", "motorcycle", or "bicycle"; (2) a bounding box demarcating the precise coordinates of the vehicle in the image; and (3) a confidence indicating the certainty of the existence of the vehicle.

[0123] It is worth noting that each of the above-mentioned training data (whether general data or special data) includes at least one complete label corresponding to a task, which includes but is not limited to the following attributes: classification result, bounding box, and confidence.

[0124] S320, in each round of training, input the training data set to the multi-task object detection model.

[0125] The multi-task object detection model includes a backbone feature extraction layer, a multi-scale feature fusion layer, a domain feature alignment layer, a first detection head, a second detection head, a third detection head, and a fourth detection head.

[0126] When the training data set is input into the multi-task object detection model, the following steps are performed by the multi-task object detection model:

[0127] 1. Feature extraction and fusion processing is performed on the training data set to obtain a first feature data set corresponding to the special data and a second feature data set corresponding to the general data.

[0128] The backbone feature extraction layer includes a local feature backbone, a global semantic backbone, and an attention module. The local feature backbone is used to extract local texture features of each training data in the training data set. The global semantic backbone is used to extract global context information of each training data in the training data set. The attention module is used to perform attention fusion on the local texture features and the global context information. The multi-scale feature fusion layer is used to generate feature maps at different scales based on the initial feature data set output by the backbone feature extraction layer to obtain the first feature data set and the second feature data set.

[0129] Optionally, local feature extraction is performed on each training data in the training data set by the local feature backbone to obtain a local feature data set corresponding to the training data set. The local feature data set includes local texture features of each training data.

[0130] Context feature extraction is performed on each training data in the training data set by the global semantic backbone to obtain a global feature data set corresponding to the training data set. The global feature data set includes global context information of each training data.

[0131] For each training data, the following steps are performed to obtain an initial feature data set corresponding to the training data set: the local texture features and the global context information corresponding to the training data are spliced to obtain a spliced feature map. The attention weight of the spliced feature map is adjusted by the attention module to obtain the initial feature data corresponding to the training data. The initial feature data set includes the initial feature data corresponding to each training data.

[0132] The initial feature data set is processed by the multi-scale feature fusion layer to generate feature maps at different scales, thereby obtaining the first feature data set and the second feature data set.

[0133] Please refer to the following formula, which is used to describe the steps performed by the backbone feature extraction layer.

[0134]

[0135] wherein, Fcsp refers to the local texture features obtained by the local feature backbone after feature extraction on each training data in the training data set, which is in the form of a feature map. (Fswin) is the global context information obtained by the global semantic backbone after feature extraction on each training data in the training data set, and the form is also a feature map. Conv() is a concatenation operation, which is used to concatenate the local feature map and the global feature map; CA is an attention module, which is used to perform attention fusion on the concatenated feature map.

[0136] In the above formula, Fcsp represents the local feature map output at one scale, and Fswin represents the context / global feature map output at the scale. The two are concatenated along the channel dimension to generate Fconcat, and then the CA module generates a channel-by-channel weight and reweights it to obtain (Ffusion), and the form of Ffusion is also a feature map (with the same input spatial size, and the number of channels is the concatenated channel or the channel after dimension reduction).

[0137] For example, let (F{csp}) be the local feature map extracted by the local feature backbone, and (F{swin}) be the global feature map extracted by the global semantic backbone. The two are concatenated along the channel dimension at the same spatial scale: (F{concat}=\operatorname{Concat}(F{csp},F{swin})). After concatenation, the attention module CA is input, and CA processes the concatenated feature map through global average pooling, a fully connected layer (FC), and a sigmoid activation function to obtain a channel-by-channel weight (w{ch}), and multiply it by (F{concat}) channel by channel to output a fusion feature map (F{fusion}). In some embodiments, a 1x1 convolution layer can also be added after the attention module to reduce the channel dimension to (C{fusion}).

[0138] It is worth noting that the spatial dimensions (HxW) of Ffusion, Fcsp, and Fswin should be the same. Let the number of channels of Fcsp be Ccsp, and the number of channels of Fswin be Cswin, then the dimension of the concatenated feature map is (Ccsp+Cswin,H,W). If dimension reduction is needed, a 1x1conv can be used after the attention module to map it to a unified channel Cfusion.

[0139] The number of scales of the backbone feature extraction layer can be set according to requirements. For each scale, the initial feature data set contains initial feature data of each training data at different scales.

[0140] For example, there are 1000 training data and 5 scales, so the initial feature data of each training data contains the original output features of the training data at 5 scales.

[0141] This "double trunk + attention fusion" structure can capture fine-grained and global structure information at the same time, and adapt to highway multi-scale targets.

[0142] For example, the multi-scale feature fusion layer adopts a bidirectional feature pyramid structure, and each initial feature data in the initial feature data set includes original output features output by the trunk feature extraction layer at different scales.

[0143] For each initial feature data, the fusion output features of the initial feature data at different scales are obtained as the feature maps of the initial feature data at different scales.

[0144] The core of the multi-scale feature fusion layer is to efficiently fuse feature information from different scales through a learnable weighting method. The input of this process is the original output feature set F output by the trunk feature extraction layer at different scales, and the output is the feature map set P after fusion, which has strong representation ability at all scales.

[0145] Wherein, when obtaining the fusion output feature at the i-th scale, the following steps are performed: down-sampling the fusion output feature at the i-1-th scale to obtain a down-sampling result, i is a positive integer; up-sampling the fusion output feature at the i+1-th scale to obtain an up-sampling result; and performing weighted fusion according to the down-sampling result, the up-sampling result and the original output feature at the i-th scale to obtain the fusion output feature of the initial feature data at the i-th scale as the feature map at the i-th scale.

[0146] Please refer to the following formula, which is used to obtain the feature map of each initial feature data at the i-th scale.

[0147]

[0148] Wherein, Fi is the original output feature output by the trunk feature extraction layer at the i-th scale, Pi is the feature map at the i-th scale / the fusion output feature at the i-th scale, Pi+1 is the feature map at the i+1-th scale / the fusion output feature at the i+1-th scale, Pi-1 is the feature map at the i-1-th scale / the fusion output feature at the i-1-th scale, Up is the up-sampling operation, Down is the down-sampling operation. w1, w2, w3 are scientific weighting coefficients, so that the feature fusion of different scales is adaptive.

[0149] Optionally, Up (up-sampling) operation, with bilinear interpolation followed by 1x1 convolution. Bilinear interpolation is a smooth image magnification algorithm that can magnify high-level, low-resolution feature maps Pi+1 to match the size of the current layer; and the subsequent 1x1 convolution is responsible for adjusting the number of channels and fine-tuning the interpolated features to ensure compatibility. Down (down-sampling) operation, with stride-2 convolution or average pooling followed by 1x1 convolution, the purpose is to reduce the low-level, high-resolution feature map Pi-1, also for size matching. Step convolution can learn new features while shrinking, while average pooling retains statistical information. After fusion, to further enhance the expression ability of the features and control the amount of calculation, additional optimization steps can also be introduced. For example, applying a channel attention mechanism to the fused feature map Pi, which can let the model automatically focus on the most important feature channels for the current task and suppress irrelevant channel interference. Then, a 3x3 depth separable convolution is connected, which can greatly reduce the parameter amount and computational burden of the model while maintaining good performance, making it more suitable for deployment on resource-constrained devices. The entire process is iteratively executed, constantly optimizing the feature representation of each layer, and finally generating a high-quality multi-scale feature pyramid, providing a solid foundation for subsequent detection tasks.

[0150] Optionally, before inputting the initial feature data into the multi-scale feature fusion layer, the channel number can be aligned to the same number of channels as Pi through convolution or projection layer, to facilitate weighted fusion.

[0151] For any three adjacent scales, first up-sample the high-level Pi+1, down-sample the low-level Pi-1, and do a learnable weighted sum with the original output feature Fi of the current scale i to get the final fusion output of this scale Pi.

[0152] It is worth noting that the number of scales is the same as the number of scale layers designed by the backbone feature extraction layer, and the multi-scale feature semantic layer set can be denoted as (P3, P4…Pn), n is a positive integer, the value of n can be set according to the output layer number of the backbone feature extraction layer and the detection demand (usually n=5, corresponding to P3-P7). Each Pi corresponds to different spatial resolution and receptive field, used for detecting targets of different sizes respectively.

[0153] Different scale feature maps have obvious differences in spatial resolution and receptive field: high resolution layers are suitable for small targets, and low resolution layers are suitable for large targets. For example, high resolution layers (such as P3, larger HxW) are used for small target detection (helmet, pedestrian head and shoulder); intermediate layers (P4) are used for medium targets; low resolution layers (P5 and lower) are used for large targets (vehicles, trucks). Among them, for different targets, the target can be mapped to the corresponding scale according to the preset mapping rule according to the pixel area / aspect ratio, and then regression / classification is performed. According to the fusion output features of each initial feature data in the initial feature data set at different scales, the first feature data set and the second feature data set are obtained.

[0154] Compared with the traditional feature pyramid network (FPN), the bidirectional feature pyramid structure has the following advantages: allowing bidirectional information flow; introducing a weighting mechanism to avoid feature oversmoothing; and automatically adjusting the contribution of features at different layers.

[0155] It is worth noting that the training data set is a data set mixed with general data and special data. When the backbone feature extraction layer and the multi-scale feature fusion layer perform feature extraction and fusion processing on it, the general data or the special data will not be distinguished, but the same steps will be performed on all training data. Therefore, the multi-task target detection model will not distinguish whether the initial feature data belongs to general data or special data, and will not distinguish whether the feature maps at different scales output by the multi-scale feature fusion layer belong to general data or special data. Here, the first feature data set and the second feature data set are used for easy understanding, and the subsequent use of the domain alignment feature layer for processing is more convenient for expression. In fact, the first feature data set and the second feature data set can be regarded as a complete data set.

[0156] 2. Map the first feature data set and the second feature data set to the same feature dimension space to obtain a third feature data set, and determine the distribution difference between the feature distribution of the special data and the feature distribution of the general data according to the third feature data set.

[0157] Among them, the domain feature alignment layer includes a feature projection layer, a domain classifier, and a gradient inversion layer. The feature projection layer is used to map the first feature data set and the second feature data set to the same feature dimension space to obtain a third feature data set. The domain classifier is used to determine whether each feature data in the third feature data set belongs to special data or general data. The gradient inversion layer is used to determine the distribution difference between the feature distribution of the special data and the feature distribution of the general data according to the judgment result output by the domain classifier.

[0158] The purpose of domain feature alignment is to solve the distribution difference problem between special data and general data due to different environmental characteristics, that is, the domain bias problem, so as to improve the generalization ability and stability of the model in the unseen environment. By the feature projection layer, the two types of data are mapped to the same feature dimension space to obtain a third feature data set, and a gradient reversal layer is used to construct an adversarial learning process. In this process, the domain classifier tries to distinguish the source of the features in the third feature data set, and the gradient reversal layer generates a distribution difference value for optimizing the model according to the judgment result of the domain classifier. The difference value is used to guide the feature extraction network to learn how to generate features that cannot be distinguished by the domain classifier, so as to minimize the feature distribution difference between the special data and the general data, so that the model learns the domain-invariant features, rather than the non-essential features related to the data source.

[0159] Optionally, the first feature data set and the second feature data set are mapped to the same feature dimension space by the feature projection layer to obtain a third feature data set.

[0160] The type of each feature data in the third feature data set is judged by the domain classifier to obtain a judgment result, and the judgment result is used to guide the division of the third feature data set into a first subset and a second subset. The first subset is composed of feature data belonging to special data, and the second subset is composed of feature data belonging to general data.

[0161] As mentioned earlier, the first feature data set and the second feature data set can be regarded as the same data set, and the feature projection layer will map all feature data in the data set to the same channel dimension feature space, that is, each feature data in the third feature data set has the same channel dimension.

[0162] For example, all feature data are mapped to Cproj, and the spatial size HxW of each feature data still varies with the scale (high layer HxW is small, and low layer HxW is large). That is, the feature projection layer projects all scale feature maps to the same dimension (C{proj}) in the channel dimension, but the spatial resolution (HxW) of each scale is still different. After projection, the feature shape is ((C{proj},Hi,Wi)), where (Hi,Wi) varies with scale i.

[0163] The feature distribution of the first subset and the second subset is analyzed by the gradient reversal layer according to the judgment result output by the domain classifier to determine the distribution difference value.

[0164] Please refer to the following formula for calculating the distribution difference value.

[0165]

[0166] wherein the formula is a common method in the field of domain alignment loss (MMD), used to measure the distance between the feature distribution of the source domain (general data) and the target domain (special data), and its mathematical definition is the maximum mean difference, that is, the above formula LMMD is the square of the maximum mean difference. The prototype of the formula is as follows:

[0167]

[0168] is a kernel function that maps the sample to a reproducing kernel Hilbert control (RKHS), represents the kernel space, and if the means of two feature distributions are close in the kernel space, it means that the distributions of the two domains are similar.

[0169] The expansion of the formula is as follows:

[0170]

[0171] wherein m and n are the number of samples of the source domain and the target domain respectively, K(x, y) is a kernel function (usually an RBF kernel, and a multi-kernel or adaptive bandwidth strategy can be used to improve robustness), and the formula of the kernel function is as follows:

[0172]

[0173] Returning to the above formula LMMD for calculating the distribution difference, the formula actually takes the MMD loss as the distribution difference between the feature distribution of the first subset and the feature distribution of the second subset, fs is the mean (center point) of the general data after the feature mapping layer mapping; ft is the mean (center point) of the special data after the feature mapping layer mapping. If LMMD is minimum, it means that the class condition or overall distribution of the two domains in the mapped space is close, thereby reducing the domain bias and improving the consistency and stability of the model on different data sources.

[0174] In the training stage, the model is loss-trained for the purpose of minimizing the MMD loss, which can make the distributions of different data domains in the feature space closer, thereby enhancing the generalization ability of the model.

[0175] For example, the way to calculate fs and ft includes but is not limited to the following:

[0176] (1) Batch mean (batch mean): in each training batch, the mean of the features of all general data domain samples in the batch and the mean of the features of all special data domain samples are calculated respectively. The calculation formula is as follows:

[0177] fs = (1 / Ns) *∑{i=1}^{Ns}φ(xi^s), ft = (1 / Nt) *∑{j=1}^{Nt}φ(xj^t), where Ns and Nt represent the number of samples in the general data domain and the special data domain in the current batch respectively, and φ(xi^s) and φ(xj^t) represent the feature vectors of the ith sample in the general data domain and the jth sample in the special data domain after passing through the feature projection layer respectively.

[0178] (2) Global mean: The global feature mean of the general data domain and the special data domain is pre-calculated on the entire training data set and fixedly used in training.

[0179] (3) Sliding mean: In the training process, the feature mean of the general data domain and the special data domain is gradually updated in a manner similar to momentum update. For example, the new mean = (1-momentum coefficient) *old mean + momentum coefficient *current batch mean. The update formula is as follows: fsnew = (1-m) *fsold + m*fsbatch, ftnew = (1-m) *ftold + m*ftbatch, where fsold and ftold are the sliding means updated in the last round, fsbatch and ftbatch are the batch means calculated in the current batch, and m is the momentum coefficient, which is usually a value close to 0 (such as 0.9).

[0180] 3. Perform target detection on each data in the third feature data set respectively to obtain a first prediction result corresponding to the first task, a second prediction result corresponding to the second task, a third prediction result corresponding to the third task, and a fourth prediction result corresponding to the fourth task.

[0181] The first detection head is configured to perform target detection on the third feature data set to obtain the first prediction result. The second detection head is configured to perform target detection on the third feature data set to obtain the second prediction result. The third detection head is configured to perform target detection on the third feature data set to obtain the third prediction result. The fourth detection head is configured to perform target detection on the third feature data set to obtain the fourth prediction result.

[0182] Optionally, different tasks are performed by the multi-task detection head in parallel to perform target detection on each feature data in the third feature data set, and the result output by each task detection head can refer to the following formula:

[0183]

[0184] where bi is a bounding box indicating the position of the target, ci is a class probability indicating the probability that the target belongs to a certain class (the number of classes can be set), and oi is a target confidence indicating the probability that there is a target in the bi bounding box.

[0185] The first detection head performs target detection on the third feature data set to obtain a first prediction result, the first prediction result including a first bounding box, a first classification result and a first confidence, the first classification result being used to indicate a target type at the first bounding box, and the first confidence being used to indicate whether a target exists at the first position.

[0186] For example, after target detection is performed on one feature data in the third feature data set, the following output result is obtained: {first bounding box [120, 150, 80, 100], first classification result “95% probability of existing helmet / wearing helmet”, and first confidence 0.98}.

[0187] The second detection head performs target detection on the third feature data set to obtain a second prediction result, the second prediction result including a second bounding box, a second classification result and a second confidence, the second classification result being used to indicate a target type at the second bounding box, and the second confidence being used to indicate whether a target exists at the second position.

[0188] For example, after target detection is performed on one feature data in the third feature data set, the following output result is obtained: {second bounding box [80, 100, 60, 100], second classification result “99% probability of existing head and shoulder”, and second confidence 0.98}.

[0189] The third detection head performs target detection on the third feature data set to obtain a third prediction result, the third prediction result including a third bounding box, a third classification result and a third confidence, the third classification result being used to indicate a target type at the third bounding box, and the third confidence being used to indicate whether a target exists at the third position.

[0190] For example, after target detection is performed on one feature data in the third feature data set, the following output result is obtained: {third bounding box [80, 80, 60, 80], third classification result “94% probability of pedestrian”, and first confidence 0.96}.

[0191] The fourth detection head performs target detection on the third feature data set to obtain a fourth prediction result, the fourth prediction result including a fourth bounding box, a fourth classification result and a fourth confidence, the fourth classification result being used to indicate a target type at the fourth bounding box, and the fourth confidence being used to indicate whether a target exists at the fourth position.

[0192] For example, after target detection is performed on one feature data in the third feature data set, the following output result is obtained: {fourth bounding box [180, 120, 60, 90], fourth classification result “99% probability of vehicle”, and fourth confidence 0.95}.

[0193] S330, obtaining a total loss value according to a first difference value between the first prediction result and the first label, a second difference value between the second prediction result and the second label, a third difference value between the third prediction result and the third label, a fourth difference value between the fourth prediction result and the fourth label, and the distribution difference value, and updating parameters of the multi-task target detection model to minimize the total loss value to obtain the trained multi-task target detection model.

[0194] The first difference value is composed of a first classification difference value, a first bounding box difference value and a first confidence difference value. The second difference value is composed of a second classification difference value, a second bounding box difference value and a second confidence difference value. The third difference value is composed of a third classification difference value, a third bounding box difference value and a third confidence difference value. The fourth difference value is composed of a fourth classification difference value, a fourth bounding box difference value and a fourth confidence difference value.

[0195] Optionally, a first weight difference value is obtained according to a weighted sum of the first classification difference value, the first bounding box difference value and the first confidence difference value.

[0196] A second weight difference value is obtained according to a weighted sum of the second classification difference value, the second bounding box difference value and the second confidence difference value.

[0197] A third weight difference value is obtained according to a weighted sum of the third classification difference value, the third bounding box difference value and the third confidence difference value.

[0198] A fourth weight difference value is obtained according to a weighted sum of the fourth classification difference value, the fourth bounding box difference value and the fourth confidence difference value.

[0199] The total loss value is obtained according to a weighted sum of the first weight difference value, the second weight difference value, the third weight difference value, the fourth weight difference value and the distribution difference value, and parameters of the multi-task target detection model are updated to minimize the total loss value to obtain the trained multi-task target detection model.

[0200] For example, the following formula is used to calculate the total loss value:

[0201]

[0202] Ltotal=αLdet+βLdom, where Ltotal is the total loss value, Ldet is a detection loss (i.e., a weighted sum of difference values between output results of each task detection head and labels), Ldom is a domain alignment loss (i.e., a distribution difference value calculated by a gradient reversal layer of a domain feature alignment layer), and α and β are weight coefficients, the values of which can be defined by the user. By default, α can be set to 1.0 and β can be set to 0.25.

[0203] For example, the detection loss Ldet of each detection head includes the following three parts according to the following formula:

[0204]

[0205] Wherein, for the training data (labeled with labels) and the prediction results of the same task, Lcls is a classification loss, used to indicate the difference between the classification results in the labels and the classification results in the prediction results, Lbox is a bounding box loss, used to indicate the difference between the bounding boxes in the labels and the bounding boxes in the prediction results, and Lobj is a confidence loss, used to indicate the difference between the confidence in the labels and the confidence in the prediction results.

[0206] Wherein, the classification loss Lcls can be calculated by the following formula, which is the formula of the FocalLoss loss function, used to solve the sample imbalance problem:

[0207]

[0208] Wherein, the bounding box loss Lbox can be calculated by the following formula, which is the formula of the GIoU loss function, capable of improving the convergence stability of the bounding box overlap:

[0209]

[0210] Wherein, the confidence loss Lobj can generally be obtained by using a binary cross-entropy (BCE) or FocalLoss (for class imbalance) loss function, with the label being 1 (existence of target) or 0 (no target). Different weights can be used to balance (such as focal α, γ) or sampling strategies (such as hard negative mining) for positive samples and negative samples.

[0211] Optionally, the output format of each detection head is ([C, 4, 1]) or is expanded to ([C, 4, 1, A]), respectively corresponding to a category probability vector, a bounding box regression amount, a target confidence, and an optional attribute prediction dimension A.

[0212] For example, the first detection head is used to detect the state of a cyclist and a helmet, with an output dimension of [C1, 4, 1], and the loss functions used include Focal, GIoU, and BCE; the second detection head is used to detect the upper body / head and shoulders of a human body, with an output dimension of [C2, 4, 1], and the loss functions used include Focal and IoU; the third detection head is used to identify and detect pedestrian attributes, with an output dimension of [C3, 4, 1, A], A being an optional attribute dimension, such as “gender”, and the loss functions used include Multi-Label, BCE, and GIoU; and the fourth detection head is used to detect vehicles, with an output dimension of [C4, 4, 1], and the loss functions used include Focal and IoU.

[0213] It is worth noting that for specific application scenarios, the number of labels in the training data set can be pruned as needed when training the multi-task target detection model, and the number of prediction results obtained in each training round, the way of updating the model parameters based on the difference between the prediction results and the labels will also be adjusted accordingly. The maximum number of tasks that the trained multi-task target detection model can perform and the type are related to the number and type of task detection heads contained in the model itself. During the training phase, if the multi-task target detection model has the ability to process four tasks simultaneously after training, the corresponding task detection head can be selected to perform the task as needed during actual inference application, or each detection head outputs the result, and the output result is selected as needed. For example, the multi-task target detection model can perform the first task, the second task, the third task and the fourth task after training, and the demand of the detection business is the first task and the second task, so in the inference stage, the output results of the first detection head and the second detection head can be selected for subsequent application. For example, the multi-task target detection model can only perform the first task and the third task after training, and in the inference stage, at most, the output result is selected from the output results of the first detection head and the third detection head for subsequent application.

[0214] In summary, the method for training a multi-task target detection model provided by the present application processes labels corresponding to different tasks in the same model by obtaining a training data set containing special data and general data. The internal mechanism realizes feature sharing, that is, multiple tasks share the low-level features of the same backbone network, thereby reducing the number of parameters and improving the training efficiency. At the same time, the difference between the prediction results of different tasks and the corresponding labels, as well as the distribution difference between the special data and the general data, are used together to update the model, so that the model can have the ability to process multiple tasks simultaneously, and the complementarity between tasks is used to improve the accuracy of the detection results. During the training phase, the model learns the domain-invariant features between the general data and the special data by minimizing the distribution difference, thereby establishing a semantic bridge between the two. This makes the model pay more attention to the essential features in the data rather than the data source when performing a specific task, ultimately improving the robustness of the model and achieving the effect of deploying only one model in the traffic monitoring system to output multiple types of detection results.

[0215] Figure 4 is a flowchart of the method for multi-task target detection provided by an embodiment of the present application. The method is executed by the multi-task target detection model trained in the above-mentioned embodiment, and the method includes the following steps:

[0216] S410, obtaining a to-be-detected image.

[0217] The to-be-detected image is an image collected from a traffic road scene.

[0218] For referenceFigure 2 As shown in Figure 2 The to-be-detected image a is obtained after image acquisition on a certain traffic driving scene, and includes the following targets: a first pedestrian 231, a second pedestrian 232, a first vehicle 221, a second vehicle 222, a driver 233, and a helmet 211 worn by the driver 233.

[0219] S420, input the to-be-detected image into the multi-task target detection model to obtain a first detection result, a second detection result, a third detection result, and a fourth detection result.

[0220] The first detection result is obtained by performing a helmet wearing detection task on the to-be-detected image, the second detection result is obtained by performing a head-shoulder detection task on the to-be-detected image, the third detection result is obtained by performing a pedestrian detection task on the to-be-detected image, and the fourth detection result is obtained by performing a vehicle detection task on the to-be-detected image.

[0221] The multi-task target detection model includes a backbone feature extraction layer, a multi-scale feature fusion layer, a domain feature alignment layer, a first detection head, a second detection head, a third detection head, and a fourth detection head. The backbone feature extraction layer includes a local feature backbone, a global semantic backbone, and an attention module. The domain feature alignment layer includes a feature projection layer, a domain classifier, and a gradient reversal layer.

[0222] The multi-task target detection model in this embodiment has been trained and belongs to an inference stage. The feature projection layer used for forward mapping is retained during inference, while the domain classifier / gradient reversal layer (GRL) used during training can be removed or disabled, that is, the distribution loss (MMD loss) is not calculated during the inference stage. During the inference stage, the feature projection layer performs the same linear / convolutional mapping as during the training stage to ensure consistency of the features during inference. The feature projection layer maps feature maps of different scales to a unified channel dimension and synchronously sends the mapped results to the first detection head, the second detection head, the third detection head, and the fourth detection head.

[0223] Optionally, the multi-task target detection model performs the following processing steps on the to-be-detected image:

[0224] The local texture features of the to-be-detected image are extracted by the local feature backbone to obtain a local feature map. At the same time, the global context information of the to-be-detected image is extracted by the global semantic backbone to obtain a global feature map. The local feature map and the global feature map are spliced and sent to the attention module. The attention module performs attention fusion on the spliced feature map to obtain original output feature maps at different scales. The scale depends on the number of layers of the backbone feature extraction layer, that is, during the feature extraction of the to-be-detected image by the backbone feature extraction layer, original output features at different scales are generated.

[0225] The original output feature maps of different scales are input into the multi-scale feature fusion layer, which processes the original output feature maps to generate feature maps of different scales, and inputs the feature maps of different scales into the feature projection layer of the domain feature alignment layer; the feature projection layer maps the feature maps of different scales to a unified feature dimension space to obtain mapped feature maps. The mapped feature maps are input into the first detection head, the second detection head, the third detection head and the fourth detection head.

[0226] The first detection head performs a helmet wearing detection task on the to-be-detected image to obtain a first detection result; the second detection head performs a head-shoulder detection task on the to-be-detected image to obtain a second prediction result; the third detection head performs a pedestrian detection task on the to-be-detected image to obtain a third prediction result; and the fourth detection head performs a vehicle detection task on the to-be-detected image to obtain a fourth prediction result.

[0227] Each prediction result includes the following information: a classification result, a bounding box, and a confidence. The classification result is used to indicate the type of the detected target and the probability of belonging to the type; the bounding box is used to indicate the position of the target in the to-be-detected image, usually represented as a four-dimensional vector [tx, ty, tw, th] (representing the offset / coordinates of the center point of the bounding box in x and y, and the width and height of the bounding box, respectively); and the confidence is used to indicate the probability of the existence of the target in the bounding box.

[0228] Please refer to Figure 2 , in Figure 2 , the targets related to the helmet wearing detection task include the driver 233 and the helmet 211; the targets related to the head-shoulder detection task include the first pedestrian 231, the second pedestrian 232 and the driver 233; the targets related to the pedestrian detection task include the first pedestrian 231 and the second pedestrian 232; and the targets related to the vehicle detection task include the first vehicle 221 and the second vehicle 222.

[0229] Therefore, the first detection result includes a helmet wearing detection result related to the helmet 211 worn by the driver 233; the second detection result includes a head-shoulder detection result related to the first pedestrian 231, the second pedestrian 232 and the driver 233; the third detection result includes a pedestrian detection result related to the first pedestrian 231 and the second pedestrian 232; and the fourth detection result includes a vehicle detection result related to the first vehicle 221 and the second vehicle 222.

[0230] It is worth noting that all detection heads of the multi-task object detection model will make forward output. In the training stage, only the detection head related to the sample label contained in the current training sample will participate in loss calculation (irrelevant detection heads can be shielded by task mask mechanism, for a certain training sample, only the detection head related to the sample label is activated to participate in loss calculation, and the loss term of the remaining detection heads is shielded); in the inference stage, the detection heads of all tasks will output the prediction results of the input image, but the input image may not contain all task-related targets, so the detection heads of this part of irrelevant tasks usually output background / low-score detection results.

[0231] In the inference stage, each detection head makes forward prediction, and post-processing can select the head results of interest or combine multi-head outputs according to task requirements.

[0232] For example, at least one detection result is selected from the first detection result, the second detection result, the third detection result and the fourth detection result according to the requirements of the detection business.

[0233] That is, after the multi-task object detection model is trained, the model can be used to perform at least one task to obtain the corresponding detection result. In application, each task detection head can perform object detection on the input image to obtain the respective detection result. According to the requirements of the detection business, the multi-task object detection model can output only the detection result related to the detection business, or can also output all detection results, and then select the required detection result from all detection results according to the requirements.

[0234] For example, the requirement of the detection business is to perform pedestrian detection on the image to be detected. After the image to be detected is input into the multi-task object detection model, each detection head will output the corresponding detection result. Only the result output by the third detection head is related to the detection business, so the third detection result output by the third detection head is selected for subsequent application.

[0235] The multi-task object detection method provided in the embodiment introduces the inference application process of the model, directly associates the trained model with the actual application scenario, and forms a complete closed loop from training to deployment. The role of the model is clarified, that is, the corresponding detection result is output for different tasks, and the specific meaning of these results in the traffic scene is emphasized.

[0236] In summary, the model architecture and training method of the multi-task target detection model provided by the application cooperatively optimize different target detection tasks in a unified framework, reduce the repeated training cost, and improve the model scalability. By setting a global semantic backbone and a local feature backbone at the same time to obtain a double-backbone complementary fusion structure, the local high-resolution backbone (CSP) and the context-enhanced backbone (Swin) are fused through channel attention after channel splicing, taking into account small target details and global semantics. During training, the multi-task target detection model is trained using both special data and general data, a domain alignment multi-task training framework with multi-scale unified projection is set, and MMD / GRL and other domain alignment methods are used to project the multi-scale features to a unified channel dimension and combine them, so that the special data and the general data are aligned in a unified subspace, thereby realizing cross-domain multi-task joint learning, solving the distribution difference problem between special data and general data, and making the model maintain stable detection performance under different light, angle and environmental conditions. Through the combination of a bidirectional pyramid structure (learnable weighting) and relationship consistency constraints, learnable weighted fusion is used at the multi-scale fusion place and relationship consistency loss is added, which improves the consistency and robustness between different task outputs, outputs feature maps at different scales, and improves the cross-scale target detection capability, especially for small targets such as cyclists and helmets. The above combined architecture solves the problem of imbalance between local and global information, and improves the cross-domain generalization ability and small target detection performance through domain alignment and multi-task cooperation. Through the lightweight optimization structure of the model, the processing rate on the Jetson platform can reach 40FPS (i.e., the model can process and output detection results for 40 images per second), meeting the real-time requirements of highway monitoring. As a front-end module of the event detection system, the model provides high-precision and real-time traffic participant detection results, and also solves the problem of difficulty in fusing special data and general data, providing a unified detection framework for the landing of traffic scene AI (Artificial Intelligence, Artificial Intelligence) algorithms.

[0237] The method for training a multi-task target detection model corresponding to the above embodiment, Figure 5 The structure block diagram of the device for training a multi-task target detection model provided by the embodiment of the application is shown. For ease of illustration, only the parts related to the embodiment of the application are shown.

[0238] Reference Figure 5 The device 500 includes a data acquisition module 510 and a training module 520. The data acquisition module 510 is used to acquire a training data set, and the training data set includes special data and general data. The training module 520 is used to train a multi-task target detection model. The device 500 can be integrated in a training device, and the types of the training device include but are not limited to servers, terminals with strong computing power (computing power reaching a specified standard), and other electronic devices.

[0239] Figure 6 A structural block diagram of the device for multi-task target detection provided by the embodiment of the present application is shown. For ease of illustration, only the parts related to the embodiment of the present application are shown.

[0240] Referring to Figure 6 The device 600 comprises a data acquisition module 610 and a detection module 620. The data acquisition module 610 is configured to acquire an image to be detected. The detection module 620 is configured to perform multiple target detection tasks by using a multi-task target detection model, and obtain detection results output by each detection head. The device 600 can be integrated in an inference device. The inference device can be, but is not limited to, a server, a terminal with strong computing power (the computing power reaches a specified standard), and the like.

[0241] It should be noted that the information interaction between the above devices / modules, the execution process, and the like, are based on the same concept as the method embodiments of the present application. For specific functions and technical effects brought by the method embodiments, please refer to the method embodiments part, which will not be repeated here.

[0242] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example for illustration. In actual applications, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0243] In order to realize the above-mentioned embodiments, the present application further provides an electronic device which can be configured as an inference device or a training device in the above-mentioned embodiments.

[0244] Figure 7 A structural schematic diagram of an electronic device according to an embodiment of the present application.

[0245] As Figure 7 shown, the electronic device 700 comprises:

[0246] The memory 710 and the at least one processor 720, a bus 730 connecting different components (including the memory 710 and the processor 720), the memory 710 stores a computer program, when the processor 720 executes the program, the method for training a multi-task target detection model, the method for multi-task target detection of the embodiments of the application are realized.

[0247] The bus 730 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include, but are not limited to, industry standard architecture (ISA) bus, micro channel architecture (MAC) bus, enhanced ISA bus, video electronics standards association (VESA) local bus, and peripheral component interconnect (PCI) bus.

[0248] The electronic device 700 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by the electronic device 700 and include both volatile and nonvolatile media, removable and non-removable media.

[0249] The memory 710 can also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 740 and / or cache memory 750. The electronic device 700 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 760 can be provided for reading from and writing to a non-removable, non-volatile magnetic media (e.g., a "hard drive") Figure 7 (not shown, typically referred to as a "hard disk drive"). Although Figure 7 (not shown, typically referred to as a "hard disk drive"). Although

[0250] The program / utility 780 having a set (at least one) of program modules 770 can be stored in, for example, the memory 710 by way of example, such program modules 770 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination thereof, can include implementation of a network environment. The program modules 770 typically perform the functions and / or methodologies of the embodiments described herein.

[0251] Electronic device 700 can also communicate with one or more external devices 790 (e.g., keyboard, pointing device, display 771, etc.), and with one or more devices that enable a user to interact with electronic device 700, and / or with any device that enables electronic device 700 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 777. Furthermore, electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 773. As shown, network adapter 773 communicates with other modules of electronic device 700 via bus 730. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0252] The processor 720 executes various functional applications and data processing by running programs stored in the memory 710.

[0253] It should be noted that the implementation process and technical principles of the electronic device in this embodiment are explained in the foregoing description of the method for training a multi-task target detection model and the method for multi-task target detection in the embodiments of this application, and will not be repeated here.

[0254] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the various method embodiments above.

[0255] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0256] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the computer program for instructing the related hardware to complete all or part of the processes in the above-mentioned embodiment methods can be stored in the computer readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the photographing device / electronic device, recording medium, computer memory, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk and the like. In some jurisdictions, according to legislation and patent practice, the computer readable medium can not be an electrical carrier signal and a telecommunication signal.

[0257] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0258] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0259] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / electronic device and method can be implemented in other ways. For example, the apparatus / electronic device embodiments described above are only schematic. The division of the modules or units is only a logical function division, and there can be another division in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0260] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may also be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0261] In the above, specific details such as specific system structures, techniques, etc. are presented in order to thoroughly understand the embodiments of the present application, but not for limitation. It should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits and methods are omitted to avoid unnecessary details that hinder the description of the present application.

[0262] It should be understood that when used in the specification and the appended claims of the present application, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or sets thereof.

[0263] It should also be understood that the term "and / or" used in the specification and the appended claims of the present application means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0264] As used in the specification and the appended claims of the present application, the term "if" can be interpreted as "when" or "upon" or "in response to a determination" or "in response to detecting" depending on the context. Similarly, the phrase "if determined" or "if detected [the described condition or event]" can be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [the described condition or event]" or "in response to detecting [the described condition or event]" depending on the context.

[0265] In addition, in the description of the present application and the appended claims, the terms "first", "second", "third" and the like are only used for differentiation of description, and cannot be understood as indicating or implying relative importance.

[0266] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in a various embodiment" or "in some embodiment" or "in other embodiments" in various places throughout this specification are not necessarily all referring to the same embodiment, but can refer to one or more of the same or different embodiments. Furthermore, the term "comprising" or "containing" or "including" or "having" or "characterized by" as used herein encompasses the presence of stated features, steps or components or integers, but does not preclude the presence or addition of one or more other features, steps, components or integers. The term "consisting essentially of" or "consisting of" as used herein exclude the presence of other features, steps, components or integers.

[0267] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it still can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for training a multi-task object detection model, the method comprising: The method comprises: obtaining a training data set, wherein the training data set comprises special data and general data, the special data is image data collected for a specified traffic road scene, the general data is image data collected for multiple traffic road scenes, the specified traffic road scene has different environmental characteristics from the multiple traffic road scenes, the special data is labeled with a first label and a second label, the general data is labeled with a third label and a fourth label; the first label corresponds to a first task, the second label corresponds to a second task, the third label corresponds to a third task, and the fourth label corresponds to a fourth task; in each round of training, inputting the training data set into the multi-task object detection model, and performing the following steps by the multi-task object detection model: performing feature extraction and fusion processing on the training data set to obtain a first feature data set corresponding to the special data and a second feature data set corresponding to the general data; mapping the first feature data set and the second feature data set to the same feature dimension space to obtain a third feature data set, and determining a distribution difference between a feature distribution of the special data and a feature distribution of the general data according to the third feature data set; performing object detection on each data in the third feature data set respectively to obtain a first prediction result corresponding to the first task, a second prediction result corresponding to the second task, a third prediction result corresponding to the third task, and a fourth prediction result corresponding to the fourth task; obtaining a total loss value according to a first difference between the first prediction result and the first label, a second difference between the second prediction result and the second label, a third difference between the third prediction result and the third label, a fourth difference between the fourth prediction result and the fourth label, and the distribution difference, and updating parameters of the multi-task object detection model to minimize the total loss value to obtain a trained multi-task object detection model.

2. The method of claim 1, wherein, The multi-task target detection model comprises a backbone feature extraction layer, a multi-scale feature fusion layer, a domain feature alignment layer, a first detection head, a second detection head, a third detection head, and a fourth detection head. The backbone feature extraction layer comprises a local feature backbone, a global semantic backbone, and an attention module. The local feature backbone is configured to extract local texture features of each training data in the training data set. The global semantic backbone is configured to extract global context information of each training data in the training data set. The attention module is configured to perform attention fusion on the local texture features and the global context information. The multi-scale feature fusion layer is configured to generate feature maps at different scales based on an initial feature data set output by the backbone feature extraction layer, so as to obtain the first feature data set and the second feature data set. The domain feature alignment layer comprises a feature projection layer, a domain classifier, and a gradient inversion layer. The feature projection layer is configured to map the first feature data set and the second feature data set to the same feature dimension space to obtain the third feature data set. The domain classifier is configured to determine whether each feature data in the third feature data set belongs to the special data or the general data. The gradient inversion layer is configured to determine the distribution difference between the feature distribution of the special data and the feature distribution of the general data based on the determination result output by the domain classifier. The first detection head is configured to perform target detection on the third feature data set to obtain the first prediction result. The second detection head is configured to perform target detection on the third feature data set to obtain the second prediction result. The third detection head is configured to perform target detection on the third feature data set to obtain the third prediction result. The fourth detection head is configured to perform target detection on the third feature data set to obtain the fourth prediction result.

3. The method of claim 2, wherein, The feature extraction and fusion processing on the training data set comprises: performing local feature extraction on each training data in the training data set by the local feature backbone to obtain a local feature data set corresponding to the training data set, wherein the local feature data set comprises local texture features of each training data; performing context feature extraction on each training data in the training data set by the global semantic backbone to obtain a global feature data set corresponding to the training data set, wherein the global feature data set comprises global context information of each training data; for each training data, the following steps are performed to obtain an initial feature data set corresponding to the training data set: splicing the local texture features and the global context information corresponding to the training data to obtain a spliced feature map; adjusting the attention weight of the spliced feature map by the attention module to obtain an initial feature data corresponding to the training data; and the initial feature data set comprises the initial feature data corresponding to each training data. The initial feature dataset is processed by the multi-scale feature fusion layer to generate feature maps at different scales, so as to obtain the first feature dataset and the second feature dataset.

4. The method of claim 2, wherein, The first feature dataset and the second feature dataset are mapped to the same feature dimension space by the feature projection layer to obtain the third feature dataset, and the distribution difference between the feature distribution of the special data and the feature distribution of the general data is determined according to the third feature dataset. The first feature dataset and the second feature dataset are mapped to the same feature dimension space by the feature projection layer to obtain the third feature dataset. The type of each feature data in the third feature dataset is judged by the domain classifier to obtain the judgment result, which is used to guide the division of the third feature dataset into a first subset and a second subset, the first subset being composed of feature data belonging to the special data, and the second subset being composed of feature data belonging to the general data. The feature distribution of the first subset and the second subset is analyzed according to the judgment result output by the domain classifier by the gradient inversion layer to determine the distribution difference.

5. The method of claim 2, wherein, The first task is a helmet wearing detection task, the second task is a head and shoulder detection task, the third task is a pedestrian detection task, and the fourth task is a vehicle detection task. Each data in the third feature dataset is respectively subjected to target detection to obtain a first prediction result corresponding to the first task, a second prediction result corresponding to the second task, a third prediction result corresponding to the third task, and a fourth prediction result corresponding to the fourth task. The third feature dataset is subjected to target detection by the first detection head to obtain the first prediction result; the first prediction result includes a first bounding box, a first classification result, and a first confidence, the first classification result being used to indicate the target type at the first bounding box, and the first confidence being used to indicate whether there is a target at the first position; The third feature dataset is subjected to target detection by the second detection head to obtain the second prediction result; the second prediction result includes a second bounding box, a second classification result, and a second confidence, the second classification result being used to indicate the target type at the second bounding box, and the second confidence being used to indicate whether there is a target at the second position; The third feature dataset is subjected to target detection by the third detection head to obtain the third prediction result; the third prediction result includes a third bounding box, a third classification result, and a third confidence, the third classification result being used to indicate the target type at the third bounding box, and the third confidence being used to indicate whether there is a target at the third position; The fourth prediction result is obtained by target detection on the third feature data set by the fourth detection head; the fourth prediction result includes a fourth bounding box, a fourth classification result and a fourth confidence, the fourth classification result is used to indicate a target type at the fourth bounding box, and the fourth confidence is used to indicate whether a target exists at a fourth position.

6. The method of claim 5, wherein, The first difference value is composed of a first classification difference value, a first bounding box difference value and a first confidence difference value; the second difference value is composed of a second classification difference value, a second bounding box difference value and a second confidence difference value; the third difference value is composed of a third classification difference value, a third bounding box difference value and a third confidence difference value; and the fourth difference value is composed of a fourth classification difference value, a fourth bounding box difference value and a fourth confidence difference value; The total loss value is obtained according to the first difference value between the first prediction result and the first label, the second difference value between the second prediction result and the second label, the third difference value between the third prediction result and the third label, the fourth difference value between the fourth prediction result and the fourth label and the distribution difference value, and the parameters of the multi-task target detection model are updated to minimize the total loss value, to obtain the trained multi-task target detection model, including: The first weight difference value is obtained according to a weighted sum of the first classification difference value, the first bounding box difference value and the first confidence difference value; The second weight difference value is obtained according to a weighted sum of the second classification difference value, the second bounding box difference value and the second confidence difference value; The third weight difference value is obtained according to a weighted sum of the third classification difference value, the third bounding box difference value and the third confidence difference value; The fourth weight difference value is obtained according to a weighted sum of the fourth classification difference value, the fourth bounding box difference value and the fourth confidence difference value; The total loss value is obtained according to a weighted sum of the first weight difference value, the second weight difference value, the third weight difference value, the fourth weight difference value and the distribution difference value, and the parameters of the multi-task target detection model are updated to minimize the total loss value, to obtain the trained multi-task target detection model.

7. The method of claim 3, wherein, The multi-scale feature fusion layer adopts a bidirectional feature pyramid structure, each initial feature data in the initial feature data set includes original output features output by the backbone feature extraction layer at different scales; The first feature data set and the second feature data set are obtained by processing the initial feature data set through the multi-scale feature fusion layer, including: For each initial feature data, obtain a fusion output feature of the initial feature data at different scales as a feature map of the initial feature data at different scales; wherein when obtaining the fusion output feature at the i-th scale, the following steps are performed: performing down-sampling processing on the fusion output feature at the (i-1)-th scale to obtain a down-sampling result, i is a positive integer; performing up-sampling processing on the fusion output feature at the (i+1)-th scale to obtain an up-sampling result; performing weighted fusion on the down-sampling result, the up-sampling result, and the original output feature at the i-th scale to obtain the fusion output feature of the initial feature data at the i-th scale as the feature map at the i-th scale; According to the fusion output feature of each initial feature data in the initial feature data set at different scales, the first feature data set and the second feature data set are obtained.

8. A method of multi-task object detection, the method comprising: The method comprises: obtaining a to-be-detected image, the to-be-detected image being an image obtained by collecting a traffic road scene; inputting the to-be-detected image into a multi-task target detection model to obtain a first detection result, a second detection result, a third detection result, and a fourth detection result, the first detection result being a result obtained by performing a helmet wearing detection task on the to-be-detected image, the second detection result being a result obtained by performing a head-shoulder detection task on the to-be-detected image, the third detection result being a result obtained by performing a pedestrian detection task on the to-be-detected image, and the fourth detection result being a result obtained by performing a vehicle detection task on the to-be-detected image, the multi-task target detection model being trained by using the training method in any one of claims 1 to 7.

9. The method of claim 8, wherein, The method further comprises: selecting at least one detection result from the first detection result, the second detection result, the third detection result, and the fourth detection result according to the requirements of a detection business.

10. An electronic device comprising a memory, one or more processors, and a computer program stored in the memory and executable on the one or more processors, wherein, The one or more processors execute the computer program, so that the electronic device implements the method in any one of claims 1 to 7, or so that the electronic device implements the method in any one of claims 8 to 9.

Images