Multi-task network training method and related device

Abstract

The embodiment of the invention discloses a multi-task network training method and a related device, and the method comprises the steps: obtaining a data training set which comprises a plurality of images; performing feature extraction on the plurality of images in the data training set to obtain a first feature and a second feature; determining a comprehensive feature according to the first feature and the second feature; determining a target detection prediction result according to the second feature; determining a semantic segmentation prediction result according to the comprehensive features and the target detection prediction result; and determining a multi-task loss function according to the target detection prediction result and the semantic segmentation prediction result, and optimizing the training of the multi-task network based on the multi-task loss function. According to the method and the device, the function of multi-task information interaction can be realized based on the training framework of the multi-task network, so that the multi-task network can realize accurate identification under the condition that the data annotation is missing.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multi-task network training method and related devices. Background Art

[0002] With the development of autonomous driving technology, automatic object recognition has become the key to autonomous driving. Currently, synchronously trained object detection and semantic segmentation are generally used to automatically identify objects. Object detection is used to detect different objects and locations in the acquired image, so as to better identify and track other vehicles, pedestrians, obstacles, etc.; semantic segmentation is used to divide the acquired image into different semantic areas, such as roads, buildings, pedestrians, vehicles, etc., so as to better understand the scene. Combining object detection and semantic segmentation can provide autonomous driving with scene understanding, environmental perception, and obstacle recognition capabilities, thereby helping the autonomous driving system make safe and reasonable decisions.

[0003] The current multi-task networks for semantic segmentation and object detection have high requirements for training data. When data has missing labels, it cannot be used for training the multi-task network, which easily leads to data waste. Therefore, when labeled data resources are tight, the requirements for data labeling are high, which easily leads to difficulty in training the multi-task network. Summary of the invention

[0004] The embodiments of the present application provide a multi-task network training method and related devices, which realize the function of multi-task information interaction through the training framework of the multi-task network, flexibly adapt to the situation of misaligned sub-task labeling categories and incomplete data labeling, and make full use of existing data for training, so that the multi-task network can achieve accurate recognition in the absence of data labeling.

[0005] In a first aspect, an embodiment of the present application provides a multi-task network training method, comprising:

[0006] Acquire a data training set, wherein the data training set includes a plurality of images;

[0007] Performing feature extraction on multiple images in the data training set to obtain a first feature and a second feature, wherein the first feature includes multiple feature vectors corresponding to the multiple images at different scales, and the second feature is multiple detection-related feature vectors obtained by performing feature extraction on the multiple feature vectors at different scales;

[0008] Determine a comprehensive feature according to the first feature and the second feature, wherein the comprehensive feature is used to represent a feature obtained by fusing a plurality of feature vectors corresponding to different image scales;

[0009] Determine a target detection prediction result according to the second feature;

[0010] Determine a semantic segmentation prediction result according to the comprehensive features and the target detection prediction result;

[0011] A multi-task loss function is determined according to the target detection prediction result and the semantic segmentation prediction result, and the training of the multi-task network is optimized based on the multi-task loss function.

[0012] In a second aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory storing execution instructions, wherein the memory stores one or more programs; when the processor executes the execution instructions stored in the memory, the processor executes the method described in the first aspect.

[0013] In a third aspect, an embodiment of the present application provides a computer-readable storage medium storing an energy data management program, including execution instructions. When a processor of an electronic device executes the execution instructions, the processor executes the method described in the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the first aspect of the embodiment of the present application. The computer program product may be a software installation package.

[0015] By implementing the embodiment of the present application, firstly, a data training set is obtained, and the data training set includes multiple images; then, feature extraction is performed on the multiple images in the data training set to obtain a first feature and a second feature, wherein the first feature includes multiple feature vectors corresponding to the multiple images at different scales, and the second feature is multiple detection-related feature vectors obtained by feature extraction of the multiple feature vectors at different scales; then, a comprehensive feature is determined based on the first feature and the second feature, and the comprehensive feature is used to characterize the features obtained after the fusion of multiple feature vectors corresponding to different scales of the image; then, the target detection prediction result is determined based on the second feature; then, the semantic segmentation prediction result is determined based on the comprehensive feature and the target detection prediction result; finally, a multi-task loss function is determined based on the target detection prediction result and the semantic segmentation prediction result, and the training of the multi-task network is optimized based on the multi-task loss function. In this way, the training framework based on the multi-task network realizes the function of multi-task information interaction, flexibly adapts to the situation of misaligned sub-task labeling categories and incomplete data labeling, and makes full use of existing data for training, so that the multi-task network can achieve accurate recognition in the absence of data labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 is a system architecture diagram of an autonomous driving system provided by an embodiment of the present application;

[0018] Figure 2 is a system architecture diagram of a network training system provided by an embodiment of the present application;

[0019] Figure 3 It is a flowchart of a multi-task network training method provided in an embodiment of the present application;

[0020] Figure 4 It is a flowchart of a method for obtaining a data training set provided in an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of a feature fusion method provided in an embodiment of the present application;

[0022] Figure 6 is a schematic diagram of a partial label training method provided in an embodiment of the present application;

[0023] Figure 7 It is a functional unit composition block diagram of a multi-task network training device provided in an embodiment of the present application;

[0024] Figure 8 It is a block diagram of the functional units of another multi-task network training device provided in an embodiment of the present application;

[0025] Fig. 9 It is a structural schematic diagram of an electronic device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0028] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0029] The current multi-task network has high requirements for training data. If the data has missing labels, it cannot be used for training the multi-task network, which easily leads to data waste. There is also an imbalance in the number and types of categories of sub-task training data. The loss magnitude and convergence speed between sub-tasks are different, and the strength of the correlation between sub-tasks is uncertain. Therefore, how to optimize the multi-task network and how to assign training weights to two sub-tasks are still technical problems that need to be solved urgently.

[0030] See also Figure 1 , Figure 1 1 is a system architecture diagram of an automatic driving system provided by an embodiment of the present application. Figure 1 As shown, the automatic driving system 10 includes an environment perception and positioning module 101, a decision-making and planning module 102, and an execution control module 103. The environment perception and positioning module 101 also includes a training module 1011. The environment perception and positioning module 101 is mainly used for environment perception and positioning services, mainly through sensor technology and cameras, GPS positioning and other technologies to obtain environmental indicators during the driving process of the car, and collect data; the decision-making and planning module 102 mainly makes judgments and guides the next behavior of the vehicle through the collected data; the execution control module 103 mainly controls the throttle, brake and other related systems of the car through signal instructions according to the decision; wherein the training module 1011 may include a network training system, which is mainly used to optimize the multi-task network for target detection and semantic segmentation, so that the final prediction effect of the multi-task network is close to the true value of the manual annotation, so as to better perceive the environment around the vehicle and provide accurate guidance information for the subsequent decision-making and execution of the automatic driving system 10.

[0031] See also Figure 2 , Figure 2 1 is a system architecture diagram of a network training system provided by an embodiment of the present application. Figure 2 As shown, the network training system 20 includes a data category balancing module 201, an information interaction module 202, and a loss weight module 203, wherein the network training system 20 is the main system in the autonomous driving system architecture, and the prediction results of the multi-task network in the network training system 20 provide guidance for the subsequent decision-making and control of autonomous driving. Among them, the image category is replicated and the category is balanced sampled by the data category balancing module 201, so as to improve the generalization ability of the multi-task model for small sample categories in target detection and semantic segmentation tasks, so as to enhance the flexibility of the multi-task model; the information interaction module 202 is used to perform a complete interaction from features to outputs, so as to realize the function of multi-task information interaction; the conditional probability of the network prediction output is modeled by the loss weight module 203, and the uncertainty of each subtask is calculated, so as to dynamically allocate training weights for the subtasks, so as to make the multi-task training process converge faster. Through the network training system 20, it is flexibly adapted to the situation where the subtask annotation categories are not aligned and the data annotation is incomplete, and the existing data is fully utilized to train the entire network, so as to achieve the purpose of enabling the multi-task network to achieve better recognition performance in the case of missing data annotation.

[0032] See also Figure 3 , Figure 3 The present invention provides a flow chart of a multi-task network training method. The method includes the following steps.

[0033] S310, obtaining a data training set, where the data training set includes a plurality of images.

[0034] Among them, see Figure 4 , Figure 4 It is a flow chart of a method for obtaining a data training set provided in an embodiment of the present application.

[0035] The specific steps include:

[0036] S410, determining a category set in the data set.

[0037] Among them, as an example, any vehicle-related picture P in the existing data set can be obtained first, and then each category contained in the picture P is counted according to the segmentation and annotation, such as motor vehicles, non-motor vehicles, pedestrians, animals, etc., and a category set S is formed according to these categories.

[0038] S420: Determine the detection quantity of each category according to the category set.

[0039] For each category in the category set S, it is determined whether it has a detection annotation; if a category has a detection annotation, the number of detection boxes N of the category in the image P is counted, and the number of the category is N; if a category has no detection annotation, the number of the category is assumed to be 1. For example, the category may be pedestrians, and there are 3 detection boxes of pedestrians in the image P, so it can be confirmed that the number of pedestrians in the image P is 3.

[0040] S430, counting the categories of all images in the data set and the number of detections corresponding to the categories to obtain data category distribution.

[0041] Among them, the steps S410-S420 are looped to count the number of categories in all pictures in the entire data set, and the data category distribution of the entire data set under multi-task is accumulated, where the data category distribution can be: {category: total number}.

[0042] S440, determining the image category with the largest total number according to the data category distribution, and expanding the data set according to the image category with the largest total number.

[0043] Among them, according to the statistical distribution of multi-task data categories, the total number of the largest category is used as a benchmark, and the ratio of the total number of the remaining categories in the data set to the total number of the largest category is calculated, and the sample sets of other categories are expanded according to the ratio to obtain the expanded data set. For example, the largest category is motor vehicles, the total number of motor vehicles is 90, and the total number of non-motor vehicles in other categories is 20, then 90 / 20=4.5, but the number of non-motor vehicles is expanded by 4.5 times, so that the total number of non-motor vehicles reaches 90, thereby expanding the data set.

[0044] S450, performing balanced sampling of different image categories on the expanded data set to obtain a data training set.

[0045] During training, the same number of samples are sampled for all categories in the expanded data set. For example, 50 samples are taken for each of the categories of motor vehicles, non-motor vehicles, pedestrians, obstacles, etc.

[0046] It can be seen that the statistics of semantic segmentation categories and target detection numbers are combined, and the data set is expanded according to the above ratio, so as to achieve data category balance under multi-task.

[0047] S320, perform feature extraction on the multiple images in the data training set to obtain a first feature and a second feature, wherein the first feature includes a plurality of feature vectors corresponding to the multiple images at different scales, and the second feature is a plurality of detection-related feature vectors obtained by performing feature extraction on the multiple feature vectors at different scales.

[0048] Among them, feature extraction of vehicle-related images in the data training set is global feature extraction for multi-task features. Multi-tasks can include target detection and semantic segmentation. The first feature belongs to the global shared feature of multi-tasks. Among them, images can have multiple scales. Multi-scale refers to signal sampling at different scales. Different features can be observed at different scales. Simply put, the features perceived by people are different when the distance from a fixed object is different, that is, when the imaging size of an object in the field of view is different (different scale), the features it exhibits are also different. Images of different scales are suitable for different tasks. In this embodiment, in order to optimize the multi-task network, it is necessary to fuse features at different scales. For example, objects in vehicle-related images have similarities in local areas, such as color, texture, etc. In this embodiment, multi-scale fusion can be specifically manifested as including both global overall information and local detail information in the output feature map to better perceive the environment around the vehicle. Among them, after obtaining the first feature shared globally by multiple tasks, feature screening is performed in the above-mentioned shared features to select features related to target detection as the second feature.

[0049] S330, determining a comprehensive feature according to the first feature and the second feature, where the comprehensive feature is used to represent a feature obtained by fusing a plurality of feature vectors corresponding to different image scales.

[0050] Among them, multiple feature vectors at multiple scales in the first feature can be obtained, such as the first feature vector at the first scale, the second feature vector at the second scale, and then the first detection-related feature vector is extracted from the first feature vector at the first scale; the first detection-related feature vector is mapped to the segmentation feature space through mapping, and the first interaction parameter vector and the second interaction parameter vector at the first scale are generated respectively. Among them, the segmentation feature space can be a feature screening of the first feature globally shared by multiple tasks, selecting features related to semantic segmentation, and forming the first interaction parameter vector and the second interaction parameter vector based on the selected features related to semantic segmentation, and the first interaction parameter vector and the second interaction parameter vector are obtained by the same feature mapping. The first feature vector, the first interaction parameter vector and the second interaction parameter vector at the first scale are weighted multiple times to obtain the first comprehensive feature vector at the first scale after fusion; the feature vector related to the second detection is extracted from the second feature vector at the second scale, and the feature vector related to the second detection is mapped to the segmentation feature space through mapping, and the third interaction parameter vector and the fourth interaction parameter vector at the second scale are respectively generated, wherein the third interaction parameter vector and the fourth interaction parameter vector are obtained by the same feature mapping, and then the first comprehensive feature vector at the first scale is taken, and the first comprehensive feature vector is expanded to the same dimension as the second feature vector through an upsampling operation, and each element in the first comprehensive feature vector and the second feature vector is added element by element to generate a new second feature vector after multi-scale fusion; the new second feature vector, the third interaction parameter vector and the fourth interaction parameter vector are weighted multiple times to obtain the second comprehensive feature vector at the second scale after fusion, and the second comprehensive feature vector can be determined as a comprehensive feature.

[0051] The first scale and the second scale are just any two scales under multiple scales, and the first eigenvector and the second eigenvector are just any two eigenvectors among the multiple eigenvectors.

[0052] S340: Determine a target detection prediction result according to the second feature.

[0053] Among them, an exemplary method is to obtain an accurate candidate area according to the second feature, extract a candidate area feature map according to the accurate candidate area, and then determine the category of the target according to the candidate area feature map, so as to output the category prediction result of the target detection. An exemplary method is to detect categories such as pedestrians, motor vehicles, and traffic signal indicators near the vehicle according to the second feature.

[0054] S350, determining a semantic segmentation prediction result according to the comprehensive features and the target detection prediction result.

[0055] Among them, a semantic segmentation prediction result for reference can be determined based on the comprehensive features, and then the same categories as those in the semantic segmentation prediction result for reference are selected from the categories of the target detection prediction results, and then these same categories are added by 1 to reach the same dimension as the number of categories of the semantic segmentation prediction result for reference, so as to obtain a new target detection prediction result, and then the semantic segmentation prediction result for reference is dot-product weighted according to the new target detection prediction result to obtain the final semantic segmentation prediction result. An example of the method of adding 1 is that the detection outputs 3 categories, specifically [motor vehicle, non-motor vehicle, pedestrian], whose logical value logits is [2, 3, 5], and the segmentation outputs 5 categories, specifically [motor vehicle, non-motor vehicle, pedestrian, cone barrel, warning column], then the logits of the detection output can be added by 1, becoming [2, 3, 5, 1, 1], thereby achieving dimensional alignment.

[0056] S360, determining a multi-task loss function according to the target detection prediction result and the semantic segmentation prediction result, and optimizing the training of the multi-task network based on the multi-task loss function.

[0057] Among them, the multi-task may include multiple sub-tasks. In this embodiment, it may include two sub-tasks, namely, the target detection sub-task and the semantic segmentation sub-task. Assuming that in the case of simultaneous multi-task training, for any sub-task T i , the uncertainty caused by other tasks is δ i , for the classification task, assuming that p(y i |f w (x)) obeys the Boltzmann distribution, then:

[0058]

[0059] Where p represents the probability distribution, i = 1 or 2, y i is the prediction output result (including target detection prediction result and semantic segmentation prediction result); f w (x) is the predicted output probability of the subtask, w represents the parameter, f represents the mapping function; C represents the number of output categories (3 categories, 5 categories); the activation function Softmax is a commonly used mathematical function, usually used in multi-classification problems, which maps a set of arbitrary real values ​​to a value range with probabilistic meaning, so that each value is between 0 and 1 and the sum of all values ​​is equal to 1. In neural networks, Softmax is often used to convert network output into probability distribution for classification or probability prediction. Given a vector z = (z1, z2, ..., z k ), the Softmax function can be expressed as:

[0060]

[0061] Where e is a natural constant and k is the length of vector z. The Softmax function converts each element in vector z into a real number between 0 and 1, indicating the importance or probability of the element among all elements. The output of the Softmax function can be viewed as a probability distribution, in which each element represents the probability of the corresponding category. In neural networks, Softmax is usually used as the activation function of the last output layer. Using the Softmax function can make the probability distribution of the network output clearer and more intuitive, making it easier to perform classification or probability prediction. For regression tasks, assuming p(y i |f w (x)) obeys the Laplace distribution, then:

[0062]

[0063] Among them, further using the multi-task negative log-likelihood function can obtain the uncertainty-based multi-task loss function as follows:

[0064]

[0065] Among them, in order to adapt to the situation where the subtask is not fully labeled, the loss of the subtask can also be adaptively calculated according to the following loss function formula.

[0066]

[0067] Among them, L i is the loss value of the subtask, For subtask T i The loss function calculation formula is Softmax for classification subtasks and mean square error MSE for regression subtasks. MSE is a commonly used loss function that measures the difference between the model prediction value and the true value. In regression problems, MSE is a widely used loss function. Given a set of prediction values and the corresponding true value y=(y1,y2,...,y n ), MSE can be defined as:

[0068]

[0069] Where n is the number of samples. MSE calculates the average of the squared errors between the predicted values ​​and the true values. The smaller the value of MSE, the closer the model prediction is to the true value. MSE can be used in supervised learning, such as linear regression, polynomial regression, and neural networks. In neural networks, MSE is often used for regression problems, such as predicting house prices, predicting stock prices, etc. By optimizing the MSE loss function, the parameters of the model can be adjusted to minimize the error between the predicted values ​​and the true values, thereby improving the performance of the model.

[0070] It can be seen that in this embodiment, the function of multi-task information interaction is realized through the training framework of the multi-task network, which flexibly adapts to the situations of misaligned sub-task labeling categories and incomplete data labeling, and makes full use of existing data for training, so that the multi-task network can achieve accurate recognition in the absence of data labeling, thereby better perceiving the environment around the vehicle and providing accurate guidance information for autonomous driving tracking and decision-making.

[0071] In a possible embodiment, obtaining a data training set includes: obtaining an initial image set, determining an image category distribution for a first category of images in the initial image set through semantic segmentation annotation and target detection annotation, wherein the image category distribution includes categories and total numbers; determining an image category with the largest total number according to the image category distribution; based on a ratio of a first total number corresponding to the image category with the largest total number to a second total number corresponding to other image categories, copying images of the other image categories based on the ratio so that the total number of the other image categories reaches the first total number, wherein the copied images are second category images, and the first category images and the second category images constitute an extended image set; performing balanced sampling of different image categories on the extended image set to obtain the data training set.

[0072] Among them, each category contained in each image in the initially acquired vehicle-related image set can be counted according to the segmentation annotation, and the categories contained in each image can be counted, and then it can be determined whether each category contained in each image has a detection annotation; if there is a detection annotation, the number of detection boxes of the category in the corresponding image is counted, and the number of detection boxes is determined as the number of the category; if there is no detection annotation, the number of the category is defaulted to 1; the number of categories of all images in the initial image set is counted, and the image category distribution under multiple tasks is accumulated, where the data category distribution can be: {category: total number}. Exemplary data category distribution can include: {pedestrians: 47}, {motor vehicles: 58}, {warning columns: 20}, {cones: 18}, {animals: 10}, etc.

[0073] Among them, according to the statistical distribution of image categories, the total number of the largest category is used as a benchmark, and the total number of the remaining categories in the data set is calculated to be equal to the total number of the largest category. The image set is expanded according to the ratio so that the total number of other categories is the same as the total number of the largest category. For example, the largest category is pedestrians. The total number of pedestrians is 50, and the total number of cones is 10, so 50 / 10=5. However, the number of cones is expanded by 5 times to reach a total number of 50, thereby obtaining an expanded vehicle-related image set. Then, the above-mentioned expanded vehicle-related image set is subjected to category balanced sampling so that each category uses the same number of samples.

[0074] It can be seen that in this embodiment, on the one hand, the correlation between the data annotations of semantic segmentation and target detection is combined, and the situation of misalignment of data volume and category types between semantic segmentation and target detection is also compatible, which can effectively generate a relatively category-balanced training data set, thereby achieving the purpose of improving the generalization ability of the multi-task model for small sample categories in semantic segmentation and target detection; on the other hand, the annotation categories of semantic segmentation and target detection can be flexibly configured according to application requirements, and the annotation amounts between the two subtasks can also be different, which greatly reduces the requirements for data annotation.

[0075] In a possible embodiment, the multiple feature vectors include a first feature vector and a second feature vector, the multiple detection-related feature vectors include a first detection-related feature vector and a second detection-related feature vector, and determining the comprehensive feature based on the first feature and the second feature includes: mapping the first detection-related feature vector to a segmentation feature space to generate a first interaction parameter and a second interaction parameter, the segmentation feature space being used to characterize a set space composed of multiple segmentation-related feature vectors obtained by feature extraction of the multiple feature vectors at different scales; performing feature fusion based on the first interaction parameter, the second interaction parameter and the first feature vector to obtain a first comprehensive feature; fusing the first comprehensive feature and the second feature vector to generate a new second feature vector; mapping the second detection-related feature vector to the segmentation feature space to generate a third interaction parameter and a fourth interaction parameter; performing feature fusion based on the third interaction parameter, the fourth interaction parameter and the new second feature vector to obtain a second comprehensive feature, and determining the second comprehensive feature as the comprehensive feature.

[0076] Among them, by way of example, the feature vector f1 at the m1 scale and the feature vector f2 at the m2 scale in the first feature can be obtained, and then the detection-related feature vector d1 is extracted from the f1 feature vector at the m1 scale; the detection-related feature vector d1 is mapped to the segmentation feature space through mapping, and two interaction parameter vectors a1 and b1 at the m1 scale are generated respectively, wherein a1 and b1 are obtained by the same feature mapping, and the f1 feature vector at the m1 scale and the interaction parameter vectors a1 and b1 are weighted multiple times to obtain the fused comprehensive feature vector c1 at the m1 scale.

[0077] Among them, the multiple weightings in c1 obtained by multiple weightings can include two attention weighting operations, see Figure 5 , Figure 5 Schematic diagram of a feature fusion method provided in an embodiment of the present application. Figure 5As shown, the specific steps include: firstly, the input feature vector X is weighted with the input interaction parameter vector A to obtain the weighted feature X'; wherein, the input feature vector X can be the first feature, and the interaction parameter vector A can be one of the interaction parameters obtained by mapping the detection-related feature vector to the segmentation feature space. The feature vector X is first weighted with the input interaction parameter vector A before the convolution layer, which can be understood as providing guidance for the semantic segmentation prediction result according to the target detection prediction result, and indicating in advance which channel information is more effective. Feature extraction is performed on the weighted feature X' to obtain a new feature vector F; wherein, the feature extraction of the weighted feature X' through the convolution layer can be exemplified by vehicle texture features and vehicle color features in vehicle-related images. The new feature vector F is weighted with the incoming interaction parameter vector B to obtain a weighted new feature vector F'; wherein the interaction parameter vector B can be another interaction parameter obtained by mapping the detection-related feature vector to the segmentation feature space, which is different from A. The weighting of the new feature vector F and the incoming interaction parameter vector B is after the convolution layer, which can be understood as aligning the features of target detection and semantic segmentation in the channel dimension. The weighted new feature vector F' passes through the batch normalization layer and the activation function in turn, and the feature vector Y is output. Among them, using the batch normalization layer before the activation function can improve the output accuracy and output a more stable distribution result. Batch normalization is a commonly used neural network optimization technology used to speed up the training of neural networks and improve the generalization performance of the model. It normalizes the intermediate layers of the neural network so that the mean and variance of the input data remain stable, thereby reducing the problem of internal covariate shift in the network. The activation function is a nonlinear function that is usually used in neural networks to perform nonlinear transformations on the output of neurons, thereby increasing the network's expression and fitting capabilities. The main function of the activation function is to map the output of the neuron into a nonlinear space, so that the neural network can better handle complex nonlinear problems. The output value processed by the activation function can be used to represent the degree of excitement of the neuron. The activation function is usually applied to the hidden layer and output layer of the neural network. Commonly used activation functions include Sigmoid function, ReLU function, Tanh function and Softmax function.

[0078] Among them, after obtaining the comprehensive feature vector c1, the detection-related feature vector d2 is extracted from the feature vector f2 at the m2 scale, and the detection-related feature vector d2 is mapped to the segmentation feature space through mapping, and two interaction parameter vectors a2 and b2 at the m2 scale are generated respectively, wherein a2 and b2 are obtained by the same feature mapping, and then the comprehensive feature vector c1 at the m1 scale is taken, and c1 is expanded to the same dimension as the feature vector f2 through an upsampling operation, and each element in c1 and f2 is added element by element to generate a new feature vector f'2 after multi-scale fusion; the new feature vector f'2 and the interaction parameter vectors a2 and b2 are weighted multiple times to obtain the fused comprehensive feature vector c2 at the m2 scale, and the comprehensive feature vector c2 can be determined as a comprehensive feature.

[0079] It can be seen that in this embodiment, the interaction parameters converted from the scale features of target detection are weighted, and the correlation between the two target detection and semantic segmentation is strengthened through two attention weighting operations.

[0080] In a possible embodiment, the multiple feature vectors also include a third feature vector, and the multiple detection-related feature vectors also include a third detection-related feature vector. After obtaining the second comprehensive feature, the method also includes: fusing the second comprehensive feature and the third feature vector to generate a new third feature vector; mapping the third detection-related feature vector to the segmentation feature space to generate a fifth interaction parameter and a sixth interaction parameter; performing feature fusion according to the fifth interaction parameter, the sixth interaction parameter and the new third feature vector to obtain a third comprehensive feature, and determining the third comprehensive feature as the comprehensive feature.

[0081] Among them, the feature vector f1 at the m1 scale, the feature vector f2 at the m2 scale, and the feature vector f3 at the m3 scale in the first feature can be obtained by way of example. According to the above, after obtaining the comprehensive feature vector c2, the feature vector f3 at the m3 scale is extracted with respect to the detection-related feature vector d3, and the detection-related feature vector d3 is mapped to the segmentation feature space through mapping, and two interaction parameter vectors a3 and b3 at the m3 scale are generated respectively, wherein a3 and b3 are obtained by the same feature mapping, and then the comprehensive feature vector c2 at the m2 scale is taken, and the detection-related feature vector d3 is mapped to the segmentation feature space through the upsampling operation. Expand c2 to the same dimension as the feature vector f3, add each element in c2 and f3 element by element to generate a new feature vector after multi-scale fusion; weight the new feature vector f'3 and the interaction parameter vector a3 to obtain the weighted feature f"3; perform image feature extraction on the weighted feature f"3 to obtain a new feature vector e, weight the new feature vector F and the interaction parameter vector b3 to obtain a weighted new feature vector e'; pass the weighted new feature vector e' through the batch normalization layer and the activation function layer in sequence to obtain the comprehensive feature c3, and the comprehensive feature c3 can be determined as the comprehensive feature.

[0082] Among them, an image can have multiple different scales. In this embodiment, only any three of the multiple scales are extracted as examples for explanation. When there are only two scales, the comprehensive feature c2 can be determined as the comprehensive feature, and when there are only three scales, the comprehensive feature c3 can be determined as the comprehensive feature. The comprehensive feature is obtained by fusing the features of all scales in the image.

[0083] It can be seen that in this embodiment, by fusing multi-scale features, the performance of the model can be greatly improved.

[0084] In a possible embodiment, the feature fusion according to the first interaction parameter, the second interaction parameter and the first feature vector to obtain the first comprehensive feature includes: determining a new first feature vector according to the weight of the first interaction parameter and the weight of the first feature vector, the weight of the first interaction parameter and the weight of the first feature vector are obtained by respectively calculating according to different parts of the first interaction parameter and the first feature vector; performing feature extraction on the new first feature vector to obtain a fourth feature vector; determining a new fourth feature vector according to the weight of the second interaction parameter and the weight of the fourth feature vector, the weight of the second interaction parameter and the weight of the fourth feature vector are obtained by respectively calculating according to different parts of the second interaction parameter and the fourth feature vector, and the first comprehensive feature is determined based on the new fourth feature vector.

[0085] Among them, the weights of the above-mentioned interaction parameters and feature vectors are calculated through the attention mechanism, and different weights are assigned according to different parts of the input data to control the degree of attention of the multi-task network to different parts, thereby focusing the attention of the multi-task network on the key parts of the input data and improving the representation and generalization capabilities of the multi-task network.

[0086] Among them, the weight of the feature vector M and the weight of the interaction parameter vector N are calculated through the attention mechanism, and the above weights are weighted to obtain the weighted feature M'; image feature extraction is performed on the weighted feature M' to obtain a new feature vector O; the weight of the new feature vector O and the weight of the interaction parameter vector P are calculated through the attention mechanism, and the above weights are weighted to obtain a weighted new feature vector O'; the weighted new feature vector O' is passed through the batch normalization layer and the activation function in turn to obtain the first comprehensive feature.

[0087] It can be seen that in this embodiment, the correlation between the two target detection and semantic segmentation is strengthened through two attention weighting operations.

[0088] In a possible embodiment, determining the semantic segmentation prediction result based on the comprehensive features and the target detection prediction result includes: determining a first semantic segmentation prediction result based on the comprehensive features; determining a second category in the target detection prediction result that is the same as the first category in the first semantic segmentation prediction result; dimensionally aligning the second category based on the first category so that the number of the second categories is the same as the number of the first categories, thereby obtaining a new target detection prediction result; determining a second semantic segmentation prediction result based on the new target detection prediction result and the first semantic segmentation prediction result, and determining the second semantic segmentation prediction result as the semantic segmentation prediction result.

[0089] Among them, a semantic segmentation prediction result for reference can be determined based on the comprehensive features, and then the categories in the target detection prediction result are selected from the categories in the semantic segmentation prediction result for reference, and then these identical categories are added by 1 to reach the same dimension as the number of categories in the semantic segmentation prediction result for reference, so as to obtain a new target detection prediction result, and then the semantic segmentation prediction result for reference is weighted by dot product according to the new target detection prediction result to obtain the final semantic segmentation prediction result. An example of the method of adding 1 is that the detection outputs 3 categories, specifically [motor vehicle, non-motor vehicle, pedestrian], whose logical value logits is [2, 3, 5], and the segmentation outputs 5 categories, specifically [motor vehicle, non-motor vehicle, pedestrian, cone barrel, warning column], then the logits of the detection output can be added by 1, becoming [2, 3, 5, 1, 1], thereby achieving dimensional alignment.

[0090] Among them, based on the feature interaction and output interaction, a forward reasoning and back propagation partial label training is added, please refer to Figure 6 , Figure 6 Schematic diagram of a partial label training method provided in an embodiment of the present application. Figure 6 As shown, the solid single arrow is the forward reasoning path, and the dotted single arrow is the reverse propagation path. In the forward reasoning path, the global shared features output by the feature extraction module 601 are first successively obtained through the detection head module 602 and the detection output module 604 to obtain the detection features and the logits of the detection output classification, and the detection loss is calculated by the detection loss module 606 according to the logits of the detection output classification. Among them, the detection head module 602 is used to extract features related to detection according to the global shared features, the detection output module 604 is used to determine the logits of the detection category according to the detection features, and the detection loss module 606 is used to determine the detection loss according to the logits of the detection output classification; then the detection features and the global shared features are input into the segmentation head module 603 for fusion to obtain the comprehensive features. The segmentation head module 603 includes a convolution module and an upsampling module. The convolution module is used for mapping, mapping the detection-related features into the segmentation feature space, and strengthening the interaction between detection and segmentation. The upsampling module is used to increase the resolution of the image, improve the image quality or increase the details of the feature map. Then, the fused comprehensive features and the logits of the detection output classification are sent to the segmentation output module 605 to obtain the final segmentation output result, and the segmentation loss is determined by the segmentation loss module 607 according to the segmentation output result. The segmentation output module 605 is used for the autocorrelation weighting of the category prediction of the target detection and semantic segmentation output to achieve the category output alignment between the subtasks, and the segmentation loss module 607 is used to calculate the segmentation loss according to the segmentation output result. Figure 6 As shown, in the back propagation path, because the output of the segmentation incorporates the logits of the detection category, when the loss of the segmentation performs gradient back propagation, part of the gradient flow flows along the segmentation output module 605 to the segmentation head module 603, and part flows to the detection output module 604 through the logits of the detection category. Next, the gradient of the segmentation output module 605 is propagated to the segmentation head module 603, and the gradient of the detection output module 604 is propagated to the detection head module 602. Since the segmentation head module 603 incorporates detection-related features, there will also be a branch gradient flow in the segmentation head module 603 that back propagates to the detection head module 602 through the detection features. Therefore, the detection head module 602 will obtain two gradient flows and combine them to optimize itself. Finally, the gradient flows of the two branches of segmentation and detection will be back propagated back to the feature extraction module 601, and supervise the feature extraction module 601 together.

[0091] It can be seen that in this embodiment, by integrating the detection prediction results into the segmentation prediction results, the prediction outputs of the two subtasks are integrated, thereby strengthening the connection between the two subtasks; and through the partial label training method, even in the absence of true value annotations of the detection, while learning the segmentation, the detection will also be optimized accordingly, thereby being able to supervise the entire network and improve the efficiency of training; and incompletely labeled data can also be used for training to expand the overall data set, alleviate the problem of tight labeled data resources, and reduce the requirements and difficulty of data labeling.

[0092] In a possible embodiment, the multi-task includes a plurality of sub-tasks, the plurality of sub-tasks include target detection and semantic segmentation, the multi-task loss function is determined according to the target detection prediction result and the semantic segmentation prediction result, the method includes: determining a target detection loss value L1 according to the target detection prediction result, determining a semantic segmentation loss value L2 according to the semantic segmentation prediction result, and a sub-task loss value L i Satisfies the following formula:

[0093]

[0094] Among them, y i Prediction result for subtask, i = 1 or 2, f w (x) is the output probability of the subtask prediction, is the loss function calculation formula; according to the subtask loss value L i Determine the multi-task loss function.

[0095] Among them, Softmax is used for classification subtasks, and MSE is used for regression subtasks. The semantic segmentation task belongs to the classification subtask; the object detection task includes classification branch and regression branch, that is, it contains two types of loss functions. Softmax can be used for classification tasks, and MSE can be used for regression tasks.

[0096] It can be seen that in this embodiment, the situation where the subtask annotation is incomplete can be adapted and the loss value can be adaptively calculated.

[0097] In a possible embodiment, the subtask loss value L i Determine the multi-task loss function, including: determine the sub-task weights δ according to the uncertainty of the multi-task i ; According to the subtask loss value L through the negative log-likelihood loss function i and the subtask weight δ i Determine the multi-task loss function - log p(y1,y2|f w (x)), the multi-task loss function is calculated based on the following formula:

[0098]

[0099] Among them, in the case of simultaneous multi-task training, for any task T i , the uncertainty caused by other tasks is δ i , for the classification task, assuming that p(y i |f w (x)) obeys the Boltzmann distribution, then:

[0100]

[0101] Where p represents the probability distribution, i = 1 or 2, y i is the prediction output result (including target detection prediction result and semantic segmentation prediction result); f w (x) is the predicted output probability of the subtask, w represents the parameter, f represents the mapping function; C represents the number of output categories; for regression tasks, assuming that p(y i |f w (x)) obeys the Laplace distribution, then:

[0102]

[0103] Further using the multi-task negative log-likelihood function, we can get the uncertainty-based multi-task loss function as follows:

[0104]

[0105] It can be seen that in this embodiment, the conditional probability of the network prediction output is used for modeling and the uncertainty of each subtask is calculated, so that the training weights are dynamically allocated to the subtasks, so that the multi-task training process converges faster.

[0106] In a possible embodiment, the multiple images are vehicle-related images.

[0107] Among them, vehicle-related images can be vehicle images, vehicle surrounding environment images and road surrounding environment images. Vehicles such as cars, trucks, bicycles, etc. are detected through vehicle images, and various objects around the vehicle such as pedestrians, obstacles, etc. are detected through vehicle surrounding environment images. Road signs such as traffic signs, lane markings, parking space markings and pedestrian crossing lines are detected through road surrounding environment images.

[0108] For the above embodiments, please refer to Figure 7 , Figure 7It is a functional unit composition block diagram of a multi-task network training device provided by an embodiment of the present application. The multi-task network training device 70 includes: a first acquisition unit 71, which is used to acquire a data training set, wherein the data training set includes multiple images; a second acquisition unit 72, which is used for the second acquisition unit to extract features from multiple images in the data training set to obtain first features and second features, wherein the first feature includes multiple feature vectors corresponding to the multiple images at different scales, and the second feature is multiple detection-related feature vectors obtained by extracting features from multiple feature vectors at different scales; a first determination unit 73, which is used to determine a comprehensive feature based on the first feature and the second feature, wherein the comprehensive feature is used to characterize the features obtained by fusing multiple feature vectors corresponding to different scales of the image; a second determination unit 74, which is used to determine a target detection prediction result based on the second feature; a third determination unit 75, which is used to determine a semantic segmentation prediction result based on the comprehensive feature and the target detection prediction result; and a fourth determination unit 76, which is used to determine a multi-task loss function based on the target detection prediction result and the semantic segmentation prediction result.

[0109] In a possible embodiment, in terms of acquiring data, the first acquiring unit 71 is specifically used to: Figure 4 , Figure 4It is a flow chart of a method for obtaining a data training set provided by an embodiment of the present application. The specific steps include: S410, determining a set of categories in a data set. Among them, an exemplary method is to first obtain any vehicle-related picture P in an existing data set, and then count each category contained in the picture P according to the segmentation annotation, for example, it may be a motor vehicle, a non-motor vehicle, a pedestrian, an animal, etc., and form a category set S according to these categories. S420, determining the number of detections of each category according to the category set. Among them, for each category in the category set S, it is determined whether it has a detection annotation; if a category has a detection annotation, the number of detection frames N of the category in the picture P is counted, and the number of the category is N; if a category has no detection annotation, the number of the category is assumed to be 1. An exemplary method is that the category is a pedestrian, and there are 3 detection frames of pedestrians in the picture P, then it can be confirmed that the number of pedestrians in the picture P is 3. S430, counting the categories of all images in the data set and the number of detections corresponding to the categories, and obtaining the data category distribution. Wherein, the steps of S410-S420 are looped to count the number of categories in all images in the entire data set, and the data category distribution of the entire data set under multiple tasks is accumulated, wherein the data category distribution can be: {category: total number}. S440, determine the image category with the largest total number according to the data category distribution, and expand the data set according to the image category with the largest total number. Wherein, according to the statistical distribution of multi-task data categories, the total number of the largest category is used as a benchmark, and the ratio of the total number of the remaining categories in the data set to the total number of the largest category is calculated respectively, and the sample sets of other categories are expanded according to the ratio to obtain the expanded data set. For example, the category with the largest number of motor vehicles is motor vehicles, and the total number of motor vehicles is 90, while the total number of non-motor vehicle categories in other categories is 20, then 90 / 20=4.5, but the number of non-motor vehicle categories is expanded by 4.5 times, so that the total number of non-motor vehicles reaches 90, thereby expanding the data set. S450, perform balanced sampling of different image categories on the expanded data set to obtain a data training set. During training, the same number of samples are sampled for all categories on the expanded data set, and 50 samples may be taken for each category such as motor vehicles, non-motor vehicles, pedestrians, and obstacles. It can be seen that the statistics are combined based on the categories of semantic segmentation and the number of target detections, and the data set is expanded according to the above ratio, so as to achieve data category balance under multiple tasks.

[0110] In one possible embodiment, in terms of acquiring features, the second acquisition unit 72 is specifically used to: extract features from vehicle-related images in the data training set, which is a global feature extraction for multi-task features. Multi-tasks may include target detection and semantic segmentation, and the first feature belongs to the global shared feature of multi-tasks. Among them, images can have multiple scales, and multi-scale refers to signal sampling at different scales. Different features can be observed at different scales. In simple terms, the features perceived by people are different when the distance from a fixed object is different, that is, when the imaging size of an object in the field of view is different (different scale), the features it exhibits are also different. Images of different scales are suitable for different tasks. In this embodiment, in order to optimize the multi-task network, it is necessary to fuse features at different scales. For example, objects in vehicle-related images have similarities in local areas, such as color, texture, etc. In this embodiment, multi-scale fusion can be specifically manifested as including both global overall information and local detail information in the output feature map to better perceive the environment around the vehicle. Among them, after acquiring the first feature shared globally by multiple tasks, feature screening is performed in the above-mentioned shared features, and features related to target detection are selected as the second feature.

[0111] In a possible embodiment, in terms of feature fusion, the first determination unit 73 is specifically used to: obtain multiple feature vectors at multiple scales in the first feature, such as the first feature vector at the first scale, the second feature vector at the second scale, and then extract the first detection-related feature vector from the first feature vector at the first scale; map the first detection-related feature vector to the segmentation feature space through mapping, and generate the first interaction parameter vector and the second interaction parameter vector at the first scale respectively. Among them, the segmentation feature space can be a feature screening of the first feature globally shared by multiple tasks, selecting features related to semantic segmentation, and forming according to the selected features related to semantic segmentation; the first interaction parameter vector and the second interaction parameter vector are obtained by the same feature mapping. The first feature vector, the first interaction parameter vector and the second interaction parameter vector at the first scale are weighted multiple times to obtain the first comprehensive feature vector at the first scale after fusion; the feature vector related to the second detection is extracted from the second feature vector at the second scale, and the feature vector related to the second detection is mapped to the segmentation feature space through mapping, and the third interaction parameter vector and the fourth interaction parameter vector at the second scale are respectively generated, wherein the third interaction parameter vector and the fourth interaction parameter vector are obtained by the same feature mapping, and then the first comprehensive feature vector at the first scale is taken, and the first comprehensive feature vector is expanded to the same dimension as the second feature vector through upsampling operation, and each element in the first comprehensive feature vector and the second feature vector is added element by element to generate a new second feature vector after multi-scale fusion; the new second feature vector, the third interaction parameter vector and the fourth interaction parameter vector are weighted multiple times to obtain the second comprehensive feature vector at the second scale after fusion, and the second comprehensive feature vector can be determined as a comprehensive feature. The first scale and the second scale are just any two scales under multiple scales, and the first feature vector and the second feature vector are also just any two feature vectors among multiple feature vectors.

[0112] In a possible embodiment, in determining the detection output result, the second determination unit 74 is specifically used to: exemplarily obtain an accurate candidate area according to the second feature, extract a candidate area feature map according to the accurate candidate area, and then determine the category of the target according to the candidate area feature map, thereby outputting the category prediction result of the target detection. Exemplarily, the categories of pedestrians, motor vehicles, traffic signal indications, etc. near the vehicle can be detected according to the second feature.

[0113] In one possible embodiment, in determining the segmentation output result, the third determination unit 75 is specifically used to: determine a semantic segmentation prediction result for reference based on the comprehensive features, and then select the same category as the category in the semantic segmentation prediction result for reference from the category of the target detection prediction result, and then fill in the same category by 1 to reach the same dimension as the number of categories of the semantic segmentation prediction result for reference, and obtain a new target detection prediction result, and then perform dot product weighting on the semantic segmentation prediction result for reference according to the new target detection prediction result to obtain the final semantic segmentation prediction result. The method of filling in 1 is exemplified by the detection output of 3 categories, specifically [motor vehicle, non-motor vehicle, pedestrian], whose logical value logits is [2, 3, 5], and the segmentation output of 5 categories, specifically [motor vehicle, non-motor vehicle, pedestrian, cone barrel, warning column], then the logits of the detection output can be filled in by 1, becoming [2, 3, 5, 1, 1], thereby achieving dimensional alignment.

[0114] In a possible embodiment, in terms of determining the loss, the fourth determining unit 76 is specifically used to: a multi-task may include multiple sub-tasks. In this embodiment, it may include two sub-tasks, namely, a target detection sub-task and a semantic segmentation sub-task. Assuming that in the case of simultaneous multi-task training, for any sub-task T i , the uncertainty caused by other tasks is δ i , for the classification task, assuming that p(y i |f w (x)) obeys the Boltzmann distribution, then:

[0115]

[0116] Where p represents the probability distribution, i = 1 or 2, y i is the prediction output result (including target detection prediction result and semantic segmentation prediction result); f w (x) is the predicted output probability of the subtask, w represents the parameter, f represents the mapping function; C represents the number of output categories (3 categories, 5 categories); the activation function Softmax is a commonly used mathematical function, usually used in multi-classification problems, which maps a set of arbitrary real values ​​to a value range with probabilistic meaning, so that each value is between 0 and 1 and the sum of all values ​​is equal to 1. In neural networks, Softmax is often used to convert network output into probability distribution for classification or probability prediction. Given a vector z = (z1, z2, ..., z k ), the Softmax function can be expressed as:

[0117]

[0118] Where e is a natural constant and k is the length of vector z. The Softmax function converts each element in vector z into a real number between 0 and 1, indicating the importance or probability of the element among all elements. The output of the Softmax function can be viewed as a probability distribution, in which each element represents the probability of the corresponding category. In neural networks, Softmax is usually used as the activation function of the last output layer. Using the Softmax function can make the probability distribution of the network output clearer and more intuitive, making it easier to perform classification or probability prediction. For regression tasks, assuming p(y i |f w (x)) obeys the Laplace distribution, then:

[0119]

[0120] Among them, further using the multi-task negative log-likelihood function can obtain the uncertainty-based multi-task loss function as follows:

[0121]

[0122] Among them, in order to adapt to the situation where the subtask is not fully labeled, the loss of the subtask can also be adaptively calculated according to the following loss function formula.

[0123]

[0124] Among them, L i is the loss value of the subtask, For subtask T i The loss function calculation formula is Softmax for classification subtasks and mean square error MSE for regression subtasks. MSE is a commonly used loss function that measures the difference between the model prediction value and the true value. In regression problems, MSE is a widely used loss function. Given a set of prediction values and the corresponding true value y=(y1,y2,...,y n ), MSE can be defined as:

[0125]

[0126] Where n is the number of samples. MSE calculates the average of the squared errors between the predicted values ​​and the true values. The smaller the value of MSE, the closer the model prediction is to the true value. MSE can be used in supervised learning, such as linear regression, polynomial regression, and neural networks. In neural networks, MSE is often used for regression problems, such as predicting house prices, predicting stock prices, etc. By optimizing the MSE loss function, the parameters of the model can be adjusted to minimize the error between the predicted values ​​and the true values, thereby improving the performance of the model.

[0127] It can be understood that since the method embodiment and the device embodiment are different presentation forms of the same technical concept, the content of the method embodiment part in this application should be synchronously adapted to the device embodiment part and will not be repeated here.

[0128] In the case of integrated units, see Figure 8 , Figure 8 is a functional unit block diagram of another multi-task network training device provided in an embodiment of the present application, such as Figure 8 As shown, the multi-task network training device 70 includes: a processing module 702 and a communication module 701. The processing module 702 is used to control and manage the actions of the multi-task network training device 70, for example, to execute the steps of the first acquisition unit 71, the second acquisition unit 72, the first determination unit 73, the second determination unit 74, the third determination unit 75 and the fourth determination unit 76, and / or to perform other processes of the technology described herein. The communication module 701 is used for the interaction between the multi-task network training device 70 and other devices. Figure 8 As shown, the multi-task network training device 70 may further include a storage module 703 , and the storage module 703 is used for program codes and data of the multi-task network training device 70 .

[0129] Among them, the processing module 702 can be a processor or a controller, for example, a central processing unit (CPU), a general processor, a digital signal processor (DSP), an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements a computing function, for example, a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The communication module 701 can be a transceiver, an RF circuit or a communication interface, and the like. The storage module 703 can be a memory.

[0130] Among them, all relevant contents of each scenario involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here. The above multi-task network training device 70 can execute the above Figure 3 The multi-task network training method shown.

[0131] See also Fig. 9 , Fig. 9 is a schematic diagram of the structure of an electronic device proposed in an embodiment of the present application, such as Fig. 9As shown, the electronic device 900 includes a processor 910, a memory 920, a communication interface 930 and one or more programs 921, the one or more programs are stored in the memory and are configured to be executed by the processor, and the program includes part or all of the steps of any multi-task network training method recorded in the above method embodiment when executed, and the processor, memory and communication interface are interconnected and complete the communication work between them;

[0132] The memory may be a volatile memory such as a dynamic random access memory DRAM, or a non-volatile memory such as a mechanical hard disk. The memory is used to store a set of executable program codes, and the processor is used to call the executable program codes stored in the memory, and may execute part or all of the steps of any multi-task network training method described in the multi-task network training method embodiment.

[0133] It can be seen that the electronic device 900 described in the embodiment of the present application first obtains a data training set, which includes multiple images; then extracts features from the multiple images in the data training set to obtain a first feature and a second feature, wherein the first feature includes multiple feature vectors corresponding to the multiple images at different scales, and the second feature is a plurality of detection-related feature vectors obtained by extracting features from the multiple feature vectors at different scales; then determines a comprehensive feature based on the first feature and the second feature, and the comprehensive feature is used to characterize the features obtained after the fusion of multiple feature vectors corresponding to different scales of the image; then determines the target detection prediction result based on the second feature; then determines the semantic segmentation prediction result based on the comprehensive feature and the target detection prediction result; finally determines the multi-task loss function based on the target detection prediction result and the semantic segmentation prediction result. In this way, the training framework based on the multi-task network realizes the function of multi-task information interaction, flexibly adapts to the situation of misaligned sub-task labeling categories and incomplete data labeling, and makes full use of existing data for training, so that the multi-task network can achieve accurate recognition in the absence of data labeling, thereby better perceiving the environment around the vehicle and providing accurate guidance information for tracking and decision-making of autonomous driving.

[0134] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps of any method described in the above method embodiments, and the above computer includes an electronic device.

[0135] The embodiment of the present application also provides a computer program product, the computer program product includes a non-transitory computer-readable storage medium storing a computer program, the computer program is operable to cause a computer to execute some or all of the steps of any method described in the method embodiment. The computer program product may be a software installation package, and the computer includes an electronic device.

[0136] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0137] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0138] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0139] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0140] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0141] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer electronic device (which can be a personal computer, electronic device or network electronic device, etc.) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0142] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, etc.

[0143] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A multi-task network training method, characterized in that: include: Acquire a data training set, wherein the data training set includes a plurality of images; Performing feature extraction on multiple images in the data training set to obtain a first feature and a second feature, wherein the first feature includes multiple feature vectors corresponding to the multiple images at different scales, and the second feature is multiple detection-related feature vectors obtained by performing feature extraction on the multiple feature vectors at different scales; Determine a comprehensive feature according to the first feature and the second feature, where the comprehensive feature is used to represent a feature obtained by fusing multiple feature vectors corresponding to different scales of the image; Determine a target detection prediction result according to the second feature; Determine a semantic segmentation prediction result according to the comprehensive features and the target detection prediction result; A multi-task loss function is determined according to the target detection prediction result and the semantic segmentation prediction result, and the training of the multi-task network is optimized based on the multi-task loss function.

2. The method according to claim 1, characterized in that The step of obtaining a data training set comprises: Acquire an initial image set, and determine image category distribution of a first category of images in the initial image set through semantic segmentation annotation and object detection annotation, wherein the image category distribution includes categories and total numbers; Determine the image category with the largest total number according to the image category distribution; According to a quantity ratio of a first total number corresponding to the image category with the largest total number to a second total number corresponding to other image categories, images of the other image categories are copied based on the quantity ratio so that the total number of the other image categories reaches the first total number, wherein the copied images are images of the second category, and the images of the first category and the images of the second category constitute an extended image set; Balanced sampling of different image categories is performed on the extended image set to obtain the data training set.

3. The method according to claim 1, characterized in that The multiple feature vectors include a first feature vector and a second feature vector, the multiple detection-related feature vectors include a first detection-related feature vector and a second detection-related feature vector, and determining a comprehensive feature according to the first feature and the second feature includes: Mapping the first detection-related feature vector to a segmentation feature space to generate a first interaction parameter and a second interaction parameter, wherein the segmentation feature space is used to represent a set space composed of a plurality of segmentation-related feature vectors obtained by extracting features from the plurality of feature vectors at different scales; Perform feature fusion according to the first interaction parameter, the second interaction parameter, and the first feature vector to obtain a first comprehensive feature; Fusing the first comprehensive feature and the second feature vector to generate a new second feature vector; Mapping the second detection-related feature vector into the segmentation feature space to generate a third interaction parameter and a fourth interaction parameter; Feature fusion is performed according to the third interaction parameter, the fourth interaction parameter and the new second feature vector to obtain a second comprehensive feature, and the second comprehensive feature is determined as the comprehensive feature.

4. The method according to claim 3, characterized in that The multiple feature vectors further include a third feature vector, the multiple detection-related feature vectors further include a third detection-related feature vector, and after obtaining the second comprehensive feature, the method further includes: Fusing the second comprehensive feature with the third feature vector to generate a new third feature vector; Mapping the third detection-related feature vector into the segmentation feature space to generate a fifth interaction parameter and a sixth interaction parameter; Feature fusion is performed according to the fifth interaction parameter, the sixth interaction parameter and the new third feature vector to obtain a third comprehensive feature, and the third comprehensive feature is determined as the comprehensive feature.

5. The method according to claim 3, characterized in that: The performing feature fusion according to the first interaction parameter, the second interaction parameter and the first feature vector to obtain a first comprehensive feature includes: determining a new first feature vector according to a weight of the first interaction parameter and a weight of the first feature vector, wherein the weight of the first interaction parameter and the weight of the first feature vector are obtained by respectively calculating different parts of the first interaction parameter and the first feature vector; Performing feature extraction on the new first feature vector to obtain a fourth feature vector; A new fourth eigenvector is determined according to the weight of the second interaction parameter and the weight of the fourth eigenvector, wherein the weight of the second interaction parameter and the weight of the fourth eigenvector are obtained by respectively calculating the second interaction parameter and different parts of the fourth eigenvector, and the first comprehensive feature is determined according to the new fourth eigenvector.

6. The method according to claim 1, characterized in that The determining of the semantic segmentation prediction result according to the comprehensive feature and the target detection prediction result includes: Determine a first semantic segmentation prediction result according to the comprehensive features; Determining, in the object detection prediction result, a second category that is the same as the first category in the first semantic segmentation prediction result; Dimensionally aligning the second categories according to the first categories so that the number of the second categories is the same as the number of the first categories, thereby obtaining a new target detection prediction result; A second semantic segmentation prediction result is determined according to the new target detection prediction result and the first semantic segmentation prediction result, and the second semantic segmentation prediction result is determined as the semantic segmentation prediction result.

7. The method according to claim 6, characterized in that The multi-task includes a plurality of sub-tasks, and the plurality of sub-tasks include target detection and semantic segmentation. The multi-task loss function is determined according to the target detection prediction result and the semantic segmentation prediction result. The method includes: The target detection loss value L1 is determined according to the target detection prediction result, the semantic segmentation loss value L2 is determined according to the semantic segmentation prediction result, and the subtask loss value L i Satisfies the following formula: Among them, y i Prediction result for subtask, i = 1 or 2, f w (x) is the output probability of the subtask prediction, is the loss function calculation formula; According to the subtask loss value L i Determine the multi-task loss function.

8. The method according to claim 7, characterized in that The subtask loss value L i Determine the multi-task loss function, including: Determine the subtask weights δ according to the uncertainty of the multi-tasks i ; According to the subtask loss value L through the negative log-likelihood loss function i and the subtask weight δ i Determine the multi-task loss function - log p(y1,y2|f w (x)), the multi-task loss function is calculated based on the following formula: Where p represents the probability distribution.

9. The method according to claim 1, characterized in that: The plurality of images are vehicle-related images.

10. An electronic device, characterized in that: The method comprises a processor and a memory storing execution instructions, wherein the memory stores one or more programs; when the processor executes the execution instructions stored in the memory, the processor executes the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: An energy data management program is stored, including execution instructions. When a processor of an electronic device executes the execution instructions, the processor executes the method according to any one of claims 1-9.

Images