Multi-task network training method and electronic device

Abstract

Disclosed herein are a multi-task network training method and an electronic device, the method comprising: obtaining a data training set, the data training set comprising a plurality of images; performing feature extraction on the plurality of images in the data training set, and obtaining a first feature and a second feature; according to the first feature and the second feature, determining a comprehensive feature; according to the second feature, determining a target detection prediction result; according to the comprehensive feature and the target detection prediction result, determining a semantic segmentation prediction result; and, according to the target detection prediction result and the semantic segmentation prediction result, determining a multi-task loss function, and, on the basis of the multi-task loss function, optimizing training of a multi-task network.

Description

Multi-task network training method and electronic equipment

[0001] This disclosure claims priority to the Chinese patent disclosure with application number 202410373734.8 filed with the Patent Office of China on March 28, 2024, and with the disclosure name “Multi-task network training method and related device”, the entire content of which is incorporated by reference into this disclosure. Technical Field

[0002] The embodiments of the present application relate to, but are not limited to, the field of image processing technology, and in particular to a multi-task network training method and electronic device. Background Art

[0003] With the development of autonomous driving technology, automatic target recognition has become the key to autonomous driving. Currently, synchronously trained target detection and semantic segmentation are generally used to automatically recognize targets.

[0004] The current multi-task networks for semantic segmentation and object detection have high requirements for training data. When the data is missing labels, it cannot be used to train the multi-task network, which easily leads to data waste. Therefore, when the labeled data resources are tight, the requirements for data labeling are high, which easily leads to the difficulty of training the multi-task network. Technical Solutions

[0005] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0006] The embodiments of the present application provide a multi-task network training method and electronic device, which realize the function of multi-task information interaction through the training framework of the multi-task network, flexibly adapt to the situation of misaligned sub-task labeling categories and incomplete data labeling, and make full use of existing data for training, so that the multi-task network can achieve accurate recognition in the absence of data labeling.

[0007] In a first aspect, an embodiment of the present application provides a multi-task network training method, comprising:

[0008] Acquire a data training set, wherein the data training set includes a plurality of images;

[0009] Performing feature extraction on multiple images in the data training set to obtain a first feature and a second feature, wherein the first feature includes multiple feature vectors corresponding to the multiple images at different scales, and the second feature is multiple detection-related feature vectors obtained by performing feature extraction on the multiple feature vectors at different scales;

[0010] Determine a comprehensive feature based on the first feature and the second feature, where the comprehensive feature is used to represent a feature obtained by fusing multiple feature vectors corresponding to different image scales;

[0011] Determining a target detection prediction result according to the second feature;

[0012] Determine a semantic segmentation prediction result based on the comprehensive features and the target detection prediction result;

[0013] A multi-task loss function is determined according to the target detection prediction result and the semantic segmentation prediction result, and the training of the multi-task network is optimized based on the multi-task loss function.

[0014] In a second aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory storing execution instructions, wherein the memory stores one or more programs; when the processor executes the execution instructions stored in the memory, the processor executes the method described in the first aspect.

[0015] In a third aspect, an embodiment of the present application provides a computer-readable storage medium storing an energy data management program, including execution instructions. When a processor of an electronic device executes the execution instructions, the processor executes the method described in the first aspect.

[0016] In a fourth aspect, embodiments of the present application provide a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to perform some or all of the steps described in the first aspect of the embodiments of the present application. The computer program product may be a software installation package.

[0017] By implementing the embodiment of the present application, first obtain a data training set, which includes multiple images; then perform feature extraction on the multiple images in the data training set to obtain a first feature and a second feature, wherein the first feature includes multiple feature vectors corresponding to the multiple images at different scales, and the second feature is a plurality of detection-related feature vectors obtained by feature extraction of the multiple feature vectors at different scales; then determine a comprehensive feature based on the first feature and the second feature, the comprehensive feature is used to characterize the features obtained after the fusion of multiple feature vectors corresponding to different scales of the image; then determine the target detection prediction result based on the second feature; then determine the semantic segmentation prediction result based on the comprehensive feature and the target detection prediction result; finally determine a multi-task loss function based on the target detection prediction result and the semantic segmentation prediction result, and optimize the training of the multi-task network based on the multi-task loss function. In this way, the training framework based on the multi-task network realizes the function of multi-task information interaction, flexibly adapts to the situation where the sub-task labeling categories are not aligned and the data labeling is incomplete, and makes full use of the existing data for training, so that the multi-task network can achieve accurate recognition in the absence of data labeling.

[0018] Still other aspects will become apparent upon reading and understanding the accompanying drawings and detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] FIG1 is a system architecture diagram of an autonomous driving system provided by an embodiment of the present application;

[0021] FIG2 is a system architecture diagram of a network training system provided by an embodiment of the present application;

[0022] FIG3 is a flow chart of a multi-task network training method provided in an embodiment of the present application;

[0023] FIG4 is a flow chart of a method for obtaining a data training set according to an embodiment of the present application;

[0024] FIG5 is a schematic diagram of a feature fusion method provided in an embodiment of the present application;

[0025] FIG6 is a schematic diagram of a partial label training method provided in an embodiment of the present application;

[0026] FIG7 is a block diagram of functional units of a multi-task network training device provided in an embodiment of the present application;

[0027] FIG8 is a block diagram of functional units of another multi-task network training device provided in an embodiment of the present application;

[0028] FIG9 is a schematic structural diagram of an electronic device proposed in an embodiment of the present application.

[0029] Implementation Methods of the Application

[0030] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0032] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0033] Current multi-task networks have high requirements for training data. When data has missing labels, they cannot be used for training multi-task networks, which easily leads to data waste. There is also an imbalance in the number and types of categories of sub-task training data. The loss magnitude and convergence speed between sub-tasks are different, and the strength of the correlation between sub-tasks is also uncertain. Therefore, how to optimize the multi-task network and how to assign training weights to the two sub-tasks are very critical.

[0034] Please refer to Figure 1, which is a system architecture diagram of an autonomous driving system provided by an embodiment of the present application. As shown in Figure 1, the autonomous driving system 10 includes an environment perception and positioning module 101, a decision-making and planning module 102, and an execution control module 103. The environment perception and positioning module 101 also includes a training module 1011. The environment perception and positioning module 101 is mainly used for environmental perception and positioning services, mainly through sensor technology, cameras, GPS positioning and other technologies to obtain environmental indicators during the vehicle's driving process and collect data; the decision-making and planning module 102 mainly uses the collected data to make judgments and guide the vehicle's next behavior; the execution control module 103 mainly controls the vehicle's throttle, brake and other related systems through signal instructions based on the decision; wherein the training module 1011 may include a network training system, which is mainly used to optimize the multi-task network for target detection and semantic segmentation, so that the final prediction effect of the multi-task network is close to the manually labeled true value, so as to better perceive the environment around the vehicle and provide accurate guidance information for the subsequent decision-making and execution of the autonomous driving system 10.

[0035] Please refer to Figure 2, which is a system architecture diagram of a network training system provided by an embodiment of the present application. As shown in Figure 2, the network training system 20 includes a data category balancing module 201, an information interaction module 202, and a loss weight module 203, wherein the network training system 20 is the main system in the autonomous driving system architecture, and provides guidance for subsequent autonomous driving decision-making and control through the prediction results of the multi-task network in the network training system 20. Among them, the image category is replicated and the category is balanced sampled by the data category balancing module 201, so as to improve the generalization ability of the multi-task model for small sample categories in target detection and semantic segmentation tasks, so as to enhance the flexibility of the multi-task model; the complete interaction from feature to output is carried out by the information interaction module 202, so as to realize the function of multi-task information interaction; the conditional probability of the network prediction output is modeled by the loss weight module 203, and the uncertainty of each subtask is calculated, so as to dynamically allocate training weights to the subtasks, so that the multi-task training process converges faster. Through the network training system 20, it is flexible to adapt to the situation of misaligned subtask labeling categories and incomplete data labeling, and fully utilize the existing data to train the entire network, so as to enable the multi-task network to achieve better recognition performance in the absence of data labeling.

[0036] Please refer to Figure 3, which is a flow chart of a multi-task network training method provided by an embodiment of the present application. The method includes the following steps.

[0037] S310: Acquire a data training set, where the data training set includes multiple images.

[0038] Please refer to Figure 4, which is a flow chart of a method for obtaining a data training set provided in an embodiment of the present application. The specific steps include:

[0039] S410: Determine a category set in the data set.

[0040] Among them, for example, you can first obtain any vehicle-related picture P in the existing data set, and then count each category contained in the segmented and labeled picture P, such as motor vehicles, non-motor vehicles, pedestrians, animals, etc., and form a category set S based on these categories.

[0041] S420: Determine the number of detections for each category according to the category set.

[0042] For each category in the category set S, determine whether it has a detection annotation. If a category has a detection annotation, count the number N of detection boxes for that category in the image P, and obtain the number of that category as N. If a category does not have a detection annotation, the default number of that category is 1. For example, if the category is pedestrians and there are 3 pedestrian detection boxes in the image P, then it can be confirmed that there are 3 pedestrians in the image P.

[0043] S430: Count the categories of all images in the data set and the number of detections corresponding to the categories to obtain data category distribution.

[0044] Among them, the steps S410-S420 are looped to count the number of categories in all pictures in the entire data set, and the data category distribution of the entire data set under multi-task is accumulated, where the data category distribution can be: {category: total number}.

[0045] S440 , determining an image category with the largest total number according to the data category distribution, and expanding the data set according to the image category with the largest total number.

[0046] Based on the statistically determined distribution of multi-task data categories, the total number of the largest category is used as a benchmark. The ratio of the total number of remaining categories in the dataset to the total number of the largest category is calculated. The sample sets of the other categories are then expanded based on these ratios to obtain an expanded dataset. For example, the largest category is motor vehicles, which has a total number of 90. Among the other categories, the total number of non-motor vehicles is 20. Therefore, 90 / 20 = 4.5. Therefore, the number of non-motor vehicles can be expanded by 4.5 times, bringing the total number of non-motor vehicles to 90, thereby expanding the dataset.

[0047] S450, performing balanced sampling of different image categories on the expanded data set to obtain a data training set.

[0048] During training, the same number of samples are sampled for all categories in the expanded data set. For example, 50 samples are taken for each of the categories of motor vehicles, non-motor vehicles, pedestrians, obstacles, etc.

[0049] It can be seen that the statistics are combined according to the categories of semantic segmentation and the number of target detections, and the dataset is expanded according to the above ratio, so as to achieve data category balance under multi-task.

[0050] S320, perform feature extraction on multiple images in the data training set to obtain a first feature and a second feature, wherein the first feature includes multiple feature vectors corresponding to the multiple images at different scales, and the second feature is multiple detection-related feature vectors obtained by performing feature extraction on the multiple feature vectors at different scales.

[0051] Feature extraction from vehicle-related images in the training data set is a global feature extraction for multi-task features. Multi-tasks can include object detection and semantic segmentation. The first feature is a globally shared feature for multiple tasks. Images can have multiple scales. Multi-scale refers to signal sampling at different scales, and different features can be observed at different scales. Simply put, the features perceived by humans vary depending on the distance from a fixed object. That is, when an object is imaged at different sizes (at different scales) in the field of view, its features exhibit different characteristics. Images of different scales are suitable for different tasks. In this embodiment, to optimize the multi-task network, features at different scales need to be fused. For example, objects in vehicle-related images have local similarities, such as color and texture. In this embodiment, multi-scale fusion can be specifically manifested as the output feature map containing both global overall information and local detailed information to better perceive the environment around the vehicle. After obtaining the globally shared first feature for multiple tasks, feature screening is performed on these shared features to select features relevant to object detection as the second feature.

[0052] S330: Determine a comprehensive feature based on the first feature and the second feature, where the comprehensive feature is used to represent a feature obtained by fusing multiple feature vectors corresponding to different image scales.

[0053] In this case, multiple feature vectors at multiple scales in the first feature can be obtained, such as a first feature vector at the first scale and a second feature vector at the second scale, and then a first detection-related feature vector is extracted from the first feature vector at the first scale; the first detection-related feature vector is mapped to the segmentation feature space through mapping, and a first interaction parameter vector and a second interaction parameter vector at the first scale are generated respectively. In this case, the segmentation feature space can be a feature screening of the first feature globally shared by multiple tasks, selecting features related to semantic segmentation, and forming the first interaction parameter vector and the second interaction parameter vector based on the selected features related to semantic segmentation, and obtained by the same feature mapping. The first eigenvector, the first interaction parameter vector and the second interaction parameter vector at the first scale are weighted multiple times to obtain the first comprehensive feature vector at the first scale after fusion; the second detection-related feature vector is extracted from the second eigenvector at the second scale, and the second detection-related feature vector is mapped to the segmentation feature space through mapping, and the third interaction parameter vector and the fourth interaction parameter vector at the second scale are generated respectively, wherein the third interaction parameter vector and the fourth interaction parameter vector are obtained by the same feature mapping, and then the first comprehensive feature vector at the first scale is taken, and the first comprehensive feature vector is expanded to the same dimension as the second feature vector through an upsampling operation, and each element in the first comprehensive feature vector and the second feature vector is added element by element to generate a new second feature vector after multi-scale fusion; the new second feature vector, the third interaction parameter vector and the fourth interaction parameter vector are weighted multiple times to obtain the second comprehensive feature vector at the second scale after fusion, and the second comprehensive feature vector can be determined as a comprehensive feature.

[0054] The first scale and the second scale are just any two scales under multiple scales, and the first eigenvector and the second eigenvector are just any two eigenvectors among the multiple eigenvectors.

[0055] S340: Determine a target detection prediction result based on the second feature.

[0056] An example of this is obtaining an accurate candidate region based on the second feature, extracting a candidate region feature map based on the accurate candidate region, and then determining the target category based on the candidate region feature map to output a target detection category prediction result. An example of this is detecting pedestrians, motor vehicles, traffic signal indicators, and other categories near the vehicle based on the second feature.

[0057] S350: Determine a semantic segmentation prediction result based on the comprehensive features and the target detection prediction result.

[0058] Among them, a semantic segmentation prediction result for reference can be determined based on the comprehensive features, and then the categories of the target detection prediction results are selected from the categories of the semantic segmentation prediction results for reference, and then these identical categories are padded by 1 to reach the same dimension as the number of categories of the semantic segmentation prediction results for reference, to obtain a new target detection prediction result, and then the semantic segmentation prediction results for reference are weighted by dot product based on the new target detection prediction result to obtain the final semantic segmentation prediction result. An example of the padded by 1 method is that the detection outputs 3 categories, specifically [motor vehicle, non-motor vehicle, pedestrian], whose logical value logits is [2, 3, 5], and the segmentation outputs 5 categories, specifically [motor vehicle, non-motor vehicle, pedestrian, cone barrel, warning column], then the logits of the detection output can be padded by 1 to become [2, 3, 5, 1, 1], thereby achieving dimensional alignment.

[0059] S360: Determine a multi-task loss function according to the target detection prediction result and the semantic segmentation prediction result, and optimize the training of the multi-task network based on the multi-task loss function.

[0060] Among them, the multi-task can include multiple sub-tasks. In this embodiment, it can include two sub-tasks, namely the target detection sub-task and the semantic segmentation sub-task. Assume that in the case of simultaneous multi-task training, for any sub-task T i , the uncertainty caused by other tasks is δ i , for the classification task, assuming p(y i |f w (x)) obeys the Boltzmann distribution, then:

[0061] Where p represents the probability distribution, i = 1 or i = 2, y i is the prediction output result (including target detection prediction result and semantic segmentation prediction result); f w (x) is the predicted output probability of the subtask, w represents the parameter, f represents the mapping function; C represents the number of output categories (3 categories, 5 categories); the activation function Softmax is a commonly used mathematical function, usually used in multi-classification problems, which maps a set of arbitrary real values ​​to a value range with probabilistic meaning, so that each value is between 0 and 1 and the sum of all values ​​is equal to 1. In neural networks, Softmax is often used to convert network output into a probability distribution for classification or probability prediction. Given a vector z = (z1, z2, ..., z k ), the Softmax function can be expressed as:

[0062] Where e is a natural constant and k is the length of vector z. The Softmax function converts each element in vector z into a real number between 0 and 1, which represents the importance or probability of the element among all elements. The output of the Softmax function can be regarded as a probability distribution, in which each element represents the probability of the corresponding category. In neural networks, Softmax is usually used as the activation function of the last output layer. Using the Softmax function can make the probability distribution of the network output more clear and intuitive, making it easier to perform classification or probability prediction. For regression tasks, assuming p(y i |f w (x)) obeys the Laplace distribution, then:

[0063] Among them, further using the multi-task negative log-likelihood function can obtain the uncertainty-based multi-task loss function as follows:

[0064] Among them, in order to adapt to the situation where subtask annotation is incomplete, the loss of the subtask can also be adaptively calculated according to the following loss function formula.

[0065] Among them, L i is the loss value of the subtask, For subtask T i The loss function calculation formula is Softmax for classification subtasks and mean square error MSE for regression subtasks. MSE is a commonly used loss function that measures the difference between the model prediction value and the true value. In regression problems, MSE is a widely used loss function. Given a set of predicted values and the corresponding true value y=(y1,y2,...,y n ), MSE can be defined as:

[0066] Where n is the number of samples. MSE calculates the average of the squared errors between the predicted values ​​and the true values. A smaller MSE value indicates that the model's predictions are closer to the true values. MSE can be used in supervised learning, such as linear regression, polynomial regression, and neural networks. In neural networks, MSE is often used for regression problems, such as predicting house prices and stock prices. By optimizing the MSE loss function, the model's parameters can be adjusted to minimize the error between the predicted values ​​and the true values, thereby improving the model's performance.

[0067] It can be seen that in this embodiment, the function of multi-task information interaction is realized through the training framework of the multi-task network, which flexibly adapts to the situation of misaligned sub-task labeling categories and incomplete data labeling, and makes full use of existing data for training, so that the multi-task network can achieve accurate recognition in the absence of data labeling, thereby better perceiving the environment around the vehicle and providing accurate guidance information for autonomous driving tracking and decision-making.

[0068] In a possible embodiment, obtaining a data training set includes: obtaining an initial image set, determining an image category distribution for a first category of images in the initial image set through semantic segmentation annotation and target detection annotation, wherein the image category distribution includes categories and total numbers; determining an image category with the largest total number based on the image category distribution; based on a quantity ratio of a first total number corresponding to the image category with the largest total number to a second total number corresponding to other image categories, copying images of the other image categories based on the quantity ratio so that the total number of the other image categories reaches the first total number, wherein the copied images are second category images, and the first category images and the second category images constitute an extended image set; performing balanced sampling of different image categories on the extended image set to obtain the data training set.

[0069] In this case, each category contained in each image in the initially acquired vehicle-related image set can be counted based on the segmentation annotations. The categories contained in each image are then counted. Then, it is determined whether each category contained in each image has a detection annotation. If a detection annotation exists, the number of detection boxes for the category in the corresponding image is counted, and the number of detection boxes is determined as the number of categories. If no detection annotation exists, the number of categories is defaulted to 1. The number of categories of all images in the initial image set is counted and accumulated to obtain the image category distribution under multiple tasks. The data category distribution can be: {category: total number}. Exemplary data category distributions may include: {pedestrians: 47}, {motor vehicles: 58}, {warning posts: 20}, {cones: 18}, {animals: 10}, etc.

[0070] Based on the statistically determined distribution of image categories, the total number of the largest category is used as a benchmark. The ratio of the total number of the remaining categories in the dataset to the total number of the largest category is calculated. The image set is then expanded based on these ratios, so that the total number of the remaining categories is equal to the total number of the largest category. For example, the largest category is pedestrians. The total number of pedestrians counted is 50, and the total number of cones is 10. Therefore, 50 / 10 = 5. Therefore, the number of cones is increased by 5 times to a total of 50, thus obtaining an expanded vehicle-related image set. This expanded vehicle-related image set is then subjected to class-balanced sampling, ensuring that each category uses an equal number of samples.

[0071] It can be seen that in this embodiment, on the one hand, the correlation between the data annotations of semantic segmentation and target detection is combined, and the situation of misalignment of data volume and category types between semantic segmentation and target detection is also compatible, which can effectively generate a relatively category-balanced training data set, thereby achieving the purpose of improving the generalization ability of the multi-task model for small sample categories in semantic segmentation and target detection; on the other hand, the annotation categories of semantic segmentation and target detection can be flexibly configured according to application requirements, and the annotation amount between the two subtasks can also be different, which greatly reduces the requirements for data annotation.

[0072] In a possible embodiment, the multiple feature vectors include a first feature vector and a second feature vector, the multiple detection-related feature vectors include a first detection-related feature vector and a second detection-related feature vector, and determining the comprehensive feature based on the first feature and the second feature includes: mapping the first detection-related feature vector to a segmentation feature space to generate a first interaction parameter and a second interaction parameter, the segmentation feature space being used to characterize a set space composed of multiple segmentation-related feature vectors obtained by feature extraction of the multiple feature vectors at different scales; performing feature fusion based on the first interaction parameter, the second interaction parameter and the first feature vector to obtain a first comprehensive feature; fusing the first comprehensive feature and the second feature vector to generate a new second feature vector; mapping the second detection-related feature vector to the segmentation feature space to generate a third interaction parameter and a fourth interaction parameter; performing feature fusion based on the third interaction parameter, the fourth interaction parameter and the new second feature vector to obtain a second comprehensive feature, and determining the second comprehensive feature as the comprehensive feature.

[0073] Among them, by way of example, the feature vector f1 at the m1 scale and the feature vector f2 at the m2 scale in the first feature can be obtained, and then the detection-related feature vector d1 is extracted from the f1 feature vector at the m1 scale; the detection-related feature vector d1 is mapped to the segmentation feature space through mapping, and two interaction parameter vectors a1 and b1 at the m1 scale are generated respectively, where a1 and b1 are obtained by the same feature mapping, and the f1 feature vector at the m1 scale and the interaction parameter vectors a1 and b1 are weighted multiple times to obtain the fused comprehensive feature vector c1 at the m1 scale.

[0074] Among them, the multiple weightings in c1 obtained by multiple weightings may include two attention weighting operations, please refer to Figure 5, which is a schematic diagram of a feature fusion method provided by an embodiment of the present application. As shown in Figure 5, the specific steps include: first, the input feature vector X is attention-weighted with the incoming interaction parameter vector A to obtain the weighted feature X'; wherein, the input feature vector X may be the first feature, and the interaction parameter vector A may be one of the interaction parameters obtained by mapping the detection-related feature vector to the segmentation feature space. The feature vector X is first weighted with the incoming interaction parameter vector A before the convolution layer, which can be understood as providing guidance for the semantic segmentation prediction result based on the target detection prediction result, and pre-pointing out which channels' information is more effective. Feature extraction is performed on the weighted feature X' to obtain a new feature vector F; wherein, feature extraction of the weighted feature X' is performed through the convolution layer, and examples thereof may be vehicle texture features and vehicle color features in vehicle-related images. The new feature vector F is weighted with the incoming interaction parameter vector B, yielding a new weighted feature vector F'. The interaction parameter vector B can be another interaction parameter, different from A, obtained by mapping the detection-related feature vectors to the segmentation feature space. The weighting of the new feature vector F and the incoming interaction parameter vector B occurs after the convolutional layer, which can be understood as aligning the object detection features with the semantic segmentation features along the channel dimension. The weighted new feature vector F' is passed through a batch normalization layer and an activation function, outputting a feature vector Y. Applying a batch normalization layer before the activation function improves output accuracy and produces a more stable distribution. Batch normalization is a commonly used neural network optimization technique used to accelerate neural network training and improve model generalization. It normalizes the intermediate layers of a neural network to stabilize the mean and variance of the input data, thereby reducing internal covariate shift within the network. An activation function is a nonlinear function commonly used in neural networks to perform nonlinear transformations on neuron outputs, thereby increasing the network's expressive power and fitting capabilities. The main function of an activation function is to map the output of a neuron into a nonlinear space, enabling the neural network to better handle complex nonlinear problems. The output value processed by the activation function can be used to represent the excitation level of the neuron. Activation functions are typically applied to the hidden and output layers of a neural network. Common activation functions include the Sigmoid function, the ReLU function, the Tanh function, and the Softmax function.

[0075] Among them, after obtaining the comprehensive feature vector c1, the detection-related feature vector d2 is extracted from the feature vector f2 at the m2 scale, and the detection-related feature vector d2 is mapped to the segmentation feature space through mapping, and two interaction parameter vectors a2 and b2 at the m2 scale are generated respectively, where a2 and b2 are obtained by the same feature mapping, and then the comprehensive feature vector c1 at the m1 scale is taken, and c1 is expanded to the same dimension as the feature vector f2 through an upsampling operation, and each element in c1 and f2 is added element by element to generate a new feature vector f'2 after multi-scale fusion; the new feature vector f'2 and the interaction parameter vectors a2 and b2 are weighted multiple times to obtain the fused comprehensive feature vector c2 at the m2 scale, and the comprehensive feature vector c2 can be determined as the comprehensive feature.

[0076] It can be seen that in this embodiment, the interaction parameters converted from the scale features of target detection are weighted, and the correlation between the two target detection and semantic segmentation is strengthened through two attention weighting operations.

[0077] In a possible embodiment, the multiple feature vectors also include a third feature vector, and the multiple detection-related feature vectors also include a third detection-related feature vector. After obtaining the second comprehensive feature, the method also includes: fusing the second comprehensive feature and the third feature vector to generate a new third feature vector; mapping the third detection-related feature vector to the segmentation feature space to generate a fifth interaction parameter and a sixth interaction parameter; performing feature fusion based on the fifth interaction parameter, the sixth interaction parameter and the new third feature vector to obtain a third comprehensive feature, and determining the third comprehensive feature as the comprehensive feature.

[0078] Among them, the exemplary feature vector f1 at the m1 scale, the feature vector f2 at the m2 scale, and the feature vector f3 at the m3 scale can be obtained in the first feature. According to the above, after obtaining the comprehensive feature vector c2, the feature vector f3 at the m3 scale is extracted to detect the relevant feature vector d3, and the detection-related feature vector d3 is mapped to the segmentation feature space through mapping, and two interaction parameter vectors a3 and b3 at the m3 scale are generated respectively, where a3 and b3 are obtained by the same feature mapping, and then the comprehensive feature vector c2 at the m2 scale is taken, and the detection-related feature vector d3 is mapped to the segmentation feature space through upsampling operation. Expand c2 to the same dimension as the feature vector f3, add each element in c2 and f3 element by element to generate a new feature vector after multi-scale fusion; weight the new feature vector f'3 and the interaction parameter vector a3 to obtain the weighted feature f"3; perform image feature extraction on the weighted feature f"3 to obtain a new feature vector e, weight the new feature vector F and the interaction parameter vector b3 to obtain a weighted new feature vector e'; pass the weighted new feature vector e' through the batch normalization layer and the activation function layer in sequence to obtain the comprehensive feature c3, and the comprehensive feature c3 can be determined as the comprehensive feature.

[0079] An image can have multiple scales. In this embodiment, only three of the multiple scales are selected as examples for illustration. When there are only two scales, the comprehensive feature c2 can be determined as the comprehensive feature. When there are only three scales, the comprehensive feature c3 can be determined as the comprehensive feature. The comprehensive feature is obtained by fusing features at all scales in the image.

[0080] It can be seen that in this embodiment, by fusing multi-scale features, the performance of the model can be greatly improved.

[0081] In a possible embodiment, the feature fusion is performed based on the first interaction parameter, the second interaction parameter and the first eigenvector to obtain the first comprehensive feature, including: determining a new first eigenvector based on the weight of the first interaction parameter and the weight of the first eigenvector, the weight of the first interaction parameter and the weight of the first eigenvector being obtained by respectively calculating based on different parts of the first interaction parameter and the first eigenvector; performing feature extraction on the new first eigenvector to obtain a fourth eigenvector; determining a new fourth eigenvector based on the weight of the second interaction parameter and the weight of the fourth eigenvector, the weight of the second interaction parameter and the weight of the fourth eigenvector being obtained by respectively calculating based on different parts of the second interaction parameter and the fourth eigenvector, and the first comprehensive feature is determined based on the new fourth eigenvector.

[0082] Among them, the weights of the above-mentioned interaction parameters and feature vectors are calculated through the attention mechanism, and different weights are assigned according to different parts of the input data to control the degree of attention of the multi-task network to different parts, thereby focusing the attention of the multi-task network on the key parts of the input data and improving the representation and generalization capabilities of the multi-task network.

[0083] Among them, the weight of the feature vector M and the weight of the interaction parameter vector N are calculated through the attention mechanism, and the above weights are weighted to obtain the weighted feature M'; image feature extraction is performed on the weighted feature M' to obtain a new feature vector O; the weight of the new feature vector O and the weight of the interaction parameter vector P are calculated through the attention mechanism, and the above weights are weighted to obtain a weighted new feature vector O'; the weighted new feature vector O' is passed through the batch normalization layer and the activation function in sequence to obtain the first comprehensive feature.

[0084] It can be seen that in this embodiment, the correlation between target detection and semantic segmentation is strengthened through two attention weighting operations.

[0085] In a possible embodiment, determining the semantic segmentation prediction result based on the comprehensive features and the target detection prediction result includes: determining a first semantic segmentation prediction result based on the comprehensive features; determining a second category in the target detection prediction result that is the same as the first category in the first semantic segmentation prediction result; dimensionally aligning the second category based on the first category so that the number of the second categories is the same as the number of the first categories, thereby obtaining a new target detection prediction result; determining a second semantic segmentation prediction result based on the new target detection prediction result and the first semantic segmentation prediction result, and determining the second semantic segmentation prediction result as the semantic segmentation prediction result.

[0086] Among them, a semantic segmentation prediction result for reference can be determined based on the comprehensive features, and then the categories in the target detection prediction result are selected from the categories in the semantic segmentation prediction result for reference, and then these identical categories are padded by 1 to reach the same dimension as the number of categories in the semantic segmentation prediction result for reference, to obtain a new target detection prediction result, and then the semantic segmentation prediction result for reference is weighted by dot product based on the new target detection prediction result to obtain the final semantic segmentation prediction result. An example of the padded by 1 method is that the detection outputs 3 categories, specifically [motor vehicle, non-motor vehicle, pedestrian], whose logical value logits is [2, 3, 5], and the segmentation outputs 5 categories, specifically [motor vehicle, non-motor vehicle, pedestrian, cone barrel, warning column], then the logits of the detection output can be padded by 1 to become [2, 3, 5, 1, 1], thereby achieving dimensional alignment.

[0087] Among them, on the basis of feature interaction and output interaction, a forward reasoning and back propagation partial label training is added, please refer to Figure 6, which is a schematic diagram of a partial label training method provided by an embodiment of the present application. As shown in Figure 6, the solid single arrow is the forward reasoning path, and the dotted single arrow is the back propagation path. In the forward reasoning path, the global shared features output by the feature extraction module 601 are first successively obtained through the detection head module 602 and the detection output module 604 to obtain the detection features and the detection output classification logits, and the detection loss is calculated by the detection loss module 606 according to the detection output classification logits. The detection head module 602 is used to extract detection-related features based on global shared features. The detection output module 604 is used to determine the logits of the detection category based on the detection features. The detection loss module 606 is used to determine the detection loss based on the logits of the detection output classification. The detection features and the global shared features are then input into the segmentation head module 603 for fusion to obtain comprehensive features. The segmentation head module 603 includes a convolution module and an upsampling module. The convolution module is used to map the detection-related features into the segmentation feature space, strengthening the interaction between detection and segmentation. The upsampling module is used to increase the image resolution, improve image quality, or increase the details of the feature map. The fused comprehensive features and the logits of the detection output classification are then input into the segmentation output module 605 to obtain the final segmentation output result. The segmentation loss is then determined based on the segmentation output result by the segmentation loss module 607. The segmentation output module 605 is used to weight the autocorrelation of the category predictions output by the target detection and semantic segmentation to achieve category output alignment between subtasks. The segmentation loss module 607 is used to calculate the segmentation loss based on the segmentation output result. As shown in Figure 6, in the backpropagation path, because the segmentation output incorporates the detection class logits, when the segmentation loss undergoes gradient backpropagation, part of the gradient flows along the segmentation output module 605 to the segmentation head module 603, while part flows through the detection class logits to the detection output module 604. Next, the gradients from the segmentation output module 605 propagate to the segmentation head module 603, and the gradients from the detection output module 604 propagate to the detection head module 602. Because the segmentation head module 603 incorporates detection-related features, a branch of the gradient flow in the segmentation head module 603 also propagates back to the detection head module 602 via the detection features. Therefore, the detection head module 602 receives two gradient flows and combines them to optimize itself. Ultimately, the gradient flows from both the segmentation and detection branches propagate back to the feature extraction module 601, jointly supervising the feature extraction module 601.

[0088] It can be seen that in this embodiment, by integrating the detection prediction results into the segmentation prediction results, the prediction outputs of the two subtasks are integrated, thereby strengthening the connection between the two subtasks; and through the partial label training method, even in the absence of true value annotations of the detection, while learning the segmentation, the detection will also be optimized accordingly, thereby being able to supervise the entire network and improve the efficiency of training; and incompletely labeled data can also be used for training to expand the overall data set, alleviate the problem of tight labeled data resources, and reduce the requirements and difficulty of data labeling.

[0089] In one possible embodiment, the multi-task includes multiple sub-tasks, and the multiple sub-tasks include target detection and semantic segmentation. The multi-task loss function is determined according to the target detection prediction result and the semantic segmentation prediction result. The method includes: determining a target detection loss value L1 according to the target detection prediction result, determining a semantic segmentation loss value L2 according to the semantic segmentation prediction result, and a sub-task loss value L i Satisfies the following formula:

[0090] Among them, y i Predict the result for the subtask, i = 1 or 2, f w (x) is the output probability of the subtask prediction, is the loss function calculation formula; according to the subtask loss value L i Determine the multi-task loss function.

[0091] Softmax is used for classification subtasks, while MSE is used for regression subtasks. Semantic segmentation is a classification subtask; object detection tasks include classification and regression branches, meaning they contain two types of loss functions. Softmax can be used for classification tasks, while MSE can be used for regression tasks.

[0092] It can be seen that in this embodiment, the loss value can be calculated adaptively to the situation where the subtask annotation is incomplete.

[0093] In a possible embodiment, the subtask loss value L i Determine the multi-task loss function, including: determining the sub-task weights δ according to the uncertainty of the multi-task i ; According to the subtask loss value L through the negative log-likelihood loss function i and the subtask weight δ i Determine the multi-task loss function -log p(y1,y2|f w (x)), the multi-task loss function is calculated based on the following formula:

[0094] Where p represents the probability distribution.

[0095] Among them, in the case of simultaneous multi-task training, for any task T i , the uncertainty caused by other tasks is δ i , for the classification task, assuming p(y i |f w (x)) obeys the Boltzmann distribution, then:

[0096] Where p represents the probability distribution, i = 1 or i = 2, y i is the prediction output result (including target detection prediction result and semantic segmentation prediction result); f w (x) is the predicted output probability of the subtask, w represents the parameter, f represents the mapping function; C represents the number of output categories; for the regression task, assuming p(y i |f w (x)) obeys the Laplace distribution, then:

[0097] Further using the multi-task negative log-likelihood function, we can get the uncertainty-based multi-task loss function as follows:

[0098] It can be seen that in this embodiment, the conditional probability of the network prediction output is used for modeling and the uncertainty of each subtask is calculated, so that the training weights are dynamically allocated to the subtasks, which makes the multi-task training process converge faster.

[0099] In a possible embodiment, the multiple images are vehicle-related images.

[0100] Among them, vehicle-related images can be vehicle images, vehicle surrounding environment images and road surrounding environment images. Vehicles such as cars, trucks, bicycles, etc. are detected through vehicle images, and various objects around the vehicle such as pedestrians, obstacles, etc. are detected through vehicle surrounding environment images. Road signs such as traffic signs, lane markings, parking space markings and pedestrian crossing lines are detected through road surrounding environment images.

[0101] Consistent with the above embodiment, please refer to FIG7 , which is a functional unit block diagram of a multi-task network training device provided in an embodiment of the present application. The multi-task network training device 70 includes: a first acquisition unit 71 for acquiring a data training set, wherein the data training set includes multiple images; a second acquisition unit 72 for performing feature extraction on the multiple images in the data training set by the second acquisition unit to obtain first features and second features, wherein the first features include multiple feature vectors corresponding to the multiple images at different scales, and the second features are multiple detection-related feature vectors obtained by feature extraction of the multiple feature vectors at different scales; a first determination unit 73 for determining a comprehensive feature based on the first feature and the second feature, wherein the comprehensive feature is used to represent the feature obtained by fusing the multiple feature vectors corresponding to the image at different scales; a second determination unit 74 for determining a target detection prediction result based on the second feature; a third determination unit 75 for determining a semantic segmentation prediction result based on the comprehensive feature and the target detection prediction result; and a fourth determination unit 76 for determining a multi-task loss function based on the target detection prediction result and the semantic segmentation prediction result.

[0102] In one possible embodiment, regarding data acquisition, the first acquisition unit 71 is specifically configured to: Refer to FIG. 4 , which is a flow chart illustrating a method for acquiring a data training set provided by an embodiment of the present application. The specific steps include: S410, determining a set of categories in the dataset. For example, a vehicle-related image P from an existing dataset can be obtained. Then, based on the segmentation annotations, each category contained in image P is counted. For example, these categories may be motor vehicles, non-motor vehicles, pedestrians, animals, etc., and these categories can be combined to form a set of categories S. S420, determining the number of detections for each category based on the set of categories. For each category in the set of categories S, a determination is made as to whether it has a detection annotation. If a category has a detection annotation, the number of detection boxes N for that category in image P is counted, resulting in a value of N for that category. If a category does not have a detection annotation, the number of that category is assumed to be 1. For example, if the category is a pedestrian and there are three pedestrian detection boxes in image P, then the number of pedestrians in image P can be determined to be three. S430, counting the categories of all images in the dataset and the number of detections corresponding to each category is counted to obtain a data category distribution. Steps S410-S420 are looped to count the number of categories in all images in the entire dataset, and the data category distribution of the entire multi-task dataset is accumulated. The data category distribution can be: {category:total number}. S440 determines the image category with the largest total number based on the data category distribution, and expands the dataset based on the image category with the largest total number. Based on the statistically calculated multi-task data category distribution, the total number of the largest category is used as a benchmark to calculate the ratio of the total number of the remaining categories in the dataset to the total number of the largest category. The sample sets of the other categories are expanded based on the ratios to obtain the expanded dataset. For example, the largest category is motor vehicles, with a total number of 90 motor vehicles, while the total number of non-motor vehicles among the other categories is 20. Therefore, 90 / 20 = 4.5. Therefore, the number of non-motor vehicles is expanded by 4.5 times, bringing the total number of non-motor vehicles to 90, thereby expanding the dataset. At step S450, balanced sampling of different image categories is performed on the expanded dataset to obtain a training data set. During training, the same number of samples is sampled for all categories in the expanded dataset. For example, 50 samples are taken for each of the motor vehicle, non-motor vehicle, pedestrian, and obstacle categories. As can be seen, by combining statistics based on the semantic segmentation categories and the number of object detections, and expanding the dataset according to the aforementioned ratio, balanced data categories are achieved for multiple tasks.

[0103] In one possible embodiment, regarding feature acquisition, the second acquisition unit 72 is specifically configured to extract features from vehicle-related images in the training data set, which is global feature extraction for multi-task features. The multi-tasks may include object detection and semantic segmentation, and the first feature is a globally shared feature for the multi-tasks. Images can have multiple scales. Multi-scale refers to signal sampling at different scales, and different features can be observed at different scales. Simply put, different distances from a fixed object result in different perceived features. That is, different scales of an object in the field of view result in different features. Images of different scales are suitable for different tasks. In this embodiment, to optimize the multi-task network, features at different scales need to be fused. For example, objects in vehicle-related images have local similarities, such as color and texture. In this embodiment, multi-scale fusion can be specifically manifested as the output feature map containing both global information and local details to better perceive the vehicle's surrounding environment. After obtaining the globally shared first feature for the multi-tasks, feature screening is performed within the shared features to select features relevant to object detection as the second feature.

[0104] In one possible embodiment, in terms of feature fusion, the first determination unit 73 is specifically used to: obtain multiple feature vectors at multiple scales in the first feature, such as the first feature vector at the first scale and the second feature vector at the second scale, and then extract the first detection-related feature vector from the first feature vector at the first scale; map the first detection-related feature vector to the segmentation feature space through mapping, and generate the first interaction parameter vector and the second interaction parameter vector at the first scale respectively. The segmentation feature space can be a feature screening of the first feature globally shared by multiple tasks, selecting features related to semantic segmentation, and forming the feature space based on the selected features related to semantic segmentation; the first interaction parameter vector and the second interaction parameter vector are obtained by the same feature mapping. The first eigenvector, the first interaction parameter vector, and the second interaction parameter vector at the first scale are weighted multiple times to obtain a fused first comprehensive eigenvector at the first scale; the second detection-related eigenvector is extracted from the second eigenvector at the second scale, and the second detection-related eigenvector is mapped to the segmentation feature space to generate a third interaction parameter vector and a fourth interaction parameter vector at the second scale, respectively, where the third interaction parameter vector and the fourth interaction parameter vector are obtained by the same feature mapping; then the first comprehensive eigenvector at the first scale is taken, and the first comprehensive eigenvector is expanded to the same dimension as the second eigenvector through an upsampling operation; each element in the first comprehensive eigenvector and the second eigenvector is added element by element to generate a new second eigenvector after multi-scale fusion; the new second eigenvector, the third interaction parameter vector, and the fourth interaction parameter vector are weighted multiple times to obtain a fused second comprehensive eigenvector at the second scale, and the second comprehensive eigenvector can be determined as a comprehensive feature. The first scale and the second scale are just any two scales among the multiple scales, and the first eigenvector and the second eigenvector are just any two eigenvectors among the multiple eigenvectors.

[0105] In one possible embodiment, in determining the detection output result, the second determination unit 74 is specifically configured to: exemplarily obtain a precise candidate region based on the second feature, extract a candidate region feature map based on the precise candidate region, and then determine the target category based on the candidate region feature map, thereby outputting a target detection category prediction result. Exemplarily, the second feature may be used to detect categories such as pedestrians, motor vehicles, and traffic signal signs near the vehicle.

[0106] In one possible embodiment, in terms of determining the segmentation output result, the third determination unit 75 is specifically used to: determine a semantic segmentation prediction result for reference based on the comprehensive features, and then select the same categories as the categories in the semantic segmentation prediction result for reference from the categories of the target detection prediction result, and then fill in these same categories by 1 to achieve the same dimension as the number of categories of the semantic segmentation prediction result for reference, to obtain a new target detection prediction result, and then perform dot product weighting on the semantic segmentation prediction result for reference based on the new target detection prediction result to obtain the final semantic segmentation prediction result. The method of filling in 1 can be exemplified by the detection output of 3 categories, specifically [motor vehicle, non-motor vehicle, pedestrian], whose logical value logits is [2, 3, 5], and the segmentation output of 5 categories, specifically [motor vehicle, non-motor vehicle, pedestrian, cone barrel, warning column], then the logits of the detection output can be filled in by 1, becoming [2, 3, 5, 1, 1], thereby achieving dimensional alignment.

[0107] In one possible embodiment, in terms of determining the loss, the fourth determining unit 76 is specifically configured to: a multi-task may include multiple sub-tasks. In this embodiment, it may include two sub-tasks, namely, a target detection sub-task and a semantic segmentation sub-task. Assuming that in the case of simultaneous multi-task training, for any sub-task T i , the uncertainty caused by other tasks is δ i , for the classification task, assuming p(y i |f w (x)) obeys the Boltzmann distribution, then:

[0108] Where p represents the probability distribution, i = 1 or i = 2, y i is the prediction output result (including target detection prediction result and semantic segmentation prediction result); f w (x) is the predicted output probability of the subtask, w represents the parameter, f represents the mapping function; C represents the number of output categories (3 categories, 5 categories); the activation function Softmax is a commonly used mathematical function, usually used in multi-classification problems, which maps a set of arbitrary real values ​​to a value range with probabilistic meaning, so that each value is between 0 and 1 and the sum of all values ​​is equal to 1. In neural networks, Softmax is often used to convert network output into a probability distribution for classification or probability prediction. Given a vector z = (z1, z2, ..., z k ), the Softmax function can be expressed as:

[0109] Where e is a natural constant and k is the length of vector z. The Softmax function converts each element in vector z into a real number between 0 and 1, which represents the importance or probability of the element among all elements. The output of the Softmax function can be regarded as a probability distribution, in which each element represents the probability of the corresponding category. In neural networks, Softmax is usually used as the activation function of the last output layer. Using the Softmax function can make the probability distribution of the network output more clear and intuitive, making it easier to perform classification or probability prediction. For regression tasks, assuming p(y i |f w (x)) obeys the Laplace distribution, then:

[0110] Among them, further using the multi-task negative log-likelihood function can obtain the uncertainty-based multi-task loss function as follows:

[0111] Among them, in order to adapt to the situation where subtask annotation is incomplete, the loss of the subtask can also be adaptively calculated according to the following loss function formula.

[0112] Among them, L i is the loss value of the subtask, For subtask T i The loss function calculation formula is Softmax for classification subtasks and mean square error MSE for regression subtasks. MSE is a commonly used loss function that measures the difference between the model prediction value and the true value. In regression problems, MSE is a widely used loss function. Given a set of predicted values and the corresponding true value y=(y1,y2,...,y n ), MSE can be defined as:

[0113] Where n is the number of samples. MSE calculates the average of the squared errors between the predicted values ​​and the true values. A smaller MSE value indicates that the model's predictions are closer to the true values. MSE can be used in supervised learning, such as linear regression, polynomial regression, and neural networks. In neural networks, MSE is often used for regression problems, such as predicting house prices and stock prices. By optimizing the MSE loss function, the model's parameters can be adjusted to minimize the error between the predicted values ​​and the true values, thereby improving the model's performance.

[0114] It can be understood that since the method embodiment and the device embodiment are different presentation forms of the same technical concept, the content of the method embodiment part in this application should be synchronously adapted to the device embodiment part and will not be repeated here.

[0115] In the case of adopting an integrated unit, please refer to Figure 8, which is a block diagram of the functional unit composition of another multi-task network training device provided in an embodiment of the present application. As shown in Figure 8, the multi-task network training device 70 includes: a processing module 702 and a communication module 701. The processing module 702 is used to control and manage the actions of the multi-task network training device 70, for example, executing the steps of the first acquisition unit 71, the second acquisition unit 72, the first determination unit 73, the second determination unit 74, the third determination unit 75 and the fourth determination unit 76, and / or for executing other processes of the technology described herein. The communication module 701 is used for interaction between the multi-task network training device 70 and other devices. As shown in Figure 8, the multi-task network training device 70 may also include a storage module 703, which is used for program code and data of the multi-task network training device 70.

[0116] The processing module 702 may be a processor or controller, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The communication module 701 may be a transceiver, an RF circuit, or a communication interface, and the like. The storage module 703 may be a memory.

[0117] All relevant contents of each scenario involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here. The above multi-task network training device 70 can execute the multi-task network training method shown in FIG3.

[0118] Please refer to Figure 9, which is a schematic diagram of the structure of an electronic device proposed in an embodiment of the present application. As shown in Figure 9, the electronic device 900 includes a processor 910, a memory 920, a communication interface 930, and one or more programs 921. The one or more programs are stored in the memory and are configured to be executed by the processor. When the program is executed, it includes some or all of the steps of any one of the multi-task network training methods described in the above method embodiments. The processor, memory, and communication interface are interconnected and complete communication work with each other.

[0119] The memory may be a volatile memory such as a dynamic random access memory (DRAM) or a non-volatile memory such as a mechanical hard disk. The memory is used to store a set of executable program codes, and the processor is used to call the executable program codes stored in the memory to execute some or all of the steps of any multi-task network training method described in the above multi-task network training method embodiment.

[0120] It can be seen that the electronic device 900 described in the embodiment of the present application first obtains a data training set, which includes multiple images; then performs feature extraction on the multiple images in the data training set to obtain a first feature and a second feature, wherein the first feature includes multiple feature vectors corresponding to the multiple images at different scales, and the second feature is a plurality of detection-related feature vectors obtained by feature extraction of the multiple feature vectors at different scales; then determines a comprehensive feature based on the first feature and the second feature, and the comprehensive feature is used to characterize the feature obtained by fusing the multiple feature vectors corresponding to the image at different scales; then determines the target detection prediction result based on the second feature; then determines the semantic segmentation prediction result based on the comprehensive feature and the target detection prediction result; finally, determines the multi-task loss function based on the target detection prediction result and the semantic segmentation prediction result. In this way, the training framework based on the multi-task network realizes the function of multi-task information interaction, flexibly adapts to the situation where the sub-task labeling categories are not aligned and the data labeling is incomplete, and makes full use of the existing data for training, so that the multi-task network can achieve accurate recognition in the absence of data labeling, thereby better perceiving the environment around the vehicle and providing accurate guidance information for autonomous driving tracking and decision-making.

[0121] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any method described in the above method embodiments, and the above computer includes an electronic device.

[0122] The present application also provides a computer program product comprising a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments. The computer program product may be a software installation package, and the computer may comprise an electronic device.

[0123] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all optional embodiments, and the actions and modules involved are not necessarily required by this application.

[0124] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0125] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0126] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0127] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0128] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions for enabling a computer electronic device (which can be a personal computer, electronic device or network electronic device, etc.) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present application. The aforementioned memory includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0129] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory may include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, etc. Optionally, all or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiments can be implemented in the form of hardware, for example, by implementing its corresponding functions through an integrated circuit, or in the form of a software functional module, for example, by having a processor execute a program / instruction stored in a memory to implement its corresponding function. This application is not limited to any specific form of combination of hardware and software.

[0130] The above is a detailed introduction to the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core ideas of the present application. At the same time, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application, and should be included in the scope of the claims of the present application. In summary, the contents of this specification should not be understood as limiting the present application.

Claims

1. A multi-task network training method, wherein: include: Acquire a data training set, wherein the data training set includes a plurality of images; Performing feature extraction on multiple images in the data training set to obtain a first feature and a second feature, wherein the first feature includes multiple feature vectors corresponding to the multiple images at different scales, and the second feature is multiple detection-related feature vectors obtained by performing feature extraction on the multiple feature vectors at different scales; Determine a comprehensive feature based on the first feature and the second feature, where the comprehensive feature is used to represent a feature obtained by fusing multiple feature vectors corresponding to different image scales; Determining a target detection prediction result according to the second feature; Determine a semantic segmentation prediction result based on the comprehensive features and the target detection prediction result; A multi-task loss function is determined according to the target detection prediction result and the semantic segmentation prediction result, and the training of the multi-task network is optimized based on the multi-task loss function.

2. The method according to claim 1, wherein The obtaining of the data training set includes: Obtaining an initial image set, and determining image category distribution of a first category of images in the initial image set through semantic segmentation annotation and object detection annotation, wherein the image category distribution includes categories and total numbers; Determining the image category with the largest total number according to the image category distribution; According to a quantity ratio of a first total number corresponding to the image category with the largest total number to a second total number corresponding to other image categories, copying images of the other image categories based on the quantity ratio so that the total number of the other image categories reaches the first total number, wherein the copied images are images of the second category, and the images of the first category and the images of the second category constitute an extended image set; Balanced sampling of different image categories is performed on the extended image set to obtain the data training set.

3. The method according to claim 1, wherein The multiple feature vectors include a first feature vector and a second feature vector, the multiple detection-related feature vectors include a first detection-related feature vector and a second detection-related feature vector, and determining a comprehensive feature based on the first feature and the second feature includes: Mapping the first detection-related feature vector to a segmentation feature space to generate a first interaction parameter and a second interaction parameter, wherein the segmentation feature space is used to represent a set space composed of multiple segmentation-related feature vectors obtained by feature extraction of the multiple feature vectors at different scales; Performing feature fusion according to the first interaction parameter, the second interaction parameter, and the first feature vector to obtain a first comprehensive feature; Fusing the first comprehensive feature and the second feature vector to generate a new second feature vector; Mapping the second detection-related feature vector into the segmentation feature space to generate a third interaction parameter and a fourth interaction parameter; Feature fusion is performed according to the third interaction parameter, the fourth interaction parameter, and the new second feature vector to obtain a second comprehensive feature, and the second comprehensive feature is determined as the comprehensive feature.

4. The method according to claim 3, wherein the plurality of feature vectors further include a third feature vector, the plurality of detection-related feature vectors further include a third detection-related feature vector, and after obtaining the second comprehensive feature, further comprising: Fusing the second comprehensive feature and the third feature vector to generate a new third feature vector; Mapping the third detection-related feature vector into the segmentation feature space to generate a fifth interaction parameter and a sixth interaction parameter; Feature fusion is performed according to the fifth interaction parameter, the sixth interaction parameter, and the new third feature vector to obtain a third comprehensive feature, and the third comprehensive feature is determined as the comprehensive feature.

5. The method according to claim 3, wherein: The performing feature fusion according to the first interaction parameter, the second interaction parameter, and the first feature vector to obtain a first comprehensive feature includes: determining a new first eigenvector according to the weight of the first interaction parameter and the weight of the first eigenvector, wherein the weight of the first interaction parameter and the weight of the first eigenvector are obtained by respectively calculating different parts of the first interaction parameter and the first eigenvector; Performing feature extraction on the new first eigenvector to obtain a fourth eigenvector; A new fourth eigenvector is determined based on the weight of the second interaction parameter and the weight of the fourth eigenvector, the weight of the second interaction parameter and the weight of the fourth eigenvector are obtained by respectively calculating the second interaction parameter and different parts of the fourth eigenvector, and the first comprehensive feature is determined based on the new fourth eigenvector.

6. The method according to claim 1, wherein Determining the semantic segmentation prediction result according to the comprehensive features and the target detection prediction result includes: Determine a first semantic segmentation prediction result according to the comprehensive features; Determining, in the object detection prediction result, a second category that is the same as the first category in the first semantic segmentation prediction result; Dimensionally aligning the second categories according to the first categories so that the number of the second categories is the same as the number of the first categories, thereby obtaining a new target detection prediction result; A second semantic segmentation prediction result is determined according to the new object detection prediction result and the first semantic segmentation prediction result, and the second semantic segmentation prediction result is determined as the semantic segmentation prediction result.

7. The method according to claim 6, wherein: The multi-task includes a plurality of sub-tasks, the plurality of sub-tasks include target detection and semantic segmentation, the multi-task loss function is determined according to the target detection prediction result and the semantic segmentation prediction result, and the method includes: The target detection loss value L1 is determined according to the target detection prediction result, the semantic segmentation loss value L2 is determined according to the semantic segmentation prediction result, and the subtask loss value L i Satisfies the following formula: Among them, y i Predict the result for the subtask, i = 1 or 2, f w (x) is the output probability of the subtask prediction, The loss function calculation formula; According to the subtask loss value L i Determine the multi-task loss function.

8. The method according to claim 7, wherein: The subtask loss value L i Determine the multi-task loss function, including: Determine the subtask weights δ according to the uncertainty of the multi-task i ; According to the subtask loss value L through the negative log-likelihood loss function i and the subtask weight δ i Determine the multi-task loss function -log p(y1,y2|f w (x)), the multi-task loss function is calculated based on the following formula: Where p represents the probability distribution.

9. The method according to claim 1, wherein: The plurality of images are vehicle-related images.

10. An electronic device, wherein: The invention comprises a processor and a memory storing execution instructions, wherein the memory stores one or more programs; when the processor executes the execution instructions stored in the memory, the processor executes the following steps; Acquire a data training set, wherein the data training set includes a plurality of images; Performing feature extraction on multiple images in the data training set to obtain a first feature and a second feature, wherein the first feature includes multiple feature vectors corresponding to the multiple images at different scales, and the second feature is multiple detection-related feature vectors obtained by performing feature extraction on the multiple feature vectors at different scales; Determine a comprehensive feature based on the first feature and the second feature, where the comprehensive feature is used to represent a feature obtained by fusing multiple feature vectors corresponding to different image scales; Determining a target detection prediction result according to the second feature; Determine a semantic segmentation prediction result based on the comprehensive features and the target detection prediction result; A multi-task loss function is determined according to the target detection prediction result and the semantic segmentation prediction result, and the training of the multi-task network is optimized based on the multi-task loss function.

11. The electronic device according to claim 10, wherein: The obtaining of the data training set includes: Obtaining an initial image set, and determining image category distribution of a first category of images in the initial image set through semantic segmentation annotation and object detection annotation, wherein the image category distribution includes categories and total numbers; Determining the image category with the largest total number according to the image category distribution; According to a quantity ratio of a first total number corresponding to the image category with the largest total number to a second total number corresponding to other image categories, copying images of the other image categories based on the quantity ratio so that the total number of the other image categories reaches the first total number, wherein the copied images are images of the second category, and the images of the first category and the images of the second category constitute an extended image set; Balanced sampling of different image categories is performed on the extended image set to obtain the data training set.

12. The electronic device according to claim 10, wherein: The multiple feature vectors include a first feature vector and a second feature vector, the multiple detection-related feature vectors include a first detection-related feature vector and a second detection-related feature vector, and determining a comprehensive feature based on the first feature and the second feature includes: Mapping the first detection-related feature vector to a segmentation feature space to generate a first interaction parameter and a second interaction parameter, wherein the segmentation feature space is used to represent a set space composed of multiple segmentation-related feature vectors obtained by feature extraction of the multiple feature vectors at different scales; Performing feature fusion according to the first interaction parameter, the second interaction parameter, and the first feature vector to obtain a first comprehensive feature; Fusing the first comprehensive feature and the second feature vector to generate a new second feature vector; Mapping the second detection-related feature vector into the segmentation feature space to generate a third interaction parameter and a fourth interaction parameter; Feature fusion is performed according to the third interaction parameter, the fourth interaction parameter, and the new second feature vector to obtain a second comprehensive feature, and the second comprehensive feature is determined as the comprehensive feature.

13. The electronic device according to claim 12, wherein the plurality of feature vectors further include a third feature vector, the plurality of detection-related feature vectors further include a third detection-related feature vector, and after obtaining the second comprehensive feature, the processor further executes instructions for the following steps: Fusing the second comprehensive feature and the third feature vector to generate a new third feature vector; Mapping the third detection-related feature vector into the segmentation feature space to generate a fifth interaction parameter and a sixth interaction parameter; Feature fusion is performed according to the fifth interaction parameter, the sixth interaction parameter, and the new third feature vector to obtain a third comprehensive feature, and the third comprehensive feature is determined as the comprehensive feature.

14. The electronic device according to claim 11, wherein The performing feature fusion according to the first interaction parameter, the second interaction parameter, and the first feature vector to obtain a first comprehensive feature includes: determining a new first eigenvector according to the weight of the first interaction parameter and the weight of the first eigenvector, wherein the weight of the first interaction parameter and the weight of the first eigenvector are obtained by respectively calculating different parts of the first interaction parameter and the first eigenvector; Performing feature extraction on the new first eigenvector to obtain a fourth eigenvector; A new fourth eigenvector is determined according to the weight of the second interaction parameter and the weight of the fourth eigenvector, wherein the weight of the second interaction parameter and the weight of the fourth eigenvector are determined according to the weight of the second interaction parameter and the fourth eigenvector. Different parts of the eigenvector are calculated separately and obtained, and the first comprehensive feature is determined according to the new fourth eigenvector.

15. The electronic device according to claim 10, wherein Determining the semantic segmentation prediction result according to the comprehensive features and the target detection prediction result includes: Determine a first semantic segmentation prediction result according to the comprehensive features; Determining, in the object detection prediction result, a second category that is the same as the first category in the first semantic segmentation prediction result; Dimensionally aligning the second categories according to the first categories so that the number of the second categories is the same as the number of the first categories, thereby obtaining a new target detection prediction result; A second semantic segmentation prediction result is determined according to the new object detection prediction result and the first semantic segmentation prediction result, and the second semantic segmentation prediction result is determined as the semantic segmentation prediction result.

16. The electronic device according to claim 15, wherein The multi-task includes a plurality of sub-tasks, the plurality of sub-tasks include target detection and semantic segmentation, the multi-task loss function is determined according to the target detection prediction result and the semantic segmentation prediction result, and the method includes: The target detection loss value l1 is determined according to the target detection prediction result, the semantic segmentation loss value l2 is determined according to the semantic segmentation prediction result, and the subtask loss value L i Satisfies the following formula: Among them, y i Predict the result for the subtask, i = 1 or 2, f w (x) is the output probability of the subtask prediction, The loss function calculation formula; According to the subtask loss value L i Determine the multi-task loss function.

17. The electronic device according to claim 16, wherein: The subtask loss value L i Determine the multi-task loss function, including: Determine the subtask weights δ according to the uncertainty of the multi-task i ; According to the subtask loss value L through the negative log-likelihood loss function i and the subtask weight δ i Determine the multi-task loss function -log p(y1,y2|f w (x)), the multi-task loss function is calculated based on the following formula: Where p represents the probability distribution.

18. The electronic device according to claim 10, wherein The plurality of images are vehicle-related images.

Images