Attention weight balancing method, system and device for multi-task learning and storage medium

Abstract

The invention discloses a multi-task learning attention weight balancing method, system and device and a storage medium, and the method comprises the steps: inputting a video containing pedestrians into a model, and generating a prediction result: obtaining multi-modal data according to the video containing the pedestrians; performing feature extraction on the multi-modal data by using a multi-modal feature encoder to obtain a plurality of feature vectors; calculating discriminant ability scores of each feature vector for different tasks; the task type is automatically identified according to the discriminative force score; dynamically adjusting temperature parameters according to task types; obtaining a task-feature incidence matrix according to the temperature parameters; generating a multi-task attention weight matrix according to the temperature parameters and the task-feature incidence matrix; the multi-task attention weight matrix is optimized through a multi-level constraint mechanism; and according to the optimized multi-task attention weight matrix, generating final prediction output of each task. According to the method, the technical problem of mutual sacrifice among tasks in traditional multi-task learning is fundamentally solved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a method, system, terminal device, and computer-readable storage medium for balancing attention weights in multi-task learning. Background Technology

[0002] Machine learning, as a core technology of artificial intelligence, improves algorithm performance through experiential knowledge to achieve autonomous intelligent decision-making. However, current mainstream machine learning methods (especially deep learning) heavily rely on large-scale labeled samples, while in practical applications, labels for many tasks are difficult to obtain or the labeling cost is extremely high. Multi-task learning (MTL) can fully explore shared information in limited samples by jointly learning multiple related tasks, thereby alleviating the problem of insufficient samples for a single task.

[0003] Multi-task learning aims to improve the generalization ability of a model by leveraging the correlation between tasks, and has been widely applied in fields such as natural language processing and computer vision. Its task types cover supervised tasks (such as classification and regression), unsupervised tasks (such as clustering), reinforcement learning, and multi-view learning. In recent years, the combination of deep learning and multi-task learning—that is, deep multi-task learning—has significantly improved the generalization performance of models and reduced computational and storage overhead, but the following key problems still exist: (1) Insufficient information interaction between different tasks makes it difficult to achieve effective knowledge transfer; (2) The loss function often adopts a fixed weighting method set manually, lacking an adaptive balancing mechanism between tasks.

[0004] Existing deep multi-task learning methods mainly revolve around two directions: network structure and optimization strategies. Regarding network structure, two mainstream mechanisms exist: hard parameter sharing and soft parameter sharing. Hard parameter sharing allows all tasks to share hidden layers while retaining only task-specific output layers. While this effectively suppresses overfitting, it struggles to adapt to differences between tasks. Soft parameter sharing, on the other hand, sets an independent model for each task and improves generalization ability by constraining the parameter distance between models through regularization. In terms of optimization strategies, most studies use a fixed-weight approach to construct multi-task loss functions, with weight settings heavily reliant on prior experience. Incorrect weights can easily lead to convergence difficulties or performance degradation on certain tasks. To alleviate this problem, some studies have attempted to introduce uncertain weights, gradient normalization, or dynamic weight averaging mechanisms to achieve more balanced multi-task optimization. In summary, how to achieve adaptive loss balance and information interaction among multiple tasks remains a key challenge in deep multi-task learning and has significant research implications. Summary of the Invention

[0005] In view of this, the present invention provides a method, system, terminal device, and computer-readable storage medium for balancing attention weights in multi-task learning.

[0006] The first objective of this invention is to provide a method for balancing attention weights in multi-task learning.

[0007] The second objective of this invention is to provide an attention weight balancing system for multi-task learning.

[0008] The third objective of this invention is to provide a terminal device.

[0009] A fourth objective of this invention is to provide a computer-readable storage medium.

[0010] The first objective of this invention can be achieved by adopting the following technical solution:

[0011] A method for balancing attention weights in multi-task learning, the method comprising:

[0012] Inputting video containing pedestrians into the model generates prediction results, including:

[0013] Obtain multimodal data from videos containing pedestrians;

[0014] A multimodal feature encoder is used to extract features from multimodal data to obtain multiple feature vectors;

[0015] Calculate the discriminative power score of each feature vector for different tasks; automatically identify the task type based on the discriminative power score; dynamically adjust the temperature parameter based on the task type; obtain the task-feature correlation matrix based on the temperature parameter; where different tasks include trajectory prediction task, activity recognition task, and grid localization task.

[0016] Generate a multi-task attention weight matrix based on temperature parameters and the task-feature correlation matrix;

[0017] The multi-task attention weight matrix is ​​optimized through a multi-level constraint mechanism; based on the optimized multi-task attention weight matrix, the final prediction output for each task is generated.

[0018] Preferably, the multi-level constraint mechanism includes entropy regularization constraints and hard boundary constraints, wherein:

[0019] The entropy regularization constraint is:

[0020]

[0021] In the formula, λ ent ε is the entropy regularization coefficient, with a value range of [0.05, 0.5]; ε is the numerical stability term; W[k][i] is the element in the k-th row and i-th column of the multi-task attention weight matrix, where k = 1, 2, 3;

[0022] Hard boundary constraints are:

[0023] W clipped [k][i]=clamp(w[k][i],0.05,0.7)

[0024] In the formula, W clipped [k][i] represents the elements in the task attention weight matrix after hard boundary constraints, and clamp() is the numerical truncation function.

[0025] Preferably, after applying hard boundary constraints, a renormalization process is performed:

[0026]

[0027] In the formula, W final [k][i] represents the elements in the optimized multi-task attention weight matrix.

[0028] Preferably, generating the final prediction output for each task based on the optimized multi-task attention weight matrix includes:

[0029] Using an optimized multi-task attention weight matrix W optimized Weighted fusion of feature vectors:

[0030]

[0031] in, and These are fusion features for trajectory prediction, activity recognition, and grid localization tasks, respectively. i Let i be the i-th eigenvector;

[0032] Based on the fusion features of the trajectory prediction task, the predicted trajectory coordinates are output using the trajectory prediction head; the trajectory prediction head adopts a two-layer fully connected network to output the future two-dimensional trajectory coordinate sequence.

[0033] Based on the fusion features of the activity recognition task, the activity recognition prediction head is used to output the predicted activity category probability; wherein, the activity recognition prediction head uses a classifier to output the activity category probability distribution.

[0034] Based on the fusion features of the grid localization task, the grid localization prediction head outputs the predicted grid class probability distribution and coordinate offset. The grid localization prediction head contains two parallel fully connected layer branches: one branch is used to output the class probability of the grid cell, and the other branch is used to output the continuous coordinate offset within the grid.

[0035] Preferably, generating a multi-task attention weight matrix based on temperature parameters and the task-feature correlation matrix includes:

[0036] Based on the task-feature correlation matrix and temperature parameters, calculate the attention weights of the multimodal feature encoder:

[0037]

[0038] Among them, P k (i) represents the attention weight of task k to the i-th feature encoder, and A[k][i] is the element in the k-th row and i-th column of the task-feature association matrix. Let be the temperature parameters of task k and the i-th feature vector;

[0039] P k (i) is the element in the k-th row and i-th column of the multi-task attention weight matrix W.

[0040] Preferably, the discriminant score is:

[0041]

[0042] In the formula, Let f be the learnable parameter matrix for task k, where k = 1, 2, 3; i Let i be the i-th eigenvector, where i = 1, 2, 3, 4, 5;

[0043] The automatic identification of task types based on the discriminant score includes:

[0044] When satisfied If the output is a continuous value, then it is a comprehensive balanced task:

[0045] When satisfied If the output is a classification probability distribution, then it is a key-focused task;

[0046] The dynamic adjustment of temperature parameters based on task type includes:

[0047] For comprehensive balanced tasks, calculate the temperature parameter τ. balance :

[0048]

[0049] Where, τ base The base temperature parameter is α, the entropy weighting coefficient is β, the epoch decay coefficient is epoch, and the training round is epoch.

[0050] For critical, focused tasks, calculate the temperature parameter τ. focus :

[0051]

[0052] Where, τ min γ is the minimum temperature parameter, and γ is the scaling factor. The maximum value among the feature discrimination scores. This represents the average of the feature discrimination scores.

[0053] Preferably, the step of acquiring multimodal data from videos containing pedestrians includes:

[0054] The video containing pedestrians is sampled sequentially at a set sampling rate to obtain T sequence frames;

[0055] The trajectory coordinate sequence of a target pedestrian in T consecutive video frames is obtained by using target detection and multi-target tracking algorithms;

[0056] The coordinates of all key points of the pedestrian in each sequence frame are summarized to obtain the key point sequence matrix.

[0057] Perform pixel-level semantic classification on the sequence frames to extract common scene categories; perform semantic feature pooling within the neighborhood of the pedestrian's current position to generate scene context feature vectors;

[0058] Extract fixed-size CNN appearance features from the target pedestrian bounding box to capture visual information about clothing color and body shape features;

[0059] Identify dynamic or static objects in the sequence frames that may interact with pedestrians; based on the identification results, calculate the geometric relationship between the pedestrian and each interacting object, and encode it into a four-dimensional feature vector.

[0060] Preferably, before inputting video containing pedestrians into the model, the model is trained, including:

[0061] Design a multi-task joint loss function to quantify prediction error;

[0062] Gradient conflicts between tasks are detected, and when a gradient conflict is detected, a gradient projection mitigation algorithm is used to coordinate the optimization directions between tasks.

[0063] Use parameter update mechanisms, including dynamic weight adjustment mechanisms and optimization strategies, to improve the weight generation process.

[0064] The second objective of this invention can be achieved by adopting the following technical solution:

[0065] An attention weight balancing system for multi-task learning, the system comprising:

[0066] The prediction module is used to input videos containing pedestrians into the model and generate prediction results;

[0067] The prediction module includes:

[0068] The multimodal data acquisition unit is used to acquire multimodal data based on videos containing pedestrians.

[0069] The feature extraction unit is used to extract features from multimodal data using a multimodal feature encoder to obtain multiple feature vectors;

[0070] The temperature parameter adjustment unit is used to calculate the discriminative power score of each feature vector for different tasks; automatically identify the task type based on the discriminative power score; dynamically adjust the temperature parameter based on the task type; and obtain the task-feature correlation matrix based on the temperature parameter. Among them, different tasks include trajectory prediction tasks, activity recognition tasks, and grid localization tasks.

[0071] The generation unit is used to generate a multi-task attention weight matrix based on temperature parameters and the task-feature correlation matrix;

[0072] The optimization and prediction unit is used to optimize the multi-task attention weight matrix through a multi-level constraint mechanism; based on the optimized multi-task attention weight matrix, it generates the final prediction output for each task.

[0073] The third objective of this invention can be achieved by adopting the following technical solution:

[0074] A terminal device includes a processor and a memory for storing a processor-executable program, wherein when the processor executes the program stored in the memory, it implements the above-described attention weight balancing method for multi-task learning.

[0075] The fourth objective of this invention can be achieved by adopting the following technical solution:

[0076] A computer-readable storage medium storing a program that, when executed by a processor, implements the above-described attention weight balancing method for multi-task learning.

[0077] The present invention has the following advantages over the prior art:

[0078] (1) Achieve true task collaborative optimization: Through a differentiated temperature control mechanism, each task can obtain an attention distribution pattern that perfectly matches its characteristics, fundamentally solving the technical problem of mutual sacrifice between tasks in traditional multi-task learning.

[0079] (2) Completely solve the problem of imbalanced attention weights: Through the synergistic effect of multi-level weight constraint mechanism, the extremely unbalanced distribution of attention weights can be effectively adjusted into a scientific and reasonable distribution pattern, which significantly enhances the robustness and generalization ability of the model.

[0080] (3) Significantly enhances the stability and controllability of the training process: The combined use of entropy regularization constraints and hard boundary constraints effectively prevents the occurrence of weight collapse during the training process, making the training process more stable and predictable. Attached Figure Description

[0081] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0082] Figure 1 This is a simplified flowchart of the attention weight balancing method for multi-task learning in Embodiment 1 of the present invention;

[0083] Figure 2 This is a detailed flowchart of the attention weight balancing method for multi-task learning in Embodiment 1 of the present invention;

[0084] Figure 3 This is a comparative schematic diagram of adjusting the attention weight distribution using temperature parameters under different task types in Embodiment 1 of the present invention;

[0085] Figure 4 This is a schematic diagram illustrating the change process of attention weight distribution before and after multi-level constraint (entropy constraint and hard constraint) processing in Embodiment 1 of the present invention;

[0086] Figure 5 This is a structural block diagram of the attention weight balancing system for multi-task learning according to Embodiment 2 of the present invention;

[0087] Figure 6 This is a structural block diagram of the terminal device according to Embodiment 3 of the present invention. Detailed Implementation

[0088] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be understood that the specific embodiments described are merely used to explain this application and are not intended to limit this application.

[0089] Example 1:

[0090] like Figure 1 , 2As shown, this embodiment provides a method for balancing attention weights in multi-task learning, including the following steps:

[0091] S101. Obtain multimodal data based on videos containing pedestrians.

[0092] The video containing pedestrians is sampled sequentially at a set sampling rate to obtain T sequence frames.

[0093] For example, a raw video with a duration of 3.2 seconds is acquired; the raw video is then temporally sampled at a sampling rate of 2.5fps to obtain 8 sequence frames.

[0094] Multimodal data includes the following five types:

[0095] (1) Original trajectory data.

[0096] By using target detection and multi-target tracking algorithms, the two-dimensional plane trajectory coordinate sequence of the target pedestrian in 8 consecutive video frames is obtained: {(x1,y1),(x2,y2),...,(x8,y8)}.

[0097] (2) Human posture key point data.

[0098] For each pedestrian in the sequence of frames, the coordinates of 17 keypoints in each frame are summarized to obtain a keypoint sequence matrix K∈R^(T×17×2). The three dimensions of this matrix represent: the number of time frames, the number of keypoints, and the two-dimensional planar coordinates (x,y) of each keypoint.

[0099] The coordinates of 17 body keypoints were extracted using a pose estimator pre-trained on the MSCOCO dataset, specifically including:

[0100] Head (5 parts): Nose, left and right eyes, left and right ears;

[0101] Upper limbs (6): left and right shoulders, left and right elbows, left and right wrists;

[0102] Trunk (2): Left and right hips

[0103] The chest area is usually derived from key points on the left and right shoulders and hips, and is not labeled separately in datasets such as COCO.

[0104] Lower limbs (4): left and right knees, left and right ankles.

[0105] (3) Scene segmentation feature data.

[0106] A pre-trained scene segmentation model is used to perform pixel-level semantic classification of video frames, extract common scene categories, and perform semantic feature pooling within a 3×3 neighborhood of the pedestrian's current position to generate a scene context feature vector.

[0107] Common scene categories include roads, sidewalks, grass, and buildings.

[0108] (4) Data on the appearance characteristics of the person.

[0109] A pre-trained FPN (Feature Pyramid Network) object detection model is used, and RoIAlign technology is used to extract fixed-size CNN appearance features from the bounding boxes of target pedestrians to capture visual information such as clothing color and body shape features.

[0110] (5) Environment interaction object data.

[0111] An object detection model is used to identify other dynamic or static objects in video frames that may interact with the pedestrian. Based on the detection results, the geometric relationship between the target pedestrian (represented by bounding box b) and each interacting object (represented by bounding box k) is calculated and encoded into a four-dimensional feature vector. This vector is calculated as follows:

[0112] Relative horizontal position:

[0113] Relative vertical position:

[0114] Relative width ratio:

[0115] Relative height ratio:

[0116] Objects include vehicles, doors, and other objects that can be pushed or pulled.

[0117] S102. Use a multimodal feature encoder to extract features from multimodal data.

[0118] Five specialized feature encoders were constructed to extract deep features from the data of the corresponding modalities, and each encoder output a feature vector of the same dimension.

[0119] (1) Use a trajectory sequence encoder to extract features from the original trajectory data and output the trajectory feature vector.

[0120] The encoder employs a recurrent neural network structure, preferably a bidirectional long short-term memory network (Bi-LSTM). By processing historical trajectory coordinate sequences, it extracts motion patterns and velocity variation features, outputting a trajectory feature vector. Where d is the feature dimension.

[0121] (2) Use a structured keypoint encoder to extract features from human posture keypoint data and output posture feature vectors.

[0122] This encoder employs either a Graph Convolutional Network (GCN) or a Graph Attention Network (GAT). It semantically groups multiple body keypoints (4 points on the head, 3 points each on the left and right arms, 2 points on the torso, and 3 points each on the left and right legs), models the structured relationships between body parts using a graph neural network, and captures the differences in contribution of different body parts to activity recognition through an attention mechanism, outputting a pose feature vector.

[0123] (3) Use the scene semantic encoder to extract features from the scene segmentation feature data and output the scene feature vector.

[0124] The encoder uses a convolutional neural network structure to encode scene segmentation features, preferably a convolutional neural network containing multiple convolutions and downsampling, to generate a hierarchical representation of the scene context and output a scene feature vector:

[0125] (4) Use the human appearance encoder to perform time-series modeling of human appearance feature data and output appearance feature vector.

[0126] This encoder employs a temporal feature extraction network, preferably a Long Short-Term Memory (LSTM) network, to model the extracted CNN appearance features in order to obtain / capture appearance change patterns and output an appearance feature vector.

[0127] (5) Use the environmental interaction encoder to extract features from the environmental interaction object data and output the interaction feature vector.

[0128] This encoder employs a temporal modeling network, preferably a Long Short-Term Memory (LSTM) network. By fusing and embedding geometric relationship features and object type one-hot encoding, it models temporal interaction patterns and outputs interaction feature vectors.

[0129]

[0130] This embodiment utilizes a multimodal feature encoder (five dedicated feature encoders) to extract features from multimodal data, resulting in five independent feature vectors f. i The set is: {f1,f2,f3,f4,f5}, i = 1, 2, 3, 4, 5.

[0131] S103. Based on the extracted feature vector, identify the task type; based on the identified task type, calculate the temperature parameter; based on the temperature parameter, obtain the task-feature correlation matrix.

[0132] There are three tasks in total: the first task is trajectory prediction, the second task is activity recognition, and the third task is grid localization, i.e., task k = 1, 2, 3.

[0133] Using the five independent feature vectors {f1,f2,f3,f4,f5} extracted in step S102, feature importance analysis and dynamic temperature parameter setting are performed through the task-specific temperature control module.

[0134] (1) Feature discrimination evaluation: Calculate the discrimination score of each feature vector for different tasks.

[0135] For task k and feature vector f i The discriminant score is:

[0136]

[0137] in, Let be the learnable parameter matrix for task k, with dimension d×1, which is automatically updated during training through backpropagation.

[0138] Specifically, Use Xavier initialization or He initialization.

[0139] (2) Automatic task type identification: The task type is automatically identified based on the discrimination score.

[0140] The feature is defined using entropy H as:

[0141]

[0142] Task types include comprehensive balanced tasks and key focused tasks.

[0143] A task is identified as a comprehensive balanced task (such as trajectory prediction or grid localization) when both of the following conditions are met:

[0144]

[0145] A task is identified as a key-focused task (such as activity recognition) when both of the following conditions are met:

[0146]

[0147] in, This represents the maximum value among all feature discrimination scores.

[0148] (3) Dynamic temperature parameter calculation: The temperature parameters are dynamically adjusted according to the task type.

[0149] For calculating the temperature parameter τ in a comprehensive balanced task balance :

[0150] τ balance =τ base +α·H·e -β·epoch

[0151] Where, τ base The base temperature parameter is α, the entropy weighting coefficient is β, the epoch decay coefficient is epoch, and epoch is the training round.

[0152] For critical, focused tasks, calculate the temperature parameter τ. focus :

[0153]

[0154] Where, τ min γ is the minimum temperature parameter, and γ is the scaling factor. The maximum value among the feature discrimination scores. This is the average value among the feature discrimination scores.

[0155] In this embodiment, the feature vector f is calculated for different task types. i Temperature parameters related to task k

[0156] (4) Construction of task-feature correlation matrix: Based on the temperature parameters, construct the task-feature correlation matrix.

[0157] The task-feature correlation matrix A is a 3×5 (k×i) matrix, where A[k][i] is an element of matrix A, representing the relationship between task k and feature vector f. i Dependence:

[0158]

[0159] Where A[k][i] represents the relationship between task k and f i The relevance score of the task e mbedding k For the embedding representation of task k, [f i ;task e mbedding k ] represents the concatenation operation of two vectors, MLP stands for Multilayer Perceptron, and softmax is the softmax normalization function.

[0160] The temperature parameters calculated in this step It is a core control parameter that affects the distribution of attention weights. For example... Figure 3 As shown, the temperature parameter directly alters the distribution of attention weights for different task types. The temperature parameter is not a direct weight value, but rather a "regulator" for the softmax function.

[0161] when When the softmax distribution becomes "sharp," attention is highly focused on the few features with the strongest discriminative power (focusing effect), which is suitable for key focus-oriented tasks such as activity recognition.

[0162] when When the softmax distribution becomes "flat", attention is relatively evenly distributed among multiple features (balance effect), which is suitable for comprehensive balanced tasks such as trajectory prediction and grid localization.

[0163] when At this time, the distribution approximates the standard softmax distribution, and attention allocation is determined by the original relevance score.

[0164] The temperature parameter will be used as the denominator in the attention weight calculation formula in step S104, directly controlling the final weight distribution. It should be noted that the temperature parameter calculated here is a fixed value in a single forward propagation, and its optimization will be achieved through the parameter update mechanism during the training phase (step S107).

[0165] S104. Generate a multi-task attention weight matrix based on temperature parameters and task-feature correlation matrix.

[0166] This step is the weight generation stage in forward propagation.

[0167] The temperature parameters calculated using step S103 The task-feature association matrix A is normalized using softmax to generate corresponding attention weights for each task. This step is a purely forward computation process, generating weights based on the current model parameters (MLP weights, temperature generation mechanism parameters, etc.). These weights remain fixed in the current inference / training iteration and are used in the subsequent feature fusion step S106.

[0168] The optimization of attention weights is not achieved by directly modifying the weight values, but by updating the upstream parameters (temperature parameters) that influence weight generation during the training phase. MLP parameters, feature encoder parameters, etc., are used to generate a better weight distribution in the next iteration.

[0169] The specific calculation process is as follows:

[0170] (1) Calculate the attention weights of the multimodal feature encoder based on the task-feature correlation matrix and temperature parameters:

[0171]

[0172] Among them, P k(i) represents the attention weights of task k on the i-th feature encoder.

[0173] (2) Generate a multi-task attention weight matrix W based on the attention weights of the multimodal feature encoder:

[0174] W[3][5]=[[P1(1),P1(2),P1(3),P1(4),P1(5)]

[0175] [P2(1),P2(2),P2(3),P2(4),P2(5)]

[0176] [P3(1),P3(2),P3(3),P3(4),P3(5)] ]

[0178] The three row vectors in the multi-task attention weight matrix W represent: the weight of trajectory prediction task, the weight of activity recognition task, and the weight of grid localization task, respectively.

[0179] S105. Optimize the multi-task attention weight matrix through a multi-level constraint mechanism.

[0180] For each task weight vector W[k][i] in the 3×5 attention weight matrix W, apply the triple constraint mechanism:

[0181] (1) Entropy regularization constraint: Prevents excessive concentration of weights in a single task.

[0182]

[0183] Where, λ ent ε is the entropy regularization coefficient, with a value range of [0.05, 0.5]; ε is a numerical stability term to prevent numerical instability in logarithmic operations.

[0184] Specifically, λ ent The typical value for ε is 0.1; the typical value for ε is 1e-7.

[0185] By increasing entropy loss, the weight distribution becomes more uniform, preventing the model from over-relying on a particular feature.

[0186] (2) Hard boundary constraints: restrict the range of weight values ​​for each task.

[0187] W clipped [k][i]=clamp(W[k][i],0.05,0.7)

[0188] In the formula, clamp(x,min,max) represents the numerical truncation function. When the input value x is less than min, min is taken; when it is greater than max, max is taken; otherwise, x itself is taken.

[0189] The weight values ​​of each vector in the multi-task attention weight matrix W are limited to the range of [0.05, 0.7] to ensure that each feature has a minimum contribution (not less than 5%), while avoiding any feature from completely dominating (not exceeding 70%).

[0190] (3) Renormalization: Preserving the probability distribution characteristics of each task weight vector:

[0191]

[0192] After boundary constraints are applied, renormalization ensures that the weight vector for each task still meets the basic requirements of the probability distribution.

[0193] The optimized multi-task attention weight matrix is ​​finally obtained:

[0194] W optimized [3][5]=[[W final [1][1],W final [1][2],W final [1][3],W final [1][4],W final [1][5]],

[0195] [W final [2][1],W final [2][2],W final [2][3],W final [2][4],W final [2][5]],

[0196] [W final [3][1],W final [3][2],W final [3][3],W final [3][4],W final [3][5]] ]

[0198] For the five features, even if the softmax output has an extreme distribution such as [0.92, 0.02, 0.02, 0.02, 0.02], after boundary clipping it becomes [0.7, 0.05, 0.05, 0.05, 0.05] (summing to 0.9), thus ensuring the synergistic effect of the multimodal features.

[0199] like Figure 4As shown, without constraints, attention weights may become extremely unbalanced. After applying entropy regularization constraints and hard boundary constraints in Example 1 and then renormalizing, the weights are pulled back to a reasonable range: that is, after boundary clipping of extreme distributions, the sum of the weight vectors may no longer be equal to 1 (0.9), but after renormalization, it becomes [0.778, 0.056, 0.056, 0.056, 0.056] (sum of 1.0).

[0200] S106. Based on the optimized multi-task attention weight matrix, generate the final prediction output for each task.

[0201] (1) The extracted feature vectors are weighted and fused using the optimized multi-task attention weight matrix to obtain the fused features of each task.

[0202] Using matrix W optimized [3][5] The five independent feature vectors in step S102 are weighted and fused:

[0203]

[0204] in, and These are fusion features for trajectory prediction, activity recognition, and grid localization tasks, respectively.

[0205] (2) Based on the fusion characteristics of each task, the prediction head is used to predict the final prediction output of each task.

[0206] (2-1) Based on the fusion features of the trajectory prediction task, the predicted trajectory coordinates are output using the trajectory prediction head.

[0207] The trajectory prediction head is used to handle comprehensive balanced tasks, and it uses a two-layer fully connected network to output a two-dimensional trajectory coordinate sequence for the next 12 frames:

[0208]

[0209] Among them, FC traj This indicates a two-layer fully connected network.

[0210] (2-2) Based on the fusion features of the activity recognition task, the activity recognition prediction head is used to output the predicted activity category probability.

[0211] The activity recognition prediction head is used to handle critical focus tasks, and uses a classifier to output an 8+1 class probability distribution of activity categories (including the background class):

[0212]

[0213] Among them, FC act This represents a classifier.

[0214] (2-3) Based on the fusion characteristics of the grid positioning task, the grid positioning prediction head outputs the predicted grid category probability distribution and coordinate offset.

[0215] The grid localization prediction head is used to handle comprehensive balanced tasks, and adopts a structure that combines classification and regression outputs.

[0216] Preferably, the grid localization prediction head contains two parallel fully connected layer branches: one branch FC cls Output the class probability of the grid cells (via the softmax activation function), another branch is FC. reg Output the continuous coordinate offsets within the grid (via a linear activation function):

[0217]

[0218] (3) Post-processing optimization.

[0219] The prediction results for each task are post-processed, including trajectory smoothing filtering, activity probability threshold adjustment, and conversion from grid coordinates to pixel coordinates.

[0220] A complete forward propagation process was completed using steps S101-S106:

[0221] Acquire multimodal data → Extract feature vectors → Generate attention weights → Weighted fusion → Output prediction results.

[0222] After obtaining the prediction outputs for each task, the model training and optimization phase begins (step S107). In this phase, the learnable parameters of the entire model are updated using the backpropagation algorithm by calculating the loss between the predicted values ​​and the true labels.

[0223] It is understandable that the attention weights W[k][i] are not directly learnable parameters of the model, but are dynamically calculated from the following learnable parameters:

[0224] Temperature parameter τ (calculated from task type and discrimination score);

[0225] MLP parameters in the task-feature correlation matrix A;

[0226] Parameters of the feature encoder (Bi-LSTM, GCN, CNN, etc.);

[0227] Therefore, the optimization of attention weights follows an "indirect optimization" mechanism:

[0228] Training phase → Update upstream learnable parameters → Influence the generation of multi-task attention weight matrix in the next step S104 → Generate a better multi-task attention weight matrix; This design allows the weight distribution to be adaptively adjusted during the training process, rather than being a fixed manually set value.

[0229] S107, Model Training and Optimization Stage.

[0230] This step is the core of model training, optimizing the parameters of the entire model through the backpropagation algorithm. Unlike the forward propagation phase (S104-S106), which uses fixed weights for inference, the training phase continuously optimizes the weights through the following mechanism:

[0231] The three-layer structure for training optimization:

[0232] ① Multi-task joint loss function: quantifies prediction error;

[0233] ② Gradient conflict detection and handling: coordinating optimization directions between tasks;

[0234] ③ Parameter update mechanism: Improve the weight generation process.

[0235] The specific optimization process is as follows:

[0236] (1) Design of joint loss function for multiple tasks:

[0237] Basic loss function definition:

[0238] Trajectory prediction loss (mean squared error):

[0239]

[0240] in, Let i be the predicted trajectory coordinates at the i-th time step. The coordinates of the original trajectory at the i-th time step.

[0241] Activity recognition loss (cross-entropy):

[0242]

[0243] in, Let i be the probability of the i-th type of activity (i = 0, 1, ..., 8, corresponding to 8 target activities and 1 background class). The original activity category label (one-hot encoded, 1 for the true category i, and 0 for the rest).

[0244] Specifically, the original activity category labels were obtained by parsing the corresponding manually labeled files (named Annotation Files) in the ActEV / VIRAT training dataset. During the data preprocessing stage, these text-based category labels (such as 'Running', 'Carrying') were mapped to numeric indices and further converted into one-hot encoded vectors for use in the cross-entropy loss function.

[0245] Grid localization loss (joint loss of classification and regression):

[0246] L grid =L cls +λ reg ·L reg

[0247] in:

[0248]

[0249] Here, SmoothL1 is the smoothing L1 loss function, used to prevent gradient explosion in regression tasks. Its calculation formula is: squared loss is used when the absolute value of the prediction error is less than 1, otherwise linear loss is used.

[0250] in, Let be the predicted class probability of the i-th grid cell; The original grid cell category label (one-hot encoded); λ reg This is the regression loss weighting coefficient, used to balance the contributions of classification loss and regression loss.

[0251] Specifically, the original grid cell category labels are generated during the data preprocessing stage by mapping the real location coordinates of the targets in the training set to a preset grid coordinate system: if the center point of the target exists in a certain grid, the category label corresponding to that grid is set to 1, otherwise it is 0.

[0252] Specifically, λ reg The value range is [0.5, 1.0], with a typical value of 0.8.

[0253] Dynamically weighted total loss function:

[0254]

[0255] Among them, w k (t) represents the dynamic weights of task k at training step t, w r This is the entropy regularization coefficient (typical value 0.1).

[0256] (2) Gradient conflict detection and handling between tasks:

[0257] Calculate the gradient:

[0258]

[0259] Gradient similarity calculation:

[0260]

[0261] Collision detection condition: sim(g i ,g j When ) < -0.3, gradient conflict is considered to exist.

[0262] When gradient conflicts are detected, the gradient projection mitigation algorithm (PCGrad) is used:

[0263]

[0264] Where g1, g2, and g3 are the gradient vectors for the three tasks, namely the trajectory prediction gradient, the activity recognition gradient, and the grid localization gradient; ||·||2 represents the L2 norm; This is an indicator function; projection is only performed when gradient conflicts exist.

[0265] (3) Dynamic weight adjustment mechanism:

[0266] Task performance evaluation metrics:

[0267] Trajectory prediction: Average displacement error (ADE);

[0268] Activity recognition: Classification accuracy;

[0269] Grid location: Intersection over Union (IoU);

[0270] Performance improvement calculation:

[0271]

[0272] Weight dynamic update strategy:

[0273] w k (t+1)=w k (t)×exp(-η×max(0,perf_improvement k (t)))

[0274] Weight normalization:

[0275]

[0276] The evaluation window is T = 100 steps, the learning rate is η = 0.01, and the initial weights are...

[0277] (4) Training optimization strategy:

[0278] Phased training strategy:

[0279] (4-1) Warm-up phase (epoch 1-20): Fixed weights Focus on feature learning;

[0280] (4-2) Weight learning phase (epoch 21-80): Enable dynamic weight adjustment mechanism;

[0281] (4-3) Fine-tuning phase (epoch 81-100): Reduce the learning rate and stabilize the training.

[0282] The attention weight update mechanism updates the temperature parameter based on the current loss situation to optimize the weight distribution.

[0283] For comprehensive and balanced tasks:

[0284]

[0285] For key-focused tasks:

[0286] τ new =τ current ×(1-v·C attn )

[0287] Where, τ new For the corrected temperature parameter, τ current The current temperature parameter is the one before adjustment; μ and v are both fine-tuning coefficients. C is the variance of the training loss values ​​over the most recent N steps, used to measure training stability; attn This represents the level of attention concentration (the ratio of the maximum weight to the average weight).

[0288] In this embodiment, μ = 0.1 and v = 0.2.

[0289] Through the above mechanism, collaborative optimization among multiple tasks is achieved, which not only ensures the performance improvement of each task, but also maintains a reasonable distribution of attention weights, thus solving the technical problem of mutual sacrifice between tasks in traditional multi-task learning.

[0290] After multiple rounds of iterative training (typically 100 epochs), the model parameters converge, the attention weight distribution stabilizes, and the performance of each task reaches an optimal balance. At this point, the trained model can be used in real-world inference scenarios to perform the forward propagation process S101-S106 on new input data and generate prediction results.

[0291] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware, and the corresponding program can be stored in a computer-readable storage medium.

[0292] It should be noted that although the method operations of the above embodiments are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the order of execution of the described steps may be changed. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0293] Example 2:

[0294] like Figure 5 As shown, this embodiment provides an attention weight balancing system for multi-task learning. The system includes a prediction module for inputting videos containing pedestrians into the model to generate prediction results. The prediction module includes:

[0295] The multimodal data acquisition unit 501 is used to acquire multimodal data based on videos containing pedestrians;

[0296] The feature extraction unit 502 is used to extract features from multimodal data using a multimodal feature encoder to obtain multiple feature vectors;

[0297] The temperature parameter adjustment unit 503 is used to calculate the discriminative power score of each feature vector for different tasks; automatically identify the task type based on the discriminative power score; dynamically adjust the temperature parameter based on the task type; and obtain the task-feature correlation matrix based on the temperature parameter; where different tasks include trajectory prediction task, activity recognition task, and grid positioning task.

[0298] The generation unit 504 is used to generate a multi-task attention weight matrix based on temperature parameters and task-feature correlation matrix;

[0299] The optimization and prediction unit 505 is used to optimize the multi-task attention weight matrix through a multi-level constraint mechanism; and to generate the final prediction output of each task based on the optimized multi-task attention weight matrix.

[0300] The specific implementation of each module in this embodiment can be found in Embodiment 1 above, and will not be repeated here. It should be noted that the system provided in this embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above.

[0301] Example 3:

[0302] This embodiment provides a terminal device, which can be a computer, such as... Figure 6 As shown, it is connected via a system bus 601 to a processor 602, a memory, an input device 603, a display 604, and a network interface 605. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium 606 and an internal memory 607. The non-volatile storage medium 606 stores an operating system, computer programs, and a database. The internal memory 607 provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. When the processor 602 executes the computer programs stored in the memory, it implements the attention weight balancing method for multi-task learning in Embodiment 1 described above.

[0303] Example 4:

[0304] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the attention weight balancing method for multi-task learning described in Embodiment 1 above.

[0305] It should be noted that the computer-readable storage medium in this embodiment can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0306] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope disclosed in the present invention, based on the technical solution and inventive concept of the present invention, shall fall within the scope of protection of the present invention.

Claims

1. A multi-task learning attention weight balancing method, characterized in that, The method comprises: inputting a video containing pedestrians into a model, generating a prediction result, comprising: obtaining multi-modal data from the video containing pedestrians; using a multi-modal feature encoder to extract features from the multi-modal data to obtain a plurality of feature vectors; calculating a discriminative score of each feature vector for different tasks; automatically identifying the type of task according to the discriminative score; dynamically adjusting the temperature parameter according to the type of task; obtaining a task-feature association matrix according to the temperature parameter; wherein the different tasks include trajectory prediction task, activity recognition task and grid positioning task; generating a multi-task attention weight matrix according to the temperature parameter and the task-feature association matrix; optimizing the multi-task attention weight matrix through a multi-level constraint mechanism; generating the final prediction output of each task according to the optimized multi-task attention weight matrix.

2. The attention weight balancing method of claim 1, wherein, The multi-level constraint mechanism comprises an entropy regularization constraint and a hard boundary constraint, wherein: The entropy regularization constraint is: In the formula, λ ent is an entropy regularization coefficient, and the value range is [0.05, 0.5]; ε is a numerical stability term; W[k][i] is an element in the kth row and the ith column of the multi-task attention weight matrix, and task k = 1, 2, 3; The hard boundary constraint is: W clipped [k][i] = clamp(W[k][i], 0.05, 0.7) In the formula, W clipped [k][i] is an element in the hard boundary constraint task attention weight matrix, and clamp() is a numerical truncation function.

3. The attention weight balancing method of claim 2, wherein, After the hard boundary constraint, the normalization processing is carried out: In the formula, W final [k][i] is an element in the optimized multi-task attention weight matrix.

4. The attention weight balancing method of claim 1, wherein, The final prediction output of each task according to the optimized multi-task attention weight matrix comprises: Using an optimized multitask attention weight matrix W optimized Weighted fusion of feature vectors: wherein, and are trajectory prediction task fusion features, activity recognition task fusion features, and grid localization task fusion features, respectively, f i is the i-th feature vector; According to the trajectory prediction task fusion feature, the predicted trajectory coordinates are output by using the trajectory prediction head; wherein the trajectory prediction head adopts a double-layer full connection network, which is used to output a two-dimensional trajectory coordinate sequence in the future; According to the activity recognition task fusion feature, the predicted activity category probability is output by using the activity recognition prediction head; wherein the activity recognition prediction head adopts a classifier, which is used to output an activity category probability distribution; According to the grid positioning task fusion feature, the predicted grid category probability distribution and coordinate offset are output by using the grid positioning prediction head; the grid positioning prediction head contains two parallel full connection layer branches: one branch is used to output the category probability of the grid unit, and the other branch is used to output the continuous coordinate offset in the grid.

5. The attention weight balancing method of claim 1, wherein, The multi-task attention weight matrix is generated according to the temperature parameter and the task-feature association matrix, comprising: According to the task-feature association matrix and the temperature parameter, the attention weight of the multi-modal feature encoder is calculated: where P k (i) is the attention weight of the i-th feature encoder for task k, A[k][i] is the element in the i-th column and the k-th row of the task-feature association matrix, is the temperature parameter of the i-th feature vector for task k. P k (i) as the element of the k-th row and i-th column of the multi-task attention weight matrix W.

6. The attention weight balancing method of claim 1, wherein, The discriminative score is: wherein is a learnable parameter matrix for task k, k = 1, 2, 3; f i is the i-th feature vector, i = 1, 2, 3, 4, 5; The type of task is automatically identified according to the discriminative score, comprising: When the conditions are met and the output is a continuous value, it is a comprehensive balancing task: When the following conditions are met and the output is a classification probability distribution, it is a key focus task. The temperature parameter is dynamically adjusted according to the type of task, comprising: For the integrated balancing task, the temperature parameter τ is calculated balance : where τ base is the base temperature parameter, a is the entropy weight coefficient, β is the epoch decay coefficient, and epoch is the training round. For critical focus-type tasks, a temperature parameter τ is calculated focus : where τ min is a minimum temperature parameter, γ is a scaling coefficient, is a maximum value in the feature discriminative score, is an average value of the feature discriminative score.

7. The attention weight balancing method of claim 1, wherein, The multi-modal data is obtained according to the video containing pedestrians, comprising: The video containing pedestrians is time-sampled at a set sampling rate to obtain T sequence frames; The trajectory coordinate sequence of the target pedestrian in the T continuous video frames is obtained by target detection and multi-target tracking algorithm; All key point coordinates of the pedestrian in each sequence frame are summarized to finally obtain a key point sequence matrix; The sequence frame is pixel-level semantic classified to extract common scene categories; the semantic feature vector is generated by performing semantic feature pooling in the neighborhood range of the current position of the pedestrian; The fixed size CNN appearance feature is extracted from the target pedestrian bounding box to capture the visual information of clothing color and posture features; Dynamic or static objects that may interact with the pedestrian in the sequence frame are identified; based on the identification result, the geometric relationship between the pedestrian and each interactive object is calculated and encoded into a four-dimensional feature vector.

8. The attention weight balancing method according to any one of claims 1 to 7, characterized in that, Before inputting the video containing pedestrians into the model, the model is trained, including: a multi-task joint loss function is designed to quantify prediction errors; gradient conflicts between tasks are detected, and when a gradient conflict is detected, a gradient projection relief algorithm is used to coordinate the optimization direction between tasks; a parameter updating mechanism is used, including a weight dynamic adjustment mechanism and an optimization strategy, to improve the weight generation process.

9. A multi-task learning attention weight balancing system, comprising: The system comprises: a prediction module for inputting the video containing pedestrians into the model to generate a prediction result; wherein the prediction module comprises: a multi-modal data acquisition unit for acquiring multi-modal data from the video containing pedestrians; a feature extraction unit for extracting features from the multi-modal data using a multi-modal feature encoder to obtain a plurality of feature vectors; a temperature parameter adjustment unit for calculating a discriminant score of each feature vector for different tasks; automatically identifying the type of task according to the discriminant score; dynamically adjusting the temperature parameter according to the type of task; obtaining a task-feature association matrix according to the temperature parameter; wherein the different tasks include trajectory prediction tasks, activity recognition tasks and grid positioning tasks; a generation unit for generating a multi-task attention weight matrix according to the temperature parameter and the task-feature association matrix; an optimization and prediction unit for optimizing the multi-task attention weight matrix through a multi-level constraint mechanism; generating the final prediction output of each task according to the optimized multi-task attention weight matrix.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the attention weight balancing method of any one of claims 1-8.

Images