A multitask visual inspection gradient optimization method and system

Abstract

The present application relates to the technical field of computer vision and artificial neural network training, and discloses a multi-task visual detection gradient optimization method and system. Training samples with directional bounding boxes and key point annotations are input into a multi-task visual detection network, directional target detection results and key point positioning results of the same target instance are associated, task loss, geometric consistency state and task effective markers are generated; shared parameters and task-specific parameters are divided according to network layer ownership, the gradient similarity of the shared parameters for the effective task is calculated, the target value corresponding to the geometric consistency state is weighted with the historical target value to obtain the target gradient similarity; when the gradient similarity is lower than the target value, the back projection component is removed and the compensation is made along the reference gradient direction, the shared parameters are updated after aggregation, each task-specific parameter is updated independently, and the model is output when the termination condition is met. The present application can make the output geometric state and the shared gradient correspond to each other in coordination, and reduce the parameter update interference between tasks.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial neural network training technology, and in particular to a multi-task visual detection gradient optimization method and system. Background Technology

[0002] In the fields of computer vision and artificial neural network training, multi-task visual detection networks typically employ a structure combining a shared feature extraction network with multiple task prediction heads. These networks perform object detection, bounding box regression, keypoint localization, or pose recognition on the same training samples. During training, each task generates its own task loss, which is then combined with fixed or dynamic weights to form a joint loss. The network parameters are then updated via backpropagation.

[0003] When directional target detection and keypoint localization tasks share network parameters, the task gradients generated by the two tasks for the shared network parameters may have directional differences. Existing solutions often weight, crop, or project the task gradients based on task loss, gradient direction angle, or a fixed similarity threshold. However, they typically do not consider the geometric relationship between directional target detection results and keypoint localization results to distinguish different gradient states, making it difficult to ensure that gradient coordination conditions correspond to the output structural state of the same target instance.

[0004] Furthermore, existing multi-task training schemes often employ a unified joint update method for shared feature extraction network parameters and task prediction head parameters, or fail to explicitly define the parameter range for gradient coordination. When cross-task gradient adjustments simultaneously affect task-specific parameters, the parameter update paths of the directional target detection task and the keypoint localization task are prone to interference. Moreover, there is a lack of clear constraints on whether the corresponding task gradients should participate in the joint update when keypoints are invisible, task annotations are missing, or target instance matching is abnormal.

[0005] For joint training of network parameters shared by directional target detection and keypoint localization tasks, existing techniques still suffer from insufficient coordination between output geometry and gradient conditions, and unclear boundaries between the updates of shared parameters and task-specific parameters. Therefore, it is necessary to address the problem of coordinating the gradients of shared parameters based on the output states of multiple tasks while maintaining independent updates of task-specific parameters. Summary of the Invention

[0006] To address the above problems, this invention provides a multi-task visual detection gradient optimization method and system, which solves the problem of coordinating the gradients of shared parameter tasks and maintaining the independent updating of task-specific parameters in the joint training of directional target detection tasks and key point localization tasks that share network parameters, based on geometric consistency state.

[0007] To address the aforementioned technical problems, this invention provides a multi-task visual detection gradient optimization method, comprising:

[0008] S100. Obtain training samples with oriented bounding box annotations and key point annotations, input them into the multi-task visual detection network, generate oriented target detection results and key point localization results, associate them according to the target instance identifier, and obtain the instance association prediction set;

[0009] S200. Generate a task loss set based on the instance association prediction set; generate geometric consistency state and task valid markers based on the major axis direction of the oriented bounding box of the same target instance, the structural direction of key points, the positional inclusion state of key points and visibility.

[0010] S300. Divide the shared parameter set and the task-specific parameter set according to the network layer affiliation, obtain the task gradient of the effective task to the shared parameter set and the task-specific parameter gradient of each task to the corresponding task-specific parameter set, calculate the cosine similarity of the task gradient, and weight the similarity target value corresponding to the geometric consistency state with the historical target gradient similarity to obtain the target gradient similarity.

[0011] S400. When the cosine similarity is lower than the target gradient similarity, remove the back projection component of the gradient to be adjusted relative to the reference gradient, and add a compensation component along the direction of the reference gradient; aggregate the adjusted or unadjusted task gradients into a joint gradient, update the shared parameter set with the joint gradient, and update the task-specific parameters with each dedicated gradient; update the geometric consistency state and historical target gradient similarity based on the updated network, and output the multi-task visual detection model when the training termination condition is met.

[0012] Furthermore, the training samples include image labels, target instance labels, target category annotations, oriented bounding box annotations, and multiple key point annotations;

[0013] The directional bounding box annotation includes the target center coordinates, target width, target height, and target rotation angle;

[0014] Each keypoint annotation includes a keypoint number, keypoint coordinates, and keypoint visibility marker.

[0015] Furthermore, the association based on the target instance identifier includes:

[0016] Match the predicted directional bounding box in the directional target detection result with the directional bounding box annotation;

[0017] Match the key point coordinates in the key point localization results with the key point annotations;

[0018] Based on the target instance identifier, the directional target detection result and the key point localization result that match the same directional bounding box annotation are combined into a target instance record;

[0019] Multiple target instance records are combined into the instance association prediction set.

[0020] Furthermore, the generation of the geometrically consistent state includes:

[0021] The major axis direction of the directional bounding box is generated based on the long side of the predicted directional bounding box in the directional target detection result;

[0022] The key point structure direction is generated based on the key point coordinates of two key points with a preset structural relationship;

[0023] Calculate the directional deviation between the major axis of the oriented bounding box and the structural direction of the key point;

[0024] Obtain the positional state of the keypoints relative to the predicted orientation bounding box;

[0025] The directional deviation, the position inclusion state, and the key point visibility marker are combined according to preset state rules to generate a first consistent state, a second consistent state, a conflict state, or an invalid state, which serve as the geometric consistency state.

[0026] Furthermore, the generation of the task valid marker includes:

[0027] When the directional bounding box annotation exists, the directional target detection result matches the directional bounding box annotation, and the number of visible key points used to generate the key point structure direction reaches a preset number, a valid task valid label is generated for the corresponding target instance;

[0028] When the directional bounding box annotation is missing, the directional target detection result does not match the directional bounding box annotation, or the number of visible key points is lower than the preset number, an invalid task valid label is generated for the corresponding target instance.

[0029] A task loss mask is generated based on the target instances with valid task labels. The task gradients that participate in the shared parameter update are obtained based on the task loss filtered by the task loss mask. Tasks without valid target instances do not participate in the similarity calculation and aggregation of the task gradients.

[0030] Furthermore, the task loss set includes the directional target detection task loss and the key point localization task loss;

[0031] The loss for the directional target detection task includes target classification loss and directional bounding box regression loss;

[0032] The loss in the key point localization task includes key point coordinate loss and key point visibility loss;

[0033] Adaptive task weight coefficients are generated based on the quantization scalar of the task uncertainty corresponding to each task. The adaptive task weight coefficients are then weighted with the corresponding task loss to generate a joint loss value.

[0034] Furthermore, the multi-task visual detection network includes a shared feature extraction network, a directional target detection head, and a key point localization head;

[0035] The shared parameter set includes network parameters in the shared feature extraction network that simultaneously provide features to the directional target detection head and the key point localization head;

[0036] The task-specific parameter set includes the network parameters of the directional target detection head and the network parameters of the key point localization head;

[0037] Establish parameter attribution relationships according to network layer identifiers, perform backpropagation on the loss of the directional target detection task and the loss of the key point localization task respectively, and obtain the task gradient of each task with respect to the shared parameter set and the task-specific parameter gradient with respect to the corresponding task-specific parameter set.

[0038] Furthermore, the tasks include directional target detection tasks and key point localization tasks;

[0039] The task gradient corresponding to the directional target detection task and the task gradient corresponding to the key point localization task are both task gradients generated by the corresponding tasks for the shared parameter set.

[0040] The cosine similarity is the cosine similarity between the task gradient corresponding to the directional target detection task and the task gradient corresponding to the key point localization task.

[0041] The gradient similarity target value is selected based on the geometric consistency state, wherein the first consistency state, the second consistency state, and the conflict state correspond to the first target value, the second target value, and the third target value that increase sequentially, respectively, and the invalid state retains the gradient similarity of the previous valid historical target; the gradient similarity target value is weighted with the historical target gradient similarity to generate the target gradient similarity;

[0042] When the cosine similarity is lower than the target gradient similarity, one task gradient is taken as the task gradient to be adjusted and the other task gradient is taken as the reference task gradient. The back projection component of the task gradient to be adjusted in the direction of the reference task gradient is removed. When the cosine similarity after removal is still lower than the target gradient similarity, a compensation component is added along the direction of the reference task gradient until the target gradient similarity or a preset compensation upper limit is reached, and a coordinated task gradient is generated.

[0043] When the cosine similarity is not lower than the target gradient similarity, the original task gradient is retained; the coordinated task gradient is aggregated with the reference task gradient, or the retained original task gradient is aggregated to generate the joint gradient.

[0044] Furthermore, based on the updated multi-task visual detection network, the directional target detection results and key point localization results for the next training cycle are generated, and the geometric consistency state is regenerated according to the directional target detection results and key point localization results for the next training cycle.

[0045] The historical target gradient similarity is updated based on the gradient similarity target value corresponding to the regenerated geometric consistency state and the target gradient similarity of the current training cycle; when the regenerated geometric consistency state is an invalid state, the previous valid historical target gradient similarity is maintained.

[0046] When the task gradient is zero or contains non-numerical values, an abnormal training state marker is generated, and the corresponding parameter update is skipped; the updated verification loss value of the multi-task visual detection network is obtained within a continuous verification cycle; when the change in the verification loss value is lower than the convergence tolerance, the geometric consistency state remains stable, and the proportion of target instances corresponding to conflict states is lower than a preset proportion, the training termination condition is generated; the geometric consistency state remains stable means that the change in the proportion of target instances corresponding to each state is lower than a preset proportion, and the change in the average directional deviation of effective target instances is lower than a preset angle; when the maximum number of training rounds is reached, the training termination condition is also generated.

[0047] The present invention also provides a multi-task visual detection gradient optimization system, comprising:

[0048] The multi-task prediction and association module is used to acquire training samples with oriented bounding box annotations and key point annotations, input them into the multi-task visual detection network to generate oriented target detection results and key point localization results, and associate them according to the target instance identifier to obtain the instance association prediction set;

[0049] The task training state generation module is connected to the multi-task prediction and association module. It is used to generate a task loss set based on the instance association prediction set, and generate geometric consistency state and task valid label according to the long axis direction of the oriented bounding box of the same target instance, the structural direction of the key point, the key point position inclusion state and visibility.

[0050] The gradient coordination parameter generation module, connected to the task training state generation module, is used to divide the shared parameter set and the task-specific parameter set according to the network layer affiliation, obtain the task gradient of the effective task to the shared parameter set and the task-specific parameter gradient of each task, calculate the cosine similarity between task gradients, and weight the gradient similarity target value corresponding to the geometric consistency state with the historical target gradient similarity to generate the target gradient similarity.

[0051] The network parameter update and model output module, connected to the gradient coordination parameter generation module, is used to remove the back projection component of the gradient of the task to be adjusted and add a compensation component along the direction of the reference task gradient when the cosine similarity is lower than the target gradient similarity. It aggregates the adjusted or unadjusted task gradients to generate a joint gradient, updates the shared parameter set with the joint gradient, and updates the corresponding task-specific parameter set with the gradients of each task-specific parameter. Based on the updated network, it updates the geometric consistency state and historical target gradient similarity, and outputs a multi-task visual detection model when the training termination condition is met.

[0052] The key innovations of this invention include:

[0053] (1) Associate the directional target detection results and key point localization results of the same target instance, form a hierarchical geometric consistency state based on the long axis direction of the directional bounding box, the structural direction of the key point, the positional inclusion state of the key point and visibility, and weight the corresponding gradient similarity target value with the historical target gradient similarity so that the spatial structural state of the visual output directly participates in the shared parameter gradient coordination.

[0054] (2) The shared parameter set and the task-specific parameter set are divided according to the network layer affiliation. The cross-task gradient adjustment only applies to the shared parameters. When the gradient similarity is insufficient, the back projection component is removed first, and then the compensation component constrained by the upper limit is added along the gradient direction of the reference task. The task-specific parameters are still updated independently along their respective task-specific parameter gradients.

[0055] (3) Generate effective task labels and task loss masks according to target instances, and limit the effective tasks participating in the calculation and aggregation of shared parameter gradient similarity; after parameter update, re-form geometric consistency state and update historical target gradient similarity, so that the prediction state, gradient coordination conditions and the next training cycle form a feedback loop.

[0056] The following are its main beneficial effects:

[0057] (1) By converting orientation deviation, position containing state and key point visibility into hierarchical gradient similarity target values, the task gradient adjustment intensity corresponds to the degree of geometric conflict of the same target instance, reducing the coordination deviation caused by relying solely on fixed thresholds or gradient angles.

[0058] (2) By removing back projection components and compensating for reference gradient direction, the ability of shared parameter task gradient to meet the current target similarity is improved; at the same time, the boundary between joint update of shared parameters and independent update of task-specific parameters is defined to reduce parameter update interference between task prediction heads.

[0059] (3) By using the task valid label and task loss mask, invalid gradients corresponding to missing labels, invisible key points or abnormal instance matching are eliminated, and the updated geometric consistency state is used to smoothly update the historical target gradient similarity, thereby reducing the impact of abnormal batches and short-term state fluctuations on subsequent training. Attached Figure Description

[0060] Figure 1 A flowchart illustrating a multi-task visual detection gradient optimization method provided in an embodiment of this application;

[0061] Figure 2 This is a structural block diagram of a multi-task visual inspection gradient optimization system provided in an embodiment of this application. Detailed Implementation

[0062] To enable those skilled in the art to better understand the technical solution, the present invention will be described in detail below with reference to embodiments. The description in this part is only exemplary and explanatory, and should not be used to limit the scope of protection of the present invention in any way.

[0063] Example 1: Refer to Figure 1 This is a flowchart illustrating a multi-task visual detection gradient optimization method provided in an embodiment of the present invention. The flowchart may include at least steps S100-S400:

[0064] S100. Obtain training samples with oriented bounding box annotations and keypoint annotations, input them into the multi-task visual detection network, generate oriented target detection results and keypoint localization results, associate them according to target instance identifiers, and obtain the instance association prediction set:

[0065] In this embodiment, the multi-task visual detection gradient optimization process is executed by a model training device. This device is equipped with a processor, memory, and a graphics processing unit. The memory stores training samples, the multi-task visual detection network, network parameters, and training state. The multi-task visual detection network includes a shared feature extraction network, a directional target detection head, and a keypoint localization head. The shared feature extraction network receives training images and outputs shared visual features; the directional target detection head outputs the target category and predicted directional bounding box based on the shared visual features; and the keypoint localization head outputs multiple predicted keypoint coordinates and corresponding keypoint visibility prediction results based on the shared visual features. The model training device reads training samples in training batches, and the forward propagation process of this step is triggered after each training batch arrives.

[0066] Image identifiers in the training samples are used to distinguish different training images, while target instance identifiers are used to distinguish different targets within the same training image. Each target instance corresponds to a target category label, an oriented bounding box label, and multiple keypoint labels. The oriented bounding box label is represented by the target center coordinates, target width, target height, and target rotation angle. Keypoint labels include keypoint number, keypoint coordinates, and keypoint visibility markers. Keypoint visibility markers indicate whether the corresponding keypoint is visible, partially occluded, or invisible in the current training image. In one implementation, a visible state is recorded as 1, and an invisible state as 0; a partially occluded state is recorded as 0 or 1 according to the labeling rules, or configured as an independent state value. Keypoints participating in the keypoint structure orientation calculation should have valid coordinates and visibility markers.

[0067] A shared feature extraction network extracts features layer by layer from the training image, outputting shared visual features containing target contours, local edges, and spatial location information. The directional target detection head reads these shared visual features and calculates the target category probability, target center offset, target width and height, and target rotation angle to form the directional target detection result. The keypoint localization head extracts local structural responses from the same shared visual features, outputting the predicted coordinates and visibility probability of each keypoint to form the keypoint localization result. The directional target detection head and the keypoint localization head share the front-end visual features, but their output layer parameters are independent. Subsequent parameter updates are processed separately according to the shared parameter set and the task-specific parameter set.

[0068] After the prediction results are generated, the multi-task prediction and association module first matches the predicted oriented bounding boxes with the oriented bounding box annotations in the current training image. During matching, the target category is checked first; for predicted oriented bounding boxes with the same category, the overlap degree is calculated with each oriented bounding box annotation. When a predicted oriented bounding box meets the matching conditions with multiple annotation boxes, the annotation box with the largest overlap degree is selected. When multiple predicted oriented bounding boxes point to the same annotation box, the prediction result with the higher target category probability and the larger overlap degree is retained, and the remaining prediction results do not participate in the geometric consistency calculation of the current target instance, but can still participate in the target classification loss calculation according to the positive and negative sample rules of the detection task.

[0069] In implementations where the directional target detection head and keypoint localization head output results based on the same spatial location or the same detection query, the keypoint localization result directly inherits the target instance identifier of the corresponding predicted directional bounding box. In implementations where the two task heads output candidate results separately, the keypoint localization results are associated according to the distance between the predicted keypoint center position and the predicted directional bounding box center position; when the distances are the same or close, the number of keypoints falling inside the predicted directional bounding box is compared. directional target detection results and keypoint localization results matching the same directional bounding box label are combined into a single target instance record.

[0070] The target instance record stores the target instance identifier, directional target detection results, key point localization results, and corresponding annotation data. When multiple target instances exist, the multi-task prediction and association module establishes separate target instance records and combines these records into an instance association prediction set. If a predicted directional bounding box cannot match any directional bounding box annotation, the prediction result is treated as a negative sample in the detection task, and no target instance record is established for geometric consistency calculation. If the directional bounding box has matched but the key point localization result is missing, the target instance record is still retained, and the key point localization status is marked as invalid for the S200 to generate a valid task marker.

[0071] The target instance identifiers, oriented target detection results, and keypoint localization results from the instance association prediction set are passed to S200. The target instance identifiers limit subsequent geometric consistency calculations to be performed within the same target instance, avoiding comparisons between the orientation bounding box direction of one target and the keypoint structural direction of another target. The labeled data corresponding to the prediction results then enters the task loss calculation process.

[0072] S200. Generate a task loss set based on the instance association prediction set; generate geometric consistency state and task valid markers based on the major axis direction of the oriented bounding box, the structural direction of key points, the positional inclusion state of key points, and visibility of the same target instance:

[0073] After receiving the instance association prediction set formed by S100, the task training state generation module first calculates the task loss for directional target detection and the task loss for keypoint localization according to the task type. Then, it forms a geometrically consistent state based on the geometric relationship of each target instance. The task loss and the geometrically consistent state use the same target instance association basis, but they play different roles: the task loss enters the backpropagation process, while the geometrically consistent state enters the target gradient similarity generation process.

[0074] The loss in a directional target detection task includes target classification loss and directional bounding box regression loss. The target classification loss is calculated based on the difference between the predicted target class probability and the target class label. The directional bounding box regression loss is calculated based on the differences between the predicted target center, predicted target width, predicted target height, and predicted target rotation angle and their corresponding labels. For rotation angles with periodic equivalence, the angle error is calculated using the minimum equivalent angle difference to avoid large numerical differences in angles representing the same direction due to crossing angle boundaries.

[0075] The loss in keypoint localization includes keypoint coordinate loss and keypoint visibility loss. Keypoint coordinate loss is formed by the distance between the predicted keypoint coordinates and the labeled keypoint coordinates. To avoid the direct impact of different target sizes on the magnitude of coordinate error, in one implementation, the coordinate distance is normalized using the width, height, or area of ​​the corresponding orientation bounding box. Keypoint visibility loss is calculated based on the difference between the predicted keypoint visibility and the keypoint visibility label. Invisible keypoints are not included in the keypoint coordinate loss but still participate in the keypoint visibility loss; when partially occluded keypoints have valid coordinates, they are included in the keypoint coordinate loss according to a preset weight.

[0076] The target detection task loss and the keypoint localization task loss together form a task loss set. In one implementation, a task uncertainty quantization scalar is configured for each of the two tasks. The task uncertainty quantization scalar serves as an updatable parameter during training, and its value is exponentially mapped to form adaptive task weight coefficients. The target detection task loss is multiplied by its corresponding adaptive task weight coefficient, and the keypoint localization task loss is also multiplied by its corresponding adaptive task weight coefficient. These are then combined with the constraint term corresponding to the task uncertainty quantization scalar to form a joint loss value. This weighting process occurs before task gradient extraction; subsequently, the task gradients of each task loss relative to the shared parameter set are still retained, rather than only retaining the single gradient corresponding to the joint loss value.

[0077] Geometric consistency states are calculated one by one according to the target instance records. For a target instance, the task training state generation module reads the width, height, and rotation angle of the predicted oriented bounding box. The larger of the width and height values ​​corresponds to the long side of the oriented bounding box, and the major axis direction of the oriented bounding box is generated based on the long side and the rotation angle. The major axis direction is an undirected axis, and the positive and negative directions represent the same geometric axis. Therefore, the subsequent direction deviation calculation compares the angles between the key point structure direction and the major axis direction, and between the key point structure direction and the opposite direction of the major axis, and takes the smaller angle as the direction deviation.

[0078] The keypoint structural orientation is generated from two keypoints with a preset structural relationship. This preset structural relationship is associated with the keypoint number in the training configuration. For example, in a fruit detection scenario, the stem connection point and the fruit vertex can be selected as directional keypoints; in an industrial part detection scenario, the assembly reference point and the end positioning point can be selected as directional keypoints. The task training state generation module reads the coordinates of the two directional keypoints and uses the coordinate difference between the first and second keypoints to form the keypoint structural orientation. If the visibility of either directional keypoint is marked as invalid, or if the distance between the two keypoints is less than a preset distance, a valid keypoint structural orientation is not calculated, and the geometric orientation state of the corresponding target instance is marked as invalid.

[0079] The location inclusion state is obtained by transforming the keypoint coordinates to the local coordinate system of the oriented bounding box. The local coordinate system is centered at the center of the predicted oriented bounding box, with the horizontal axis along the major axis and the vertical axis along the minor axis. A keypoint is considered "inside the box" if the absolute value of its local horizontal coordinate is no greater than half the length of the major side of the predicted oriented bounding box and the absolute value of its local vertical coordinate is no greater than half the length of the minor side. If the keypoint is outside the bounding box but within a preset extended area, its location inclusion state is considered "border neighborhood." If it is outside the extended area, its location inclusion state is considered "outside the box." The extended area is set proportionally to the width and height of the predicted oriented bounding box, without using a fixed pixel distance to avoid using the same boundary margin for targets of different scales.

[0080] The task training state generation module combines orientation deviation, location inclusion state, and keypoint visibility markers into a geometrically consistent state. In one implementation, when the orientation deviation is within a first preset interval and all directional keypoints are located inside the predicted orientation bounding box, the geometrically consistent state is recorded as the first consistent state; when the orientation deviation is within a second preset interval, or when there are directional keypoints located in the boundary neighborhood, it is recorded as the second consistent state; when the orientation deviation exceeds the second preset interval, or when the directional keypoints are located outside the predicted orientation bounding box, it is recorded as a conflict state; when the keypoint structural orientation cannot be calculated, it is recorded as an invalid state. The geometrically consistent state can also be represented by a state vector containing the orientation deviation value, location inclusion state, and visibility state, and is not limited to a discrete hierarchical form.

[0081] The task training state set also includes valid task labels generated for each target instance. Valid task labels are generated for the corresponding target instance when the directional bounding box annotation exists, the predicted directional bounding box matches the labeled box, and the number of visible keypoints used to form the keypoint structure orientation reaches a preset number. Invalid task labels are generated when the directional bounding box annotation is missing, the prediction result fails to match, or the number of visible keypoints is less than a preset number. A task loss mask is formed based on the valid target instances. The keypoint coordinate loss and the task loss used for shared parameter gradient coordination are calculated only based on the target instances within the mask. When the detection task labels are still valid, the task-specific parameter gradients of the directional target detection head can be retained.

[0082] When a training batch contains multiple target instances, the task training state generation module first obtains the geometric consistency state of each target instance, and then aggregates them according to the number of valid target instances. During aggregation, the statistics of directional deviation, the number of target instances corresponding to each geometric state, and the task valid label are retained. When there are no valid target instances, the current training batch does not use the geometric consistency state to adjust the target gradient similarity. S300 continues to use the historical target gradient similarity and records the current batch as a batch with missing geometric states.

[0083] The final task training state set includes the task loss set, joint loss value, valid task labels, task loss mask, geometric consistency state, and target instance association information. The task loss filtered by the task loss mask enters the backpropagation process of S300, the valid task labels limit the tasks participating in the calculation and aggregation of shared parameter gradient similarity, and the geometric consistency state participates in the generation of target gradient similarity.

[0084] S300. Divide the shared parameter set and the task-specific parameter set according to the network layer affiliation. Obtain the task gradient of the effective task to the shared parameter set and the task-specific parameter gradient of each task to the corresponding task-specific parameter set. Calculate the cosine similarity of the task gradients. Weight the similarity target value corresponding to the geometric consistency state with the historical target gradient similarity to obtain the target gradient similarity.

[0085] After receiving the task training state set, the gradient coordination parameter generation module first divides the network parameters according to the affiliation of the network layers. Network parameters in the shared feature extraction network that simultaneously provide features to both the directional target detection head and the keypoint localization head are assigned to the shared parameter set; network parameters within the directional target detection head that only participate in target classification and directional bounding box regression are assigned to the first task-specific parameter set; and network parameters within the keypoint localization head that only participate in keypoint coordinate and visibility prediction are assigned to the second task-specific parameter set. When a cross-task feature fusion layer exists, this fusion layer receives information from both task branches simultaneously, and its network parameters can be assigned to the shared parameter set.

[0086] Parameter attribution is established based on network layer identifiers during model initialization and remains consistent with the network structure version during training. When the model structure changes, the gradient coordination parameter generation module rereads the network parameter names and their respective network layers, updating the parameter attribution relationships. If a network parameter is simultaneously marked as both a shared parameter set and a task-specific parameter set, parameter updates for the current training batch are paused, and parameter attribution is regenerated based on network connectivity relationships to prevent the same network parameter from receiving both joint gradients and task-specific gradients within a single training cycle.

[0087] The gradients for each task are obtained by performing backpropagation separately. For the oriented target detection task, the gradient coordination parameter generation module uses the oriented target detection task loss as the starting point for backpropagation, calculates its first task gradient relative to the shared parameter set, and calculates its first task-specific parameter gradient relative to the first task-specific parameter set. For the keypoint localization task, the keypoint localization task loss is used as the starting point for backpropagation, calculates its second task gradient relative to the shared parameter set, and calculates its second task-specific parameter gradient relative to the second task-specific parameter set.

[0088] When extracting gradients for different tasks within a training batch, the forward computation graph is preserved, and the automatic differentiation process is called separately to prevent intermediate computation results from being released during the first backpropagation. After each task gradient extraction is completed, the gradient is copied to the corresponding gradient buffer without immediately updating the network parameters. The task gradients used for coordinating shared parameter gradients are obtained based on the task loss filtered by the task loss mask; tasks without valid target instances are not included in the shared parameter task gradient set, and whether their task-specific parameter gradients are retained is determined according to the labeled valid state of the corresponding task.

[0089] Gradient similarity is calculated for a shared parameter set. The gradients of the first and second tasks are expanded into gradient vectors according to the parameter order in the shared parameter set. The inner product of the two gradient vectors and their respective gradient norms are calculated. Then, the inner product is divided by the product of the two gradient norms to obtain the cosine similarity, which is used as the current gradient similarity. A current gradient similarity close to 1 indicates that the update directions of the two tasks for the shared parameters are similar; close to 0 indicates a weak directional correlation; and less than 0 indicates that the two tasks contain mutually canceling update components.

[0090] When the gradient norm of a certain task is 0, there is no valid divisor for the cosine similarity. The gradient coordination parameter generation module generates a zero-gradient aberration training state marker, and the current training batch does not use this task gradient to update shared parameters. When the task gradient contains non-numerical elements, a non-numerical aberration training state marker is generated, and the current network parameters are retained. When the gradient norm exceeds the gradient norm threshold, the original task gradient is not directly used to calculate the similarity. Instead, the task gradient is first subjected to norm pruning, and then the current gradient similarity is calculated based on the pruned gradient.

[0091] The target gradient similarity is generated jointly by the current geometrically consistent state and the historical target gradient similarity. The first consistent state, the second consistent state, and the conflicting state correspond to the first target value, the second target value, and the third target value, respectively, which increase sequentially. Specifically, the first consistent state is formed when the directional deviation is small, the directional keypoint is visible, and it is located inside the predicted orientation bounding box; the second consistent state is formed when the directional deviation is in the middle range or the keypoint is located in the boundary neighborhood; and the conflicting state is formed when the directional deviation is large or the keypoint is located outside the predicted orientation bounding box. This results in training batches with higher geometric conflict levels corresponding to stronger shared gradient direction constraints.

[0092] For example, the first target value can be set to 0.1, the second target value can be set to 0.4, and the third target value can be set to 0.7. The above values ​​correspond to only one implementation method; the specific values ​​are configured based on changes in task loss and the geometric consistency state in the verification batch. When the geometric consistency state is invalid, a new gradient similarity target value is not selected from the current batch; instead, the previously valid historical target gradient similarity is maintained.

[0093] After obtaining the target gradient similarity value, it is weighted and summed with historical target gradient similarities to form the current target gradient similarity. Let the historical weight be λ, then the current target gradient similarity equals the product of the historical target gradient similarity and λ, plus the product of the target gradient similarity value and 1-λ; in one implementation, λ is set to 0.8. When multiple consecutive training batches are in the same geometric consistency state, the current target gradient similarity gradually approaches the target gradient similarity value corresponding to that state; when the geometric state fluctuates in the short term, the historical weight restricts the target value from abruptly changing between adjacent batches. When historical target gradient similarities are lacking in the first training cycle, the target gradient similarity value corresponding to the current geometric consistency state is used as the initial historical value.

[0094] S300 outputs the current gradient similarity, the target gradient similarity, the shared parameter task gradient set, and the task-specific parameter gradient. The current gradient similarity is compared with the target gradient similarity in S400; the shared parameter task gradient set enters the gradient adjustment and aggregation process; and the task-specific parameter gradients enter the parameter update process of the corresponding task prediction head.

[0095] S400. When the cosine similarity is lower than the target gradient similarity, remove the back projection component of the gradient to be adjusted relative to the reference gradient, and add a compensation component along the direction of the reference gradient; aggregate the adjusted or unadjusted task gradients into a joint gradient, update the shared parameter set with the joint gradient, and update the task-specific parameters with each dedicated gradient; based on the updated network, update the geometric consistency state and historical target gradient similarity, and output the multi-task visual detection model when the training termination condition is met.

[0096] The network parameter update and model output module reads the current gradient similarity, target gradient similarity, first task gradient, and second task gradient from the S300 output. When the current gradient similarity is not lower than the target gradient similarity, the update relationship of the two tasks for shared parameters satisfies the orientation constraint corresponding to the current geometric state, and the gradients of both tasks are retained. When the current gradient similarity is lower than the target gradient similarity, the orientation of one task gradient is adjusted. The task to be adjusted can be selected alternately according to training batches or according to the change in task loss; when the loss of one task decreases less in consecutive training batches while the loss of another task changes steadily, the former is selected as the task to be adjusted.

[0097] When performing gradient adjustment, the projected overlap component of the gradient of the task to be adjusted in the direction of the gradient of the reference task is first calculated. When the projected overlap component is negative, this component corresponds to the reverse update component between the two tasks, and the reverse update component is first deleted from the gradient of the task to be adjusted. After deletion, the cosine similarity between the adjusted gradient and the gradient of the reference task is recalculated. If the recalculated cosine similarity is still lower than the target gradient similarity, a compensation component is added along the unit direction of the gradient of the reference task. The compensation coefficient is determined based on the difference between the current gradient similarity and the target gradient similarity, and the norm of the compensation component is limited to a preset proportion of the original norm of the gradient of the task to be adjusted.

[0098] In another implementation, the compensation coefficient is determined through a one-dimensional search. The network parameter update and model output module increments the compensation coefficient starting from 0. Each time, the compensation coefficient is multiplied by the unit vector of the reference task gradient and added to the gradient of the task to be adjusted, and the cosine similarity is recalculated. The search stops when the cosine similarity reaches the target gradient similarity or the compensation coefficient reaches a preset upper limit. This yields the coordinated task gradient. This process only applies to the gradient of the shared parameter task and does not change the gradients of the first and second task-specific parameters.

[0099] The joint gradient of shared parameters is formed by aggregating the coordinated task gradient with the gradient of another task. When configuring adaptive task weight coefficients, each task gradient is first multiplied by its corresponding weight, and then added parameter by parameter; if no task weights are configured, they are added according to a preset base ratio. When a task's valid value is marked as invalid, the joint gradient is formed by the gradient of the other valid task. When both tasks are invalid, the current shared parameter update is skipped, but the loss and abnormal states of the current training batch are still recorded.

[0100] After the joint gradient is generated, the preset optimizer updates the shared parameter set based on the joint gradient. The first task-specific parameter gradient only updates the task-specific parameter set of the directional target detection head; the second task-specific parameter gradient only updates the task-specific parameter set of the keypoint localization head. The shared parameter set will not receive independent updates from task-specific parameter gradients again, nor will it receive cross-task joint gradients. After the parameter updates are completed, the model parameter update results are generated.

[0101] When the task gradient norm exceeds a threshold, the gradients exceeding the threshold are scaled proportionally according to the norm before participating in joint gradient generation. If the task gradient contains non-numerical elements, the corresponding parameter updates are not performed in the current training batch, and the temporary state corresponding to the abnormal gradient is cleared from the optimizer. If consecutive non-numerical gradients occur, the training device restores the most recent valid training checkpoint and reduces the parameter update step size for the next training cycle. This restoration path is an optional implementation and does not change the processing order of task gradient coordination.

[0102] After the model parameters are updated, the network parameter update and model output module inputs the next training or validation batch into the updated multi-task visual detection network to regenerate oriented target detection results and keypoint localization results, and establishes a new instance association prediction set according to the target instance association method in S100. Subsequently, according to the major axis direction, keypoint structural direction, and positional inclusion state calculation method in S200, the geometric consistency state is regenerated. The regenerated geometric consistency state reflects the structural relationship between the outputs of the two tasks under the current network parameters.

[0103] The historical target gradient similarity is updated based on the target gradient similarity value corresponding to the regenerated geometrically consistent state and the target gradient similarity of the current training cycle. When the regenerated geometrically consistent state changes towards the first consistent state, the historical target gradient similarity is updated smoothly according to the historical weights, without directly decreasing to the first target value. When the geometrically consistent state remains in a conflict state, the number of consecutive gradient conflicts increases, and the third target value is selected again in the next training cycle. When the geometrically consistent state is invalid, the previously valid historical target gradient similarity is maintained, and the invalid state is not used to overwrite the historical record.

[0104] The training termination condition is determined by combining the validation loss value and the geometric consistency state. After the validation cycle ends, the network parameter update and model output module obtains the validation loss value corresponding to the updated multi-task visual detection network and calculates the change of the validation loss value relative to the previous validation cycle. A training termination condition is generated when the change is below the convergence tolerance for multiple consecutive validation cycles, the geometric consistency state remains stable within the same validation cycle, and the proportion of target instances corresponding to conflict states is below a preset proportion. Stable geometric consistency means that the change in the proportion of target instances corresponding to each state is below a preset proportion, and the change in the average directional deviation of effective target instances is below a preset angle. A training termination condition is also generated when the maximum number of training epochs is reached.

[0105] If the training termination condition is not met, the updated historical target gradient similarity is returned to S300 in the next training batch and continues to participate in target gradient similarity calculation. When the training termination condition is met, the network parameter update and model output module solidifies the network parameters of the shared feature extraction network, the directional target detection head, and the keypoint localization head, saves the keypoint number configuration, the directional bounding box representation configuration, and the network structure configuration, and outputs the multi-task visual detection model. After receiving the image to be detected, the multi-task visual detection model outputs the directional target detection result and the keypoint localization result through one forward propagation; the task gradient calculation, gradient similarity comparison, and gradient projection used in the training phase are not included in the model inference process.

[0106] Example 2: Figure 2 A structural block diagram of a multi-task visual detection gradient optimization system according to an embodiment of the present invention is shown. Figure 2 As shown, the structure may include:

[0107] The multi-task prediction and association module 01 is used to acquire training samples with directional bounding box annotations and keypoint annotations. The training samples are input into the multi-task visual detection network to generate directional target detection results and keypoint localization results, and are associated according to target instance identifiers to obtain an instance association prediction set. This multi-task prediction and association module is located in the model training device and connected to the training sample storage area and the forward computation interface of the multi-task visual detection network. The training samples are input in training batches. Each training sample includes an image identifier, target instance identifier, target category annotation, directional bounding box annotation, and multiple keypoint annotations. The directional bounding box annotation includes the target center coordinates, target width, target height, and target rotation angle. Each keypoint annotation includes a keypoint number, keypoint coordinates, and keypoint visibility marker. After a training batch is loaded into the model training device, the multi-task prediction and association module checks the correspondence between the image data and the annotation data. If an image identifier is missing, the image size cannot be read, or the target instance identifier is duplicated, the forward computation of the corresponding training sample is paused, and an abnormal sample status is set for that training sample. The remaining training samples continue to enter the current training batch.

[0108] The multi-task visual detection network includes a shared feature extraction network, a directional target detection head, and a keypoint localization head. The shared feature extraction network performs layer-by-layer feature extraction on the training images, feeding the resulting shared visual features into the directional target detection head and the keypoint localization head. The directional target detection head outputs the target category probability, predicted target center, predicted target width, predicted target height, and predicted target rotation angle based on the shared visual features, and forms a predicted directional bounding box accordingly. The keypoint localization head outputs multiple predicted keypoint coordinates and corresponding keypoint visibility prediction results based on the same shared visual features. Both task heads read the same batch of shared visual features during the forward computation phase, and the network parameters within each task head are stored in their respective task-specific parameter regions, enabling the subsequent gradient extraction process to distinguish between shared network parameters and task-specific network parameters.

[0109] After the prediction results are generated, the multi-task prediction and association module first matches the predicted oriented bounding boxes with the oriented bounding box annotations in the current training image. During matching, the target categories are compared first, and then the overlap between the predicted oriented bounding boxes and the oriented bounding box annotations with the same category is calculated. When one predicted oriented bounding box corresponds to multiple oriented bounding box annotations, the oriented bounding box annotation with the largest overlap is selected. When multiple predicted oriented bounding boxes correspond to the same oriented bounding box annotation, the prediction results whose target category probabilities and overlap degrees conform to the preset matching rules are retained, and the remaining prediction results are used as unmatched detection results in the calculation of target classification loss, and are not included in the calculation range of geometric consistency state.

[0110] Keypoint localization results are associated with the matched directional target detection results according to the target instance identifier. In the implementation where the directional target detection head and keypoint localization head output results according to the same detection query, the keypoint localization results inherit the target instance identifier of the corresponding detection query. In the implementation where the two task heads output candidate results separately, the multi-task prediction and association module compares the distance between the center of the keypoint set and the center of the predicted directional bounding box, and checks the number of keypoints falling within the predicted directional bounding box. Keypoint localization results whose distance and position inclusion relationships meet the association conditions are assigned to the corresponding target instance. After association, the directional target detection results, keypoint localization results, and their annotation data for the same target instance constitute a target instance record, and multiple target instance records constitute the instance association prediction set. When a keypoint localization result is missing, the corresponding target instance record is retained, and the keypoint localization status is set to invalid. The invalid status is passed to the task training state generation module along with the instance association prediction set, and the task training state generation module controls whether the corresponding task enters the gradient aggregation process.

[0111] The task training state generation module 02, connected to the multi-task prediction and association module, is used to generate a task loss set based on the instance association prediction set. It generates geometric consistency states and valid task labels based on the major axis direction, keypoint structure direction, keypoint position inclusion state, and visibility of the same target instance's oriented bounding box. The task training state generation module receives multiple target instance records through the instance association prediction set interface and calls the oriented target detection results, keypoint localization results, and corresponding annotation data according to the task type. The difference between the oriented target detection results and the target category annotations forms the target classification loss; the difference between the predicted oriented bounding box and the oriented bounding box annotations in terms of center position, width, height, and rotation angle forms the oriented bounding box regression loss. The target classification loss and the oriented bounding box regression loss are combined to form the oriented target detection task loss. The distance between the predicted keypoint coordinates and the keypoint coordinate annotations in the keypoint localization results forms the keypoint coordinate loss; the difference between the keypoint visibility prediction results and the keypoint visibility annotations forms the keypoint visibility loss. The two are combined to form the keypoint localization task loss.

[0112] For keypoints marked as invisible, the task training state generation module does not include the corresponding predicted coordinates in the keypoint coordinate loss, but retains the keypoint visibility loss. For partially occluded keypoints with valid coordinates, they are included in the keypoint coordinate loss according to preset weights. In one implementation, the keypoint coordinate difference is normalized using the width and height of the corresponding oriented bounding box, so that the keypoint coordinate errors at different target scales use the same dimension. The oriented target detection task loss and the keypoint localization task loss together constitute the task loss set. When configuring the task uncertainty quantization scalar, the task training state generation module forms adaptive task weight coefficients based on the task uncertainty quantization scalar corresponding to each task, and weights the corresponding task losses to obtain a joint loss value; the original task losses of each task are still retained separately for the gradient coordination parameter generation module to extract the task gradient.

[0113] Geometric consistency states are formed one by one according to the target instance records. The task training state generation module selects the long side of the orientation bounding box based on the width, height, and rotation angle of the predicted orientation bounding box, and uses the extension direction of the long side as the major axis direction of the orientation bounding box. The key point structural direction is formed by two key points with a preset structural relationship. The task training state generation module reads the coordinates of the two key points, and uses the coordinate difference between the first key point and the second key point to represent the key point structural direction. Since the two extension directions of the major axis of the orientation bounding box represent the same geometric axis, the module calculates the angle between the key point structural direction and the positive and negative directions of the major axis, respectively, and uses the smaller angle as the direction deviation.

[0114] The task training state generation module also transforms the keypoint coordinates to the local coordinate system of the predicted orientation bounding box, compares the local coordinates of the keypoints with the boundary ranges of the long and short sides of the predicted orientation bounding box, and forms a position inclusion state. When the keypoint is inside the predicted orientation bounding box, the position inclusion state is recorded as "inside the box"; when the keypoint is in the boundary neighborhood formed by expanding the orientation bounding box size, the position inclusion state is recorded as "boundary neighborhood"; when the keypoint exceeds the boundary neighborhood, the position inclusion state is recorded as "outside the box". The orientation deviation, position inclusion state, and keypoint visibility marker together constitute the geometric consistency state. If one of the two keypoints used to form the keypoint structural orientation is invisible, or if the distance between the two keypoints is less than a preset distance, the geometric consistency state of the corresponding target instance is set to an invalid state.

[0115] The task training state set also includes valid task labels generated according to target instances. A valid task label is formed when the directional bounding box annotation exists, the predicted directional bounding box matches the annotation, and the number of visible keypoints reaches a preset number; an invalid task label is formed when the directional bounding box annotation is missing, the match fails, or the number of visible keypoints is less than the preset number. The task training state generation module generates a task loss mask according to target instances with valid task labels, and obtains the task gradients participating in shared parameter updates based on the task loss filtered by the task loss mask; tasks without valid target instances do not participate in the shared parameter gradient similarity calculation and aggregation of the current training batch.

[0116] The gradient coordination parameter generation module 03, connected to the task training state generation module, is used to divide the shared parameter set and the task-specific parameter set according to network layer affiliation, obtain the task gradient of the effective task with respect to the shared parameter set and the task-specific parameter gradient of each task with respect to the corresponding task-specific parameter set, calculate the cosine similarity between the task gradients, and weight the gradient similarity target value corresponding to the geometric consistency state with the historical target gradient similarity to generate the target gradient similarity. The gradient coordination parameter generation module is connected to the parameter storage area of ​​the multi-task visual detection network and establishes parameter mapping relationships according to network layer identifiers during the training initialization phase. Network parameters that simultaneously output shared visual features to the directional target detection head and the key point localization head in the shared feature extraction network are assigned to the shared parameter set; network parameters within the directional target detection head and the key point localization head that only participate in the prediction of this task are assigned to the corresponding task-specific parameter sets. When the network structure changes, the gradient coordination parameter generation module rereads the network layer identifiers and updates the parameter mapping relationships; when the same network parameter appears in both the shared parameter set and the task-specific parameter set, the parameter update of the current training batch is paused, and the parameter affiliation is re-divided according to the network connection relationship.

[0117] After the task loss set arrives, the gradient coordination parameter generation module uses the directional target detection task loss and the keypoint localization task loss as the starting point for backpropagation, respectively. When performing backpropagation with the directional target detection task loss, it extracts the first task gradient relative to the shared parameter set and the first task-specific parameter gradient relative to the directional target detection head parameters. When performing backpropagation with the keypoint localization task loss, it extracts the second task gradient relative to the shared parameter set and the second task-specific parameter gradient relative to the keypoint localization head parameters. The forward computation graph is preserved between the two backpropagations, and the obtained gradients are stored in the corresponding gradient cache so that the subsequent gradient extraction does not overwrite the previous result. When a task is marked as invalid, the corresponding task gradient relative to the shared parameter set is not included in the gradient similarity calculation.

[0118] The gradients of the first and second tasks are expanded according to the parameter order in the shared parameter set. The gradient coordination parameter generation module calculates the inner product of the two gradient vectors and their respective gradient norms, and forms a cosine similarity between the inner product and the gradient norms. This cosine similarity is used as the gradient similarity. When the gradient norm of a task is 0, a zero gradient anomaly is marked for the corresponding training batch, and this task is not included in the current joint gradient. When the task gradient contains non-numerical values, the current network parameters are retained, and the anomaly is passed to the network parameter update and model output module. When the task gradient norm exceeds the gradient norm threshold, gradient clipping is performed first according to the gradient norm threshold, and then the gradient similarity is calculated.

[0119] The formation of target gradient similarity is constrained by both the current geometric consistency state and historical target gradient similarity. The gradient coordination parameter generation module selects a first target value, a second target value, or a third target value that increases sequentially based on the first consistency state, the second consistency state, or the conflict state, respectively. The state is jointly determined by directional deviation, keypoint location inclusion state, and keypoint visibility. When the geometric consistency state is invalid, the gradient similarity target value is not updated using the current target instance, but rather the previously valid historical target gradient similarity is maintained.

[0120] The selected gradient similarity target value and the historical target gradient similarity are weighted according to preset weights to form the current target gradient similarity. When multiple consecutive training batches are in the same geometric consistency state, the current target gradient similarity gradually approaches the corresponding gradient similarity target value; when the current geometric consistency state differs significantly from the previous training batch, the historical target gradient similarity restricts the target value from abruptly changing between adjacent training batches. The gradient coordination parameter generation module transmits the task gradient, task-specific parameter gradient, gradient similarity, target gradient similarity, and abnormal states to the network parameter update and model output module.

[0121] The network parameter update and model output module 04, connected to the gradient coordination parameter generation module, is used to remove the back projection component of the gradient of the task to be adjusted in the direction of the reference task gradient when the cosine similarity is lower than the target gradient similarity, and to add a compensation component along the direction of the reference task gradient when the removed cosine similarity is still lower than the target gradient similarity; to aggregate the coordinated task gradient and the reference task gradient, or to aggregate the unadjusted original task gradient, to generate a joint gradient and update the shared parameter set, and to update the corresponding task-specific parameter set with the gradient of each task-specific parameter; to regenerate the geometric consistency state based on the updated network and update the historical target gradient similarity, and to output the multi-task visual detection model when the training termination condition is met.

[0122] When adjusting the direction of the gradient of the task to be adjusted, the network parameter update and model output module calculates the projection overlap component of the gradient of the task to be adjusted in the direction of the reference task gradient. When the projection overlap component represents a reverse update component, the reverse update component is removed from the gradient of the task to be adjusted; after removal, the gradient similarity between the gradient of the task to be adjusted and the gradient of the reference task is recalculated. If the recalculated gradient similarity is still lower than the target gradient similarity, a compensation component is added along the direction of the reference task gradient until the recalculated gradient similarity reaches the target gradient similarity or the compensation component reaches a preset upper limit. The direction adjustment only applies to the task gradient corresponding to the shared parameter set and does not change the task-specific parameter gradients corresponding to the directional target detection head and the key point localization head.

[0123] The task gradient after orientation adjustment is aggregated with the gradient of another task according to task weights to form a joint gradient. Without an adaptive task weight coefficient, aggregation is performed according to a preset base ratio. When a task's valid marker is invalidated, the joint gradient is formed from the task gradient of the other valid task; when both tasks are invalid, the shared parameter update for the current training batch is paused, while the shared parameters from the previous training batch are retained. After the joint gradient is formed, the network parameter update and model output module calls the parameter update process to update the shared parameter set; the first task-specific parameter gradient only updates the task-specific parameter set of the directional target detection head, and the second task-specific parameter gradient only updates the task-specific parameter set of the keypoint localization head. This forms a joint update path for shared parameters and an independent update path for task-specific parameters.

[0124] After the parameter update is completed, the network parameter update and model output module inputs the next training batch or validation batch into the updated multi-task visual detection network to re-obtain the oriented target detection results and keypoint localization results. It then calls the same major axis direction, keypoint structural direction, and positional inclusion state calculation rules as the task training state generation module to re-form a geometrically consistent state. The re-formed geometrically consistent state, along with the current gradient similarity and the current target gradient similarity, participates in the historical target gradient similarity update. When the geometrically consistent state is invalid, the previous valid historical target gradient similarity is maintained, and the invalid state is not used to replace the historical state.

[0125] When the change in the joint loss value within consecutive validation periods is less than the convergence tolerance, and the geometric consistency state remains unchanged within the consecutive validation periods, the network parameter update and model output module establishes a training termination condition. If the training termination condition is not met, the updated historical target gradient similarity is returned to the gradient coordination parameter generation module as input for generating the target gradient similarity in the next training batch. When the task gradient is zero, contains non-numerical values, or gradient anomalies occur in multiple consecutive training batches, the network parameter update and model output module skips the corresponding parameter update; in one implementation, after consecutive anomalies reach a preset number, the most recently valid network parameters are restored, and gradient extraction is re-executed from subsequent training batches.

[0126] After the training termination condition is met, the network parameter update and model output module solidifies the network parameters of the shared feature extraction network, the directional target detection head, and the keypoint localization head, and saves the keypoint number configuration and directional bounding box representation configuration corresponding to the network parameters, forming a multi-task visual detection model. During the inference phase, the multi-task visual detection model receives the image to be detected and outputs the directional target detection results and keypoint localization results; the task gradient extraction, gradient similarity comparison, task gradient adjustment, and historical target gradient similarity update during the training phase do not proceed to the inference phase.

Claims

1. A multi-task visual detection gradient optimization method, characterized in that, include: S100. Obtain training samples with oriented bounding box annotations and key point annotations, input them into the multi-task visual detection network, generate oriented target detection results and key point localization results, associate them according to the target instance identifier, and obtain the instance association prediction set; S200. Generate a task loss set based on the instance association prediction set; Generate geometrically consistent states and effective task markers based on the major axis direction of the oriented bounding box of the same target instance, the structural direction of key points, the state and visibility of key point locations; S300. Divide the shared parameter set and the task-specific parameter set according to the network layer affiliation, obtain the task gradient of the effective task to the shared parameter set and the task-specific parameter gradient of each task to the corresponding task-specific parameter set, calculate the cosine similarity of the task gradient, and weight the similarity target value corresponding to the geometric consistency state with the historical target gradient similarity to obtain the target gradient similarity. S400. When the cosine similarity is lower than the target gradient similarity, remove the back projection component of the gradient to be adjusted relative to the reference gradient, and add a compensation component along the direction of the reference gradient; aggregate the adjusted or unadjusted task gradients into a joint gradient, update the shared parameter set with the joint gradient, and update the task-specific parameters with each dedicated gradient; update the geometric consistency state and historical target gradient similarity based on the updated network, and output the multi-task visual detection model when the training termination condition is met.

2. The method according to claim 1, characterized in that, The training samples include image labels, target instance labels, target category annotations, directional bounding box annotations, and multiple key point annotations; The directional bounding box annotation includes the target center coordinates, target width, target height, and target rotation angle; Each keypoint annotation includes a keypoint number, keypoint coordinates, and keypoint visibility marker.

3. The method according to claim 2, characterized in that, The association based on the target instance identifier includes: Match the predicted directional bounding box in the directional target detection result with the directional bounding box annotation; Match the key point coordinates in the key point localization results with the key point annotations; Based on the target instance identifier, the directional target detection result and the key point localization result that match the same directional bounding box annotation are combined into a target instance record; Multiple target instance records are combined into the instance association prediction set.

4. The method according to claim 2, characterized in that, The generation of the geometrically consistent state includes: The major axis direction of the directional bounding box is generated based on the long side of the predicted directional bounding box in the directional target detection result; The key point structure direction is generated based on the key point coordinates of two key points with a preset structural relationship; Calculate the directional deviation between the major axis of the oriented bounding box and the structural direction of the key point; Obtain the positional state of the keypoints relative to the predicted orientation bounding box; The directional deviation, the position inclusion state, and the key point visibility marker are combined according to preset state rules to generate a first consistent state, a second consistent state, a conflict state, or an invalid state, which serve as the geometric consistency state.

5. The method according to claim 4, characterized in that, The generation of the task valid marker includes: When the directional bounding box annotation exists, the directional target detection result matches the directional bounding box annotation, and the number of visible key points used to generate the key point structure direction reaches a preset number, a valid task valid label is generated for the corresponding target instance; When the directional bounding box annotation is missing, the directional target detection result does not match the directional bounding box annotation, or the number of visible key points is lower than the preset number, an invalid task valid label is generated for the corresponding target instance. A task loss mask is generated based on the target instances with valid task labels. The task gradients that participate in the shared parameter update are obtained based on the task loss filtered by the task loss mask. Tasks without valid target instances do not participate in the similarity calculation and aggregation of the task gradients.

6. The method according to claim 1, characterized in that, The task loss set includes the task loss for directional target detection and the task loss for key point localization. The loss for the directional target detection task includes target classification loss and directional bounding box regression loss; The loss in the key point localization task includes key point coordinate loss and key point visibility loss; Adaptive task weight coefficients are generated based on the quantization scalar of the task uncertainty corresponding to each task. The adaptive task weight coefficients are then weighted with the corresponding task loss to generate a joint loss value.

7. The method according to claim 6, characterized in that, The multi-task visual detection network includes a shared feature extraction network, a directional target detection head, and a key point localization head; The shared parameter set includes network parameters in the shared feature extraction network that simultaneously provide features to the directional target detection head and the key point localization head; The task-specific parameter set includes the network parameters of the directional target detection head and the network parameters of the key point localization head; Establish parameter attribution relationships according to network layer identifiers, perform backpropagation on the loss of the directional target detection task and the loss of the key point localization task respectively, and obtain the task gradient of each task with respect to the shared parameter set and the task-specific parameter gradient with respect to the corresponding task-specific parameter set.

8. The method according to claim 4, characterized in that, The tasks include directional target detection and key point localization. The task gradient corresponding to the directional target detection task and the task gradient corresponding to the key point localization task are both task gradients generated by the corresponding tasks for the shared parameter set. The cosine similarity is the cosine similarity between the task gradient corresponding to the directional target detection task and the task gradient corresponding to the key point localization task. The gradient similarity target value is selected based on the geometric consistency state, wherein the first consistency state, the second consistency state, and the conflict state correspond to the first target value, the second target value, and the third target value that increase sequentially, respectively, and the invalid state retains the gradient similarity of the previous valid historical target; the gradient similarity target value is weighted with the historical target gradient similarity to generate the target gradient similarity; When the cosine similarity is lower than the target gradient similarity, one task gradient is taken as the task gradient to be adjusted and the other task gradient is taken as the reference task gradient. The back projection component of the task gradient to be adjusted in the direction of the reference task gradient is removed. When the cosine similarity after removal is still lower than the target gradient similarity, a compensation component is added along the direction of the reference task gradient until the target gradient similarity or a preset compensation upper limit is reached, and a coordinated task gradient is generated. When the cosine similarity is not lower than the target gradient similarity, the original task gradient is retained; the coordinated task gradient is aggregated with the reference task gradient, or the retained original task gradient is aggregated to generate the joint gradient.

9. The method according to claim 8, characterized in that, Based on the updated multi-task visual detection network, the directional target detection results and key point localization results for the next training cycle are generated, and the geometric consistency state is regenerated based on the directional target detection results and key point localization results for the next training cycle. The historical target gradient similarity is updated based on the gradient similarity target value corresponding to the regenerated geometric consistency state and the target gradient similarity of the current training cycle; when the regenerated geometric consistency state is an invalid state, the previous valid historical target gradient similarity is maintained. When the task gradient is zero or contains non-numerical values, an abnormal training state marker is generated, and the corresponding parameter update is skipped; the updated verification loss value of the multi-task visual detection network is obtained within a continuous verification cycle; when the change in the verification loss value is lower than the convergence tolerance, the geometric consistency state remains stable, and the proportion of target instances corresponding to conflict states is lower than a preset proportion, the training termination condition is generated; the geometric consistency state remains stable means that the change in the proportion of target instances corresponding to each state is lower than a preset proportion, and the change in the average directional deviation of effective target instances is lower than a preset angle; when the maximum number of training rounds is reached, the training termination condition is also generated.

10. A multi-task visual inspection gradient optimization system, applied to the method of any one of claims 1 to 9, characterized in that, include: The multi-task prediction and association module is used to acquire training samples with oriented bounding box annotations and key point annotations, input them into the multi-task visual detection network to generate oriented target detection results and key point localization results, and associate them according to the target instance identifier to obtain the instance association prediction set; The task training state generation module is connected to the multi-task prediction and association module. It is used to generate a task loss set based on the instance association prediction set, and generate geometric consistency state and task valid label according to the long axis direction of the oriented bounding box of the same target instance, the structural direction of the key point, the key point position inclusion state and visibility. The gradient coordination parameter generation module, connected to the task training state generation module, is used to divide the shared parameter set and the task-specific parameter set according to the network layer affiliation, obtain the task gradient of the effective task to the shared parameter set and the task-specific parameter gradient of each task, calculate the cosine similarity between task gradients, and weight the gradient similarity target value corresponding to the geometric consistency state with the historical target gradient similarity to generate the target gradient similarity. The network parameter update and model output module, connected to the gradient coordination parameter generation module, is used to remove the back projection component of the gradient of the task to be adjusted and add a compensation component along the direction of the reference task gradient when the cosine similarity is lower than the target gradient similarity. It aggregates the adjusted or unadjusted task gradients to generate a joint gradient, updates the shared parameter set with the joint gradient, and updates the corresponding task-specific parameter set with the gradients of each task-specific parameter. Based on the updated network, it updates the geometric consistency state and historical target gradient similarity, and outputs a multi-task visual detection model when the training termination condition is met.

Images