Perception task model optimization method, system and equipment and storage medium

Abstract

The invention discloses a perception task model optimization method, system and device and a storage medium, and belongs to the technical field of intelligent driving, and the perception task model optimization method comprises the steps: obtaining target data; wherein the target data is used for representing environment perception original data; determining a prediction category result and a target feature according to the target data and a pre-constructed reference model; and determining a target perception task model according to the prediction category result, the target feature, the target optimization strategy and a pre-constructed reinforcement learning agent. According to the perception task model optimization method provided by the invention, the reference model and the reinforcement learning agent are constructed, and the target optimization strategy and the reinforcement learning agent are utilized to dynamically optimize and adjust the reference model to obtain the optimal perception task model, so that the problem that the existing perception task model cannot realize dynamic parameter adjustment is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, in particular to a perception task model optimization method, system, device and storage medium. BACKGROUND

[0002] The perception task is a core link of an automatic driving system, which relies on a deep learning model to understand the environment around the vehicle body and detect the category and attribute of the target obstacle, so as to realize the function of avoiding obstacles during driving. However, the deep learning model optimization method mainly adopts a static optimization strategy, such as adjusting the hyperparameters statically, using a class-balanced weighted loss function, and using a data enhancement strategy, so as to be unable to adjust the parameters in real time according to the training state. SUMMARY

[0003] The present application provides a perception task model optimization method, system, device and storage medium, aiming at solving the problem that the existing model cannot dynamically adjust the parameters.

[0004] In a first aspect, a perception task model optimization method is provided, comprising: obtaining target data; wherein the target data is used to represent the environment perception original data; determining a predicted category result and a target feature according to the target data and a pre-constructed benchmark model; determining a target perception task model according to the predicted category result, the target feature, a target optimization strategy, and a pre-constructed reinforcement learning agent.

[0005] In some embodiments, the reinforcement learning agent comprises a state encoder, a target policy network, and an action executor. determining a target perception task model according to the predicted category result, the target feature, a target optimization strategy, and a pre-constructed reinforcement learning agent, comprising: determining a target state vector according to the state encoder, the predicted category result, and the target feature; determining the target perception task model according to the target state vector, the target policy network, the action executor, and a preset iteration update condition.

[0006] In some embodiments, determining a target state vector according to the state encoder, the predicted category result, and the target feature comprises: determining a plurality of initial state vectors according to the state encoder, the predicted category result, and the target feature; and splicing the plurality of initial state vectors to determine the target state vector. The plurality of initial state vectors comprise a target feature statistical quantity mean value, a target feature statistical quantity variance, an intersection over union of each category, an average intersection over union of the intersection over union of each category, and a training progress.

[0007] In some embodiments, the preset iterative update condition comprises a preset optimization condition and a preset iteration condition; the target perception task model is determined according to the target state vector, the target policy network, the action executor and the preset iterative update condition, comprising: The target action vector and the target value estimate are determined according to the target state vector and the target policy network; The basic parameters of the baseline model are adjusted according to the target action vector and the action executor; The target reward is determined according to the predicted category results before and after the parameter adjustment; The target policy network is updated according to the target state vector, the target action vector, the target reward, the target value estimate and the preset optimization condition; Return to the step of determining the target state vector according to the state encoder, the predicted category result and the target feature, until the preset iteration condition is met, the target perception task model is determined.

[0008] In some embodiments, the baseline model comprises a backbone network and a prediction head; the target action vector comprises a first action, a second action, a third action and a fourth action; The first action is used to adjust the learning rate of the backbone network by the first learning rate scaling factor; the second action is used to adjust the learning rate of the prediction head by the second learning rate scaling factor; the third action is used to adjust the feature map intensity by the global feature enhancement scaling factor; the fourth action is used to adjust the category weight according to the category balance intensity coefficient; The basic parameters of the baseline model are adjusted according to the target action vector and the action executor, comprising: The learning rate of the backbone network and the learning rate of the prediction head are respectively set according to the first learning rate scaling factor, the second learning rate scaling factor and the preset basic learning rate; The target feature is multiplied by the global feature enhancement scaling factor to adjust the feature activation intensity; The weights of all categories are updated according to the category balance intensity coefficient and the preset mapping function.

[0009] In some embodiments, the target reward is determined according to the predicted category results before and after the parameter adjustment, comprising: The basic performance reward, the feature stability reward, the category balance reward and the policy entropy reward are determined; The basic performance reward, the feature stability reward, the category balance reward and the policy entropy reward are weighted to obtain the target reward.

[0010] In some embodiments, the target policy network is updated according to the target state vector, the target action vector, the target reward, the target value estimate and the preset optimization condition, comprising: collect the target state vector, the target action vector, the target reward, and the target value estimate, and store them in the cache area; update the target policy network when the storage capacity of the cache area meets a preset experience quantity.

[0011] In a second aspect, a perception task model optimization system is also provided, comprising: An acquisition module is configured to acquire target data, wherein the target data is used to represent environment perception raw data. A first determination module is configured to determine a predicted category result and a target feature according to the target data and a pre-constructed benchmark model. A second determination module is configured to determine a target perception task model according to the predicted category result, the target feature, a target optimization strategy, and a pre-constructed reinforcement learning agent.

[0012] In a third aspect, an electronic device is also provided, comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the method of the first aspect.

[0013] In a fourth aspect, a computer readable storage medium is also provided, which stores a computer program, and the computer program is loaded by a processor to execute the steps in the method of the first aspect.

[0014] Advantages: The perception task model optimization method, system, device, and storage medium provided in the embodiments of the present application comprise: acquiring target data, wherein the target data is used to represent environment perception raw data; determining a predicted category result and a target feature according to the target data and a pre-constructed benchmark model; and determining a target perception task model according to the predicted category result, the target feature, a target optimization strategy, and a pre-constructed reinforcement learning agent. The perception task model optimization method provided in the present application solves the problem that the existing perception task model cannot be dynamically adjusted by constructing a benchmark model and a reinforcement learning agent, and dynamically optimizing and adjusting the benchmark model by using a target optimization strategy and a reinforcement learning agent to obtain an optimal perception task model. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 is a flowchart of a perception task model optimization method provided in the embodiments of the present application; Figure 2is a sensing task model optimization principle schematic diagram provided in an embodiment of the present application; Figure 3 is a principle structure block diagram of a sensing task model optimization system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person skilled in the art without creative work fall within the scope of protection of the present application.

[0018] In the description of the present application, it should be understood that the orientations or positional relationships indicated by the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features limited by "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0019] "A and / or B" includes the following three combinations: only A, only B, and a combination of A and B.

[0020] The use of "adapted for" or "configured for" in the present application means open and inclusive language that does not exclude devices adapted for or configured for performing additional tasks or steps. In addition, the use of "based on" means open and inclusive, because the process, step, calculation or other action "based on" one or more stated conditions or values can be based on additional conditions or values beyond those stated in practice.

[0021] In this application, the word "exemplary" is used to mean "serving as an example, instance, or illustration." Any embodiment described as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. The following description is presented to enable any person skilled in the art to make and use the application. In the following description, for purposes of explanation, specific details are set forth. It is apparent to one skilled in the art that the present application can be practiced without the specific details presented. In other instances, well-known structures and processes are not shown in detail to avoid obscuring the application. Thus, the present application is not intended to be limited by the embodiments shown, but is to be accorded with the full scope consistent with the principles and features presented herein.

[0022] It is found by the applicant that the perception task is a crucial part of the automatic driving system, which needs to rely on a deep learning model to understand the environment around the vehicle body, detect the category and attribute of the target obstacle, and achieve the functions of automatic driving system such as driving and obstacle avoidance. However, the perception task model faces the following challenges: Firstly, the class imbalance problem: the rare class (such as traffic cone, special vehicle) in the road scene has few samples, which leads to low recognition accuracy of the model.

[0023] Secondly, low training efficiency: traditional fixed parameter training needs to be adjusted repeatedly by trial and error, which is low in efficiency and consumes a lot of computing resources.

[0024] Thirdly, model hyperparameter sensitivity problem: the performance of the model is highly sensitive to the setting of hyperparameters (such as learning rate, loss weight, etc.). The traditional grid search to adjust the learning rate and class weight is low in efficiency and cannot adapt to the dynamic training process.

[0025] Fourthly, global optimization bias: the traditional model optimization method uses gradient descent method to treat all samples and classes equally, ignoring the special needs of difficult example samples and weak classes.

[0026] Fifthly, easy to fall into local optimum: fixed optimization strategy cannot be adjusted adaptively according to the real-time state of the model, which leads to slow convergence speed and easy to fall into local optimum, and it is difficult to further improve the performance.

[0027] Sixthly, long tail problem: for the long tail distribution problem (i.e. a large number of samples belong to common classes, and rare class samples are extremely few), the model is difficult to learn and generalize these rare scenes, which leads to poor recognition effect of rare classes in semantic segmentation, and static loss function is difficult to balance the learning progress of each class.

[0028] It can be seen that the deep learning model optimization method mainly adopts static optimization strategies such as static hyperparameter adjustment, class balanced weighted loss function, and data enhancement strategy, which cannot adjust the parameters in real time according to the training state.

[0029] Therefore, the embodiment of the present application provides a perception task model optimization method, system, device and storage medium. The method comprises the following steps: constructing a benchmark model and a reinforcement learning agent, and dynamically optimizing and adjusting the benchmark model by using a target optimization strategy and the reinforcement learning agent to obtain an optimal perception task model, so as to solve the problem that the perception task model cannot be dynamically adjusted.

[0030] It should be noted that the perception task model described in the present application includes a perception task model in the field of automatic driving. The perception task model can be used to implement target detection, segmentation task and static lane line detection, and the specific implementation can be set according to the actual situation, which is not limited here.

[0031] Figure 1 is a flowchart of a perception task model optimization method provided in the embodiment of the present application. The method can be applied to the process of dynamically adjusting the parameters of the perception task model in the vehicle control system. The method can be executed by a perception task model optimization system, which can be implemented by software and / or hardware, and can be configured in the processor or controller of the vehicle control system. Please refer to Figure 1 , which comprises the following steps: Step 110, obtaining target data.

[0032] The target data is used to represent the environment perception raw data. The raw data includes image data and / or point cloud data. The image data and / or point cloud data are specific environment perception raw data. For example, taking target detection (for example, the lane line obstacle is the target) as an example, the image data can be the actual image of the lane line captured by the camera, and the point cloud data can be the actual point cloud data of the lane line detected by the vehicle laser radar. For example, taking the segmentation task as an example, the image data can be the actual image of the environment around the vehicle captured by the camera, and the point cloud data can be the actual point cloud data of the environment around the vehicle detected by the vehicle laser radar.

[0033] Step 120, determining the predicted category result and the target feature according to the target data and the pre-constructed benchmark model.

[0034] The benchmark model is a complete deep learning network model, mainly including data preprocessing, backbone network, prediction head, loss function and other standard components. First, standard supervised pre-training is performed on the labeled data set.

[0035] The data preprocessing includes normalization, standardization, data enhancement and uniform size processing of the target data.

[0036] The specific implementation process of normalization and standardization is that: the data input is image data and / or point cloud data. For example, in image semantic segmentation, the input image + corresponding label mask is preprocessed. The pixel value of the image is normalized to [0, 1] or standardized using the mean and standard deviation. The purpose of normalizing or standardizing the target data is to stabilize the input data distribution, accelerate model convergence, and avoid gradient explosion.

[0037] The specific implementation process of data augmentation is that: the normalized or standardized image and mask are subjected to the same data augmentation. Data augmentation includes random rotation and flipping. Data augmentation can increase the diversity of data and improve the generalization ability and robustness of the model.

[0038] The specific implementation process of uniform size is that: the image and the true value mask are unified to a fixed size.

[0039] Finally, the preprocessed image batch is processed as the input of the backbone network, and the mask is used to calculate the loss with the predicted result (i.e., the predicted class result).

[0040] The backbone network is used for feature extraction. The backbone network can adopt a U-Net convolutional network (UNet). UNet is an encoder-decoder structure. The encoder is usually a pre-trained convolutional neural network (CNN), such as a residual network (ResNet). The encoder is used for feature extraction to obtain target features. The decoder is a transposed convolution for upsampling operation. The purpose is to map the semantic features extracted by the encoder back to the original image size. These features (i.e., target features) are transmitted to the prediction head, which outputs pixel-level classification results (i.e., predicted class results).

[0041] The prediction head is used for task-specific output. The structure of the prediction head is one or more convolutional layers. The output channel of the last convolutional layer is the number of classes, followed by a softmax function, which is applied to the channel dimension to convert the output value of each pixel in all classes to a probability. The form is [B*num_cls*H*W], where B represents the batch size, num_cls represents the number of classes, H represents the height of the output feature map, and W represents the width of the output feature map. The output is sent to the loss function together with the true value mask for calculation.

[0042] Wherein, the loss function is cross-entropy loss (CELoss), the true value mask and the prediction output (i.e. the prediction category result) are calculated loss, and then the gradient is updated by back propagation.

[0043] Step 130, determining the target perception task model according to the prediction category result, the target feature, the target optimization strategy and the pre-built reinforcement learning agent.

[0044] Wherein, the target feature includes a graph feature. Wherein, the graph feature is a graph feature output by the backbone network, for example, a feature obtained after a backbone network decoder, and of course other layers of output (such as the output of the encoder) can also be selected.

[0045] Figure 2 Figure 1 is a schematic diagram of a perception task model optimization principle provided in an embodiment of the present application. As an example, the structure of the fine-tuning module composed of the benchmark model and the reinforcement learning agent is as shown in Figure 2 Wherein, the reinforcement learning agent includes a state encoder, a RL policy network (i.e. a target policy network), an action executor and policy optimization.

[0046] In some embodiments, the reinforcement learning agent includes a state encoder, a target policy network and an action executor; determining the target perception task model according to the prediction category result, the target feature, the target optimization strategy and the pre-built reinforcement learning agent, specifically including the following steps: Step one, determining the target state vector according to the state encoder, the prediction category result and the target feature.

[0047] Specifically, the target feature and the prediction category result are obtained from the benchmark model, and the target state vector is obtained from the state encoder according to the prediction category result and the target feature, so that the current state of the benchmark model can be obtained, which is conducive to subsequent adjustment and optimization of the parameters of the benchmark model, and further conducive to obtaining the optimized perception task model (i.e. the target perception task model).

[0048] In some embodiments, determining the target state vector according to the state encoder, the prediction category result and the target feature includes: determining a plurality of initial state vectors according to the state encoder, the prediction category result and the target feature; and splicing the plurality of initial state vectors to determine the target state vector.

[0049] Wherein, the plurality of initial state vectors include target feature statistical quantity mean, target feature statistical quantity variance, each category's intersection over union, the average intersection over union of each category's intersection over union and training progress. It should be noted that the initial state vector can also include other evaluation index vectors, which can be set according to actual conditions, and are not limited here.

[0050] Wherein, the initial state vector is an initial state vector of the baseline model.

[0051] Wherein, the specific calculation process of the target feature statistics mean and the target feature statistics variance is as follows: first, set the feature map (i.e. the target feature) output by the backbone network. Then, fix the channel dimension of the feature map to C through convolution. Second, calculate the mean [B, C] and variance [B, C] of the feature map, and calculate a mean and a variance for each channel, so the mean and the variance are both C-dimensional vectors, and then flatten, so the state dimension of this part is 2*C.

[0052] Wherein, the specific calculation process of the intersection over union of each category and the average intersection over union of the intersection over union of each category is as follows: obtain the prediction category result of the output of the baseline model, then count the distribution of each category, and calculate the intersection over union (IoU) of each category and the mean intersection over union (mIoU) of all categories, i.e. the average intersection over union, through the prediction category result and the true value. Assuming that the number of categories is N, the state dimension of this part is N+1 (i.e. the IoU of N categories and the mIoU of 1). The trend of mIoU is the slope [B, 1] of the mIoU of the last 3 iterations. The variance of the IoU of each category is [B, 1], which is used to measure the difference between categories. The category performance Gini coefficient [B, 1] can be directly used for reward calculation.

[0053] Wherein, the training progress includes the training iteration progress, the normalized learning rate and the normalized gradient clipping. Wherein, the training progress is set as the training iteration progress epoch (dimension [B, 1]), the normalized learning rate (dimension [B, 1]) and the normalized gradient clipping (dimension [B, 1]) these three parameters, and the state dimension of this part is 3.

[0054] Specifically, the target feature and the prediction category result are obtained from the baseline model, and the initial state vector of the baseline model is calculated by the state encoder according to the prediction category result and the target feature. Then, the initial state vectors are spliced to obtain the final state vector (i.e. the target state vector). For example, the target feature statistics mean, the target feature statistics variance, the intersection over union of each category, the average intersection over union of the intersection over union of each category and the training progress are spliced (cat) to obtain the final state vector [B, Cs]. Wherein, B is the batch size, and Cs is the state vector dimension, i.e. the sum of the dimensions of the feature map mean and variance, the category IoU and mIoU and the training progress, i.e. its dimension is: 2*C+N+7.

[0055] Step two, determining the target perception task model according to the target state vector, the target policy network, the action executor and the preset iteration update condition.

[0056] The target policy network is constructed by using a preset proximal policy optimization (PPO) algorithm.

[0057] In some embodiments, the preset iterative update condition includes a preset optimization condition and a preset iteration condition; the target perception task model is determined according to the target state vector, the target policy network, the action executor and the preset iterative update condition, including: the target action vector and the target value estimate are determined according to the target state vector and the target policy network; the basic parameters of the baseline model are adjusted according to the target action vector and the action executor; the target reward is determined according to the predicted category result before and after the parameter adjustment; the target policy network is updated according to the target state vector, the target action vector, the target reward, the target value estimate and the preset optimization condition; the step of determining the target state vector according to the state encoder, the predicted category result and the target feature is returned until the target perception task model is determined when the preset iteration condition is met.

[0058] In the technical scheme of the embodiments of the present application, the target action vector ([B, 4]) includes 4 dimensions, and each dimension is a numerical value (scalar). The 4 dimensions are the first action, the second action, the third action and the fourth action.

[0059] The target value estimate refers to that each state vector in the target state vector will calculate a corresponding value to evaluate whether the state vector is conducive to model optimization or counter-optimization of the model, which is the value estimate.

[0060] In some embodiments, the baseline model includes a backbone network and a prediction head; the target action vector includes a first action, a second action, a third action and a fourth action; wherein the first action is used to adjust the learning rate of the backbone network by a first learning rate scaling factor; the second action is used to adjust the learning rate of the prediction head by a second learning rate scaling factor; the third action is used to adjust the feature map intensity by a global feature enhancement scaling factor; the fourth action is used to adjust the category weight according to a category balance intensity coefficient; the basic parameters of the baseline model are adjusted according to the target action vector and the action executor, including: the learning rate of the backbone network and the learning rate of the prediction head are respectively set according to the first learning rate scaling factor, the second learning rate scaling factor and the preset basic learning rate; the target feature is multiplied by the global feature enhancement scaling factor to adjust the feature activation intensity; the weights of all categories are updated according to the category balance intensity coefficient and a preset mapping function.

[0061] The first action is a learning rate scaling factor (scale1) of the backbone network, which is mapped using a sigmoid function and then scaled to the range [0.1, 3.0]; the second action is a learning rate scaling factor (scale2,) of the prediction head, which is mapped using a sigmoid function and then scaled to the range [0.1, 5.0]. The effects of the first action and the second action are to differentially adjust the parameter update speed of the feature extraction layer and the classifier of the baseline model.

[0062] The third action is a global feature enhancement scaling factor, which equally scales all feature channels to adjust the overall activation intensity of the feature map. The use of a sigmoid function mapping and then scaling to the range [0.5, 2.0] indicates that the overall activation of the feature map can be reduced by half or enhanced by one time.

[0063] The fourth action is a class balance intensity coefficient, which is mapped using a sigmoid function and then scaled to the range [0, 2]. According to the current performance of each class IoU, a weight is generated for each class, and the weight of the class with poorer performance is compensated more. The effect is to increase the compensation intensity of the class weight of all weak classes while avoiding excessive compensation.

[0064] It should be noted that the target action vector has action data and the specific setting of the action, which can be adjusted according to different task requirements, and is not specifically limited here.

[0065] Specifically, the specific implementation process of adjusting the basic parameters of the baseline model according to the target action vector and the action executor is: adjusting the corresponding parameters according to the different action settings described above. First, the first action and the second action multiply the preset learning rate value by the corresponding scaling coefficient to obtain the learning rate of the hierarchical network. Then, the third action multiplies the feature map by the corresponding scaling factor. Second, the class weight compensation coefficient of the fourth action is mapped to [1, 3] through a mapping function, then the weights of all classes are updated, and the updated weights of all classes are applied in calculating the loss and the backbone network of the baseline model.

[0066] In some embodiments, the target reward is determined according to the predicted class results before and after parameter adjustment, including: determining a basic performance reward, a feature stability reward, a class balance reward, and a strategy entropy reward; and weighting the basic performance reward, the feature stability reward, the class balance reward, and the strategy entropy reward to obtain the target reward.

[0067] Specifically, the target reward is obtained by weighting the basic performance reward, the feature stability reward, the class balance reward, and the strategy entropy reward. These reward weighting designs ensure that the RL agent realizes adaptive parameter adjustment in the long-tail scene and dynamically optimizes to maximize the highest indicators such as mIoU.

[0068] where the base performance reward is the reward of the base mIoU improvement, i.e. The scaling factor is multiplied by the relative value of mIoU improvement, and the specific calculation formula is: ; wherein, is a scaling factor, used to encourage continuous performance improvement. represents the base performance reward, represents the current mIoU, represents the previous mIoU.

[0069] wherein the feature stability reward is to encourage feature distribution consistency and punish the dramatic fluctuation of feature statistics. The specific calculation formula is: ; wherein, represents the feature stability reward; represents the scaling factor (or weight coefficient), ; Σ represents the feature covariance matrix, which punishes the dramatic fluctuation of feature distribution; represents the current feature covariance matrix; represents the previous feature covariance matrix, which is used to punish the dramatic fluctuation of feature distribution; represents the Frobenius norm.

[0070] wherein the class balance reward is to use the Gini coefficient to measure the performance balance of the IoU of all classes, and the smaller the value is, the more balanced it is. When the Gini coefficient decreases (i.e. more balanced), a positive reward is given, and vice versa.

[0071] wherein the calculation of the Gini coefficient: for a set of IoU values, first sort them in ascending order, and then calculate the Gini coefficient. The specific calculation formula is: G = (2 * sum(i * sorted_iou_i) / (n * sum(sorted_iou))) - (n+1) / n; R_balance = γ ×(previous_gini - current_gini); wherein R_balance represents the class balance reward; γ represents the scaling factor, for example, γ = 5; G represents the Gini coefficient; sorted_iou represents the sorted IoU value; i represents the index (from 1 to n); n represents the number of classes; previous_gini represents the previous Gini coefficient; current_gini represents the current Gini coefficient.

[0072] The policy entropy reward is: to encourage policy exploration and prevent premature convergence. The specific calculation formula is: R_entropy = δ × H(π(·|s)); Wherein, R_entropy represents the policy entropy reward; H represents the policy entropy, which is used to encourage moderate exploration; δ represents the scaling factor of the policy entropy reward, δ = 0.3; s represents the current state.

[0073] Finally, the final comprehensive immediate reward (i.e. target reward) is obtained by weighted sum of these rewards. The calculation formula is: R_total = w1×R_perf + w2×R_stable + w3×R_balance + w4× R_entropy; Wherein, R_total represents the target reward, and the basic weight w1:w2:w3:w4 = 1:1:1.5:0.5.

[0074] In some embodiments, the target policy network is updated according to the target state vector, the target action vector, the target reward, the target value estimate and the preset optimization condition, including: collecting the target state vector, the target action vector, the target reward, the target value estimate, and storing them in the cache area; in the case that the storage amount of the cache area meets the preset experience quantity, the target policy network is updated.

[0075] Wherein, the preset optimization condition is: updating the target policy network in the case that the storage amount of the cache area meets the preset experience quantity.

[0076] Specifically, the target state vector, the target action vector, the target reward, the target value estimate and other parameters obtained in the above process are collected and stored in the experience replay buffer (i.e. cache area), and the target state vector, the target action vector, the target reward, the target value estimate and other parameters are cached. Wherein, the cache area capacity is usually 10,00 experience, and after the cache area stores enough sample experience, the PPO policy network is updated (i.e. updating the target policy network).

[0077] Wherein, the specific implementation process of updating the target policy network is: first, sampling the cache area experience. Then, calculate the generalized advantage estimation (GAE) advantage. Finally, introduce Kalman filter as GAE enhanced correction value estimate, reduce noise to support dynamic parameter adjustment and accelerate convergence, achieve better optimization effect.

[0078] Wherein, the steps of Kalman filter correction target value estimate include: State space model: the true value V_true(t) is taken as the state, and the observation value is the V_pred(t) output by the Critic network. Wherein, V_pred(t) represents the predicted value estimate of the Critic network at time step t.

[0079] State equation: V_true(t) = V_true(t-1) + w(t), w(t) ~ N(0, Q). Wherein, t represents the time step; w(t) represents the process noise; Q represents the process noise covariance matrix.

[0080] Observation equation: V_pred(t) = V_true(t) + v(t), v(t) ~ N(0, R) Through Kalman filtering iteration, the smoothed value sequence V_smooth is obtained. Wherein, v(t) represents the observation noise; R represents the observation noise covariance matrix.

[0081] Wherein, the calculation of GAE advantage step includes: Step one, improve the value sequence by Kalman filtering smoothing to obtain V_smooth.

[0082] Step two, for each time step t, calculate the adaptive discount factor γ_t.

[0083] Step three, calculate the temporal difference (TD) error. The specific calculation formula is: δ_t = r_t + γ_t * V_smooth(s_{t+1}) - V_smooth(s_t); Wherein, δ_t represents the temporal difference error; r_t represents the immediate reward, that is, the reward obtained at time step t; γ_t represents the discount factor; V_smooth(s_{t+1}) represents the smoothed value estimate of the next state; V_smooth(s_t) represents the smoothed value estimate of the current state.

[0084] Step four, calculate the GAE advantage, and the specific calculation formula is: A_t^{GAE} = Σ_{k=0}^{T-t-1} (γ_t λ)^k δ_{t+k}; Wherein, A_t^{GAE} represents the GAE advantage of time step t; λ represents the GAE smoothing coefficient; δ_{t+k} represents the temporal difference error at time step t+k; T represents the total time step number of the trajectory.

[0085] For example, the target state vector sequence is: s0, s1,..., sT. The target reward sequence is: r0, r1,..., rT 1. The target value estimates are: V(s0), V(s1),..., V(sT), which are output by the Critic network. The discount factor γ typically ranges from 0.9 to 0.99. The smoothing coefficient λ typically ranges from 0.8 to 0.95.

[0086] After updating the target policy network, it also includes evaluating the current policy, calculating the policy loss, value function loss, entropy loss, etc. The total loss of the policy network is obtained by weighted sum of the three types of loss. The specific calculation process is: I. Policy Loss: PPO uses the clipped surrogate objective to update the policy to prevent the update step from being too large. First, calculate the probability ratio of the new and old policies, the specific calculation formula is: r_t(θ) = π_θ(a_t | s_t) / π_θ_old(a_t | s_t); Where r_t(θ) represents the probability ratio of the new and old policies; π_θ represents the current policy network (parameters θ); π_θ_old represents the old policy network (policy before updating); a_t represents the action taken at time step t; s_t represents the state at time step t.

[0087] Then, calculate the two losses, the specific calculation formula is: L1 = r_t(θ) * A_t; L2 = clip(r_t(θ), 1-ε, 1+ε) * A_t; Where L1 represents the unclipped surrogate objective loss; L2 represents the clipped surrogate objective loss; A_t represents the advantage estimate at time step t; clip represents the clipping function, which limits r_t(θ) within [1-ε, 1+ε]; ε represents the clipping parameter.

[0088] Finally, the policy loss takes the smaller value of the two (because we want to maximize the expected return, but use negative loss so it is minimized), the specific calculation formula is: L_policy = - E[ min(L1, L2) ]; Where L_policy represents the policy loss; E represents the expected value, which is usually approximated by averaging over the batch samples.

[0089] II. Value Loss: Value loss is the difference between the current value function estimate and the target value, usually using mean squared error. The target value can be calculated in various ways, such as using n-step returns or GAE (Generalized Advantage Estimation). For example, let V_target be the target value, then the value loss is: L_value = (V(s_t) - V_target)^2; where L_value represents the value loss; V(s_t) represents the Critic network's value prediction at state s_t.

[0090] In PPO, the target value is usually calculated using GAE, which is: V_target = A_t + V(s_t); III. Entropy Loss: Entropy loss is used to encourage exploration and prevent the policy from converging to a local optimum too early. The entropy of the current policy is calculated as: H(π(·|s_t)) = - Σ π(a|s_t) * log(π(a|s_t)); where H(π(·|s_t)) represents the entropy of the current policy at state s_t; π(·|s_t) represents the policy distribution at state s_t; π(a|s_t) represents the probability of taking action a at state s_t.

[0091] The entropy loss is the negative entropy value (since we want to maximize entropy, but in the loss function we usually minimize, so we add a negative sign), which is calculated as: L_entropy = - H(π(·|s_t)); where L_entropy represents the entropy loss.

[0092] In the total loss, these three parts of the loss are weighted and summed as: L_total = L_policy + c1 * L_value + c2 * L_entropy; where c1 and c2 are hyperparameters, c1 represents the value function loss coefficient, usually c1=0.5; c2 represents the entropy coefficient, usually c2=0.01.

[0093] Then, use an optimizer (e.g., Adam) to update the parameters of the target policy network.

[0094] Specifically, the target policy network is updated according to the target state vector, the target action vector, the target reward, the target value estimation, and the preset optimization condition; and the step of determining the target state vector according to the state encoder, the predicted category result, and the target feature is returned until the preset iteration condition is met, and the specific implementation process of the target perception task model is determined as follows: repeating the cycle of “state encoding-RL policy network decision-action execution-reward calculation-experience storage-policy optimization (when the buffer is full)”.

[0095] For example, the preset iteration condition of the embodiment of the application includes setting the iteration number to 100 rounds, or other preset termination conditions, such as mIoU being greater than a preset threshold, or mIoU improving by less than 0.1% for ten consecutive iterations. After the iteration is completed, the final optimized model and the trained RL policy network model are obtained, that is, the target perception task model is obtained.

[0096] It can be understood that the perception task model optimization method provided by the embodiment of the application effectively solves the model learning difficulty problem caused by the long-tail distribution of data in the perception task by constructing a benchmark model and a reinforcement learning agent, using the continuous optimization capability and target optimization strategy of reinforcement learning, and adaptively and dynamically adjusting the model parameters, and simultaneously realizes adaptive and dynamic adjustment of the model parameters to obtain the best model, so as to obtain the optimal perception performance index in a complex driving scene. Moreover, the application introduces a reinforcement learning agent to construct a closed-loop model optimization framework. The agent can automatically decide and execute parameter adjustment actions (such as hierarchical learning rate, feature enhancement, and category weight) according to the real-time state of the model in the training process (such as feature distribution, category performance, and training progress), to dynamically respond to the change of data distribution, especially the learning difficulty of rare categories in the long-tail distribution, so as to realize adaptive model optimization and finally improve the overall performance and convergence effect of the model. Therefore, the application can solve the problem that the related art lacks the ability to adaptively adjust parameters according to the training state and real-time data distribution, especially the problem that it is difficult to effectively cope with the long-tail data distribution problem commonly existing in real scenes, resulting in poor recognition performance of rare categories and difficulty in improving the overall index of the model.

[0097] The application constructs an independent RL agent, and the core goal of the agent is to dynamically decide and adjust the training parameters (hyperparameters and structure parameters) of the model itself by optimizing the set reward signal (directly related to the model performance index, such as mIoU, category balance, and feature stability). The RL agent learns how to maximize the long-term cumulative reward (i.e., the final model performance) by adjusting these parameters, rather than directly calculating the gradient to update the model weights. This is a higher-level, performance feedback-based meta-optimization process.

[0098] In summary, the embodiments of the present application can achieve the following effects: First, automatic parameter adjustment: through reinforcement learning agent, key training parameters are automatically and dynamically adjusted, and the cost of manual parameter adjustment is significantly reduced.

[0099] Second, improve long-tail performance: through adaptive parameter adjustment and dynamic parameter adjustment, the learning weights of various categories are dynamically balanced, effectively solving the learning difficulty problem of long-tail distribution, improving the recognition rate of rare categories, and achieving the highest performance index.

[0100] Third, accelerate convergence: adaptive hierarchical learning rate strategy optimizes the training process, accelerates model convergence, and improves training efficiency.

[0101] Fourth, fine gradient control: hierarchical gradient control mechanism (such as different learning rates for backbone and prediction head, channel weight adjustment) optimizes the update process of network parameters.

[0102] Fifth, stable and efficient training: the experience replay mechanism is introduced to improve sample utilization efficiency and enhance the stability of the reinforcement learning training process.

[0103] Figure 3 is a principle structure block diagram of a perception task model optimization system provided in an embodiment of the present application. The present application also provides a perception task model optimization system, please refer to Figure 3 , the perception task model optimization system 100 includes: an acquisition module 101, configured to acquire target data; wherein the target data is used to represent environment perception original data; a first determination module 102, configured to determine a prediction category result and a target feature according to the target data and a pre-constructed benchmark model; a second determination module 103, configured to determine a target perception task model according to the prediction category result, the target feature, a target optimization strategy, and a pre-constructed reinforcement learning agent.

[0104] The technical scheme of the present application provides a perception task model optimization system, by constructing a benchmark model and a reinforcement learning agent, and dynamically optimizing and adjusting the benchmark model using a target optimization strategy and a reinforcement learning agent to obtain an optimal perception task model, solving the problem that the perception task model cannot realize dynamic parameter adjustment.

[0105] In some embodiments, the reinforcement learning agent includes a state encoder, a target policy network, and an action executor; the second determination module 103 is further configured to: determine a target state vector according to the state encoder, the prediction category result, and the target feature; and determine the target perception task model according to the target state vector, the target policy network, the action executor, and a preset iterative update condition.

[0106] In some embodiments, the second determining module 103 is further configured to determine a plurality of initial state vectors according to the state encoder, the predicted category result and the target feature; and splice the plurality of initial state vectors to determine a target state vector; wherein the plurality of initial state vectors include a target feature statistical quantity mean value, a target feature statistical quantity variance, an intersection over union of each category, an average intersection over union of the intersection over union of each category, and a training progress.

[0107] In some embodiments, the preset iterative update condition includes a preset optimization condition and a preset iteration condition; the second determining module 103 is further configured to determine a target action vector and a target value estimate according to the target state vector and the target policy network; adjust the basic parameters of the baseline model according to the target action vector and the action executor; determine a target reward according to the predicted category result before and after the parameter adjustment; update the target policy network according to the target state vector, the target action vector, the target reward, the target value estimate and the preset optimization condition; and return to the step of determining the target state vector according to the state encoder, the predicted category result and the target feature until the preset iteration condition is met, and then determine the target perception task model.

[0108] In some embodiments, the baseline model includes a backbone network and a prediction head; the target action vector includes a first action, a second action, a third action and a fourth action; wherein the first action is used to adjust the learning rate of the backbone network by a first learning rate scaling factor; the second action is used to adjust the learning rate of the prediction head by a second learning rate scaling factor; the third action is used to adjust the feature map intensity by a global feature enhancement scaling factor; and the fourth action is used to adjust the category weight according to a category balance intensity coefficient; the second determining module 103 is further configured to set the learning rate of the backbone network and the learning rate of the prediction head respectively according to the first learning rate scaling factor, the second learning rate scaling factor and a preset basic learning rate; multiply the target feature by the global feature enhancement scaling factor to adjust the feature activation intensity; and update the weight of all categories according to the category balance intensity coefficient and a preset mapping function.

[0109] In some embodiments, the second determining module 103 is further configured to determine a basic performance reward, a feature stability reward, a category balance reward and a strategy entropy reward; and weight the basic performance reward, the feature stability reward, the category balance reward and the strategy entropy reward to obtain the target reward.

[0110] In some embodiments, the second determining module 103 is further configured to collect the target state vector, the target action vector, the target reward, the target value estimate and store them in a cache area; and update the target policy network when the storage amount of the cache area meets a preset experience quantity.

[0111] The embodiment further provides an electronic device including a memory and a processor, the memory storing a computer program, and the computer program being executed by the processor to implement the method of any one of the above embodiments.

[0112] The embodiment further provides a computer readable storage medium storing a computer program, and the computer program being loaded by a processor to execute the arrangement in the method of any one of the above embodiments.

[0113] In the embodiment of the present application, the storage medium can be a magnetic disk, an optical disk, a read only memory (ROM), or a random access memory (RAM), etc.

[0114] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0115] The above describes in detail the method, system, device and storage medium provided by the embodiment of the present application for optimizing a perception task model. The principle and implementation mode of the present application are described by applying specific examples. The above embodiment is only used to help understand the method and core idea of the present application. Meanwhile, for those skilled in the art, the specific implementation mode and application range can be changed according to the idea of the present application. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for perception task model optimization, the method comprising: The method comprises the following steps: acquiring target data; wherein the target data is used to represent environmental perception raw data; determining a predicted category result and a target feature according to the target data and a pre-constructed reference model; determining a target perception task model according to the predicted category result, the target feature, a target optimization strategy, and a pre-constructed reinforcement learning intelligent agent, wherein the target perception task model is obtained by adaptively and dynamically adjusting model parameters to solve the learning difficulty problem caused by long-tail distribution of data, and the target perception task model is optimized.

2. The perception task model optimization method of claim 1, wherein, The reinforcement learning intelligent agent comprises a state encoder, a target policy network, and an action executor. The determination of the target perception task model according to the predicted category result, the target feature, the target optimization strategy, and the pre-constructed reinforcement learning intelligent agent comprises: determining a target state vector according to the state encoder, the predicted category result, and the target feature; determining the target perception task model according to the target state vector, the target policy network, the action executor, and a preset iteration update condition.

3. The perception task model optimization method of claim 2, wherein, The determination of the target state vector according to the state encoder, the predicted category result, and the target feature comprises: determining a plurality of initial state vectors according to the state encoder, the predicted category result, and the target feature; and splicing the plurality of initial state vectors to determine the target state vector; wherein the plurality of initial state vectors comprise a target feature statistical quantity mean value, a target feature statistical quantity variance, an intersection over union of each category, an average intersection over union of the intersection over union of each category, and a training progress.

4. The method of claim 2, wherein, The preset iteration update condition comprises a preset optimization condition and a preset iteration condition; and the determination of the target perception task model according to the target state vector, the target policy network, the action executor, and the preset iteration update condition comprises: determining a target action vector and a target value estimate according to the target state vector and the target policy network; adjusting basic parameters of the reference model according to the target action vector and the action executor; determining a target reward according to the predicted category result before and after parameter adjustment; updating the target policy network according to the target state vector, the target action vector, the target reward, the target value estimate, and the preset optimization condition; returning to the step of determining the target state vector according to the state encoder, the predicted category result, and the target feature until the target perception task model is determined when the preset iteration condition is met.

5. The method of claim 4, wherein, The reference model comprises a backbone network and a prediction head; and the target action vector comprises a first action, a second action, a third action, and a fourth action. The first action is used to adjust a learning rate of the backbone network by a first learning rate scaling factor; the second action is used to adjust a learning rate of the prediction head by a second learning rate scaling factor; the third action is used to adjust feature map intensity by a global feature enhancement scaling factor; and the fourth action is used to adjust category weight according to a category balance intensity coefficient. The adjustment of the basic parameters of the reference model according to the target action vector and the action executor comprises: According to the first learning rate scaling coefficient, the second learning rate scaling coefficient, and a preset base learning rate, learning rates of the backbone network and the prediction head are respectively set; The target feature is multiplied by the global feature enhancement scaling factor to adjust the feature activation intensity; According to the category balance intensity coefficient and a preset mapping function, weights of all categories are updated.

6. The method of claim 4, wherein, The target reward is determined according to the predicted category result before and after parameter adjustment, including: A base performance reward, a feature stability reward, a category balance reward, and a strategy entropy reward are determined. The base performance reward, the feature stability reward, the category balance reward, and the strategy entropy reward are weighted to obtain the target reward.

7. The perception task model optimization method of claim 4, wherein, The target policy network is updated according to the target state vector, the target action vector, the target reward, the target value estimate, and the preset optimization condition, including: The target state vector, the target action vector, the target reward, and the target value estimate are collected and stored in a cache area. When the storage amount of the cache area meets a preset experience quantity, the target policy network is updated.

8. A perception task model optimization system, comprising: It includes: An acquisition module is configured to acquire target data, wherein the target data is used to represent environment perception raw data; A first determination module is configured to determine a predicted category result and a target feature according to the target data and a pre-constructed reference model; A second determination module is configured to determine a target perception task model according to the predicted category result, the target feature, a target optimization strategy, and a pre-constructed reinforcement learning intelligent agent.

9. An electronic device, comprising: It includes a memory and a processor, the memory has a computer program stored thereon, and the computer program is executed by the processor to implement the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It has a computer program stored thereon, and the computer program is loaded by a processor to execute the steps in the method of any one of claims 1-7.

Images