An incremental learning method and system for autonomous driving environment perception

Abstract

This invention discloses an incremental learning method and system for environmental perception in autonomous driving, comprising: classifying targets in an environmental image dataset into known and unknown classes; dividing a training set and a test set to remove images of unknown classes; constructing a mixture density network using a Gaussian mixture model, and optimizing the mixture density network to minimize the loss function to obtain a target detection model; modeling each known class using the known class feature space of the target detection model and a preset Gaussian mixture model, and fitting the parameters of the Gaussian mixture model for each known class using the expectation-maximization algorithm; estimating the cognitive semantic uncertainty of all classes using the log-likelihood function and the Gaussian mixture model for each known class, cyclically and incrementally manually labeling images with high cognitive semantic uncertainty, and further training the target detection model using the labeled images to obtain the optimal target detection model. This invention can reduce the cost of manual labeling and training.

Description

Technical Field

[0001] This invention belongs to the field of image target detection technology, specifically relating to an incremental learning method and system for environmental perception in autonomous driving. Background Technology

[0002] Currently, Artificial Intelligence (AI), especially machine learning, has become an indispensable technology in fields such as computer vision, natural language processing, big data analysis, robotics, and software testing, and has made rapid progress in the past decade. This has had a significant impact on academia, industry, and even the lives of ordinary people. Intelligent unmanned systems are one typical application, and environmental perception plays a crucial role in these systems. Intelligent unmanned systems such as robots, self-driving cars, and drones need to constantly perceive their surroundings while in motion; accurate environmental perception and cognition are the foundation of safety.

[0003] Traditional object detection techniques and machine learning are designed for closed-set scenarios. However, in the environmental perception of intelligent unmanned systems, due to environmental changes or modeling deficiencies, these systems, such as autonomous driving systems, inevitably face scenarios that are inaccurately or unrecognizable—scenarios that fall under the category of open-set scenarios. For such scenarios, the intelligent brain should be able to distinguish between familiar and unfamiliar or even never-before-seen scenarios. Traditional object detection and machine learning techniques cannot solve this problem. The training process for existing environmental perception object detection largely relies on labeled training datasets, while unfamiliar and unknown scenarios are mostly unlabeled data. Labeling this data incurs enormous human and time costs. Even with new scenario data, while the intelligent brain can autonomously learn from existing technologies using only new scenario data, it often forgets old scenarios and knowledge, leading to catastrophic amnesia. Summary of the Invention

[0004] To address the above problems, this invention proposes an incremental learning method and system for environmental perception in autonomous driving. It primarily focuses on how intelligent systems can automatically detect unfamiliar scenarios and prompt users to label an appropriate number of samples, thereby initiating incremental learning to improve model performance. This is an uncertainty-driven incremental learning method. The technical solution adopted by this invention to solve the above technical problems is as follows:

[0005] An incremental learning method for environmental perception in autonomous driving includes the following steps:

[0006] S1, based on the object detection task, the environmental image dataset The target in the process is labeled with a class, and the labeled classes are divided into known classes and unknown classes, and the known classes and unknown classes belong to different classes;

[0007] S2, the segmented environmental image dataset Divided into training set and test set training set and test set The training set is obtained by removing images marked with unknown classes. and test set

[0008] S3 uses a Gaussian mixture model to construct a mixture density network whose output includes the mean, variance, and mixture weights. The output of the mixture density network is then used to compute the training set. The uncertainty of each image in the model is used to establish a first training input image set. The first training input image set is then input into a hybrid density network, and the hybrid density network is optimized with the goal of minimizing the loss function to obtain the target detection model.

[0009] S4. Use the target detection model obtained in step S3 to generate the feature space of the known classes. Model each known class according to the feature space of the known classes and the preset Gaussian mixture model. Use the expectation-maximization algorithm to fit the parameters of the Gaussian mixture model of each known class.

[0010] S5, training set Input the object detection model, estimate the cognitive semantic uncertainty of all classes using the log-likelihood function and the Gaussian mixture model of each known class, establish a set of images to be labeled based on the cognitive semantic uncertainty, manually label the images in the set of images to be labeled, and further train the object detection model using the manually labeled images to obtain the optimal object detection model.

[0011] Step S3 includes the following steps:

[0012] S3.1, construct a mixture density network using the parameters of the Gaussian mixture model as output, execute t=1, and set a precision difference threshold. And precision difference ratio threshold

[0013] S3.2, from the training set S1 images are randomly selected from the dataset to form the first training input image set;

[0014] S3.3, Input the images from the first training input image set into the hybrid density network and train it with the goal of minimizing the loss function to obtain the target detector;

[0015] S3.4, test set The image input target detector calculates the mean accuracy and determines whether it meets the requirements. Head If yes, proceed to step S3.5; otherwise, output the target detector for round t-1 as the target detection model and proceed to step S4, where MAP t MAP represents the mean accuracy of the target detector in round t. t-1 MAP represents the mean accuracy of the target detector in round t-1. t-2 This represents the mean accuracy of the target detector in round t-2;

[0016] S3.5, determine if S2 < S1; if so, use the training set. Update the first training input image set and proceed to step S3.3; otherwise, compute the training set using the parameters of the Gaussian mixture model. The arbitrary uncertainty and cognitive uncertainty of the remaining images excluding the first training input image set, where S2 represents the training set. The number of images remaining in the dataset excluding the first training input image set;

[0017] S3.6, use the Z score to normalize the arbitrary uncertainty and cognitive uncertainty calculated in step S3.5, and determine the uncertainty of each remaining image based on the normalized arbitrary uncertainty and cognitive uncertainty;

[0018] S3.7 Sort the uncertainties of each remaining image obtained in step S3.6 in descending order to obtain a set of remaining image uncertainties. Add the images corresponding to the first S1 uncertainties in the set of remaining image uncertainties to the first training input image set to obtain the updated first training input image set. Execute t = t + 1 and return to step S3.3.

[0019] In step S3.3, the formula for calculating the loss function is as follows:

[0020]

[0021] In the formula, L Total Let L represent the total loss, N represent the number of positive matches, and L represent the total loss. loc (λ, l, g) represents the localization loss based on the negative log-likelihood function in the regression task, where λ represents the index parameter indicating whether the anchor box matches the ground truth bounding box, g represents the ground truth bounding box parameters, l represents the Gaussian mixture model parameters corresponding to the predicted output bounding box, and η represents the weighted hyperparameters. This represents the contribution of positive matching in a classification task. This represents the contribution of negative matching in a classification task, where c represents the class parameter.

[0022] In the regression task, the localization loss L based on the negative log-likelihood function loc The formula for calculating (λ, l, g) is:

[0023]

[0024] In the formula, Pos represents the set of anchor boxes that are positively matched. This represents the matching metric, and K represents the number of Gaussian mixture models. This represents the blending weight of the k-th component in the bounding box parameter b of the i-th anchor box. Let b represent the mean value of the bounding box parameter b of the i-th anchor box in the k-th component. Let represent the variance of the k-th component of the bounding box parameter b of the i-th anchor box. ε represents the offset of the bounding box parameter b of the j-th true bounding box from the corresponding anchor box parameter, and ε represents the hyperparameter.

[0025] Contribution of positive matching in the classification task The calculation formula is:

[0026]

[0027] In the formula, Pos represents the set of anchor boxes that are positively matched. This represents the matching index, K represents the number of Gaussian mixture models, and π represents the number of Gaussian mixture models. ik This represents the blending weights output by the k-th component of the i-th anchor box. This represents the true class of the j-th true bounding box of category G. Let C represent the class probability distribution of the p-th class of the k-th component of the i-th anchor box, and let C represent the number of classes.

[0028] The contribution of negative matching in the classification task The calculation formula is:

[0029]

[0030] In the formula, M represents the ratio of hard negative samples mined, and Neg represents the set of negative matching anchor boxes. This represents the background class of the i-th anchor box.

[0031] Class probability distribution of class p in the k-th component The calculation formula is:

[0032]

[0033] In the formula, Let $\mathbf{k}$ represent the mean of the $p$-th class of the $k$-th component. Let γ represent the variance of the k-th component in class p, and let γ represent the auxiliary noise variable. Represents a normal distribution;

[0034] When the number of parameters in the classification header is reduced, the class probability distribution of the p-th class in the k-th component is... The calculation formula has been updated to:

[0035]

[0036] In step S3.5, the formula for calculating the arbitrary uncertainty is:

[0037]

[0038] In the formula, u al Representing arbitrary uncertainty, π k ∑ represents the blending weight of the k-th component. k This represents the variance of the k-th component;

[0039] The formula for calculating cognitive uncertainty is:

[0040]

[0041] In the formula, u ep μ represents cognitive uncertainty. k π represents the mean of the k-th component. k″ μ represents the blending weight of the k″-th component. k″ This represents the mean of the k″-th component.

[0042] An incremental learning system for environmental perception in autonomous driving includes:

[0043] The category labeling module is used to classify environmental image datasets according to the object detection task. The target in the process is labeled with a class, and the labeled classes are divided into known classes and unknown classes, and the known classes and unknown classes belong to different classes;

[0044] The image segmentation module is used to segment the labeled environmental image dataset. Divided into training set and test set Then the training set and test set The training set is obtained by removing images marked with unknown classes. and test set

[0045] The uncertainty calculation module is used to calculate the training set. Cognitive uncertainty and arbitrary uncertainty in images;

[0046] The target detection model training module constructs a hybrid density network based on the output parameters of the Gaussian mixture model. It uses the cognitive uncertainty and arbitrary uncertainty calculated by the uncertainty calculation module to optimize the hybrid density network with the goal of minimizing the loss function to obtain the target detection model.

[0047] The object detection model optimization module constructs a Gaussian mixture model for each known class based on the expectation-maximization algorithm. It estimates the cognitive semantic uncertainty of the known and unknown classes using the log-likelihood function and the Gaussian mixture model of each known class. Based on the cognitive semantic uncertainty, it establishes a set of images to be labeled, manually annotates the images to be labeled, and finally uses the manually annotated images to further train the object detection model generated by the object detection model training module to obtain the optimal object detection model.

[0048] The beneficial effects of this invention are:

[0049] This invention can effectively distinguish between known and unknown classes in open set scenarios, thereby identifying unknown classes (i.e., new classes) in open sets and achieving incremental learning. It can effectively learn known and unknown classes in open set scenarios in stages, while minimizing manual labeling and training costs. During the incremental learning of unknown classes, it can avoid catastrophic forgetting of known classes. It has universal applicability, applicable to both first-order and second-order object detectors, and compared with other existing methods, it has better average accuracy and better performance. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a diagram illustrating the training comparison of known classes on the IDD dataset.

[0052] Figure 2 This is a diagram illustrating the training comparison of unknown classes on the IDD dataset.

[0053] Figure 3 This is a sample visualization view when conducting experiments using the IDD dataset.

[0054] Figure 4 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Example 1: An incremental learning method for environmental perception in autonomous driving, such as Figure 4 As shown, it includes the following steps:

[0057] S1, based on the object detection task, the environmental image dataset The target in the process is labeled with a class, and the labeled classes are divided into known classes and unknown classes, and the known classes and unknown classes belong to different classes;

[0058] An environmental image dataset is obtained by collecting images captured in an autonomous driving environment. Based on the preset target detection task, the environmental image dataset is analyzed. In each image, targets are labeled with classes. Based on the attributes of the labeled classes, these classes are divided into known classes and unknown classes. The known classes have at least two categories, and the unknown classes have at least one category. The categories of the known classes and the unknown classes are different. For example, the categories of the known classes are bicycles and cars, and the categories of the unknown classes are people.

[0059] S2, the environmental image dataset divided in step S1 Divided into training set and test set training set and test set The training set is obtained by removing images marked with unknown classes. and test set Initialize the first round training number t = 1 and the second round training number

[0060] By removing images containing unknown classes, the labeled unknown classes become truly unknown classes, thus ensuring the effective use of the training set. This category will not be seen by the object detector during training, and the test set... The training set also does not contain target objects of that category, which can form an approximately closed set, while the original training set... and the original test set This is equivalent to an open set. Of course, unlabeled unknown classes—those other than the known and unknown classes—still exist in the background of the dataset. They are equivalent to the background in the open set and are trained to ignore these classes during the object detector's training process. Furthermore, the environmental image dataset can also be divided into training, testing, and validation sets. The added validation set is mainly used to adjust the model's hyperparameters.

[0061] S3. A mixture density network is constructed using Gaussian Mixture Models (GMMs), whose output includes the mean, variance, and mixture weights. The output of the mixture density network is then used to compute the training set. The uncertainty of each image in the dataset is considered. Based on this uncertainty, a first training input image set is established. This first training input image set is then input into a hybrid density network. The hybrid density network is optimized to minimize the loss function to obtain the object detection model, including the following steps:

[0062] S3.1, construct a mixture density network using the parameters of the Gaussian mixture model as output, and set a precision difference threshold. And precision difference ratio threshold

[0063] The parameters of the Gaussian mixture model are mean, variance, and mixture weights.

[0064] S3.2, from the training set S1 images are randomly selected from the dataset to form the first training input image set;

[0065] S1 is a positive integer, and Head S represents the new training set. The number of samples.

[0066] S3.3, Input the images from the first training input image set into the hybrid density network and train it with the goal of minimizing the loss function to obtain the target detector;

[0067] The formula for calculating the loss function is as follows:

[0068]

[0069] In the formula, L Total Let L represent the total loss, and N represent the number of positive matches. A positive match is a correct match, meaning the IoU between the model's predicted bounding box and the ground truth bounding box is greater than a certain threshold (0.5 in this example), and the model's predicted category for the object within the predicted bounding box is the same as the true category. loc(λ, l, g) represents the localization loss based on the negative log-likelihood function in the regression task, and η represents the weighted hyperparameter used to balance the localization loss and the classification loss. In this embodiment, η = 2. This represents the contribution of positive matching in a classification task. This represents the contribution of negative matching in the classification task. Negative matching is an incorrect match, meaning the IoU between the predicted bounding box and the ground truth bounding box is less than a certain threshold (0.5 in this example), or the model's predicted class of the object within the predicted bounding box differs from the true class. c represents the class parameter, λ represents the index parameter indicating whether the anchor box matches the ground truth bounding box, g represents the ground truth bounding box parameters (which can be the center x-coordinate, center y-coordinate, width w, or height h of the ground truth bounding box), and l represents the Gaussian mixture model parameters corresponding to the predicted bounding box. When the bounding box parameter is b, the corresponding Gaussian mixture model parameters are... This represents the blending weight of the k-th component in the bounding box parameter b of the i-th anchor box. Let b represent the mean value of the bounding box parameter b of the i-th anchor box in the k-th component. Let represent the variance of the k-th component of the bounding box parameter b of the i-th anchor box.

[0070] In the regression task, the localization loss L based on the negative log-likelihood function loc The formula for calculating (λ, l, g) is:

[0071]

[0072] In the formula, Let represent the matching metric used to match the i-th anchor box with the j-th ground truth bounding box of category G, and Pos represent the set of positively matching anchor boxes. Let represent the offset of the bounding box parameter b of the j-th ground truth bounding box from the corresponding anchor box parameter. The bounding box parameter b is defined by its center coordinates (x and y), width (w), and height (h), i.e., b = {x, y, w, h}. Therefore... Offset including the center x-coordinate offset of the center y-coordinate offset of width w Or the offset of height h Right now ε represents a hyperparameter, and K represents the number of Gaussian mixture models. In this embodiment, ε is set to 10 to ensure the numerical stability of the logarithmic function. -9 The regression localization loss is mainly used to regress the accurate offset of the true bounding box of an approximating object relative to the anchor box.

[0073] The matching metric used to match the i-th anchor box and the j-th ground truth bounding box of category G. The calculation formula is:

[0074]

[0075] In the formula, IoU represents the intersection over union ratio, which is the ratio of the anchor box to the ground truth bounding box. The intersection over union ratio is a concept used in object detection, which calculates the overlap rate between the "predicted bounding box" and the "ground truth bounding box", that is, the ratio of their intersection to their union.

[0076] Offset of the center x-coordinate The calculation formula is:

[0077]

[0078] In the formula, This represents the x-coordinate of the center of the j-th ground truth bounding box. This represents the x-coordinate of the center of the i-th anchor frame. This represents the width of the i-th anchor frame.

[0079] offset of the center y-coordinate The calculation formula is:

[0080]

[0081] In the formula, This represents the y-coordinate of the center of the j-th ground truth bounding box. This represents the center ordinate of the i-th anchor frame. This represents the height of the i-th anchor frame.

[0082] offset of width w The calculation formula is:

[0083]

[0084] In the formula, This represents the width of the j-th real bounding box.

[0085] offset of height h The calculation formula is:

[0086]

[0087] In the formula, This represents the height of the j-th ground truth bounding box.

[0088] Traditional bounding box regression loss, namely smoothed L1 loss, only considers the coordinates of the predicted and ground truth bounding boxes. Therefore, it cannot take into account the fuzziness or arbitrary uncertainty of the bounding boxes. This application, however, also considers the anchor box (default box), thus taking into account the fuzziness of the bounding boxes and calculating their corresponding arbitrary uncertainties.

[0089] Contribution of positive matching in the classification task The calculation formula is:

[0090]

[0091] In the formula, π ik This represents the blending weights output by the k-th component of the i-th anchor box. This represents the real class of the j-th real bounding box. Let C represent the class probability distribution of the p-th class of the k-th component of the i-th anchor box, where C represents the number of classes and N represents the number of positively matching anchor boxes.

[0092] The contribution of negative matching in the classification task The calculation formula is:

[0093]

[0094] In the formula, M represents the hard negative sample mining ratio. In this embodiment, not all negative matches are used. Instead, the values ​​of the proposed hybrid classification loss (i.e., the loss function) are used to rank them, and the top M×N are selected as the final negative matches for training. In the experiment, M is set to 3, and Neg represents the set of negative matching anchor boxes. This represents the background class of the i-th anchor box.

[0095] The class probability distribution of the k-th component is obtained using the mean and variance of Gaussian noise, and the corresponding calculation formula is:

[0096]

[0097] In the formula, This represents the class probability distribution of the p-th class of the k-th component, and its calculation result is assigned to... Let $\mathbf{k}$ represent the mean of the $p$-th class of the $k$-th component. Let γ represent the variance of the class p of the k-th component, and let γ represent the auxiliary noise variable. The magnitude of γ is related to... and same, This represents a normal distribution.

[0098] In object detection tasks, each bounding box is defined by its center coordinates, width, and height. Since a Gaussian mixture model consists of K components, each component is equivalent to a Gaussian distribution, representing the sample distribution within a cluster. The K components, i.e., the linear superposition of Gaussian distributions, form the Gaussian mixture model. In regression tasks, each model predicts three sets of parameters for each bounding box: mean... variance and mixed weights In classification tasks, the output of each class is modeled as a Gaussian mixture model, which outputs the mean, variance, and mixture weights of each component for each class. Where p represents the p-th category.

[0099] The formula for calculating the mean of the Gaussian mixture model is as follows:

[0100]

[0101] In the formula, Let represent the mean value of the bounding box parameter b output by the k-th component of the Gaussian mixture model. Let represent the predicted mean of the bounding box parameter b output by the k-th component of the Gaussian mixture model.

[0102] The formula for calculating the variance of the Gaussian mixture model is as follows:

[0103]

[0104] In the formula, Let represent the variance of the bounding box parameter b output by the k-th component of the Gaussian mixture model. Let represent the prediction variance of the bounding box b output by the k-th component of the Gaussian mixture model. This variance represents the arbitrary uncertainty. σ represents the sigmoid function, which satisfies the positive constraint on variance.

[0105] The formula for calculating the mixture weights of the Gaussian mixture model is as follows:

[0106]

[0107] In the formula, This represents the mixing weight of the bounding box b output by the k-th component of the Gaussian mixture model. This represents the predicted mixing weight of the bounding box b output by the k-th component of the Gaussian mixture model. By using a softmax function, the mixed weights are kept in the probability space, i.e., the sum of the probabilities of all components is 1.

[0108] In this embodiment, the Gaussian mixture model parameters are used as the output of the mixture density network. The output layer of the mixture density neural network is used to predict the probability distribution, rather than predicting a single value for each output of the network. For regression tasks, the network outputs the Gaussian mixture model for the mean, variance, and mixture weights of each bounding box parameter (including the x-coordinate, y-coordinate, width, and height). For classification tasks, the network outputs the Gaussian mixture model for the mean, variance, and mixture weights of each component for each class.

[0109] S3.4, test set The image input target detector calculates the mean accuracy and determines whether it meets the requirements. and If yes, proceed to step S3.5; otherwise, output the target detector for round t-1 as the target detection model and proceed to step S4, where MAP t MAP represents the mean accuracy of the target detector in round t. t-1 MAP represents the mean accuracy of the target detector in round t-1. t-2 This represents the mean accuracy of the target detector in round t-2;

[0110] In this embodiment,

[0111] S3.5, determine if S2 < S1; if so, use the training set. Update the first training input image set and proceed to step S3.3; otherwise, compute the training set using the parameters of the Gaussian mixture model. The uncertainty of the remaining images excluding the first training input image set, including arbitrary uncertainty and cognitive uncertainty, where S2 represents the training set. The number of images remaining in the dataset excluding the first training input image set;

[0112] training set The remaining images in the training set refer to the images in the training set. The images other than those in the first training input image set are used to calculate the uncertainty of the remaining images, which provides a reference index for the subsequent selection of training samples. This is especially important in real open set scenarios. Compared with training all samples at once, continuously selecting a certain number of training samples in a loop will greatly reduce training costs such as manual labeling. Compared with continuously randomly selecting the same number of training samples, the training samples selected based on uncertainty will result in better model training performance and higher accuracy.

[0113] The formula for calculating the arbitrary uncertainty is:

[0114]

[0115] In the formula, u al Representing arbitrary uncertainty, π k Let represent the blending weight of the k-th component, and ∑ k Let represent the variance of the k-th component, and

[0116] The formula for calculating cognitive uncertainty is:

[0117]

[0118] In the formula, u ep μ represents cognitive uncertainty. k Let represent the mean of the k-th component, and π k″ μ represents the blending weight of the k″-th component. k″ This represents the mean of the k″-th component.

[0119] To predict the probability distribution of the output values, a Gaussian mixture model is used to modify the output layer of the mixture density network, leading to an increase in the number of parameters, especially in the classification head. Specifically, for an output feature map of size F×F, there are C classes, D anchor boxes, and bounding boxes defined using 4 coordinates each. Therefore, the modified output layer adds F×F×D×(4×3×K) localization parameters and F×F×D×(C×2×K+K) classification parameters. Optimally, the algorithm's efficiency is improved by reducing the number of parameters in the classification head. In this embodiment, the number of classification parameters is reduced to F×F×D×(C×K+K). Therefore, the class probability distribution of the p-th class in the k-th component... The calculation formula has been updated to:

[0120]

[0121] Clearly, formula (16) is more relaxed than formula (10). Factors to consider.

[0122] Meanwhile, the formula for calculating any uncertainty in the classification task is updated as follows:

[0123]

[0124] In the formula, diag(q) represents a vector q and A diagonal matrix of elements. At this point, μ al It is a C×C matrix, where the value of each diagonal element can be interpreted as an arbitrary uncertainty specific to the class. Similarly, based on the updated class probability distribution... The value of the positive match is used to update the classification loss function of the object detection model, which is also the contribution of the positive match in the classification task. Contribution of negative matching in classification tasks

[0125] S3.6, use the Z-score to normalize the arbitrary uncertainty and cognitive uncertainty calculated in step S3.5, and determine the uncertainty of each remaining image based on the normalized arbitrary uncertainty and cognitive uncertainty;

[0126] The formula for normalization is:

[0127]

[0128] In the formula, u represents the normalized value of the uncertainty of the j′-th object in the i′-th image. i′j′ μ represents the uncertainty (arbitrary uncertainty or cognitive uncertainty) of the j′-th target, i.e., the object, in the i′-th image. U σ represents the mean of the uncertainty of all objects in a set of images. U This represents the variance of the uncertainty for all objects in a set of images. Normalization is used to compensate for the unbounded coordinate values ​​of the bounding boxes, as each uncertainty in the image may have a different range of values.

[0129] The uncertainty of each image is determined based on the normalized arbitrary uncertainty and cognitive uncertainty. Specifically, firstly, the maximum value of the cognitive uncertainty and the maximum value of the arbitrary uncertainty of all objects in each image in the classification task are selected as the cognitive uncertainty and arbitrary uncertainty of the image classification task, respectively. Secondly, the maximum value of the cognitive uncertainty and the maximum value of the arbitrary uncertainty of all objects in each image in the regression task are selected as the cognitive uncertainty and arbitrary uncertainty of the image regression task, respectively. These four values ​​constitute the set of image uncertainties, that is... This represents the cognitive uncertainty of the i′-th image in a classification task. Let represent any uncertainty in the i′-th image during the classification task. This represents the cognitive uncertainty of the i′-th image in the regression task. Let represent any uncertainty in the i′-th image during the regression task. This represents the normalized value of the cognitive uncertainty of the j′-th object in the i′-th image during a classification task. Represents the arbitrary uncertainty normalized value of the j′-th object in the i′-th image in a classification task. This represents the normalized value of the cognitive uncertainty of the j′-th object in the i′-th image during the regression task. Let represent any normalized uncertainty value of the j′-th object in the i′-th image in the regression task. Then, the maximum value is selected from the uncertainty set of the images as the uncertainty value of that object.

[0130] The uncertainty set U of the i′-th image i′ The expression is:

[0131]

[0132] S3.7 Sort the uncertainties of each remaining image obtained in step S3.6 in descending order to obtain a set of remaining image uncertainties. Add the images corresponding to the first S1 uncertainties in the set of remaining image uncertainties to the first training input image set to obtain the updated first training input image set. Execute t = t + 1 and return to step S3.3.

[0133] The original first training input image set remains unchanged. By adding samples with high uncertainty to the first training input image set, the training samples can be enriched. In addition, the added samples can be considered as difficult samples. Using these difficult samples, the object detection model can be trained to learn known classes better.

[0134] S4. Using the target detection model obtained in step S3, generate the feature space of the known classes. Based on the feature space of the known classes and the preset Gaussian mixture model, model each known class. Use the expectation-maximization algorithm to fit the parameters of the Gaussian mixture model of each known class to optimize the target detection model. This includes the following steps:

[0135] S4.1, based on the training set generated by the object detection model The known class of the image and its corresponding feature vectors are used to construct the feature space of the known class;

[0136] The feature space refers to the n-dimensional space in which the feature vectors of known classes exist, where n represents the number of features of all known classes.

[0137] S4.2, set several fixed feature points in the feature space of the known classes, and each fixed feature point corresponds to a known class;

[0138] S4.3, using the training set The image is used to train the object detection model, and the loss function of the object detection model is updated to minimize the distance between the feature vector of each known class of object and the corresponding fixed feature point.

[0139] Updating the loss function of the object detection model refers to updating the classification loss function. Specifically, the anchor loss term is weighted by a hyperparameter and then added directly to the original classification loss function to obtain the updated classification loss function. The anchor loss term represents the minimum distance between the feature vector of a known class and its corresponding fixed feature point. The selection of fixed feature points for each known class depends on which known class the class feature vector belongs to. By minimizing the distance between the feature vector of a known class and its fixed feature point, the feature vector of the known class can be mapped to the known class feature space better and more strictly. The hyperparameter reduces the size of the anchor loss term to maintain balance with the original classification loss function and weights the clustering constraints imposed by the anchor loss term during training. Replacing the original classification loss function with the updated classification loss function during training allows for the learning of a more structured feature space. The term "structured" mainly refers to its ability to be modeled by a Gaussian mixture model.

[0140] S4.4, a Gaussian mixture model is preset for each known class;

[0141] S4.5, the confidence score for the collection category is not less than [amount missing]. The feature vectors corresponding to each known class, whose IoU value between the predicted bounding box and the ground truth bounding box is not less than Θ, constitute the second training input image set for each known class;

[0142] In this embodiment, the class confidence score threshold IoU threshold Θ = 0.5.

[0143] S4.6 Input the second training input image set of each known class into the corresponding Gaussian mixture model, and use the expectation-maximization algorithm to fit the parameters of the Gaussian mixture model of each known class to obtain the parameters of the Gaussian mixture model of each known class, and then model the feature vector distribution of the second training input image set in the feature space.

[0144] In the object detection model, a structured feature space is constructed and modeled using a Gaussian mixture model. This lays the groundwork for subsequent computation of cognitive semantic uncertainty to identify unknown classes, enabling the object detection model to perform incremental learning on unknown classes while avoiding catastrophic forgetting.

[0145] S5, training set Input the object detection model, use the log-likelihood function and the Gaussian mixture model of each known class to estimate the cognitive semantic uncertainty of all classes, establish a set of images to be labeled based on the cognitive semantic uncertainty, manually label the images to be labeled, and use the manually labeled images to further train the object detection model to obtain the optimal object detection model;

[0146] S5.1, the training set Input the target detection model, use the log-likelihood function and the Gaussian mixture model of each known class to calculate the log-likelihood value of each feature vector belonging to each known class, and determine the cognitive semantic uncertainty of the detected object, i.e. the target, belonging to each known class based on the log-likelihood value;

[0147] The formula for calculating the log-likelihood value is as follows:

[0148]

[0149] In the formula, Let G represent the feature vector of the object being detected, l represent the feature vector in the structured feature space, and G represent the feature vector in the structured feature space. r Let π represent the r-th Gaussian mixture model. r，j′ μ represents the mixing weight of the j′-th component in the r-th Gaussian mixture model. r，j′ Let ∑ represent the mean of the j′-th component of the r-th Gaussian mixture model. r，j′ Let represent the variance of the j′th component of the r-th Gaussian mixture model.

[0150] By calculating the log-likelihood value P of the Gaussian mixture model for each known class, P = log(p(l;G)). r This allows us to obtain a measure of cognitive uncertainty, unc, for each known class. One of the low log-likelihood values ​​indicates high cognitive semantic uncertainty regarding whether the detected objects belong to their respective known classes.

[0151] S5.2, Based on the cognitive semantic uncertainty of the detected object belonging to each known class, determine the final value of the cognitive semantic uncertainty of each detected object, and then determine the cognitive semantic uncertainty of the image;

[0152] Specifically, a set of cognitive semantic uncertainty values ​​for the detected object across all known classes is established, and the minimum value of this set is taken as the final cognitive semantic uncertainty value for the detected object. Among them, unc i′j′ This represents the final value of the cognitive semantic uncertainty of the j′-th detected object in the i′-th image. Let represent the cognitive semantic uncertainty of the j′-th detected object in the i′-th image relative to the q-th known class. The maximum value of the final cognitive semantic uncertainty of all detected objects in an image is taken as the cognitive semantic uncertainty of that image, i.e. in This represents the cognitive semantic uncertainty of the i′-th image.

[0153] S5.3, Sort the cognitive semantic uncertainty of each image obtained in step S5.2 in descending order to obtain a set of cognitive semantic uncertainties for the images, and select the images with higher cognitive semantic uncertainties. The images corresponding to each value constitute the image set to be labeled. The images in the image set to be labeled are manually labeled with categories. The manually labeled image set is added to the training image set to obtain the updated training image set. The updated training image set is used to train the object detection model.

[0154] in, It is a positive integer, and and Indicates training set The number of samples, the training image set was initially an empty set.

[0155] S5.4, test set Calculate the MAP using the trained object detection model and determine if it meets the requirements. and If yes, proceed to step S5.5; otherwise, output... The target detection model for wheels is considered the optimal target detection model, where... Indicates the first MAP of the target detection model in the wheel, Indicates the first MAP of the target detection model in the wheel, Indicates the first MAP of the target detection model in the wheel;

[0156] S5.5, Judgment If so, use the training set Update the training image set and proceed to step S5.3; otherwise, based on the training set... The remaining images besides the training image set are used to update the training set. Return to step S5.1, where, Indicates training set And the number of images remaining outside the training image set.

[0157] This paper utilizes a structured feature space and a Gaussian mixture model to calculate the cognitive semantic uncertainty of images to identify unknown classes. In open-set scenarios, the higher the final value of the cognitive semantic uncertainty of the detected object, the less likely it is to belong to a known class and the greater the likelihood of it belonging to an unknown class (i.e., a new class). This further indicates that the object detector needs to learn this new class. To this end, a certain number of images containing unknown classes are cyclically selected for manual annotation, labeling all categories. The labeled images are then used to continuously train and optimize the object detection model. This allows the object detection model to effectively identify unknown classes (i.e., new classes) in open-set scenarios, thereby achieving incremental learning while avoiding catastrophic forgetting of known classes. Moreover, it minimizes the costs of manual labeling and training during the training process.

[0158] Example 2: An incremental learning system for environmental perception in autonomous driving, comprising:

[0159] The category labeling module is used to classify environmental image datasets according to the object detection task. The target in the process is labeled with a class, and the labeled classes are divided into known classes and unknown classes, and the known classes and unknown classes belong to different classes;

[0160] The image segmentation module is used to segment the labeled environmental image dataset. Divided into training set and test set Then the training set and test set The training set is obtained by removing images marked with unknown classes. and test set

[0161] The uncertainty calculation module is used to calculate the training set. Cognitive uncertainty and arbitrary uncertainty in images;

[0162] The target detection model training module constructs a hybrid density network based on the output parameters of the Gaussian mixture model. It uses the cognitive uncertainty and arbitrary uncertainty calculated by the uncertainty calculation module to optimize the hybrid density network with the goal of minimizing the loss function to obtain the target detection model.

[0163] The object detection model optimization module constructs a Gaussian mixture model for each known class based on the expectation-maximization algorithm. It estimates the cognitive semantic uncertainty of the known and unknown classes using the log-likelihood function and the Gaussian mixture model of each known class. Based on the cognitive semantic uncertainty, it establishes a set of images to be labeled, manually annotates the images to be labeled, and finally uses the manually annotated images to further train the object detection model generated by the object detection model training module to obtain the optimal object detection model.

[0164] In this embodiment, all calculation methods and model training methods are the same as in Embodiment 1.

[0165] This application also provides an electronic device, which includes a processor and a memory. The memory stores a computer program, and when the computer program is executed by the processor, it performs the steps of the above-described incremental learning method for autonomous driving environment perception.

[0166] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it performs the steps of the incremental learning method for autonomous driving environment perception described above. Specifically, the storage medium can be a general-purpose storage medium, such as a removable disk or hard disk. When the computer program on the storage medium is run, it can execute the embodiments of the incremental learning method for autonomous driving environment perception described above.

[0167] The network architecture of this application is implemented using a Python environment, the PyTorch deep learning framework, and MMDetection and MMCV. For the basic feature extractor, ResNet50 is used as the backbone; it is powerful yet simple. The new feature extractor is the same as the basic feature extractor. For the object detection framework, experiments are conducted on both second-order object detection (Faster R-CNN) and first-order object detection (RetinaNet) to demonstrate the general applicability of the method. All experiments were performed using an NVIDIA Corporation GP102 TITAN Xp GPU with 12GB of memory and an Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz. The following describes the implementation of uncertainty-driven incremental learning for autonomous driving environmental perception using this application, specifically experiments on the IDD dataset.

[0168] For the IDD dataset, the first seven classes—car, person, rider, truck, motorcycle, bus, and bicycle—are defined as labeled known classes, while the remaining three classes—animal, traffic sign, and traffic light—are labeled unknown classes. The experiment used 31,569 images from the IDD training set. The IDD validation set consists of 3741 images. The IDD test set contains 4794 images. In both second-order and first-order object detection training, the RGB images were first resized to 1000×600 and enhanced by random cropping. Network parameters were trained using stochastic gradient descent (SGD) over 7 epochs. For the second-order object detector Faster R-CNN, this experiment used SGD with a learning rate of 0.005, momentum of 0.9, and weight decay of 0.0001, employing a stepped learning rate decay strategy between the 5th and 6th epochs. For the object detection model RetinaNet, the SGD learning rate was 0.01, with other parameters consistent with the second-order object detector, and the batch size was 2 throughout the training process.

[0169] In the IDD dataset, n1 is set to 8000 and n2 to 6000, as follows: Figure 1 As shown in the figure, the upper curve represents the mAP value obtained after training with different numbers of samples using the method proposed in this application, and the lower curve represents the mAP value obtained after training with different numbers of samples using a random extraction method. It can be observed that before the number of labeled samples is 8000, the mAP value of the detection result increases significantly with the increase of the number of labeled samples. However, after the number of labeled samples is 8000, the mAP value of the detection result no longer shows a significant increase with the increase of the number of labeled samples. Therefore, n1 is set to 8000. Similarly, on the IDD dataset... Figure 2 Therefore, n² is set to 6000. Here, Random represents the learning method that randomly selects samples for training, FourUnc represents the active learning method proposed in this application for known classes, and LogitUnc represents the incremental learning method proposed in this application for unknown classes.

[0170] For the IDD dataset, on a second-order object detector, this application selects the first n1 samples with high uncertainty to train the known classes. Compared with randomly selecting n1 samples to train the known classes, this method performs better and has higher accuracy, as shown in the first and second rows of Table 2-1, with an average accuracy improvement of approximately 4%. Building upon the efficient training of the known classes, incremental learning using this application is compared with randomly selecting n2 samples, demonstrating the effectiveness of the proposed method and its higher accuracy, as shown in the third and fourth rows of Table 2-1, with an average accuracy improvement of approximately 5%. For the unknown classes of animals, traffic signs, and traffic lights that should be known in the open set, the average accuracy improvements are approximately 9%, 14%, and 15%, respectively. Furthermore, a comparison of the second and fourth rows of Table 2-1 shows that the accuracy of the known class detector did not decrease significantly during incremental learning, thus avoiding catastrophic forgetting. This is because when manually labeling the selected n² samples, not only were unknown classes in the open set environment labeled, but also known classes were labeled if the samples contained them. This is equivalent to partially replaying known classes during incremental learning, thereby avoiding catastrophic forgetting of known classes. Moreover, this method is also applicable to first-order object detectors, as shown in Table 2-2. In summary, the effectiveness and universal applicability of this invention are evident, and its visualization results are shown in the figure below. Figure 3 As shown.

[0171] Table 2-1 Evaluation on the IDD dataset (Faster R-CNN)

[0172]

[0173] Table 2-2 Evaluation on the IDD dataset (RetinaNet)

[0174]

[0175] A comparison of this application with other advanced methods and techniques is shown in Table 2-3. The iOD method refers to the incremental object detection method proposed by KjJ et al. using meta-learning; the BNC method refers to the incremental object detection method proposed by Na Dong et al., which bridges incompatible and unlabeled field data; and the Unc method refers to the incremental learning method for autonomous driving environment perception proposed in this application. A comparison of these three methods applied to the IDD dataset is shown in Table 2-3. Under IDD, "base" refers to the average accuracy of the 7 known classes, "novel" refers to the average accuracy of the 3 unknown classes, and "all" refers to the average accuracy of all 10 classes. The comparison shows that for known classes, the mAP of this application is approximately 14% higher than the iOD method and approximately 6% higher than the BNC method; for unknown classes, the mAP of this application is approximately 8% higher than the iOD method and approximately 4% higher than the BNC method; and for all classes, the mAP of this application is approximately 10% higher than the iOD method and approximately 4% higher than the BNC method.

[0176] Table 2-3 Comparison with other advanced methods

[0177]

[0178] As can be seen from the above, although the iOD method and BNC method performed well in their original papers, their results were obtained from experiments on the VOC and COCO datasets. When applied to autonomous driving datasets, their performance was not significant. Therefore, it can be said that this application is more suitable for environmental perception scenarios of intelligent unmanned systems such as autonomous driving datasets.

[0179] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An incremental learning method for autonomous driving environment perception, characterized in that, Includes the following steps: S1, based on the object detection task, the environmental image dataset The target in the process is labeled with a class, and the labeled classes are divided into known classes and unknown classes, and the known classes and unknown classes belong to different classes; S2, the segmented environmental image dataset Divided into training set and test set , training set and test set The training set is obtained by removing images marked with unknown classes. and test set ; S3 uses a Gaussian mixture model to construct a mixture density network whose output includes the mean, variance, and mixture weights. The output of the mixture density network is then used to compute the training set. The uncertainty of each image in the model is used to establish a first training input image set. The first training input image set is then input into a hybrid density network, and the hybrid density network is optimized with the goal of minimizing the loss function to obtain the target detection model. S4. Use the target detection model obtained in step S3 to generate the feature space of the known classes. Model each known class according to the feature space of the known classes and the preset Gaussian mixture model. Use the expectation-maximization algorithm to fit the parameters of the Gaussian mixture model of each known class. S5, training set Input the object detection model, estimate the cognitive semantic uncertainty of all classes using the log-likelihood function and the Gaussian mixture model of each known class, establish a set of images to be labeled based on the cognitive semantic uncertainty, manually label the images in the set of images to be labeled, and further train the object detection model using the manually labeled images to obtain the optimal object detection model.

2. The incremental learning method for autonomous driving environment perception according to claim 1, wherein, Step S3 includes the following steps: S3.1, construct a mixture density network using the parameters of the Gaussian mixture model as output, and execute... And set a threshold for accuracy difference. And precision difference ratio threshold ; S3.2, randomly select from the training set a first training input image set consisting of images;​ S3.3, Input the images from the first training input image set into the hybrid density network and train it with the goal of minimizing the loss function to obtain the target detector; S3.4, test set The image input target detector calculates the mean accuracy and determines whether it meets the requirements. and If yes, proceed to step S3.5; otherwise, output... The target detector of the wheel is used as the target detection model, and step S4 is executed, where, Indicates the first The mean accuracy of the target detector for the wheel. Indicates the first The mean accuracy of the target detector for the wheel. Indicates the first The mean accuracy of the target detector for the wheel; S3.5, Judgment If so, use the training set Update the first training input image set and proceed to step S3.3; otherwise, compute the training set using the parameters of the Gaussian mixture model. The arbitrary uncertainty and cognitive uncertainty of the remaining images excluding the first training input image set, where, Indicates training set The number of images remaining in the dataset excluding the first training input image set; S3.6, use the Z score to normalize the arbitrary uncertainty and cognitive uncertainty calculated in step S3.5, and determine the uncertainty of each remaining image based on the normalized arbitrary uncertainty and cognitive uncertainty; S3.7, Sort the uncertainties of each remaining image obtained in step S3.6 in descending order to obtain a set of remaining image uncertainties. Then, select the top uncertainties from the remaining image uncertainty set... The image corresponding to each uncertainty is added to the first training input image set to obtain the updated first training input image set, and then the process is executed. Then return to step S3.

3.

3. The incremental learning method for autonomous driving environment perception according to claim 2, wherein, In step S3.3, the formula for calculating the loss function is as follows: ； In the formula, Indicates the total loss. Indicates the number of positive matches. This represents the localization loss based on the negative log-likelihood function in regression tasks. A metric parameter indicating whether the anchor box matches the actual bounding box. Represents the actual bounding box parameters. This represents the parameters of the Gaussian mixture model corresponding to the predicted output bounding box. Indicates the weighted hyperparameter. This represents the contribution of positive matching in a classification task. This represents the contribution of negative matching in a classification task. Indicates class parameters.

4. The incremental learning method for autonomous driving environment perception according to claim 3, wherein, The localization loss based on the negative log-likelihood function in the regression task The calculation formula is: ； In the formula, This represents the set of anchor boxes that are positively matching. Indicates the matching metric. This indicates the number of Gaussian mixture models. Indicates the first The bounding box parameters of each anchor box The The mixed weights of the components, Indicates the first The bounding box parameters of each anchor box The The mean of each component, Indicates the first The bounding box parameters of each anchor box The The variance of each component Indicates the first Bounding box parameters of a real bounding box The offset is returned to the corresponding anchor frame parameter. This represents hyperparameters.

5. The incremental learning method for autonomous driving environment perception according to claim 3, wherein, Contribution of positive matches in the classification task The formula for calculating is: ； In the formula, This represents the set of anchor boxes that are matching. Indicates the matching metric. This indicates the number of Gaussian mixture models. Indicates the first The first anchor frame The mixed weights output by each component Indicate category The A real class for a real bounding box. Indicates the first The first anchor frame The first component Class probability distribution of a class Indicates the number of classes; Contribution of negative matches in the classification task The calculation formula is: ； In the formula, represents the difficulty of sample mining, represents the set of negative matching anchor frames, represents the background class of the th anchor frame.

6. The incremental learning method for autonomous driving environment perception according to claim 5, wherein, The formula for calculating the class probability distribution of the first class in the first component is: The formula for calculating the class probability distribution of the first class in the first component is: The formula for calculating the class probability distribution of the first class in the first component is: The formula for ； wherein denotes the mean of the first class of the first component, denotes the variance of the first class of the first component, denotes an auxiliary noise variable, denotes a normal distribution; When the number of parameters in the classification head is reduced, the formula for calculating the class probability distribution of the th component in the th group is updated as: ​ 。 7. The incremental learning method for autonomous driving environment perception according to claim 2, wherein, In step S3.5, the formula for calculating the arbitrary uncertainty is: ； wherein represents an arbitrary uncertainty, represents a mixing weight of the th component, represents a variance of the th component; The formula for calculating cognitive uncertainty is: ； wherein represents a cognitive uncertainty, represents a mean value of the th component, represents a mixing weight of the th component, represents a mean value of the th component.

8. An incremental learning system for autonomous driving environment perception, characterized in that, include: The category labeling module is used to classify environmental image datasets according to the object detection task. The target in the process is labeled with a class, and the labeled classes are divided into known classes and unknown classes, and the known classes and unknown classes belong to different classes; an image division module configured to divide the labeled environment image dataset into a training set and a test set The images in the training set and the test set that are labeled with the unknown class are then removed to obtain a training set and a test set ; an uncertainty computation module for computing cognitive uncertainty and aleatoric uncertainty of images in the training set an uncertainty computation module for computing cognitive uncertainty and aleatoric uncertainty of images in the training set The target detection model training module constructs a hybrid density network based on the output parameters of the Gaussian mixture model. It uses the cognitive uncertainty and arbitrary uncertainty calculated by the uncertainty calculation module to optimize the hybrid density network with the goal of minimizing the loss function to obtain the target detection model. The object detection model optimization module constructs a Gaussian mixture model for each known class based on the expectation-maximization algorithm. It estimates the cognitive semantic uncertainty of the known and unknown classes using the log-likelihood function and the Gaussian mixture model of each known class. Based on the cognitive semantic uncertainty, it establishes a set of images to be labeled, manually annotates the images to be labeled, and finally uses the manually annotated images to further train the object detection model generated by the object detection model training module to obtain the optimal object detection model.

Images