An unbalanced incremental learning method based on complementary relationship mining

Abstract

The application relates to an unbalanced incremental learning method based on complementary relationship mining, comprising the following steps: constructing an unbalanced incremental learning model, including an old task model, a current task model, a feature filter, an old-new encoder, a head-tail encoder and a classifier; constructing an unbalanced data set; inputting the data set into the unbalanced incremental learning model to perform incremental learning; wherein, in the incremental learning process, the head-tail encoder is used to mine the relationship between the majority class data and the minority class data, and the old-new encoder is used to mine the relationship between the features generated by the old task model and the current task model. The current task model after learning is used to realize classification recognition. The application provides a new incremental learning method, which can be applied to various practical scenes such as automatic driving, and solves the performance decline problem caused by unbalanced data distribution in the traditional incremental learning process and the safety problem caused by the poor classification recognition accuracy due to catastrophic forgetting.

Description

Technical Field

[0001] This invention belongs to the field of incremental learning technology, specifically relating to an imbalanced incremental learning method based on complementary relationship mining. Background Technology

[0002] With the development of technology, autonomous driving technology is becoming increasingly mature, and more and more cars are equipped with assisted autonomous driving functions. These cars are pre-programmed with certain road scenario scenarios before leaving the factory, such as waiting at traffic lights and driving on normal roads, to facilitate automatic obstacle avoidance and driving. However, once on the road, cars will encounter a variety of familiar and unfamiliar road conditions. Therefore, they need to learn and adapt to new road conditions while retaining the pre-programmed scenarios.

[0003] Incremental learning aims to mitigate the forgetting of old knowledge by balancing the plasticity and stability of the model. Old knowledge is not forgotten (stability in the face of change), while new input data is learned (adaptive plasticity). Currently, most incremental learning methods assume that the data distribution is balanced across different tasks. However, real-world data is often imbalanced, typically exhibiting a long-tailed distribution.

[0004] New road condition scenarios also exhibit long-tail distribution characteristics. For example, most encounters are traffic jams, waiting at traffic lights, and normal driving scenarios, but some rarer situations may occur, such as strong winds, glare from the sun, people holding umbrellas, people carrying boxes behind cars, fallen trees in the middle of the road, and other road condition scenarios. For imbalanced data, existing incremental learning methods may lead to a significant performance degradation, especially for classes with a small number of samples, which may experience more severe forgetting. In addition, existing incremental learning methods, such as knowledge distillation, only use distance loss to narrow the gap between the output features of the old and new models, without considering their sufficient interaction, resulting in low efficiency of cross-task information distillation. In the field of autonomous driving, these problems will seriously affect the safety of autonomous driving applications. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, this invention provides an imbalanced incremental learning method based on complementarity relationship mining. The technical problem to be solved by this invention is achieved through the following technical solution:

[0006] This invention provides an imbalanced incremental learning method based on complementarity relationship mining, comprising:

[0007] Step 1: Construct an imbalanced incremental learning model, which includes: an old task model, a current task model, feature filters, an old-to-new encoder, a head-to-tail encoder, and a classifier; wherein,

[0008] The old task model, the feature filter, and the old-new encoder are cascaded in sequence.

[0009] The output of the current task model is connected to the input of the old-new encoder and the input of the head-tail encoder, respectively.

[0010] The output terminals of the old-new encoder and the head-tail encoder are both connected to the input terminal of the classifier.

[0011] Step 2: Construct an imbalanced dataset;

[0012] Step 3: Input the imbalanced dataset into the imbalanced incremental learning model for incremental learning; wherein,

[0013] During incremental learning, knowledge distillation is used to extract the knowledge contained in the old task model into the current task model; a head-tail encoder is used to mine the relationship between the majority and minority class data in the imbalanced dataset, and an old-new encoder is used to mine the relationship between the features generated by the old task model and the current task model.

[0014] Step 4: Use the current task model and classifier after incremental learning to achieve classification and recognition.

[0015] In one embodiment of the present invention, both the old task model and the current task model are ResNet18 networks.

[0016] In one embodiment of the present invention, the old-new encoder and the head-to-tail encoder have the same structure, both including a first normalization layer, a self-attention layer, a second normalization layer, and a multilayer perceptron, wherein,

[0017] The first normalization layer, the self-attention layer, the second normalization layer, and the multilayer perceptron are cascaded in sequence.

[0018] The input of the first normalization layer is fused with the output of the self-attention layer and used as the input of the second normalization layer;

[0019] The input of the second normalization layer is fused with the output of the multilayer perceptron and used as the output of the encoder.

[0020] In one embodiment of the present invention, the imbalanced dataset includes road condition scene images of multiple categories with category labels, and the distribution of the number of road condition scene images of all categories conforms to the long-tail distribution characteristic.

[0021] In one embodiment of the present invention, step 3 includes:

[0022] Step 3.1: Input the road condition scene images in the imbalanced dataset into the imbalanced incremental learning model according to their categories;

[0023] Step 3.2: The old task model extracts features from the input samples to obtain old model features, and the current task model extracts features from the input samples to obtain new model features;

[0024] Step 3.3: The feature filter performs feature filtering on the old model features to obtain key features;

[0025] Step 3.4: The old-new encoder performs feature fusion on the new model features and the key features to obtain the old-new fused features;

[0026] Step 3.5: The head-tail encoder performs feature fusion on the currently input new model features and the previously input new model features to obtain head-tail fused features;

[0027] Step 3.6: The classifier classifies and identifies the input samples based on the old-new fusion features and the head-tail fusion features, and calculates the model loss function based on the classification and identification results and the classification labels;

[0028] Step 3.7: Update the parameters of the current task model, old-new encoder, head-tail encoder, and classifier through backpropagation based on the model loss function.

[0029] In one embodiment of the present invention, step 3.3 includes:

[0030] For each category of road condition scene images, the mean value of the old model features is calculated.

[0031] Calculate the distance between the old model features and the feature mean of the road condition scene images in the corresponding category;

[0032] Based on a preset threshold, features in the old model whose distance exceeds the threshold are removed to obtain the corresponding key features.

[0033] In one embodiment of the present invention, the model loss function is expressed as:

[0034]

[0035] In the formula, Represents classification loss, This indicates semantic compensation loss. Indicates batch processing constraint loss. Let represent the distillation loss, and α, β, γ, and δ represent the weights of each type of loss; where,

[0036]

[0037]

[0038]

[0039] In the formula, n t D represents the input sample D for the current task. t The number of samples in C old C represents the number of categories of all old classes learned incrementally in stage t. all =C old +C t C represents the number of classes in the incremental learning phase t. t This represents the number of new classes in the incremental learning phase t. This represents the weight matrix of the last fully connected layer, where d represents the dimension. Let z represent the bias vector of the last fully connected layer, T denote the transpose, |||| denotes the norm operation, and z i y represents the feature output by the i-th sample after passing through the current task model. i Let x represent the label of the i-th sample. i This represents the i-th sample. This represents the old task model. Indicates the current task model. This represents the semantic compensation loss of the header class. The semantic compensation loss for the tail class is represented by μ1 and μ2 respectively. and The weight.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] This invention presents an imbalanced incremental learning method based on complementary relationship mining. It achieves incremental learning of imbalanced data by constructing an imbalanced incremental learning model. A head-to-tail encoder mines the relationships between current mini-batches, facilitating the extraction of rich feature information between the minority and majority classes. An old-to-new encoder fully utilizes the relationships between output features generated by different models to reduce catastrophic forgetting between tasks. Furthermore, knowledge distillation, semantic loss, and fine-tuning techniques are employed to improve the model's learning performance. This invention provides a novel incremental learning method applicable to various practical scenarios such as autonomous driving, solving the performance degradation problem caused by imbalanced data distribution and the security issues arising from decreased classification accuracy due to catastrophic forgetting in traditional incremental learning.

[0042] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of an imbalanced incremental learning method based on complementary relationship mining provided in an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of an imbalanced incremental learning model provided in an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram of an encoder provided in an embodiment of the present invention;

[0046] Figure 4 This is a graph showing the top-1 accuracy results of a CIFAR100 dataset divided into five tasks with different imbalance rates, as provided in this embodiment of the invention.

[0047] Figure 5 This is a graph showing the top accuracy results of a CIFAR100 dataset divided into 10 tasks with different imbalance rates, as provided in an embodiment of the present invention.

[0048] Figure 6 This is a baseline visualization comparison chart of Grad-Cam demo with a new and old encoder added in a 5-task setup, with an imbalance rate of 0.05. Detailed Implementation

[0049] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following describes in detail an unbalanced incremental learning method based on complementary relationship mining proposed according to the present invention, in conjunction with the accompanying drawings and specific embodiments.

[0050] The foregoing and other technical contents, features, and effects of the present invention will be clearly presented in the following detailed description of specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a more in-depth and concrete understanding can be gained of the technical means and effects adopted by the present invention to achieve its intended purpose. However, the accompanying drawings are for reference and illustration only and are not intended to limit the technical solutions of the present invention.

[0051] Example 1

[0052] Please see Figure 1 and Figure 2 , Figure 1 This is a schematic diagram of an imbalanced incremental learning method based on complementary relationship mining provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of an imbalanced incremental learning model provided in an embodiment of the present invention. As shown in the figure, the imbalanced incremental learning method based on complementarity relationship mining in this embodiment includes:

[0053] Step 1: Construct an imbalanced incremental learning model;

[0054] like Figure 2 As shown, the imbalanced incremental learning model in this embodiment includes: an old task model, a current task model, a feature filter, an old-to-new encoder, a head-to-tail encoder, and a classifier. The old task model, feature filter, and old-to-new encoder are cascaded sequentially. The output of the current task model is connected to the inputs of both the old-to-new encoder and the head-to-tail encoder. The outputs of both the old-to-new encoder and the head-to-tail encoder are connected to the input of the classifier.

[0055] For incremental learning models, a supervised imbalanced incremental learning setting is typically considered, where one model needs to learn sequentially. There are several different tasks. Each task contains several classes, and these classes do not overlap. It is the set of categories for task t. Let represent a training sample, where x is a sample in the input space X, and y is its corresponding label. This is the sample space. However, because the dataset is imbalanced, the number of samples per class varies across tasks. The majority class has a large number of samples, while the minority class has a small number of samples.

[0056] The incremental learning model is divided into two parts: a feature extractor and a unified classifier. Specifically, it consists of a feature extractor parameterized by θ. Map the input x to a deep feature space Features Depend on Parameterized unified classifier Generate a probability distribution As a prediction of x, the model must classify all seen classes at any point during training.

[0057] In this embodiment, the old task model and the current task model are used as feature extractors to extract features from the input samples.

[0058] In one optional implementation, both the old task model and the current task model are ResNet18 networks.

[0059] In this embodiment, the old-new encoder is used to mine the relationship between the features generated by the stored old model and the current new model, and the head-tail encoder is used to mine the feature relationship between the majority class and the minority class.

[0060] Please see Figure 3 The schematic diagram of an encoder provided by the embodiment of the present invention shown shows that, in an optional implementation, the old-new encoder and the head-tail encoder have the same structure, both including a first normalization layer, a self-attention layer, a second normalization layer, and a multilayer perceptron. The first normalization layer, the self-attention layer, the second normalization layer, and the multilayer perceptron are cascaded in sequence. The input of the first normalization layer is fused with the output of the self-attention layer to serve as the input of the second normalization layer. The input of the second normalization layer is fused with the output of the multilayer perceptron to serve as the output of the encoder.

[0061] In this embodiment, the encoder works as follows:

[0062]

[0063] Where x represents the input, MLP (Multilayer Perceptron) is a multilayer perceptron, SA (Self-Attention) is a self-attention layer, Norm (LayerNorm) is layer normalization, l represents the l-th layer of the transformer, and x′ l Let x represent the result of each self-attention. l This represents the result of each transformer block.

[0064] For ease of description, an encoder can be represented as:

[0065] y = E(x) (2);

[0066] Where x and y represent the encoder input and output, respectively, and E represents the encoder.

[0067] Step 2: Construct an imbalanced dataset;

[0068] In this embodiment, the imbalanced dataset includes road condition scene images with category labels for multiple categories, and the distribution of the number of road condition scene images for all categories conforms to the long-tail distribution characteristic.

[0069] Optionally, an image acquisition device can be used to collect all road condition scenes of multiple vehicles during their driving process. After filtering, the data can be manually classified and assigned corresponding category labels to obtain an imbalanced dataset.

[0070] Step 3: Input the imbalanced dataset into the imbalanced incremental learning model for incremental learning;

[0071] In the incremental learning process, knowledge distillation is used to extract knowledge contained in the old task model into the current task model; a head-to-tail encoder is used to mine the relationship between the majority and minority classes in the imbalanced dataset; and an old-to-new encoder is used to mine the relationship between the features generated by the old task model and the current task model. Specifically, step 3 includes:

[0072] Step 3.1: Input the road condition scene images from the imbalanced dataset into the imbalanced incremental learning model according to their categories;

[0073] Step 3.2: The old task model extracts features from the input samples to obtain old model features, and the current task model extracts features from the input samples to obtain new model features;

[0074] Step 3.3: The feature filter performs feature filtering on the old model features to obtain key features;

[0075] In an optional implementation, step 3.3 includes: calculating the mean value of the old model features corresponding to each category of road condition scene images; calculating the distance between the old model features and the mean value of the features of the road condition scene images in the corresponding category; and removing features in the old model features whose distance exceeds the threshold according to a preset threshold to obtain the corresponding key features.

[0076] In this embodiment, instead of using all the features of the samples output by the old task model, the old model features are filtered through a feature filter to filter out the more representative parts of the feature space, i.e., key features, which can reduce the model's arithmetic ability and the probability of overfitting.

[0077] Step 3.4: The old-new encoder performs feature fusion on the new model features and key features to obtain the old-new fused features;

[0078] In this embodiment, the working principle of the old-new encoder is as follows:

[0079]

[0080] Where, z′ old This represents the key features obtained after feature filtering of the old model features, z. new This represents the new model features output by the current task model. Cat (concat) indicates concatenation and merging. cat E represents the output of the old-to-new encoder. Old-New This indicates the old-new encoder.

[0081] It should be noted that during incremental learning, the unavailability of samples from past tasks leads to a severe imbalance between the old and new tasks, causing the model to suffer from catastrophic forgetting of past tasks. While knowledge distillation can address this imbalance, current knowledge distillation methods merely use distance loss to reduce the distance between the output features of the old and new models, without considering the full interaction between these features. This simplistic distillation strategy results in low efficiency. In this embodiment, the old-new encoder fully leverages the relationship between the output features generated by the stored old task model and the current task model, effectively facilitating the distillation process.

[0082] In this embodiment, the old-new encoder learns the correlation between the output features of the old and new models and attempts to retain the old model as much as possible, which can mitigate catastrophic forgetting.

[0083] Step 3.5: The head-tail encoder performs feature fusion on the new model features of the current input and the new model features of the previous input to obtain the head-tail fused features;

[0084] In this embodiment, the working principle of the head-to-tail encoder is as follows:

[0085] z′=E Head-Tail (z) (4);

[0086] In the formula, m represents the batch size, d represents the dimension, z′ represents the new model features output by the current task model, and E represents the output of the head-to-tail encoder. Head-Tail Indicates head-to-tail encoder,

[0087] In this embodiment, to overcome the imbalance within the task and uncover the relationships between small batches of data, the new model features output by the current task model are fed into the head-tail encoder to obtain the same dimensionality output. By using the head-tail encoder, relevant information between small batches of data can be captured, and the rich feature information of the majority class data can be used to assist the learning of the minority class, thereby solving the problem of intra-class imbalance.

[0088] Step 3.6: The classifier classifies and identifies the input samples based on the old-new fusion features and the head-tail fusion features, and calculates the model loss function based on the classification and identification results and the classification labels;

[0089] In this embodiment, the model loss function includes the distillation loss during the knowledge distillation process. The output of the old-new encoder is fed into the classifier, where the cross-entropy loss is calculated and renamed the batch constraint loss. The output z′ of the head-to-tail encoder is fed into the classifier to calculate the CE loss, which is also known as the cross-entropy loss, and is renamed the classification loss. Meanwhile, in order to maximize the effectiveness of the model, the semantic compensation (SC) loss is calculated to further mitigate forgetting. In imbalanced incremental learning, special attention should be paid to the forgetting of the minority class. Therefore, the SC loss of the majority class and the minority class need to be calculated separately. Since the forgetting problem of the minority class is more serious, the weight of the tail class SC loss is increased.

[0090] Specifically, the model loss function can be expressed as:

[0091]

[0092] In the formula, Represents classification loss, This indicates semantic compensation loss. Indicates batch processing constraint loss. The distillation loss is represented by α, β, γ, and δ, which represent the weights of each type of loss.

[0093] in,

[0094]

[0095]

[0096]

[0097] In the formula, n t D represents the input sample D for the current task. t The number of samples in C old C represents the number of categories of all old classes learned incrementally in stage t. all =C old +C t C represents the number of classes in the incremental learning phase t. t This represents the number of new classes in the incremental learning phase t. Let d represent the weight matrix of the last fully connected layer (i.e., the fully connected layer of the classifier), where d represents the dimension. Let z represent the bias vector of the last fully connected layer, T denote the transpose, |||| denotes the norm operation, and z i y represents the output result of the head-to-tail encoder after the features of the i-th sample are processed by the current task model. i Let x represent the label of the i-th sample. i This represents the i-th sample. This represents the old task model. Indicates the current task model. This represents the semantic compensation loss of the header class. The semantic compensation loss for the tail class is represented by μ1 and μ2 respectively. and The weight.

[0098] The semantic compensation loss can be calculated using the following formula.

[0099]

[0100] In the formula, γ is a non-negative coefficient, and μ represents the average value of the feature vectors for each category.

[0101] It should be noted that the batch processing constraint loss is calculated. With classification loss Similar, the difference is z in i The output of the old-new encoder is the feature output of the i-th sample after passing through the old task model and the current task model.

[0102] Step 3.7: Update the parameters of the current task model, old-new encoder, head-tail encoder, and classifier through backpropagation based on the model loss function.

[0103] It should be noted that after a phase of incremental learning is completed, the current task model obtained is used as the old task model for the next phase of incremental learning.

[0104] For example, taking the incremental learning stage t as an example, the above incremental learning process is as shown in Algorithm 1, and the specific code is as follows:

[0105]

[0106] The algorithm described above is the forward pass process of the I3Net framework, proposed for imbalanced incremental learning. This algorithm takes a mini-batch feature vector z as input and outputs the encoder output z′. The algorithm consists of a loop, from step 1 to step 2. Iteration is performed in the incremental phase (denoted by t).

[0107] At each stage, the algorithm first uses a feature extractor parameterized by θ. The mini-batch feature vector z is applied to the input to obtain a new feature representation z. new Then, the head-to-tail encoder is applied to z. new To explore the feature relationships between the majority and minority classes, the encoder output is z′ new express.

[0108] If the old model is available, use This indicates that the algorithm will use the new feature z.new Compared with old feature z old Connect them to obtain z cat Then, apply the old-new encoder to z. cat The encoder outputs z′ by mining the relationships between the stored old model and the features generated by the current new model. cat This indicates that the algorithm finally returns the encoder output z′. cat The goal of this algorithm is to enhance learning in the minority class and reduce catastrophic forgetting of old data when learning new incoming data.

[0109] Step 4: Use the current task model and classifier after incremental learning to achieve classification and recognition.

[0110] In this embodiment, the classification and recognition of road conditions in autonomous driving can be achieved by using the current task model and classifier after incremental learning.

[0111] It's important to note that the old-to-new encoder and head-to-tail encoder are only used during the incremental learning phase. After model training is complete, the classification and recognition of all types of road conditions in autonomous driving are achieved solely through the current task model and classifier. Since there may be shifts in features before and after using the head-to-tail encoder / old-to-new encoder, a shared classifier, identical to that used in the previous incremental learning phase, can be used to prevent this. During the testing phase, an additional classifier with the same parameters as the classifier after incremental learning can be added.

[0112] This invention presents an imbalanced incremental learning method based on complementary relationship mining. It achieves incremental learning of imbalanced data by constructing an imbalanced incremental learning model. A head-to-tail encoder mines the relationships between current mini-batches, facilitating the extraction of rich feature information between the minority and majority classes. An old-to-new encoder fully utilizes the relationships between output features generated by different models to reduce catastrophic forgetting between tasks. Furthermore, knowledge distillation, semantic loss, and fine-tuning techniques are employed to improve the model's learning performance. This invention provides a novel incremental learning approach for autonomous driving technology, avoiding the safety issues arising from the decreased classification accuracy due to catastrophic forgetting in traditional incremental learning.

[0113] Furthermore, the constructed imbalanced incremental learning model was evaluated on the CIFAR100 dataset.

[0114] 1. Evaluation criteria:

[0115] The standard incremental setup includes 5 tasks (20 new classes per task) and 10 tasks (10 new classes per task).

[0116] In addition, each task was partitioned according to an imbalance rate (IR) of 0.05 (the largest class is 50 times the smallest class) and 0.1 (the largest class is 25 times the smallest class).

[0117] To evaluate the model's performance, the data is divided into majority, intermediate, and minority classes when calculating the highest accuracy. When the imbalance rate is 0.05, classes with more than 350 samples in the training set are considered the majority class, classes with fewer than 40 samples are considered the minority class, and the remaining classes are considered the intermediate class. When the imbalance rate is 0.1, classes with more than 350 samples in the training set are considered the majority class, classes with fewer than 100 samples are considered the minority class, and the remaining classes are considered the intermediate class.

[0118] 2. Experimental parameters

[0119] Following existing incremental learning methods, ResNet18 was used as the feature extractor for all different settings, Adam was used to optimize the network, the batch size was set to 128, and all experiments were trained for 101 epochs. The initial learning rate was set to 0.001 and decreased by a factor of 10 every 45 epochs. α = 1, β = 10, γ = 30, δ = 10, μ1, μ2 = 1.5. During the incremental learning phase, the memory size was set to 2000 to fine-tune the model.

[0120] 3. Experimental Results

[0121] Please see Figure 4 The CIFAR100 dataset shown is divided into five tasks with different imbalance rates, displaying the top-1 accuracy results. Figure 4 As shown in the first row, when the dataset was divided into 5 tasks and the imbalance rate (IR) was set to 0.05, the final task accuracy was 31.7%, an improvement of 5.7% compared to IL2A. Figure 4 In the second line, the incremental learning method of this invention performs well in all 5 tasks, with IR=0.1: the final accuracy is 34.6%, an improvement of 12.6%.

[0122] Please see Figure 5 The CIFAR100 dataset shown is divided into 10 tasks with different imbalance rates, displaying the top-1 accuracy results. Figure 5 In the first row, when set to 10 tasks and IR = 0.05, the final accuracy was 27%, an improvement of 9%. Figure 5In the second row, when set to 10 tasks and IR=0.1, the final result is 31.1%, an improvement of 9.2%. Furthermore, the incremental learning method of this invention focuses only on the accuracy of the minority class, outperforming all comparable methods. Under different settings, the final accuracy of the minority class is improved by an average of 7%, demonstrating that the method of this invention is indeed effective for the minority class. By improving the performance of the minority class, the overall performance is improved. The success of this method benefits from both the old and new encoders and the head and tail encoders.

[0123] Please see Figure 6 The Grad-Cam demonstration shown adds a new-old encoder to a 5-task setup, with a baseline visualization comparison of baselines without and with the new-old encoder at an imbalance rate of 0.05. The first row represents the baseline without any tricks, and the second row represents the baseline with the new-old encoder. This was performed with an IR of 0.05 across all 5 tasks. The same samples on the left are the outputs of the model trained in the second task, and the samples on the right are the outputs of the model trained in the third task. The results show that after adding the proposed new-old encoder, the new and old models pay more similar attention to the same data. This means that mining the relationships between features generated by the new and old models can prevent changes in the old feature space, retain more information from the old model, and thus mitigate catastrophic forgetting.

[0124] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that an article or apparatus comprising a list of elements includes not only those elements but also other elements not expressly listed. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or apparatus that includes said element. Terms such as "connected" or "linked" are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0125] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. An imbalanced incremental learning method based on complementary relationship mining, characterized in that, include: Step 1: Construct an imbalanced incremental learning model, which includes: an old task model, a current task model, feature filters, an old-to-new encoder, a head-to-tail encoder, and a classifier; wherein, The old task model, the feature filter, and the old-new encoder are cascaded in sequence. The output of the current task model is connected to the input of the old-new encoder and the input of the head-tail encoder, respectively. The output terminals of the old-new encoder and the head-tail encoder are both connected to the input terminal of the classifier. Step 2: Construct an imbalanced dataset; Step 3: Input the imbalanced dataset into the imbalanced incremental learning model for incremental learning; wherein, during the incremental learning process, knowledge distillation is used to extract the knowledge contained in the old task model into the current task model; a head-to-tail encoder is used to mine the relationship between the majority and minority class data in the imbalanced dataset, and an old-to-new encoder is used to mine the relationship between the features generated by the old task model and the current task model; Step 3 includes: Step 3.1: Input the road condition scene images in the imbalanced dataset into the imbalanced incremental learning model according to their categories; Step 3.2: The old task model extracts features from the input samples to obtain old model features, and the current task model extracts features from the input samples to obtain new model features; Step 3.3: The feature filter performs feature filtering on the old model features to obtain key features; Step 3.3 includes: For each category of road condition scene images, the mean value of the old model features is calculated; the distance between the old model features and the mean value of the features in the corresponding category of road condition scene images is calculated; based on a preset threshold, features in the old model features whose distance exceeds the threshold are removed to obtain the corresponding key features. Step 3.4: The old-new encoder performs feature fusion on the new model features and the key features to obtain the old-new fused features; Step 3.5: The head-tail encoder performs feature fusion on the currently input new model features and the previously input new model features to obtain head-tail fused features; Step 3.6: The classifier classifies and identifies the input samples based on the old-new fusion features and the head-tail fusion features, and calculates the model loss function based on the classification and identification results and the classification labels; Step 3.7: Based on the model loss function, update the parameters of the current task model, old-new encoder, head-tail encoder, and classifier through backpropagation; Step 4: Use the current task model and classifier after incremental learning to achieve classification and recognition.

2. The imbalanced incremental learning method based on complementary relationship mining according to claim 1, characterized in that, Both the old task model and the current task model are ResNet18 networks.

3. The imbalanced incremental learning method based on complementary relationship mining according to claim 1, characterized in that, The old-new encoder and the head-to-tail encoder have the same structure, both including a first normalization layer, a self-attention layer, a second normalization layer, and a multilayer perceptron. The first normalization layer, the self-attention layer, the second normalization layer, and the multilayer perceptron are cascaded in sequence; The input of the first normalization layer is fused with the output of the self-attention layer and used as the input of the second normalization layer; The input of the second normalization layer is fused with the output of the multilayer perceptron and used as the output of the encoder.

4. The imbalanced incremental learning method based on complementary relationship mining according to claim 1, characterized in that, The imbalanced dataset includes road condition scene images with category labels for multiple categories, and the distribution of the number of road condition scene images for all categories conforms to the long-tail distribution characteristic.

5. The imbalanced incremental learning method based on complementary relationship mining according to claim 1, characterized in that, The model loss function is expressed as follows: ； In the formula, Represents classification loss, This indicates semantic compensation loss. Indicates batch processing constraint loss. Indicates distillation loss, , , , These represent the weights of various loss types; among them, ; ； ； In the formula, This represents the input sample for the current task. The number of samples in Indicates the first t The number of categories in all old classes during incremental learning. Indicates the first t The number of classes in all classes during incremental learning. Indicates the first t The number of new classes in incremental learning. This represents the weight matrix of the last fully connected layer. Represents dimension. This represents the bias vector of the last fully connected layer, where T represents the transpose. Represents norm operations, Indicates the first i The features output by each sample after passing through the current task model Indicates the first i The label of each sample Indicates the first i One sample, This represents the old task model. Indicates the current task model. This represents the semantic compensation loss of the header class. This represents the semantic compensation loss of the tail class. and They represent and The weight.

Images