Online sustainable evolution method based on mutual learning and data enhancement
By adopting an online continuous evolution method based on mutual learning and data augmentation in autonomous unmanned systems, the problem that autonomous unmanned system models are difficult to update quickly when new object categories appear and tasks change is solved, and the model's rapid adaptation and generalization ability improvement under single-cycle training is achieved.
Patent Information
- Application Number
- CN202510059584.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-15
- Publication Date
- 2025-05-16
AI Technical Summary
The perception and decision-making algorithms of existing autonomous unmanned systems are fixed after leaving the factory and are difficult to update dynamically, resulting in the difficulty of quickly updating the model when new object categories appear and tasks change, and it is easy to learn biased non-core features, limiting its performance and application.
Using an online continuous evolution method based on mutual learning and data augmentation, we create two models with the same structure but different parameters initialized, and combine data augmentation and mutual learning mechanisms to achieve rapid update and generalization capabilities of the model under single-cycle training.
It effectively avoids the problem of partial feature learning in single-cycle training, significantly improves the learning ability and performance of the model in the online continuous recognition scenario, and can quickly adapt to new tasks and retain old knowledge.
Smart Images
Figure CN120012865A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to technical fields such as image recognition, continuous learning, artificial intelligence, and more particularly to an online continuous evolution method based on mutual learning and data enhancement. Background Art
[0002] In recent years, autonomous unmanned systems represented by drones have shown great application value in many fields. However, the perception and decision-making algorithms of autonomous unmanned systems are fixed after leaving the factory and are difficult to update dynamically. However, in actual application environments, new object categories continue to emerge, and coupled with changes in tasks, the models of autonomous unmanned systems need to be dynamically updated to increase the ability to recognize new objects while not forgetting the knowledge of old tasks. However, due to the contradiction between storage resources and massive data, it is difficult to save the data of new tasks for model optimization for multiple training cycles, and it is necessary to achieve rapid model updates while accessing the data only once.
[0003] Online continuous evolution for image recognition is an important direction to solve this problem. It aims to sequentially learn new tasks while retaining the old knowledge, and only needs to access new data once. Compared with offline continuous learning, each data in online continuous image recognition can only be accessed once, which is more challenging. Moreover, online continuous evolution for image recognition is of great significance for autonomous unmanned equipment to quickly adapt to complex and changing environments. Therefore, it has important application significance in drones, unmanned vehicles, robots, etc.
[0004] In this field, methods based on sample subset replay and its variants have performed well and dominated. These methods save some samples of old classes for the current task through a small memory buffer and replay the cached data when learning a new task to alleviate catastrophic forgetting. Experience Replay (ER) is the first method to combine streaming data and cached samples in online continuous image recognition. It achieves replay by randomly retrieving samples from the cache. Subsequent studies are mostly based on variants of experience replay, focusing on the design of memory update strategies, sample retrieval strategies, and effective use of streaming data and cached data (for example, designing specific classifiers, loss functions, and model architectures). However, most of these methods focus on alleviating catastrophic forgetting and improving model plasticity, and generally ignore the potential problems caused by single-cycle training. Specifically, due to the limitations of single-cycle training, the model tends to learn biased non-core features, which may be sufficient to distinguish the categories currently seen. However, when new categories with similar features appear, these biased non-core features are not enough to ensure that the model outputs accurate results, thus limiting its performance and practical application.
[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0006] The present invention provides an online continuous evolution method based on mutual learning and data enhancement to solve the problems that the existing methods perform poorly, are prone to learning biased non-core features, and are limited in application under the limitation of single-cycle training.
[0007] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.
[0008] According to a first aspect of the present invention, there is provided an online continuous evolution method based on mutual learning and data enhancement, the method comprising:
[0009] Create a first model and a second model with the same structure but initialized with different parameters;
[0010] Extract a batch of samples from the training data stream of the tth task in turn, and randomly retrieve a memory batch of samples from the sample buffer, merge the two types of sample data obtained, and obtain merged batch sample data;
[0011] Perform three different enhancement processes on the merged batch sample data to obtain three types of enhanced data; combine the three types of enhanced data with the merged batch samples respectively to obtain three groups of combined data;
[0012] Input the three sets of combined data into the first model and the second model respectively, extract the first feature and the first probability distribution of the first model, and the second feature and the second probability distribution of the second model;
[0013] calculating a first distillation loss and a second distillation loss based on the first probability distribution and the second probability distribution;
[0014] Calculate a first proxy contrast loss based on the first feature, and calculate a second proxy contrast loss based on the second feature;
[0015] Calculating a first mutual contrast loss and a second mutual contrast loss based on the first feature and the second feature;
[0016] A first final loss is obtained based on the first distillation loss, the first proxy contrast loss and the first mutual contrast loss; a second final loss is obtained based on the second distillation loss, the second proxy contrast loss and the second mutual contrast loss;
[0017] Updating the first model based on the first final loss, and updating the second model based on the second final loss;
[0018] Use the reservoir update algorithm to randomly update the sample buffer with the extracted batch samples;
[0019] In the inference stage, the test sample is input into the first model and the second model, and their respective probability distributions are calculated. The final predicted probability is the average of the two probabilities.
[0020] In some exemplary embodiments, the first model and the second model are image classification models.
[0021] In some exemplary embodiments, the three different enhancement processes include:
[0022] The first enhancement process: random cropping, random horizontal flipping, color jittering and random grayscale;
[0023] The second enhancement process: large-angle rotation enhancement;
[0024] The third enhancement treatment: two random enhancements.
[0025] In some exemplary embodiments, the large angle rotation enhancement includes: rotation of 90 degrees and -90 degrees.
[0026] In some exemplary embodiments, the first distillation loss and the second distillation loss are calculated as follows:
[0027]
[0028] in, For collection The mth element in , D(·∥·) is the Kullback-Leibler divergence,
[0029] In some exemplary embodiments, the first proxy contrast loss and the second proxy contrast loss are calculated as follows:
[0030]
[0031] in, and Respectively The features obtained by the first model and the second model are and is the weight corresponding to label y in each classifier, yes Class index in β m Represents the set {β w ,β rot ,β ra The mth element in} is used to balance the effects of various data enhancements.
[0032] In some exemplary embodiments, the first mutual contrast loss is calculated as follows:
[0033] set up Represents the feature map of the lth layer, Ind is the index set of the selected layer, and the multi-layer feature set is defined as in, The first mutual comparison loss It is expressed as follows:
[0034]
[0035] in, Representation sample In the feature map at layer l, Representation and The feature set of other samples with the same category, A coefficient for balancing the influence of features at different layers.
[0036] In some exemplary embodiments, obtaining the first final loss based on the first distillation loss, the first proxy contrast loss, and the first mutual contrast loss comprises:
[0037] The first distillation loss, the first proxy contrast loss, and the first mutual contrast loss are added to obtain a first final loss.
[0038] In some exemplary embodiments, obtaining the second final loss based on the second distillation loss, the second proxy contrast loss, and the second mutual contrast loss comprises:
[0039] The second distillation loss, the second proxy contrast loss, and the second mutual contrast loss are added together to obtain a second final loss.
[0040] According to a second aspect of the present invention, an application of an online continuous evolution method based on mutual learning and data enhancement is provided, wherein the method is used for continuous online image recognition of unmanned aerial vehicles, and the training data stream is ground detection images of unmanned aerial vehicles.
[0041] The online continuous evolution method based on mutual learning and data enhancement provided by the present invention provides an effective solution to the key problems in online continuous evolution through mutual learning mechanism and data enhancement strategy. First, a mutual learning mechanism is designed to construct two sub-models during the training process. By guiding the alignment of features and distributions between different sub-models, the interaction between models is effectively captured, the co-evolution of models is promoted, and the limitations of single model learning are significantly reduced. In addition, the data feature distribution is expanded by three data enhancement strategies: default data enhancement, large-angle rotation enhancement, and two random enhancements, which significantly improves the generalization ability and feature learning ability of the model under single-cycle training. Through the combination of mutual learning mechanism and data enhancement strategy, the present invention enables the model to quickly adapt to new tasks and avoid biased feature learning problems generated in single-cycle training under the condition of accessing data only once, significantly enhancing the learning ability and performance of the model in online continuous recognition scenarios. In short, the online continuous evolution method based on mutual learning and data enhancement proposed by the present invention, compared with the traditional continuous evolution method, shows significant advantages in improving model generalization, alleviating catastrophic forgetting, and adapting to single-cycle training, and provides technical support and reliable guarantee for mission scenarios such as drone ground detection that require real-time continuous updates.
[0042] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present invention, and together with the specification are used to explain the principles of the present invention. Obviously, the accompanying drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work.
[0044] Figure 1 The overall framework of an online continuous evolution method based on mutual learning and data enhancement is schematically shown;
[0045] Figure 2 A flowchart of an online continuous evolution method based on mutual learning and data enhancement according to an exemplary embodiment of the present invention is schematically shown. DETAILED DESCRIPTION
[0046] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present invention will be more comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0047] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.
[0048] This example implementation provides an online continuous evolution method based on mutual learning and data enhancement, refer to Figure 2 As shown, the following steps may be specifically included:
[0049] Create a first model and a second model with the same structure but initialized with different parameters;
[0050] Extract a batch of samples from the training data stream of the tth task in turn, and randomly retrieve a memory batch of samples from the sample buffer, merge the two types of sample data obtained, and obtain merged batch sample data;
[0051] Perform three different enhancement processes on the merged batch sample data to obtain three types of enhanced data; combine the three types of enhanced data with the merged batch samples respectively to obtain three groups of combined data;
[0052] Input the three sets of combined data into the first model and the second model respectively, extract the first feature and the first probability distribution of the first model, and the second feature and the second probability distribution of the second model;
[0053] calculating a first distillation loss and a second distillation loss based on the first probability distribution and the second probability distribution;
[0054] Calculate a first proxy contrast loss based on the first feature, and calculate a second proxy contrast loss based on the second feature;
[0055] Calculating a first mutual contrast loss and a second mutual contrast loss based on the first feature and the second feature;
[0056] A first final loss is obtained based on the first distillation loss, the first proxy contrast loss and the first mutual contrast loss; a second final loss is obtained based on the second distillation loss, the second proxy contrast loss and the second mutual contrast loss;
[0057] Updating the first model based on the first final loss, and updating the second model based on the second final loss;
[0058] Use the reservoir update algorithm to randomly update the sample buffer with the extracted batch samples;
[0059] In the inference stage, the test sample is input into the first model and the second model, and their respective probability distributions are calculated. The final predicted probability is the average of the two probabilities.
[0060] In the following, the steps of the online continuous evolution method based on mutual learning and data enhancement in this example implementation are described in more detail with reference to the accompanying drawings and embodiments. This example implementation provides an online continuous evolution method based on mutual learning and data enhancement for continuous online image recognition of drones.
[0061] Step 1: Create two models with the same structure but initialized with different parameters, denoted as the first model and the second model.
[0062] Exemplarily, the first model includes a first feature extractor and a first classifier; wherein the first feature extractor part is f(·; Θ 1 ), the first weight of the fully connected layer of the first classifier is w 1 ;
[0063] Exemplarily, the second model includes a second feature extractor and a second classifier; wherein the second feature extractor part is f(·; Θ 2 ), the second weight of the fully connected layer of the second classifier is w 2 ;
[0064] Step 2: Training data flow for the tth task From data stream D t Draw a batch of samples from And randomly retrieve a memory batch sample from the sample buffer M The two types of data obtained are merged to obtain the merged batch sample B. Among them, Represents the nth sample, and its corresponding label is N t is the total number of all samples in the tth task.
[0065] Exemplarily, the training data stream is a drone ground detection image.
[0066] Step 3: For the batch data B obtained in step 2, use default data augmentation, large angle rotation augmentation, and two random augmentations (RandAugment) to obtain the enhanced data respectively
[0067] Exemplarily, the default data augmentation includes random cropping, random horizontal flipping, color jittering, and random grayscale;
[0068] Exemplarily, large angle rotation is enhanced to rotation of 90 degrees and -90 degrees.
[0069] Step 4: Combine the three enhanced data obtained in step 3 with the original data, respectively, and record them as and
[0070] Step 5: Input the three sets of data obtained in step 4 into the first model and the second model respectively, and extract the first feature z of the first model 1 , the first probability distribution p 1 , and the second characteristic z of the second model 2 , the second probability distribution p 2 ;
[0071] Let C t represents the set of learned categories, and denote the weights of category c in the classifiers of the first model and the second model respectively, and τ is a temperature hyperparameter. For any sample, the probability prediction calculation formula for its belonging to category c is as follows:
[0072]
[0073]
[0074] in, and are the probability predictions of category c in the first and second models, respectively.
[0075] Step 6: Based on the first probability distribution p 1 and the second probability distribution p 2 , calculate the first distillation loss and second distillation losses
[0076] Specifically, according to the formula definition in step 5, the first distillation loss in step 6 is and second distillation losses The calculation method is:
[0077]
[0078] in, For collection The mth element in , D(·∥·) is the Kullback-Leibler divergence, Note that the original data We calculate it only once, using Calculate the data in .
[0079] Step 7: Based on the first feature z 1 With the first weight w 1 , calculate the first proxy contrast loss Based on the second feature z 2 With the second weight w 2 Compute the contrastive loss for the second agent
[0080] Specifically, the first-proxy contrast loss and the second-proxy contrast loss are calculated as:
[0081]
[0082] in, and Respectively The first feature obtained by the first model and the second feature obtained by the second model, and is the weight corresponding to label y in each classifier, yes The class index in β. m Represents the set {β w ,β rot ,β ra} is used to balance the effects of each data enhancement. Also note that the original data We calculate it only once, using Calculate the data in .
[0083] Step 8: Based on the first feature z 1 With the second feature z 2 Calculate the first mutual contrast loss and the second mutual contrast loss
[0084] Specifically, if Represents the feature map of the lth layer, Ind is the index set of the selected layer, and the multi-layer feature set is defined as in, The first mutual comparison loss It is expressed as follows:
[0085]
[0086] in, Representation sample In the feature map at layer l, Representation and The feature set of other samples with the same category, To balance the coefficients of the influence of features at different layers, β m Same as the definition in step 7. Note that It contains the features obtained by inputting the original sample into the first model and the features obtained by inputting the enhanced sample into the second model. The opposite is true. Contains the features obtained by inputting the original sample into the second model and the features obtained by inputting the enhanced sample into the first model.
[0087] Step 9: Add the losses obtained in the above steps to get the first final loss and the second final loss And update the model through stochastic gradient descent;
[0088] Step 10: Next, use the reservoir update algorithm to update batch B i The samples in are randomly updated to the sample buffer middle;
[0089] Step 11: In the inference phase, the test samples are input into the two models, and their respective probability distributions are calculated. The final predicted probability is the average of the two probabilities.
[0090] Other embodiments of the invention will readily occur to those skilled in the art after considering the specification and practicing the invention herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.
[0091] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
Claims
1. An online continuous evolution method based on mutual learning and data enhancement, characterized in that: The method comprises: Create a first model and a second model with the same structure but initialized with different parameters; Extract a batch of samples from the training data stream of the tth task in turn, and randomly retrieve a memory batch of samples from the sample buffer, merge the two types of sample data obtained, and obtain merged batch sample data; Perform three different enhancement processes on the merged batch sample data to obtain three types of enhanced data; combine the three types of enhanced data with the merged batch samples respectively to obtain three groups of combined data; Input the three sets of combined data into the first model and the second model respectively, extract the first feature and the first probability distribution of the first model, and the second feature and the second probability distribution of the second model; calculating a first distillation loss and a second distillation loss based on the first probability distribution and the second probability distribution; Calculate a first proxy contrast loss based on the first feature, and calculate a second proxy contrast loss based on the second feature; Calculating a first mutual contrast loss and a second mutual contrast loss based on the first feature and the second feature; A first final loss is obtained based on the first distillation loss, the first proxy contrast loss and the first mutual contrast loss; a second final loss is obtained based on the second distillation loss, the second proxy contrast loss and the second mutual contrast loss; Updating the first model based on the first final loss, and updating the second model based on the second final loss; Use the reservoir update algorithm to randomly update the sample buffer with the extracted batch samples; In the inference stage, the test sample is input into the first model and the second model, and their respective probability distributions are calculated. The final predicted probability is the average of the two probabilities.
2. The online continuous evolution method based on mutual learning and data enhancement according to claim 1, characterized in that: The first model and the second model are image classification models.
3. The online continuous evolution method based on mutual learning and data enhancement according to claim 1, characterized in that: The three different enhancement processes include: The first enhancement process: random cropping, random horizontal flipping, color jittering and random grayscale; The second enhancement process: large-angle rotation enhancement; The third enhancement treatment: two random enhancements.
4. The online continuous evolution method based on mutual learning and data enhancement according to claim 3 is characterized in that: The large-angle rotation enhancement includes: rotation of 90 degrees and -90 degrees.
5. The online continuous evolution method based on mutual learning and data enhancement according to claim 1, characterized in that: The first distillation loss and the second distillation loss are calculated as follows: in, is the mth element in the three combined data sets, D(·∥·) is the Kullback-Leibler divergence, 6. The online continuous evolution method based on mutual learning and data enhancement according to claim 1, characterized in that: The first proxy contrast loss and the second proxy contrast loss are calculated as: in, and Respectively The features obtained by the first model and the second model are and is the weight corresponding to label y in each classifier, yes Class index in β m Represents the set {β w ,β rot ,β ra The mth element in} is used to balance the effects of various data enhancements.
7. The online continuous evolution method based on mutual learning and data enhancement according to claim 1, characterized in that: The first mutual comparison loss is calculated as follows: set up It represents the feature map of the lth layer of the first model, Ind is the index set of the selected layer, and the multi-layer feature set is defined as in, The first mutual comparison loss It is expressed as follows: in, Representation sample In the feature map at layer l, Representation and The feature set of other samples with the same category, A coefficient for balancing the influence of features at different layers.
8. The online continuous evolution method based on mutual learning and data enhancement according to claim 1, characterized in that: The obtaining of the first final loss based on the first distillation loss, the first proxy contrast loss and the first mutual contrast loss comprises: The first distillation loss, the first proxy contrast loss, and the first mutual contrast loss are added to obtain a first final loss.
9. The online continuous evolution method based on mutual learning and data enhancement according to claim 1, characterized in that: The obtaining of the second final loss based on the second distillation loss, the second proxy contrast loss and the second mutual contrast loss comprises: The second distillation loss, the second proxy contrast loss, and the second mutual contrast loss are added together to obtain a second final loss.
10. An application of an online continuous evolution method based on mutual learning and data enhancement, characterized in that: The method is used for continuous online recognition of unmanned aerial vehicle images, and the training data stream is the unmanned aerial vehicle ground detection image.