A machine learning method for multi-agent online evolutionary learning

Abstract

The present invention belongs to the field of machine learning technology, and is specifically a machine learning method for multi-agent online evolutionary learning. The method of the present invention includes: in an OEL scenario, in a local area, multiple agent models acquire perception data in real time, and continuously learn and update the agent model in combination with a small amount of labeled data; multiple agent models realize knowledge transfer through interactive learning; each agent model is facing a large amount of unlabeled non-iid perception data, and these perception data are effectively jointly processed in combination with the own capabilities of multiple agent models; wherein a mutual matching (MM) algorithm is used for knowledge sharing and continuous learning between models to stabilize the generalization ability of the model and ensure that each agent obtains better performance improvement in the OEL environment. The method of the present invention can reduce a large amount of data annotation requirements and improve the performance of the entire perception system by using knowledge sharing of multiple models.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning, and in particular relates to a machine learning method for multi-agent online evolutionary learning. Background Art

[0002] Semi-supervised learning (SSL) can use a large amount of unlabeled data for self-training and continuous evolution based on a small amount of labeled data. This feature makes SSL a potentially effective method for processing constantly changing real-time perception data. In the real-time perception environment of edge computing, the constantly changing perception data streams from the real world have the characteristics of non-independent and identically distributed (non-iid). Based on its existing knowledge, machine learning models use multi-model interactive discrimination and self-training methods to effectively utilize a large amount of newly added perception data to promote real-time tuning of each model. This type of method is called linear evolutionary learning. For example, in typical modern Internet of Things (IoT) applications, the multi-agent SSL architecture can help solve the problem of effectively utilizing a large amount of unlabeled non-iid data by sharing knowledge between models, thereby achieving the problem of multi-agent model adaptation and real-time generalization.

[0003] Semi-supervised learning (SSL) algorithms based on deep neural networks have achieved excellent results in multiple database challenge tasks. These tasks mainly study how to use unlabeled data with the same distribution for model training when there is limited labeled data. However, most advanced deep learning applications today rely on large labeled datasets to train effective machine learning models. These methods rely on huge computing power and constantly updated data annotations to ensure the generalization ability of the model. This results in poor flexibility in various applications based on machine learning models and the inability to update and tune the models in real time based on perceptual data. Compared with fully supervised methods, SSL models can be trained autonomously using a small amount of annotated data while obtaining competitive or better results, making SSL an effective method for autonomous model learning under evolving and unstable data streams.

[0004] With the development of edge intelligence technology, various edge devices with independent communication and computing capabilities are increasing, making the interaction and training of edge models possible. However, in real-world environments, the large amount of unlabeled data obtained by edge devices in real time is usually unstable, and the distribution of these data is highly time-varying. In order to maintain the generalization ability of the model in a constantly changing data environment to provide customized recognition services, the edge model needs to continuously optimize performance and generalization through collaborative judgment of multiple models when the data stream is not independent and identically distributed. Online evolutionary learning (OEL) is such a machine learning scenario. Without relying on a unified cloud model, the edge model can be trained online autonomously through real-time discrimination information interaction.

[0005] Although SSL-based edge models can be applied to OEL systems, most current advanced SSL methods do not fully consider the problem of learning with non-iid data streams. At the same time, existing methods often use pseudo-labeling and consistency regularization techniques as key components and integrate more complex data processing processes to better complete different tasks. This makes it difficult for general SSL-based models to be widely deployed and may lead to performance degradation in perception environments where data distribution changes frequently. Summary of the invention

[0006] The purpose of the present invention is to provide a machine learning method for multi-agent online evolutionary learning to overcome the shortcomings of edge models deployed in existing OEL systems.

[0007] The present invention studies the multi-agent online evolutionary SSL mechanism, which can effectively utilize the information interaction between edge models and the additional knowledge in the training process to ensure that the model obtains stable performance improvement and good generalization ability when facing non-iid unlabeled data streams. Accordingly, the present invention proposes an online evolutionary SSL algorithm based on interactive learning, soft-supervised consistency regularization and unsupervised sample mining, called the mutual matching algorithm (MutualMatch, MM), abbreviated as MM algorithm. By making full use of the additional knowledge brought by SSL and the interactive collaboration between models during the training process, MM exceeds multiple top SSL algorithms in terms of model accuracy and convergence efficiency under the same online evolution experimental scenario setting. MM simplifies the complexity of model design and follows a unified and easily scalable architecture, which is conducive to processing application tasks with insufficient labeled data and frequently changing data distribution.

[0008] The machine learning method for multi-agent online evolutionary learning provided by the present invention has a process as shown in Figure 1As shown. In the OEL scenario, in a local area, multiple edge agent models acquire perception data in real time, and continuously learn and update the agent model in combination with a small amount of labeled data. Multiple agent models can achieve knowledge transfer through interactive learning. Each agent model is facing a large amount of unlabeled non-iid perception data. How to combine the capabilities of multiple agent models to effectively jointly process these perception data is the key to solving OEL. The specific steps are as follows:

[0009] 1. Defining and establishing an OEL environment

[0010] Problem definition: Given a classification task, for each agent model, let Represents a batch of B labeled data, where x i is the training sample, y i is the corresponding label. is a batch of μB unlabeled samples, where μ is a hyperparameter, which indicates the ratio of unlabeled training samples to labeled training samples in each iteration. The amount of data compared to The amount is small, and Included in part Non-IID data. Labeled and unlabeled data are sent to the model at the same time in each training step; each batch of unlabeled data is used only once, representing a continuously updated perceptual environment, thus forming a simulation of the process of online evolutionary learning. Represented as a model This is a general formula for expressing the predicted class distribution y for input x.

[0011] The intelligent agent model is an image recognition CNN model based on deep learning. The model is deployed on the smart device at the edge. Based on the perception and collection of environmental data by the smart device, through the interaction between multiple intelligent agent models, the data is fully utilized to learn and update each model, so as to adapt to environmental changes.

[0012] (ii) A mutual matching (MM) algorithm is used for knowledge sharing and continuous learning between models to stabilize the generalization ability of the model and ensure that each agent obtains better performance improvement in the OEL environment; specifically, the SSL process for each model includes the following steps:

[0013] (1) Soft-supervisory consistency regularization processing: that is, using the unlabeled data obtained by each perception, it is subjected to soft-supervisory consistency regularization, and the soft-supervisory information predicted by the model is used to calculate the unsupervised loss, so that each model can obtain more category-related information;

[0014] (2) Adopting a dual-model interactive learning method to achieve knowledge sharing among multiple models;

[0015] (3) Introducing unsupervised sample mining methods, by statistically analyzing and evaluating some high-confidence unlabeled data, and expanding them into labeled data sets, each intelligent agent can use the perception data more efficiently, which is conducive to the better evolution of the entire system and improves the utilization rate of confidence samples;

[0016] The training process is as follows Figure 2 As shown in Figure 2, MM is an alternating training process that includes two training paths: supervised and unsupervised. Depending on the path, different enhancement methods are used to predict the labeled / unlabeled data to calculate the supervised / unsupervised loss.

[0017] Specifically:

[0018] (1) Soft supervision consistency regularization processing, the specific process is as follows:

[0019] In classification tasks, cross entropy is usually used to measure the difference between the predicted value and the target probability distribution, and is used as a loss function to guide model training. Let H(p,q) = -∑plog(q) represent the cross entropy loss of probability distribution p and q, and the model For example, is a batch of labeled and unlabeled training data, then the model The supervised loss can be expressed as:

[0020]

[0021] in, Indicates that a moderate image enhancement method is used for the labeled samples. yes by is the probability distribution of the input output, y i is x i The corresponding real annotation.

[0022] For the unsupervised loss, let: As a model For the weakly enhanced unlabeled data u i The prediction results are filtered:

[0023]

[0024] in, is the confidence threshold, argwhere is the prediction result r i Extract the function of the prediction results greater than the threshold, so as to obtain the direct probability distribution prediction r i (uns).

[0025] Therefore, the unsupervised loss can be expressed as:

[0026]

[0027] in, Indicates that unlabeled data is strongly enhanced. This formula is based on r i (uns) The consistency regularization process of cross entropy calculation for the prediction of strongly enhanced data is performed for labels.

[0028] (2) A dual-model interactive learning method is used to achieve knowledge sharing among multiple models. The specific process is as follows:

[0029] In the multi-model interactive learning stage, the Kullback-Leibler (KL) Divergence is used to measure the difference in predicted probability distribution between models, and is used as interactive information to guide the alternating training of models.

[0030] The present invention adds KL loss to the training of the supervised part and the unsupervised part, so that the model generates information interaction during the alternating training process. The unsupervised KL loss enables the model to effectively utilize the results of a part of the prediction distribution of unlabeled data that is less than the confidence threshold, and provides additional regularization constraints. We believe that in the face of OEL scenarios, this interactive training can accelerate model convergence and improve the generalization ability of the model.

[0031] First, the KL divergence formula for approximating two probability distributions p and q can be expressed as:

[0032]

[0033] like Figure 2 As shown in Figure 1, when dealing with the multi-model KL loss part under the unsupervised path, the interaction of two models is taken as an example.

[0034] First, in the model When calculating the loss of As a model Prediction results of weakly enhanced data and its use Filter and filter the results As a model The KL loss target is used to make the model The prediction results and Therefore, the model The interaction loss can be expressed as For the model The same method is used to transform the model The prediction of the target calculation model The interaction loss.

[0035] Therefore, when taking the interactive learning of two models as an example, the loss function of the overall online evolutionary learning system can be expressed as:

[0036]

[0037]

[0038] in, and Represent the overall loss function of the two models respectively. The two models will continue to be trained alternately, and the training objectives depend on the probability distribution predicted by the other party. u and λ kl are the loss weights of the unsupervised part and the interaction loss part, respectively.

[0039] (3) Introduce unsupervised sample mining methods to improve the utilization rate of confidence samples. The specific process is as follows:

[0040] Build an index counter list for the unlabeled dataset. This list contains the index of each data in the entire unlabeled dataset and the frequency counter of the data to be counted. For the unlabeled data that meets the model judgment condition, its counting frequency will be increased by 1 accordingly. When the counting frequency of a certain data reaches the index value δ, it can be used to execute subsequent functions. This list collects high-confidence prediction results in each iteration of model training and calculates its frequency of occurrence. If the frequency of some unlabeled samples reaches the index value δ, these unlabeled samples are expanded to the labeled dataset, thereby expanding the labeled dataset for calculating the supervision loss. Depending on the complexity of the training task and the size of the dataset, the index value δ is usually set to a scalar between 8-12. In particular, if the number of occurrences of a certain unlabeled data in the counter list has a high drop in confidence when it does not reach δ, the frequency count of the data in the table will be cleared and re-counted.

[0041] Since the present invention uses soft supervision loss in the consistency regularization part, while providing more information about the correlation between categories, it adds a certain uncertainty to the interactive training process, which may cause the training error to fluctuate around the local minimum. As the interactive training continues to iterate, the predictions of each model for some unlabeled samples will gradually stabilize, and some soft labels with higher confidence will present a more certain probability distribution. In order to better utilize these stable prediction samples to maintain training stability, the present invention uses the above-mentioned unsupervised sample mining process to one-hot represent these soft labels, thereby achieving the expansion of the labeled data set, so that the model can obtain more strong supervision information during the discrimination process.

[0042] In summary, the present invention first proposes an application scenario close to reality, which is an online evolutionary learning environment that is conducive to the widespread application of machine learning models. This environment is based on semi-supervised learning, and uses multiple agents to acquire and interactively process real-time changing non-iid data, so that each agent model can continuously adapt to changes in the environment and obtain common performance improvements. Compared with traditional semi-supervised learning methods that rely on fixed training data and have poor support for non-iid data, the present invention realizes real-time discrimination information sharing and interaction of multi-agent models through the proposed soft-supervised consistency regularization, multi-model real-time interactive learning and unsupervised sample mining methods, so as to ensure the stable generalization of multi-agent models in the online evolutionary learning environment.

[0043] The method of the present invention is more suitable for the deployment and application of actual Internet of Things multi-agent models. In real scenarios, the use of the method of the present invention can reduce a large amount of data annotation requirements and improve the performance of the entire perception system by utilizing knowledge sharing of multiple models. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a conceptual diagram of the semi-supervised learning of the present invention in an online evolutionary learning scenario.

[0045] Figure 2 This is the training process of the embodiment of the present invention. DETAILED DESCRIPTION

[0046] The method of the present invention is further described below through specific examples. In addition, the method of the present invention is compared with other related algorithm effects. We unified the experimental environment and re-implemented various comparison methods. The programming environment mainly uses Keras with TensorFlow as the backend.

[0047] Dataset setting: The general semi-supervised learning algorithm selects part of the labeled data from the dataset as the supervised training set, and removes the labels from the remaining data as the unlabeled training set. In order to simulate the frequently changing perceptual environment with different data distributions, this paper introduces more non-IID data into the unlabeled training set, so that the model learning process has the same data perturbations with different feature distributions as in the real scene. Specifically, we add 10% of non-IID data to the unlabeled part of each dataset.

[0048] Model structure: Since WideResNet has performed relatively stably in a large number of research experiments, we use WRN as the basic model structure and configure different variants for experiments under different databases.

[0049] Model parameter initialization:

[0050] In terms of parameter initialization, the two models used for interactive training are equipped with different parameter initialization methods. Specifically, the two models are initialized using the henormal and lecununiform methods respectively. A weight decay coefficient of 0.0005 is uniformly set for all models, and dropout is not used in any model.

[0051] Optimizer settings:

[0052] The standard SGD optimizer is used with momentum β = 0.9, and the initial learning rate η = 0.03 is used for all models with the cosine learning rate decay method.

[0053] The specific hyper parameters are as follows:

[0054]

[0055] Since different deep learning frameworks have slightly different model configurations, in order to achieve a complete control experiment, all training was performed under our experimental settings. We performed K=2 for each group of experiments. 18 We conducted multiple rounds of OEL experiments on the cifar10 / 100, SVHN, and tiny-imagenet databases based on different groups and different amounts of labeled data to evaluate the performance of Mutual Match.

[0056] The specific results of the experimental comparison (final model accuracy) of the method of the present invention and other SSL methods under multiple different data sets are shown in the following table:

[0057]

[0058]

[0059] Take the experimental results of CIFAR-10 and SVHN databases as an example:

[0060] Both CIFAR-10 and SVHN databases contain 10-class images with a resolution of 32×32, trained using the same WideResNet-28-2 model. For CIFAR-10, 50,000 training images and 10,000 test images are used. For SVHN, 73,357 training images and 26,032 test images are used. In addition, 10% non-iid data is added to the unlabeled part of each dataset to simulate real-life OEL scenarios.

[0061] In each experiment, two matching models were established, and a simulated online learning environment was constructed using unlabeled data, so that the model was in a batch learning process of continuously acquiring training data. Three groups of three different annotated images were randomly selected in the training set, and all methods were trained and evaluated three times. The average and standard deviation of the accuracy of the three training groups are reported in the above table to more accurately reflect the performance advantages of the method of the present invention.

[0062] The convergence efficiency and accuracy of the method of the present invention are higher than those of other comparative methods. In terms of the utilization of unlabeled perceptual data, the effective pseudo-labeled data generated by MM can gradually increase with the number of training rounds, and the final average accuracy of the discrimination of unlabeled data can reach about 97.5% on both data sets, which means that through multi-model knowledge sharing and interactive learning, most perceptual images can be effectively utilized and help model evolution.

[0063] Controlled variable experiment:

[0064] The overall accuracy of the model using soft-supervised consistency regularization is always higher than the conventional version, with only a slight lag in early convergence speed. Overall, soft-supervised consistency regularization can bring about a 1% performance improvement to the model.

[0065] In MM, the prediction results of filtered weakly enhanced unlabeled data are mainly used to calculate the interaction loss between models. The interaction loss plays an important role in accelerating model convergence and improving model discrimination. We tried to use the output of strongly enhanced data to calculate the interaction loss and compared it with our final method. The experimental results show that calculating the interaction loss with the output of strongly enhanced data will lead to a decrease in model performance. This is because the prediction results of strongly enhanced data are less reliable, which is equivalent to calculating the KL loss directly on the prediction results. The conflict between the two loss targets may lead to a weakening of the final performance.

[0066] We also experimented with the impact of the unsupervised sample mining process on model performance. Compared with the method without unsupervised sample mining, the performance of the two methods is basically similar in the early stage of training because the model does not generate many qualified pseudo labels. In the later stage of training, the performance of the model that uses sample mining to expand the training set will continue to be stronger than the model without sample mining.

Claims

1. A machine learning method for multi-agent online evolutionary learning, characterized in that: In the OEL scenario, in a local area, multiple intelligent agent models acquire perception data in real time, and continuously learn and update the intelligent agent models by combining a small amount of labeled data; multiple intelligent agent models realize knowledge transfer through interactive learning; each intelligent agent model is faced with a large amount of unlabeled non-iid perception data, and these perception data are effectively processed jointly by combining the capabilities of multiple intelligent agent models; the intelligent agent model is referred to as a model; the specific steps are as follows:

1. Defining and establishing an OEL environment Given a classification task, for each agent model, let Represents a batch of B labeled data, x i is the training sample, y i is the corresponding label; let is a batch of μB unlabeled samples, μ is a hyperparameter, which indicates the ratio of unlabeled training samples to labeled training samples in each iteration; in the system, each batch The amount of data compared to The amount is small, and Included in part Non-IID data; labeled and unlabeled data will be sent to the model at the same time in each training step; each batch of unlabeled data is used only once, representing a continuously updated perceptual environment, forming a process simulation of online evolutionary learning; in addition, Represented as a model The predicted category distribution y for input x is used to represent the prediction result; The intelligent agent model is a deep learning-based image recognition CNN model, which is deployed on the edge smart device. Based on the perception and collection of environmental data by the smart device, through the interaction between multiple intelligent agent models, the data is fully utilized to learn and update each model, so as to adapt to environmental changes; (ii) Using the mutual matching (MM) algorithm to share knowledge and continuously learn between models to stabilize the generalization ability of the model and ensure that each agent achieves better performance improvement in the OEL environment; Specifically, the SSL process for each model includes the following steps: (1) Soft-supervisory consistency regularization processing: that is, using the unlabeled data obtained by each perception, it is subjected to soft-supervisory consistency regularization, and the soft-supervisory information predicted by the model is used to calculate the unsupervised loss, so that each model can obtain more category-related information; (2) Adopting a dual-model interactive learning method to achieve knowledge sharing among multiple models; (3) Introducing unsupervised sample mining methods, by statistically analyzing and evaluating some high-confidence unlabeled data, and expanding them into labeled data sets, each intelligent agent can use the perception data more efficiently, which is conducive to the better evolution of the entire system and improves the utilization rate of confidence samples; MM is an alternating training process that includes two training paths: supervised and unsupervised. Different enhancement methods are used to predict labeled / unlabeled data to calculate supervised / unsupervised losses according to different paths. The specific process of the soft supervision consistency regularization process is as follows: For classification tasks, cross entropy is used to measure the difference between the predicted value and the target probability distribution, and is used as a loss function to guide model training; let H(p,q) = -∑plog(q) represent the cross entropy loss of probability distribution p and q, for the model training, is a batch of labeled and unlabeled training data, then the model The supervised loss is expressed as: in, Indicates that a moderate image enhancement method is used for the labeled samples. yes by is the probability distribution of the input output, y i is x i The corresponding real annotation; For the unsupervised loss, let: As a model For the weakly enhanced unlabeled data u i The prediction results are filtered: in, is the confidence threshold, argwhere is the prediction result r i Extract the function of the prediction results greater than the threshold, so as to obtain the direct probability distribution prediction r i (uns) ; Then, the unsupervised loss is expressed as: in, Indicates that unlabeled data is strongly enhanced. This formula is based on r i (uns) The consistency regularization process of cross entropy calculation for the prediction of strongly enhanced data is performed for labels; The specific process of using the dual-model interactive learning method to achieve knowledge sharing among multiple models is as follows: In the multi-model interactive learning stage, KL divergence is used to measure the difference in prediction probability distribution between models, and it is used as interactive information to guide model alternating training; First, the KL divergence formula for approximating two probability distributions p and q is expressed as: When dealing with the multi-model KL loss part under the unsupervised path, for the two models and The interaction is as follows: First, in the model When calculating the loss of As a model Prediction results of weakly enhanced data and its use Filter and filter the results As a model The KL loss target is used to make the model The prediction results and The distribution is approximated; therefore, the model The interaction loss can be expressed as For the model The same method is used to transform the model The prediction of the target calculation model The interaction loss; Therefore, when the two models learn interactively, the loss function of the overall online evolutionary learning system is expressed as: in, and Represent the overall loss function of the two models respectively. The two models will continue to be trained alternately, and the training objectives depend on the probability distribution predicted by the other party. u and λ kl are the loss weights of the unsupervised part and the interaction loss part, respectively.

2. The machine learning method for multi-agent online evolutionary learning according to claim 1, characterized in that: The specific process of introducing the unsupervised sample mining method to improve the utilization rate of confidence samples is as follows: Build an index counter list for the unlabeled dataset, which collects high-confidence predictions in each iteration of model training and calculates the frequency of occurrence; if those occurrence frequencies reach the index value δ, expand these unlabeled samples to the labeled dataset, thereby expanding the labeled dataset for calculating the supervision loss; depending on the complexity of the training task and the size of the dataset, the index value δ is set to a scalar between 8-12; If the number of occurrences of an unlabeled data in the counter list drops significantly before reaching δ, the frequency count of the data in the table will be cleared and recounted.

Images