Method to detect concept drift in a network

Abstract

A computer-implemented method is provided to detect a concept drift in a network. The method includes receiving data points, including a first data point that is unlabeled. The method further includes generating a first label having a first value for the first data point based on use of a first ML model; and comparing the first value of the first label to a second value of a second label of a second ML model to determine whether the values match. The method further includes, when the values are different, identifying an error including a difference between the first and second values; and calculating an aggregated weighted error rate over a range of labels and data points. The method further includes determining whether the aggregated weighted error rate exceeds a specified threshold; and detecting concept drift when the aggregated weighted error rate exceeds the specified threshold.

Description

METHOD TO DETECT CONCEPT DRIFT IN A NETWORK TECHNICAL FIELD

[0001] The present disclosure relates generally to computer-implemented methods for detecting concept drift in a network, and related methods and devices. BACKGROUND

[0002] Machine learning (ML) models can be used to make many important decisions, especially on data streams since the volume and velocity of data points typically exceed processing capacity of a human, for example. However, a relationship between features of data and a decision of a ML model may change with unknown time and / or scale. SUMMARY

[0003] There currently exist certain challenges. There may be a need to detect a change in a relationship between data features and a decision of a ML model (referred to as “concept drift” as discussed further herein), for example, to alert maintainers of the ML model of the change. Operations for doing so, however, may bae lacking. Some approaches may request labels for specific data points on demand based on certain metrics. However, this may be susceptible to a distribution of labelled data, and an ML model’s decision boundary which can affect which data points are selected and the number of data points selected. Additionally, as discussed further herein, when evaluated on common real and synthetic data sets, some approaches may require 5% of the data to be labeled on average, which may be difficult to get in production. Such approaches may be time consuming and / or may have a high computation cost, which may be problematic in a data streaming situation. Moreover, some approaches may depend on a specific architecture for an original ML model, which may not be compatible with existing systems and may be hard to update.

[0004] Certain aspects of the disclosure and their embodiments may provide solutions to these or other challenges.

[0005] Some embodiments provide a computer-implemented method performed by a computing device to detect a concept drift in a network. The method includes receiving a stream of data including a plurality of data points. A first data point from the plurality of data points is unlabeled. The method further includes generating a first label having a first value for the first data point based on use of a first ML model; and comparing the first value of the first label to a second value of a second label of a second ML model to determine whether the values of the first labeland the second label match. The method further includes, when the values of the first and second labels are different, identifying an error including a difference between the first and second values; and calculating an aggregated weighted error rate over a range of labels and data points from a respective historical label from the first and the second ML models for a historical data point through the first and second labels for the first data point. The method further includes determining whether the aggregated weighted error rate exceeds a specified threshold; and detecting the concept drift when the aggregated weighted error rate exceeds the specified threshold.

[0006] Other embodiments provide a computing device. The computing device is configured to detect a concept drift in a network. The computing device includes processing circuitry; and memory coupled with the processing circuitry. The memory includes instructions that when executed by the processing circuitry causes the computing device to perform operations including to receive a stream of data including a plurality of data points. A first data point from the plurality of data points is unlabeled. The operations further include to generate a first label having a first value for the first data point based on use of a first ML model; and to compare the first value of the first label to a second value of a second label of a second ML model to determine whether the values of the first label and the second label match. The operations further include to, when the values of the first and second labels are different, identify an error including a difference between the first and second values; and to calculate an aggregated weighted error rate over a range of labels and data points from a respective historical label from the first and the second ML models for a historical data point through the first and second labels for the first data point. The operations further include to determine whether the aggregated weighted error rate exceeds a specified threshold; and to detect the concept drift when the aggregated weighted error rate exceeds the specified threshold.

[0007] Some embodiments provide a non-transitory computer readable medium including program code to be executed by processing circuitry of a computing device configured to detect a concept drift in a network. Execution of the program code causes the program code to perform operations including to receive a stream of data including a plurality of data points. A first data point from the plurality of data points is unlabeled. The operations further include to generate a first label having a first value for the first data point based on use of a first ML model; and to compare the first value of the first label to a second value of a second label of a second ML model to determine whether the values of the first label and the second label match. The operations further include to, when the values of the first and second labels are different, identify an error including a difference between the first and second values; and to calculate an aggregated weighted error rate over a range of labels and data points from a respective historical label from the first and thesecond ML models for a historical data point through the first and second labels for the first data point. The operations further include to determine whether the aggregated weighted error rate exceeds a specified threshold; and to detect the concept drift when the aggregated weighted error rate exceeds the specified threshold.

[0008] Certain embodiments may provide one or more of the following technical advantage(s). Based on inclusion of operations for concept drift detection, fewer (e.g., significantly fewer) data points on average may be needed to achieve acceptable results. Moreover, the operations may not need labels on demand, where labels on demand can depend on availability of data labels for every sample. Rather, randomly received labels at random intervals may be processed. Another technical advantage may be that the operations do not depend on the data distribution of a data stream or an original ML model’s decision boundary to determine which label is needed, which may allow concept drift detection when other approaches may fail. Moreover, operations may have a lower computation cost compared to other approaches that need to maintain an ensemble of ML models and multiple sample sizes. Additional technical advantages may include that operations may not depend on a specific original and / or inspector ML model’s architecture or concept drift detection process, making the operations compatible with existing systems and easy to update in the future. Additionally, the operations may complement existing ML models without requiring the ML models to change their architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this application, illustrate certain non-limiting embodiments of the present disclosure. In the drawings:

[0010] Figure 1 is a block diagram of components and operations in accordance with some embodiments;

[0011] Figure 2 is a flow chart of operations for training in accordance with some embodiments;

[0012] Figure 3 is a flow chart of operations of a computing device in accordance with some embodiments;

[0013] Figure 4 is a flow chart summarizing a data flow for an example in accordance with some embodiments;

[0014] Figure 5 is a plot of results of an example in accordance with some embodiments;

[0015] Figure 6 is a plot of accuracy in accordance with some embodiments;

[0016] Figures 7-8 are flow charts illustrating example operations of a computing device in accordance with some embodiments;

[0017] Figure 9 is a block diagram of a computing device in accordance with some embodiments;

[0018] Figure 10 is a block diagram of a communication system in accordance with some embodiments; and

[0019] Figure 11 is a block diagram of a virtualization environment in accordance with some embodiments. DETAILED DESCRIPTION

[0020] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art, in which examples of embodiments of the present disclosure are shown. Inventive concepts may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present / used in another embodiment.

[0021] Detecting concept drift may be important because it can indicate when a ML model is obsolete, for example. However, some approaches to detect concept drift are supervised, meaning such approaches need labelled data to perform. This may be an unrealistic expectation given a volume and velocity of the data. As a consequence, it may be desirable to detect concept drift using partially labeled data, e.g., semi-supervised concept drift detection.

[0022] The field of semi-supervised concept drift detection, however, may be understudied. For example, some semi-supervised concept drift detectors may achieve 6% lower accuracy while requiring labels from 5% of all data points on average. See e.g., A. Haque, L. Khan, M. Baron, B. Thuraisingham, and C. Aggarwal, "Efficient handling of concept drift and concept evolution over Stream Data," 2016 IEEE 32nd International Conference on Data Engineering (ICDE), Helsinki, Finland, pp. 481-492, doi: 10.1109 / ICDE.2016.7498264 (2016); Pinagé, F., dos Santos, E.M. & Gama, J. A drift detection method based on dynamic classifier selection. Data Min. Knowl. Disc. 34, 50–74, doi.org / 10.1007 / s10618-019-00656-w (2020); Ahsanul Haque, Latifur Khan, and Michael Baron, SAND: semi-supervised adaptive novel class detection and classification over datastream. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI'16). AAAI Press, 1652–1658 (2016).

[0023] Detection may be achieved via specific combinations of an original ML model (of a classification task) and concept drift detection processes. Moreover, such approaches may rely on metrics to request specific data points on demand.

[0024] Requirements of the above approaches, however, may be difficult to meet since the ML model may already be trained and 5% labels on demand, for example, may be a significant requirement given a volume and velocity of data. Thus, there may be a need for a ML model agnostic detection method that uses less data (e.g., much less data), and the data may be received randomly.

[0025] Some literature includes different discussions of the term “concept drift”. However, at least some of the literature may not be clear about whether the literature discusses concept drift or data drift. To understand the difference between concept drift and data drift, the following three examples are presented for two data sets: a first data set ^^௧^^^^^^^^ ^^, ^^^ which represents an original data set, and a second data set ^^௧^^௧^^^^ ^^, ^^^ which is a testing data set: ^ When ^^௧^^^^^^^^ ^^^ ് ^^௧^^௧^^^^ ^^^ and ^^௧^^^^^^^^ ^^| ^^^ ൌ ^^௧^^௧^^^^ ^^| ^^^, this is an example of data drift (only) for purposes of the present disclosure. It is noted that despite being data drift (not concept drift), some literature may refer to such data drift as a dataset shift, type I concept drift, or virtual concept drift (note that the term “virtual” is sometimes omitted in the literature). ^ When ^^௧^^^^^^^^ ^^^ ൌ ^^௧^^௧^^^^ ^^^ and ^^௧^^^^^^^^ ^^| ^^^ ് ^^௧^^௧^^^^ ^^| ^^^, this is an example of concept drift (only) and is included in embodiments of the present disclosure. It is noted that some literature may refer to data of this example as having type II concept drift, or real concept drift (note, however, that the term “real” is sometimes omitted in the literature). ^When ^^௧^^^^^^^^^^^് ^^௧^^௧^^^^ ^^^ and ^^௧^^^^^^^^ ^^| ^^^ ് ^^௧^^௧^^^^ ^^| ^^^, this is anexample of both data drift and concept drift occurring and is included in embodiments of the present disclosure. It is noted that some literature may refer to data of this example as having type III concept drift, or just concept drift since it is both virtual and real at the same time.

[0026] Some examples herein include collecting partially available data labels in data streams to propagate to nearby unlabeled data points to train a first ML model, which may be referred to as an inspector model herein. The first ML model’s predictions can be used as substitutes for truelabels in a supervised concept drift detector. Thus, allowing semi-supervised concept drift detection using limited labels. Such a process may have lower false negatives and false positives, while decreasing computation cost and a number of labels needed compared to other semi- supervised approaches in terms of concept drift detection, while being agnostic to the original ML model’s (also referred to herein as a second ML model) architecture and concept drift detector.

[0027] Stratified random sampling from clusters may be included in operations herein, which may allow independence from the original ML model’s architecture and the distribution of the labels. Both of which may be issues in other approaches.

[0028] Storing a small amount of data points with true labels to mix with new data points with true labels also may be included, which may allow operations herein to overcome a need to have labels in every sample of streaming data points. This also may have a stabilizing effect of reducing false positives detection when propagating over the samples since a small amount of labeled data points are used.

[0029] Label propagation may also be included in operations herein to generate the first ML model that provides pseudo labels to substitute for true labels to detect concept drift.

[0030] Figure 1 is a block diagram of components and operations in accordance with some embodiments. For an original classification task 102, streaming data points 104 are provided to an original ML model (also referred to herein as a second ML model) 106, which predicts their labels 108 using a standard ML process understood by those of skill in the art.

[0031] Detecting concept drift with a computing device 124 can use the framework shown in Figure 1. In parallel with the original ML model 106, an inspector model 110 (also referred to herein as a first ML model) is used which also takes in the streaming data points 104 as inputs and predicts their labels (also referred to as pseudo labels). Then, concept drift detection component 112 compares the original ML model’s 106 predicted labels and the pseudo labels to check if the labels are the same or different labels, codified as 0 and 1 respectively, to detect concept drift 114. If concept drift is detected 114, the original ML model 106 is retrained 122.

[0032] As shown in Figure 1, the inspector model 110 can be trained as follow. The streaming data points 104 are grouped into a sliding sample that includes newest data points. Clustering and randomization component 116 clusters and randomly samples from these data points to obtain one data point, for example, from each of the clusters. The availability of true labels 118 in this sample is random to reflect real world conditions. If there are true labels available, using propagation and retrain component 120, these data points are propagated to the previously mentioned sample to retrain the inspector model 110. It is noted that, as discussed further herein, clustering of examples is used to assist in the sampling of data points to be labeled.

[0033] Figure 2 is a flow chart 200 of operations for training the original ML model 106 and initialization of the inspector ML model 110. Training of the original ML model 106 starts 202 and training data (with labels) 204 is accessed. A ML model with a desired architecture is trained 206 with the training data 204, and results in the original ML model 106. To initialize the inspector ML model 110, the original ML model 106 is copied 208. This copy is the initialized inspector ML model 110; and the training process ends 210.

[0034] Discussion herein includes the following symbols / parameters: ^^^ ^^, ^^^ the incoming data stream ^^ features of the incoming instance ^^ true label of the instance, which may not always be available ŷ the prediction results of the original ML model 106 and inspector model 108 respectively ŷ^^^^the prediction results of the original ML model 106 and inspector model 108 respectively ^^ ^^ ^^ ^^ ^^ ^^ ^^ maximum size of ^^ ^^ sample of w newest unlabeled data points ^^^^^^^^^௧^ௗsample of w newest data points with propagated pseudo labels ^^ the memory of true labels used for propagation ^^ the maximum distance between two samples for one to be considered as in the neighborhood of the other ^^ ^^ ^^ ^^ ^^ ^^ the number of samples in a neighborhood for a point to be considered as a core point. ^^ the concept drift detection threshold ^^ the set of labeled data points received in a current iteration

[0035] Figure 3 is a flow chart of operations of a computing device in accordance with some embodiments. As shown, the operations can be roughly divided into three parts: Part I Concept drift detection; Part II Randomization and clustering; and Part III Propagation and retraining of the inspector ML model 110.

[0036] Referring first to Part I Concept drift detection in Figure 3, operations start 300 and proceed to monitoring 302 an incoming data stream for concept drift. The monitoring can depend on the implementation and, for example, can be every single incoming data point or a sample of the incoming data points. The original ML model 106 classifies 304 the incoming data and givesits predictions (ŷ) 306 for a main task. An inspector ML model 110 works in parallel and classifies 308 the incoming data to provide its predictions (pseudo labels) (ŷ^^^^) 310. ŷ ൌ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^^^ ^Eq.1^ ŷ^^^^ൌ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^^^ ^Eq.2^

[0037] The predictions (ŷ) 306 of the original ML model 106 and the predictions (pseudo labels) (ŷ^^^^) 310 are compared 312. A difference (error) in predictions from these two ML models 106, 110 (represented as 0 if the predictions are the same and 1 otherwise) can be used as inputs to an existing supervised concept drift detection process, such as ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^, ^^ ^^ ^^, ^^ ^^ ^^ ^^, etc., to detect concept drifts in the data stream. ^^ ^^ ^^ ^^ ^^^ൌ ^0, ŷ^ ൌ ŷ^^^^,^1, ^^ ^^ℎ ^^ ^^ ^^ ^^ ^^ ^^∀ ^^ ∈ ^^ ^Eq. 3^

[0038] When therethe concept drift detection component 112 with the error. In operation 316, the supervised concept drift detection component 112 takes the difference between the original ML model’s 106 prediction and the true labels (error). This allows the supervised concept drift detection component 112, in operations 318 and 320, to compute an aggregated weighted error rate between the original ML model 106 and the inspector ML model 110 of the data points leading up to and including the current data point. In operation 322, if this error rate increases such that it surpasses a predefined threshold, the detection component 112 triggers concept drift detection (shown as record the alarmin operation 324 in Figure 3). When this happens, the instance where the concept drift detector is triggered is recorded and operations proceed to the next data point and the original ML model 106 is retrained 326. ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ൌ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^^, ^^ ^^ ^^ ^^ ^^^^ ^Eq.4^

[0039] Where there is not an error, operations proceed to operation 328 to go to the next iteration.

[0040] Since, the difference in the classification results from the original model and the inspector model (pseudo labels) is used, the concept drift detected is between the original ML model 106 and the inspector ML model 110. The inspector ML model 110, when it is trained on a mix of the original and new concept or new concept may be able to learn the new concept to provide predictions that triggers concept drift detection.

[0041] These operations may allow for the conversion of the supervised concept drift detector to semi-supervised, which may significantly reduce the number of labeled data points needed.

[0042] Referring next to Part II Randomization and clustering in Figure 3, starting from the sample ^^ 330 of newest ^^ unlabeled data points, a clustering process 332, e.g., ^^ ^^ ^^ ^^ ^^ ^^ , ^^ ^^ ^^ ^^ ^^ ^^, etc., is used to summarize the data points in the sample into groups 334, e.g., into ^^ clusters. ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ൌ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^^^^, ^^, ^^ ^^ ^^ ^^ ^^ ^^^^ ^^ ^^.5^

[0043] In operation 336, stratified random sampling is performed to generate a sample ^^ of ^^ data points for labeling. In operation 338, one random data point from each group is selected as a candidate for clustering. ^^ ൌ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^^^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^^^ ^^ ^^. 6^

[0044] In operation 340, from ^^, ^^ random points are provided with labels where 0 ^ ^^ ^^^. It is not necessary that a label is always available. Hence, the number of labels received isalways ^^ where 0 ^ ^^ ^ ^^ to model that some labels are received at some intervals. This set of ^^ points with labels may be referred to as ^^ 342. ^^ ൌ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^^^^, ^^^, 0 ^ ^^ ^|^^|^ ^^ ^^. 7^

[0045] Alternatively, as shown by the dotted line Part II Randomization and clustering in Figure 3, the clustering process can be bypassed such that for each sample ^^ of newest ^^ unlabeled data points, there are ^^ random points with labels where 0 ^ ^^ ^ ^^ directly from the sample where c would be a fixed number that was chosen instead of the number of clusters as discussed above. ^^ ൌ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^^, ^^^, 0 ^ ^^ ^ ^^, ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^5 → 7 ∗^

[0046] Referring now to Part III Propagation and retraining of the inspector ML model in Figure 3, in operations 344 and 346, any queue for a memory M is emptied, and true labels are initialized in the memory M. In operations 348 and 350 from the l randomly received labels, these labeled points are inserted into a memory M of size m containing both the previously labeled data points and the l new points. At the same time, the l oldest points are removed from M. ^^ ൌ ^^ ^^ ^^ ^^ ^^^ ^^, ^^, ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^ ^^ ^^.8^

[0047] In operation 352, these data points are propagated over the newest sample of w unlabeled data points (that is , a sample of newest unlabeled data points for clustering as described herein is used). This yields a propagation model 354 and a sample of pseudo labels 356.^^^^^^^^^௧^ௗൌ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^^, ^^^ ^ ^^ ^^. 9^

[0048] As shown in Figure 3, at this point, there are two alternatives. In a first alternative, the propagation model 354 is used as is (as shown by the dotted line in Figure 3) to replace 362 the inspector ML model 110 and classify new unlabeled data points. In the second alternative, operations 356, 358 and 360, the inspector ML model 110 is retrained with a preferred architecture using the data points and pseudo labels to obtain a new inspector ML model 110 that replaces the current inspector ML model. In the second alternative, the inspector ML model 110 is trained using the data points and pseudo labels belonging to ^^^^^^^^^௧^ௗ. ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ൌ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^൫ ^^^^^^^^^௧^ௗ൯ ^ ^^ ^^. 10^

[0049] In one non-limiting example, hyper parameters and configurations are the following: ^ Sample W size ^^ can be 500 ≤ ^^ ≤ 1000, for example. ^ Memory M size ^^ can be a fixed size of ^^ = 10; or a range 10 ≤ m ≤ 20, for example.

[0050] A non-limiting example uses case includes application of this framework on concept drift detection in telecommunications using a network traffic data set and a network traffic congestion detector trained on the network traffic data set. In this example, the operations detect when a congestion pattern changes, and an original network traffic congestion detector fails to correctly predict congestion.

[0051] Figure 4 is a flow chart summarizing the data flow for this example. This example includes a group of core-switch routers in different sites for monitoring of performance metrics, e.g., jitter, delay, packet drops, etc. The performance metrics 400 are sent to a server 402. The server 402 includes a deep learning (DL) model that was trained on historical data and predicts 404 significant congestion and large amount of packet loss within the next 24 hours.

[0052] Further, in this example, the DL model was trained such that it monitors average (Avg) jitters and Avg Delay such that if these two metrics follow some special patterns, it would indicate a major congestion is coming. It is noted that this is not just a simple linear or threshold problem, thus the need for a DL model for future prediction. The definitions of the features in this example are as follows: ^ Jitter is the fluctuation in rates of delay across the network.^ Delay is how long it takes for a packet to travel from one endpoint to another. For example, after a period of time, the bandwidth of the network is increased. This has no effect on the jitter or the delay except during congestion, hence there is no impact on the data used for predictions. This may make the prediction less reliable because when some types of congestion are predicted to happen, the increased bandwidth alleviated congestions indicated by some patterns for some cases and can cause a false positive rate to increase to, e.g., 30% from the previous false positive rate of, e.g., 5%. If the jitter or delay simply changes, this can be resolved by the data drift detector. However, in this case, the data does not change. Thus, the need for concept drift detection.

[0053] This example use case is now discussed further.

[0054] A transport network dataset includes core-switch / cell-site router performance metrics, e.g., jitter, delay, packet drops, etc., which are data features. Note that this data set does not contain any labelling for training. Thus, the following discussion describes operations for generating the labels for the original concept and the drifted concept.

[0055] In this example, the two most important and frequently changing features are Avg Delay Avg (60 min) and Avg Jitter Avg (60 min), which are used as follows to generate the labels as discussed further below. ^^^ൌ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^60 ^^ ^^ ^^^ ^ ^^ ^^.11^ ^^ଶ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^ ^^ ^^.12^

[0056] Original labels ( ^^^^^^^^^^) are generated using a circle function, e.g., an example is labeled 1 if it is inside the circle, otherwise its label is 0. ^^ ^^ ^^ ^^ ^^ ^^^^^^^^^^: ^^ ^^ ^^ ^^ ^^ ^^ ൌ^14, 7.5^, ^^ ^^ ^^ ^^ ^^ ^^ ൌ 1 ^ ^^ ^^.13^^^^^^^^^ ^1 ^ ^ଶ ^ ^ଶ^^ ൌ , ^ ^^^ െ 14 ^ ^^ଶ െ 7.5 ^ 1 ^ ^^ ^^.14^

[0057] Thethe labels belonging to the original concept. ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ൌ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^൫ ^^^^^^^^^^, ^^^^^^^^^^൯ ^ ^^ ^^.15^

[0058] the circle changes. In this example, the labels for the concept drift are generated by changing the center and the radius of the circle to the following numbers: ^^ ^^ ^^ ^^ ^^ ^^^^^^^^௧ ௗ^^^௧: ^^ ^^ ^^ ^^ ^^ ^^ ൌ^13, 8^, ^^ ^^ ^^ ^^ ^^ ^^ ൌ 1.5 ^ ^^ ^^.16^^^^^^^௧ ௗ^^^௧ ^1, ^ ^^ െ 13^ଶ ^ ^ ^ଶ^^ ൌ ^ ^ ^^ଶ െ 8 ^ 1.50, ^^ ^^ℎ ^^ ^^ ^^ ^^ ^^ ^^^ ^^ ^^.17^

[0059] In this example, a sample data stream ( ^^^ ^^, ^^^) was generated for testing as follows. First, data belonging to cell-site 0 (named ^^^) was extracted from the dataset. When there is no concept drift, the data is the following: ^^^,^^^^^^^^൫ ^^^, ^^^,^^^^^^^^൯ ^ ^^ ^^.18^

[0060] When there is concept drift, the data is as follows: ^^^,^^^^^^௧ ௗ^^^௧൫ ^^^, ^^^,^^^^^^௧ ௗ^^^௧൯^^^ ^^.19^

[0061] In this training and the detectiontask and to ensure that the data feature did not change while only the labels changed, the following was performed. The sample data stream was organized as follows: ^^^ ^^, ^^^ was composed by four instances of ^^^, while changing ^^^,^^^^^^^^to ^^^,^^^^^^௧ ௗ^^^௧in the third segment. This generated a total of 8,272 data points. By doing this, it is known that the concept drift happens between data points 4,136 and 6,204. ^^^ ^^, ^^^ ൌ ^ ^^^,^^^^^^^^, ^^^,^^^^^^^^, ^^^,^^^^^^௧ ௗ^^^௧, ^^^,^^^^^^^^^ ^ ^^ ^^.20^

[0062] the framework one-by-one.

[0063] Results from this example are shown in Figure 5, which compares concept drift detection performance of this example with some other approaches. The data points for this example contain 1000 data points for training and 7272 data points in the data stream. The bounded area in Figure 5 shows where the concept drift occurred; and the vertical dashed lines show the concept drift detection point as discussed further below. For purposes of the results, prediction accuracy was computed by comparing the prediction results to the true labels, which were available during post-mortem analysis, not while the process was running.

[0064] As shown in Figure 5, the process of the example uses true labels for 0.75% of all data points for early and correct detection of concept drift.0.75% represents 54 true labels used out of 7272 data points in the streaming data set.

[0065] Further as shown in Figure 5, this result is compared to another approach for concept drift detection: ^ Supervised Page-Hinkley test, which detected concept drift early but requires 100% of the data to be labeled.^ Naïve semi-supervised concept drift detection that uses 20% of true labels did not detect any concept drift and, thus, is not shown in Figure 5. ^ ECHO (not shown in Figure 5) did not detect concept drift using 5% of data with true labels due to reliance on clustering and data points close to the decision boundary. On the other hand, if the labels are allowed to be available on demand, ECHO requires over 75% of all data points to be labeled and provides 15 detections with at least 5 false positives.

[0066] With respect to training, some literature (e.g., H. Kingetsu and K. Kobayashi, "Born- Again Decision Boundary: Unsupervised Concept Drift Detection by Inspector Neural Network," 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China, pp.1-8, doi: 10.1109 / IJCNN52387.2021.9534340 (2021)) that uses a ML model to generate pseudo labels predictions for comparison with an original ML model’s predictions perform ML model distillation / compression of the original ML model to generate the other ML model. In contrast, the inspector ML model of the present disclosure is trained from propagated true label data on a sample of newest unlabeled data. The true label data used to train the inspector ML model is from a mix of old and new true labels received periodically. While there is an option to mix some predictions of the original ML model to the set used for propagation, this option may not be needed. Even if this option is included, the proportion is very small compared to distillation works.

[0067] As a consequence, operations of the present disclosure are semi-supervised in contrast to other approaches which are unsupervised.

[0068] With respect to the type of concept drift detected, in some other approaches (e.g., H. Kingetsu and K. Kobayashi, "Born-Again Decision Boundary: Unsupervised Concept Drift Detection by Inspector Neural Network," 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China, pp. 1-8, doi: 10.1109 / IJCNN52387.2021.9534340 (2021)), since the other ML model mimics an original ML model, the approach can only detect when ^^^ ^^^ changes, which may (or may not) coincide with ^^^ ^^| ^^^ changes. In contrast, operations of the present disclosure monitor ^^^ ^^| ^^^ changes.

[0069] With respect to sample-based concept drift detection, some other approaches may monitor some type of metric (e.g., an ML model’s uncertainty, distance from decision boundary, error rate, etc.) and generate two windows: a reference window, representing the original concept, and a validation window representing the current concept. See e.g., Ömer Gözüaçık, Alican Büyükçakır, Hamed Bonab, and Fazli Can. Unsupervised Concept Drift Detection with a Discriminative Classifier. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM '19). Association for Computing Machinery, New York, NY, USA, 2365–2368. doi.org / 10.1145 / 3357384.3358144 (2019); Baier, Lucas, et al."Detecting concept drift with neural network model uncertainty." arXiv preprint arXiv:2107.01873 (2021); Gemaque, Rosana Noronha et al. “An overview of unsupervised drift detection methods.” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 10 (2020)). In such approaches, a concept drift may occur when a statistic test concludes that the two windows are not the same. In contrast, in the present disclosure, a sample of the newest unlabeled points in the data stream is used.

[0070] It is noted that while the inspector ML model of the present disclosure may be initialized by copying the original ML model, the prediction of the inspector ML model may not be comparable to the original ML model given the semi-supervised learning. For example, Figure 6 shows that while accuracy of the inspector ML model may not be as good as the original ML model, the inspector ML model may be consistent and diverges when the concept drift occurs in accordance with some embodiments.

[0071] Figures 7 and 8 are flow charts illustrating example operations of a computing device in accordance with some embodiment.

[0072] Referring to Figure 7, in some embodiments, a computer-implemented method is provided that is performed by a computing device to detect a concept drift in a network. The method includes receiving (operation 702 in Figure 7) a stream of data including a plurality of data points. A first data point from the plurality of data points is unlabeled. The method further includes generating (operation 704 in Figure 7) a first label having a first value for the first data point based on use of a first ML model; and comparing (operation 706 in Figure 7) the first value of the first label to a second value of a second label of a second ML model to determine whether the values of the first label and the second label match. The method further includes, when the values of the first and second labels are different, identifying (operation 708 in Figure 7) an error including a difference between the first and second values; and calculating (operation 710 in Figure 7) an aggregated weighted error rate over a range of labels and data points from a respective historical label from the first and the second ML models for a historical data point through the first and second labels for the first data point. The method further includes determining (operation 712 in Figure 7) whether the aggregated weighted error rate exceeds a specified threshold; and detecting (operation 714 in Figure 7) the concept drift when the aggregated weighted error rate exceeds the specified threshold.

[0073] The second ML model may generate the second label for the first data point at a point in time before the first ML model generated the first label for the first data point. In some embodiments, the first ML model and the second ML model perform a classification-related operation; and, without limitation, respectively can be a classification ML model or a ML modelsuch as regression, clustering, association, etc. which can be converted to perform classification tasks.

[0074] In some embodiments, the method further includes recording (800) the concept drift.

[0075] Referring to Figure 8, in still other embodiments, the method further includes proceeding (operation 802) to a second data point in the plurality of data points; and retraining (operation 804) the second ML model.

[0076] In some embodiments, the plurality of data points include a plurality of unlabeled data points, and the method further includes retraining (operation 806) the first ML model.

[0077] The retraining (operation 806) the first ML model may include (i) clustering the plurality of unlabeled data points into a sample for a time period to obtain a plurality of clusters of unlabeled data points, (ii) accessing labels for unlabeled data points sampled from the plurality of clusters to obtain accessed labels, (iii) propagating the unlabeled data points sampled in the time period, and (iv) training the first ML model on the accessed labels for the unlabeled data points sampled in the time period.

[0078] The clustering may include (i) grouping the unlabeled data points into a sample comprising a group of newest unlabeled data points in the time period, and (ii) dividing the group of the newest unlabeled data points into the plurality of clusters of unlabeled data points.

[0079] In some embodiments, the method further includes sampling a respective unlabeled data point from at least one cluster to obtain a set of sampled unlabeled data points.

[0080] Training the first ML model on the accessed labels for the sampled unlabeled data points in the time period may include (i) updating the unlabeled data points with the accessed labels to obtain labeled data points, and (ii) propagating the labeled data points over the sample of unlabeled data points.

[0081] In some embodiments, propagating the labeled data points over the sample of unlabeled data points results in an updated first ML model, and the method further includes using (operation 808) the updated first ML model to detect concept drift in the network.

[0082] In still other embodiments, propagating the labeled data points over the sample is for the time period, the method further includes retraining (operation 812) the first ML model with the labeled data points over the sample for the time period.

[0083] In other embodiments, the method further includes, when the aggregated weighted error rate is less than the specified threshold, repeating the receiving (operation 702), the generating (operation 704), the comparing (operation 706), the identifying (operation 708), the calculating (operation 710), the determining (operation 712), and the detecting (operation 714).

[0084] In yet other embodiments, the method further includes initializing (operation 700) the first ML model by implementing a copy of the second ML model.

[0085] The network may include a telecommunications network.

[0086] The computing device may include a cloud-based device.

[0087] In some embodiments, concept drift includes at least one of: (i) a change in a relationship between the first data point and a label, wherein in the change is represented by the first and second labels being different for the first data point, and (ii) a change in a distribution of the plurality of data points, and a change in a relationship between at least some of the data points in the plurality of data points and a respective first label and a respective second label being different for at least some of the data points

[0088] Operations of a computing device can be performed by the computing device 900 of Figure 9. Operations of the computing device have been discussed with reference to the flow charts of Figures 7 and 8 according to some embodiments of the present disclosure. For example, modules may be stored in memory 906 or in first ML model 110, second ML model 106, concept drift detection component 112, clustering and randomization component 116, and / or propagation and retrain component 120 of Figure 9 and these modules may provide instructions so that when the instructions of a module are executed by respective computing device processing circuitry 904, computing device 900 performs respective operations of the flow charts.

[0089] Various operations from the flow charts of Figures 7 and 8 may be optional with respect to some embodiments of computing devices and related methods. For example the operations of blocks 700, 708, and 800-814 may be optional.

[0090] Figure 10 shows an example of a communication system 1000 in accordance with some embodiments.

[0091] In the example, the communication system 1000 (also referred to herein as a network) includes a telecommunication network 1002 that includes an access network 1004, such as a RAN, and a core network 1006, which includes one or more core network nodes 1008. The access network 1004 includes one or more access network nodes, such as network nodes 1010a and 1010b (one or more of which may be generally referred to as network nodes 1010), or any other similar 3GPP access node or non-3GPP access point. The network nodes 1010 facilitate direct or indirect connection of UE, such as by connecting UEs 1012a, 1012b, 1012c, and 1012d (one or more of which may be generally referred to as UEs 1012) to the core network 1006 over one or more wireless connections. A computing device (e.g., computing device 900) can be a cloud based computing device 124 or another node comprising a computing device in communication system 1000.

[0092] Example wireless communications over a wireless connection include transmitting and / or receiving wireless signals using electromagnetic waves, radio waves, infrared waves, and / or other types of signals suitable for conveying information without the use of wires, cables, or other material conductors. Moreover, in different embodiments, the communication system 1000 may include any number of wired or wireless networks, computing devices, network nodes, UEs, and / or any other components or systems that may facilitate or participate in the communication of data and / or signals whether via wired or wireless connections. The communication system 1000 may include and / or interface with any type of communication, telecommunication, data, cellular, radio network, and / or other similar type of system.

[0093] The UEs 1012 may be any of a wide variety of communication devices, including wireless devices arranged, configured, and / or operable to communicate wirelessly with the network nodes 1010 and other communication devices. Similarly, the network nodes 1010 are arranged, capable, configured, and / or operable to communicate directly or indirectly with the UEs 1012 and / or with other network nodes or equipment in the telecommunication network 1002 to enable and / or provide network access, such as wireless network access, and / or to perform other functions, such as administration in the telecommunication network 1002.

[0094] In the depicted example, the core network 1006 connects the network nodes 1010 to one or more hosts, such as host 1016. These connections may be direct or indirect via one or more intermediary networks or devices. In other examples, network nodes may be directly coupled to hosts. The core network 1006 includes one more core network nodes (e.g., core network node 1008) that are structured with hardware and software components. Features of these components may be substantially similar to those described with respect to the UEs, network nodes, and / or hosts, such that the descriptions thereof are generally applicable to the corresponding components of the core network node 1008. Example core network nodes include functions of one or more of a Mobile Switching Center (MSC), Mobility Management Entity (MME), Home Subscriber Server (HSS), Access and Mobility Management Function (AMF), Session Management Function (SMF), Authentication Server Function (AUSF), Subscription Identifier De-concealing function (SIDF), Unified Data Management (UDM), Security Edge Protection Proxy (SEPP), Network Exposure Function (NEF), and / or a User Plane Function (UPF).

[0095] The host 1016 may be under the ownership or control of a service provider other than an operator or provider of the access network 1004 and / or the telecommunication network 1002, and may be operated by the service provider or on behalf of the service provider. The host 1016 may host a variety of applications to provide one or more service. Examples of such applications include live and pre-recorded audio / video content, data collection services such as retrieving andcompiling data on various ambient conditions detected by a plurality of UEs, analytics functionality, social media, functions for controlling or otherwise interacting with remote devices, functions for an alarm and surveillance center, or any other such function performed by a server.

[0096] As a whole, the communication system 1000 of Figure 10 enables connectivity between the UEs, network nodes, and hosts. In that sense, the communication system may be configured to operate according to predefined rules or procedures, such as specific standards that include, but are not limited to: Global System for Mobile Communications (GSM); Universal Mobile Telecommunications System (UMTS); Long Term Evolution (LTE), and / or other suitable 2G, 3G, 4G, 5G standards, or any applicable future generation standard (e.g., 6G); wireless local area network (WLAN) standards, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (WiFi); and / or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Microwave Access (WiMax), Bluetooth, Z-Wave, Near Field Communication (NFC) ZigBee, LiFi, and / or any low-power wide-area network (LPWAN) standards such as LoRa and Sigfox.

[0097] In some examples, the telecommunication network 1002 is a cellular network that implements 3GPP standardized features. Accordingly, the telecommunications network 1002 may support network slicing to provide different logical networks to different devices that are connected to the telecommunication network 1002. For example, the telecommunications network 1002 may provide URLLC services to some UEs, while providing eMBB services to other UEs, and / or mMTC / Massive IoT services to yet further UEs.

[0098] In some examples, the UEs 1012 are configured to transmit and / or receive information without direct human interaction. For instance, a UE may be designed to transmit information to the access network 1004 on a predetermined schedule, when triggered by an internal or external event, or in response to requests from the access network 1004. Additionally, a UE may be configured for operating in single- or multi-RAT or multi-standard mode. For example, a UE may operate with any one or combination of Wi-Fi, NR and LTE, i.e. being configured for multi-radio dual connectivity (MR-DC), such as E-UTRAN (Evolved-UMTS Terrestrial Radio Access Network) New Radio – Dual Connectivity (EN-DC).

[0099] In the example, the hub 1014 communicates with the access network 1004 to facilitate indirect communication between one or more UEs (e.g., UE 1012c and / or 1012d) and network nodes (e.g., network node 1010b). In some examples, the hub 1014 may be a controller, router, content source and analytics, or any of the other communication devices described herein regarding UEs. For example, the hub 1014 may be a broadband router enabling access to the core network 1006 for the UEs. As another example, the hub 1014 may be a controller that sendscommands or instructions to one or more actuators in the UEs. Commands or instructions may be received from the UEs, network nodes 1010, or by executable code, script, process, or other instructions in the hub 1014. As another example, the hub 1014 may be a data collector that acts as temporary storage for UE data and, in some embodiments, may perform analysis or other processing of the data. As another example, the hub 1014 may be a content source. For example, for a UE that is a VR headset, display, loudspeaker or other media delivery device, the hub 1014 may retrieve VR assets, video, audio, or other media or data related to sensory information via a network node, which the hub 1014 then provides to the UE either directly, after performing local processing, and / or after adding additional local content. In still another example, the hub 1014 acts as a proxy server or orchestrator for the UEs, in particular if one or more of the UEs are low energy IoT devices.

[0100] The hub 1014 may have a constant / persistent or intermittent connection to the network node 1010b. The hub 1014 may also allow for a different communication scheme and / or schedule between the hub 1014 and UEs (e.g., UE 1012c and / or 1012d), and between the hub 1014 and the core network 1006. In other examples, the hub 1014 is connected to the core network 1006 and / or one or more UEs via a wired connection. Moreover, the hub 1014 may be configured to connect to an M2M service provider over the access network 1004 and / or to another UE over a direct connection. In some scenarios, UEs may establish a wireless connection with the network nodes 1010 while still connected via the hub 1014 via a wired or wireless connection. In some embodiments, the hub 1014 may be a dedicated hub – that is, a hub whose primary function is to route communications to / from the UEs from / to the network node 1010b. In other embodiments, the hub 1014 may be a non-dedicated hub – that is, a device which is capable of operating to route communications between the UEs and network node 1010b, but which is additionally capable of operating as a communication start and / or end point for certain data channels.

[0101] Figure 11 is a block diagram illustrating a virtualization environment 1100 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one or more virtual environments 1100 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a computing device, network node, UE, core network node, or host. Further,in embodiments in which the virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely virtualized. In some embodiments, the virtualization environment 1100 includes components defined by the O-RAN Alliance, such as an O-Cloud environment orchestrated by a SMO 1110 Framework via an O-2 interface.

[0102] Applications 1102 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment 1100 to implement some of the features, functions, and / or benefits of some of the embodiments disclosed herein.

[0103] Hardware 1104 includes processing circuitry, memory that stores software and / or instructions executable by hardware processing circuitry, and / or other hardware devices as described herein, such as a network interface, input / output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers 1106 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs 1108a and 1108b (one or more of which may be generally referred to as VMs 1108), and / or perform any of the functions, features and / or benefits described in relation with some embodiments described herein. The virtualization layer 1106 may present a virtual operating platform that appears like networking hardware to the VMs 1108.

[0104] The VMs 1108 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer 1106. Different embodiments of the instance of a virtual appliance 1102 may be implemented on one or more of VMs 1108, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment.

[0105] In the context of NFV, a VM 1108 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of the VMs 1108, and that part of hardware 1104 that executes that VM, be it hardware dedicated to that VM and / or hardware shared by that VM with others of the VMs, forms separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more VMs 1108 on top of the hardware 1104 and corresponds to the application 1102.

[0106] Hardware 1104 may be implemented in a standalone network node with generic or specific components. Hardware 1104 may implement some functions via virtualization.Alternatively, hardware 1104 may be part of a larger cluster of hardware (e.g. such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration (SMO) 1110, which, among others, oversees lifecycle management of applications 1102. In some embodiments, hardware 1104 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station. In some embodiments, some signaling can be provided with the use of a control system 1112 which may alternatively be used for communication between hardware nodes and radio units.

[0107] Although the computing device described herein may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and / or software needed to perform the tasks, features, functions and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the computing device, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and / or the functionality of the components may be partitioned between the processor and the network interface. In another example, non- computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.

[0108] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer- readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particularembodiments, whether executing instructions stored on a non-transitory computer-readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally.

[0109] In certain embodiments, a computing device (124, 900) is provided. The computing device is configured to detect a concept drift in a network. The computing device includes processing circuitry 904; and memory 906 coupled with the processing circuitry. The memory includes instructions that when executed by the processing circuitry causes the computing device to perform operations. The operations include to perform some or all of the functionality described herein.

[0110] In certain embodiments, a non-transitory computer readable medium 906, 106, 110, 112, 116, 120 including program code to be executed by processing circuitry 904 of a computing device 124, 900 is configured to detect a concept drift in a network. Execution of the program code causes the program code to perform operations. The operations include to perform some or all of the functionality described herein.

[0111] Further definitions and embodiments are discussed below.

[0112] In the above-description of certain embodiments of the present disclosure, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which concepts of the present disclosure belong. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0113] When an element is referred to as being “connected”, “coupled”, “responsive”, or variants thereof to another element, it can be directly connected, coupled, or responsive to the other element or intervening elements may be present. In contrast, when an element is referred to as being “directly connected”, “directly coupled”, “directly responsive”, or variants thereof to another element, there are no intervening elements present. Like numbers refer to like elements throughout. Furthermore, “coupled”, “connected”, “responsive”, or variants thereof as used herein may include wirelessly coupled, connected, or responsive. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicatesotherwise. Well-known functions or constructions may not be described in detail for brevity and / or clarity. The term “and / or” (abbreviated “ / ”) includes any and all combinations of one or more of the associated listed items.

[0114] It will be understood that although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Thus a first element / operation in some embodiments could be termed a second element / operation in other embodiments without departing from the teachings of concepts of the present disclosure. The same reference numerals or the same reference designators denote the same or similar elements throughout the specification.

[0115] As used herein, the terms “comprise”, “comprising”, “comprises”, “include”, “including”, “includes”, “have”, “has”, “having”, or variants thereof are open-ended, and include one or more stated features, integers, elements, steps, components, or functions but does not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions, or groups thereof. Furthermore, as used herein, the common abbreviation “e.g.”, which derives from the Latin phrase “exempli gratia,” may be used to introduce or specify a general example or examples of a previously mentioned item, and is not intended to be limiting of such item. The common abbreviation “i.e.”, which derives from the Latin phrase “id est,” may be used to specify a particular item from a more general recitation.

[0116] Example embodiments are described herein with reference to block diagrams and / or flowchart illustrations of computer-implemented methods, apparatus (systems and / or devices) and / or computer program products. It is understood that a block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by computer program instructions that are performed by one or more computer circuits. These computer program instructions may be provided to a processor circuit of a general purpose computer circuit, special purpose computer circuit, and / or other programmable data processing circuit to produce a machine, such that the instructions, which execute via the processor of the computer and / or other programmable data processing apparatus, transform and control transistors, values stored in memory locations, and other hardware components within such circuitry to implement the functions / acts specified in the block diagrams and / or flowchart block or blocks, and thereby create means (functionality) and / or structure for implementing the functions / acts specified in the block diagrams and / or flowchart block(s).

[0117] These computer program instructions may also be stored in a tangible computer- readable medium that can direct a computer or other programmable data processing apparatus tofunction in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the functions / acts specified in the block diagrams and / or flowchart block or blocks. Accordingly, embodiments of the present disclosure may be embodied in hardware and / or in software (including firmware, resident software, micro-code, etc.) that runs on a processor such as a digital signal processor, which may collectively be referred to as “circuitry,” “a module” or variants thereof.

[0118] It should also be noted that in some alternate implementations, the functions / acts noted in the blocks may occur out of the order noted in the flowcharts. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality / acts involved. Moreover, the functionality of a given block of the flowcharts and / or block diagrams may be separated into multiple blocks and / or the functionality of two or more blocks of the flowcharts and / or block diagrams may be at least partially integrated. Finally, other blocks may be added / inserted between the blocks that are illustrated, and / or blocks / operations may be omitted without departing from the scope of the present disclosure. Moreover, although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows.

[0119] Many variations and modifications can be made to the embodiments without substantially departing from the principles of the present disclosure. All such variations and modifications are intended to be included herein within the scope of present disclosure. Accordingly, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the examples of embodiments are intended to cover all such modifications, enhancements, and other embodiments, which fall within the spirit and scope of the present disclosure. Thus, to the maximum extent allowed by law, the scope of the present disclosure is to be determined by the broadest permissible interpretation of the present disclosure including the examples of embodiments and their equivalents, and shall not be restricted or limited by the foregoing detailed description.

Claims

Claims:

1. A computer-implemented method performed by a computing device to detect a concept drift in a network, the method comprising: receiving (702) a stream of data comprising a plurality of data points, wherein a first data point from the plurality of data points is unlabeled; generating (704) a first label having a first value for the first data point based on use of a first machine learning, ML, model; comparing (706) the first value of the first label to a second value of a second label of a second ML model to determine whether the values of the first label and the second label match; when the values of the first and second labels are different, identifying (708) an error comprising a difference between the first and second values; calculating (710) an aggregated weighted error rate over a range of labels and data points from a respective historical label from the first and the second ML models for a historical data point through the first and second labels for the first data point; determining (712) whether the aggregated weighted error rate exceeds a specified threshold; and detecting (714) the concept drift when the aggregated weighted error rate exceeds the specified threshold.

2. The method of Claim 1, wherein the second ML model generates the second label for the first data point at a point in time before the first ML model generated the first label for the first data point.

3. The method of any one of Claims 1 to 2, further comprising: recording (800) the concept drift.

4. The method of any one of Claims 1 to 3, further comprising: proceeding (802) to a second data point in the plurality of data points; and retraining (804) the second ML model.

5. The method of any one of Claims 1 to 4, wherein the plurality of data points comprise a plurality of unlabeled data points, the method further comprising: retraining (806) the first ML model.

6. The method of Claim 5, wherein the retraining (806) the first ML model comprises (i) clustering the plurality of unlabeled data points into a sample for a time period to obtain a plurality of clusters of unlabeled data points, (ii) accessing labels for unlabeled data points sampled from the plurality of clusters to obtain accessed labels, (iii) propagating the unlabeled data points sampled in the time period, and (iv) training the first ML model on the accessed labels for the unlabeled data points sampled in the time period.

7. The method of Claim 6, wherein the clustering comprises (i) grouping the unlabeled data points into a sample comprising a group of newest unlabeled data points in the time period, and (ii) dividing the group of the newest unlabeled data points into the plurality of clusters of unlabeled data points.

8. The method of any one of Claims 6 to 7, further comprising sampling a respective unlabeled data point from at least one cluster to obtain a set of sampled unlabeled data points.

9. The method of any one of Claims 6 to 8, wherein training the first ML model on the accessed labels for the sampled unlabeled data points in the time period comprises (i) updating the unlabeled data points with the accessed labels to obtain labeled data points, and (ii) propagating the labeled data points over the sample of unlabeled data points.

10. The method of Claim 9, wherein the propagating the labeled data points over the sample of unlabeled data points results in an updated first ML model, the method further comprising: using (808) the updated first ML model to detect concept drift in the network.

11. The method of Claim 9, wherein the propagating the labeled data points over the sample is for the time period, the method further comprising: retraining (812) the first ML model with the labeled data points over the sample for the time period.

12. The method of any one of Claims 1 to 11, further comprising: when the aggregated weighted error rate is less than the specified threshold, repeating the receiving (702), the generating (704), the comparing (706), the identifying (708), the calculating (710), the determining (712), and the detecting (714).

13. The method of any one of Claims 1 to 12, further comprising: initializing (700) the first ML model by implementing a copy of the second ML model.

14. The method of any one of Claims 1 to 13, wherein the network comprises a telecommunications network.

15. The method of any one of Claims 1 to 14, wherein the computing device comprises a cloud-based device.

16. The method of any one of Claims 1 to 15, wherein the concept drift comprises at least one of (i) a change in a relationship between the first data point and a label, wherein in the change is represented by the first and second labels being different for the first data point, and (ii) a change in a distribution of the plurality of data points, and a change in a relationship between at least some of the data points in the plurality of data points and a respective first label and a respective second label being different for at least some of the data points.

17. A computing device (124, 900) configured to detect a concept drift in a network, the computing device comprising: processing circuitry (904); memory (906) coupled with the processing circuitry, wherein the memory includes instructions that when executed by the processing circuitry causes the computing device to perform operations comprising: receive a stream of data comprising a plurality of data points, wherein a first data point from the plurality of data points is unlabeled; generate a first label having a first value for the first data point based on use of a first machine learning, ML, model; compare the first value of the first label to a second value of a second label of a second ML model to determine whether the values of the first label and the second label match; when the values of the first and second labels are different, identify an error comprising a difference between the first and second values; calculate an aggregated weighted error rate for a range of labels and data points from a respective historical label from the first and the second ML models for a historical data pointthrough the first and second labels for the first data point; determine whether the aggregated weighted error rate exceeds a specified threshold; and detect the concept drift when the aggregated weighted error rate exceeds the specified threshold.

18. The computing device of Claim 17, wherein the operations further comprise any of the operations of Claims 2 to 16.

19. A non-transitory computer readable medium (906, 106, 110, 112, 116, 120) including program code to be executed by processing circuitry (904) of a computing device (124, 900) configured to detect a concept drift in a network, whereby execution of the program code causes the program code to perform operations comprising: receive a stream of data comprising a plurality of data points, wherein a first data point from the plurality of data points is unlabeled; generate a first label having a first value for the first data point based on use of a first machine learning, ML, model; compare the first value of the first label to a second value of a second label of a second ML model to determine whether the values of the first label and the second label match; when the values of the first and second labels are different, identify an error comprising a difference between the first and second values; calculate an aggregated weighted error rate for a range of labels and data points from a respective historical label from the first and the second ML models for a historical data point through the first and second labels for the first data point; determine whether the aggregated weighted error rate exceeds a specified threshold; and detect the concept drift when the aggregated weighted error rate exceeds the specified threshold.

20. The non-transitory computer readable medium of Claim 19, the operations further comprising any of the operations of Claims 2 to 16.

Images