Method and electronic device for automated machine learning model retraining

Abstract

A system and / or method for automated ML model retraining by an electronic device. The system and / or method may include one or more of running a first ML model and a second ML model, predicting an accuracy degradation of the first ML model using the second ML model, determining whether the predicted accuracy degradation satisfies a predefined threshold, and / or retraining the first ML model when the predicted accuracy degradation satisfies a predefined threshold.

Description

Technical Field

[0001] Various example embodiments relate to electronic devices, and / or to methods and electronic devices for automated machine learning (ML) model retraining. Background Art

[0002] The fifth generation (5G) cellular network is a service-based architecture that is deployed to support hundreds to thousands of services. Management of 5G cellular networks and manually understanding the patterns is a very difficult task. Therefore, operators have incorporated machine learning (ML) based solutions that can pre-understand and predict problems in 5G cellular networks. Therefore, operators can take decisions to mitigate the problems in advance. There are millions of base stations and trillions of network devices in 5G cellular networks along with their respective ML models.

[0003] Network slicing is another important feature of 5G cellular networks to achieve optimal use of network resources for multiple services. There are currently many solutions for optimizing network slicing using ML models. For most of these solutions, effective training of ML models is performed for each of the network slices. This involves collecting the required amount of data for these slices for a sufficient period of time before training and deploying the ML model. When the number of services and network slices increases exponentially, there is a need to reduce ML resources by using existing, already trained models. Since many new types of slices will also be created, the amount of data available for training will be very limited.

[0004] Since closed-loop corrective / preventive actions taken on predicted problems over time may change the time series data, and new unplanned events occurring in 5G cellular networks may affect network key performance indicators (KPIs), ML models may suffer from performance degradation. In addition, new network service requests with new and diverse service requirements may be received at the 5G cellular network, which requires new network service / network slice deployment. Deploying new network services / network slices for each new service request results in a significant waste of ML resources, which may further cause non-optimized deployments in 5G cellular networks resulting in degraded Quality of Service (QoS) or Quality of Experience (QoE), failure to meet Service Level Agreements (SLA), high Operational Expenditure (OPEX), and inability of subscribers to consume network services.

[0005] The above information is presented as background information only to assist with an understanding of the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above might be applicable as prior art with respect to the present disclosure. Summary of the invention

[0006] Solution

[0007] Performance degradation of ML models can be prevented or reduced by ML model retraining by timely identifying performance degradation. In addition, ML model retraining is an expensive task because it requires specialized hardware such as graphics processing units (GPUs) or tensor processing units (TPUs), and in anticipation of deploying a large number of ML models, the process must be automated. Service providers need predictive and inexpensive means to retrain ML models to make ML deployment for 5G cellular networks economically feasible. Manual intelligent solutions exist to identify performance degradation of ML models and correct performance degradation. However, manual intelligent solutions rely on human intervention, which delays processing performance degradation of ML models. Manually detecting performance degradation of ML models and manually mitigating performance degradation may result in QoS / QoE degradation of services, which is disadvantageous to operators. Therefore, it is desirable to provide automatic prediction of the possibility of ML model performance degradation or data pattern change to meet the key requirements of all services deployed in 5G cellular networks.

[0008] Various example embodiments provide methods and / or electronic devices for automated ML model retraining. ML models need or can be designed for retraining due to performance degradation that occurs when the accuracy of the ML model's predictions based on new input values ​​degrades compared to the accuracy during the training period. The proposed example electronic device is a predictive ML training resource management and optimizer that will reduce the cost of ML model retraining.

[0009] Various example embodiments herein automatically deploy ML model retraining when performance degradation of the ML model is detected. The proposed method is used to automate the process, prevent or reduce performance degradation of the ML model, which saves time. The proposed method solves the problem of increased capital expenditures (CAPEX) and OPEX due to intelligence, because operators can afford to have limited ML resources and servers, but cannot afford to waste ML resources and servers in the case of ML service degradation. Retraining enhances the prediction accuracy of the ML model, which in turn helps optimize network services.

[0010] Various example embodiments herein intelligently and / or automatically identify a suitable trained ML super model to use for predictions for a newly created network slice for which only a minimal amount of data is available.

[0011] Therefore, various example embodiments of the present invention provide a method for automated ML model retraining by an electronic device. The method may include running a first ML model and a second ML model by an electronic device. The method may include predicting, by the electronic device, accuracy degradation of the first ML model using the second ML model. The method may include determining, by the electronic device, whether the predicted accuracy degradation satisfies a predefined threshold. The method may include retraining the first ML model by the electronic device when the predicted accuracy degradation satisfies the predefined threshold.

[0012] In an example embodiment, where the degradation in accuracy is due to an unintended event occurring in the first ML model.

[0013] In an example embodiment, wherein predicting, by the electronic device, the accuracy degradation of the first ML model using the second ML model may include receiving, by the electronic device, data on the accuracy of the first ML model, which includes at least one of model type, parameters and hyperparameters, network nodes, cell models, slice / cell configuration information, existing models that can be used for transfer learning, model training time, model prediction accuracy, resources for model training, extraction time, time window for data extraction, data generation mode, model accuracy data, and execution time of each training pipeline; storing, by the electronic device, the data on the accuracy of the first ML model in a retraining data store; and predicting, by the electronic device, the accuracy degradation of the first ML model by analyzing the data on the accuracy of the first ML model using the second ML model.

[0014] In an example embodiment, wherein retraining the first ML model by the electronic device may include estimating, by the electronic device, an expected time to complete model retraining and data extraction based on accuracy data stored in a retraining data store; extracting, by the electronic device, accuracy data of the first ML model from the retraining data store; predicting, by the electronic device, incoming requests for the first ML model; estimating, by the electronic device, resources and resource constraints; creating, by the electronic device, a plan for retraining the first ML model based on the predicted incoming requests, the expected time to complete retraining, the estimated resources and resource constraints; and scaling, by the electronic device, ML resources up / down and inward / outward based on the created plan; and triggering, by the electronic device, retraining of the first ML model based on the created plan.

[0015] In another example embodiment, retraining the first ML model by the electronic device may include receiving, by the electronic device, a request to configure an ML service using a new network slice; determining, by the electronic device, a network slice having the first ML model, the network slice having the first ML model being similar to the new network slice and capable of being used for transfer learning; predicting, by the electronic device, a super model for transfer learning of the first ML model and remaining layers of the first ML model to be retrained based on input from a model registry; and triggering, by the electronic device, retraining of the remaining layers of the first ML model.

[0016] Various example embodiments provide an electronic device for automated ML model retraining. The electronic device may include an active retraining engine, the active retraining engine including a circuit, a memory, a processor, wherein the active retraining engine is directly or indirectly coupled to the memory and the processor. The active retraining engine may be configured to run a first ML model and a second ML model. The active retraining engine may be configured to predict an accuracy degradation of the first ML model using the second ML model. The active retraining engine may be configured to determine whether the predicted accuracy degradation meets a predefined threshold. The active retraining engine may be configured to retrain the first ML model when the predicted accuracy degradation meets the predefined threshold.

[0017] These and other aspects of the example embodiments herein will be better appreciated and understood when considered in conjunction with the following description and accompanying drawings. However, it should be understood that the following description, although indicating preferred embodiments and many of their specific details, is given by way of illustration and not limitation. Many changes and modifications may be made within the scope of the embodiments, and the example embodiments herein include all such modifications. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other aspects, features and advantages of certain embodiments will become more apparent through the following detailed description in conjunction with the accompanying drawings, in which:

[0019] Figure 1 is a block diagram of an electronic device for automated ML model retraining according to an example embodiment;

[0020] Figure 2 is a block diagram of a system including an active retraining engine for retraining a first ML model having accuracy degradation according to an example embodiment;

[0021] Figure 3 is a flow chart illustrating a method for automated ML model retraining according to an example embodiment;

[0022] Figure 4is a flow chart illustrating the inner workings of an intelligent resource optimization scheduling engine for automatic accuracy degradation prediction of a first ML model in the telecommunications domain according to an example embodiment;

[0023] Figure 5 is a flow chart illustrating the inner workings of a migration learning manager for intelligent automated migration of learning for network slicing according to an example embodiment;

[0024] Figures 6A to 6C A comparison of existing methods of ML model retraining and the proposed method of automated ML model retraining according to example embodiments is shown;

[0025] Figure 7 shows different network architecture implementations with an active retraining engine according to example embodiments;

[0026] Figure 8 An example scenario of automated ML model retraining in response to degradation in accuracy of a first ML model according to an example embodiment is shown;

[0027] Fig. 9 illustrates an example scenario of automated ML model retraining due in response to receiving a new service request according to an example embodiment; and

[0028] Fig.10 A wireless communication system according to an example embodiment is shown.

[0029] The same reference numerals are used throughout the drawings to denote the same elements. DETAILED DESCRIPTION

[0030] The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. It includes various specific details to assist in understanding, but these details are considered to be exemplary only. Therefore, it will be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the various embodiments described herein without departing from the scope of the present disclosure. In addition, for the sake of clarity and conciseness, descriptions of well-known functions and structures may be omitted.

[0031] The terms and words used in the following description and claims are not limited to the bibliographical meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only and not for the purpose of limiting the present disclosure as defined by the appended claims and their equivalents.

[0032] It should be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more of such surfaces.

[0033] In various examples of the present disclosure described below, a hardware method will be described as an example. However, since various embodiments of the present disclosure may include techniques utilizing both hardware-based methods and software-based methods, they are not intended to exclude software-based methods.

[0034] As used herein, terms referring to a combination (e.g., combining, grouping, combining, aggregating, uniting, integrating, unifying), terms referring to a signal (e.g., packet, message, signal, information, signaling), terms referring to a resource (e.g., segment, symbol, slot, subframe, radio frame, subcarrier, resource element (RE), resource block (RB), bandwidth part (BWP), opportunity), terms used to refer to any operating state (e.g., step, operation, process), terms referring to data (e.g., packet, message, user flow, information, bit, symbol, codeword), terms referring to a channel, terms referring to a network entity (e.g., distribution unit (DU), radio unit (RU), central unit (CU), control plane (CU-CP), user plane (CU-UP), O-DU (Open Radio Access Network (O-RAN) DU), O-RU (O-RAN RU), O-CU (O-RAN CU), O-CU-UP (O-RAN CU-UP), O-CU-CP (O-RAN The terms CU-CP) and components of a device or apparatus are shown only for the convenience of description in the present disclosure. Therefore, the present disclosure is not limited to those terms described below, and other terms with the same or equivalent technical meanings may be used for this purpose. In addition, as used herein, terms such as "module", "unit", "component", "main body", etc. may refer to a structure or unit of at least one shape for processing a specific function.

[0035] In addition, throughout the present invention, expressions such as 'above' or 'below' may be used to determine whether a specific condition is met or satisfied, but they are merely descriptions for expressing examples and are not intended to exclude the meaning of 'greater than or equal to' or 'less than or equal to'. Respectively, a condition described as "greater than or equal to" may be replaced with an expression such as "above", a condition described as "less than or equal to" may be replaced with an expression such as "below", and a condition described as "greater than or equal to and below" may be replaced with "above and less than or equal to". In addition, hereinafter, 'A' to 'B' means at least one of the elements from A (including A) to B (including B). hereinafter, 'C' and / or 'D' means including at least one of 'C' or 'D', i.e. {'C', 'D', or 'C' and 'D'}.

[0036] The present disclosure uses terms used in some communication standards (e.g., 3rd Generation Partnership Project (3GPP), Scalable Radio Access Network (xRAN), Open Radio Access Network (O-RAN), etc.) to describe various embodiments, but they are merely examples for explanation, and various embodiments of the present disclosure can be easily modified and applied thereto even in other communication systems.

[0037] The example embodiments of this article and their various features and advantageous details are explained more fully with reference to the non-limiting embodiments shown in the accompanying drawings and described in detail in the following description. Descriptions of well-known components and processing techniques are omitted so as not to unnecessarily obscure the example embodiments of this article. In addition, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments. Unless otherwise stated, the term "or" as used herein refers to a non-exclusive or. The examples used herein are intended only to facilitate understanding of the ways in which the example embodiments of this article can be practiced, and further enable those skilled in the art to practice the example embodiments of this article. Therefore, the examples should not be interpreted as limiting the scope of the example embodiments of this article.

[0038] Embodiments may be described and illustrated in terms of blocks that perform one or more functions described. These blocks, which may be referred to herein as managers, units, modules, hardware components, etc., are physically implemented by analog and / or digital circuits (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc.), and may optionally be driven by firmware. The circuit may be embodied, for example, in one or more semiconductor chips, or on a substrate support such as a printed circuit board. The circuits that make up the blocks may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuits), or by a combination of dedicated hardware that performs some functions of the blocks and a processor that performs other functions of the blocks. Without departing from the scope, each block of the embodiment may be physically divided into two or more interactive and discrete blocks. Similarly, without departing from the scope, the blocks of the embodiment may be physically combined into more complex blocks.

[0039] The accompanying drawings are used to help easily understand various technical features, and it should be understood that the exemplary embodiments are not limited by the accompanying drawings. Therefore, various embodiments should be interpreted as extending to any changes, equivalents and substitutes other than those specifically set forth in the accompanying drawings. Although the terms first, second, etc. can be used to describe various elements in this article, these elements should not be limited by these terms. These terms are usually only used to distinguish one element from another element.

[0040] Throughout this disclosure, the terms "accuracy degradation" and "performance degradation" are used interchangeably and have the same meaning.

[0041] Therefore, the example embodiments of the present invention provide a method for automated ML model retraining by an electronic device. The method includes running a first ML model and a second ML model by an electronic device. The method includes predicting, by the electronic device, accuracy degradation of the first ML model using the second ML model. The method includes determining, by the electronic device, whether the predicted accuracy degradation meets a predefined threshold. The method includes retraining the first ML model by the electronic device when the predicted accuracy degradation meets the predefined threshold.

[0042] Therefore, example embodiments of the present invention provide an electronic device for automated ML model retraining. The electronic device includes an active retraining engine, the active retraining engine including circuitry, memory, and a processor, wherein the active retraining engine is directly or indirectly coupled to the memory and the processor. The active retraining engine is configured to run a first ML model and a second ML model. The active retraining engine is configured to predict an accuracy degradation of the first ML model using the second ML model. The active retraining engine is configured to determine whether the predicted accuracy degradation meets a predefined threshold. The active retraining engine is configured to retrain the first ML model when the predicted accuracy degradation meets the predefined threshold.

[0043] The active retraining engine can automatically predict when the first ML model will need to be retrained. The active retraining engine can automatically predict the resource requirements for training the first ML model. The active retraining engine can provide a platform for optimizing and reducing training / retraining resources. The active retraining engine can automate the use of transfer learning from previous ML models to reduce the time and cost of new ML models for network rollout (i.e., using super models).

[0044] The active retraining engine may include a data store to maintain historical information about previous training / retraining, a set of 3 neural networks for: predicting when the ML model accuracy falls below a predefined threshold, predicting the time required for data extraction and transformation to retrain a specific ML model, and predicting the time and resources required for retraining of the ML model over time.

[0045] An automated method for predicting the degradation of the accuracy of the first ML model through resource management and optimization can reduce the cost of ML model retraining. In addition, the electronic device can periodically predict the degradation of the accuracy of the first ML model and notify the changes of the top N important units and slice parameters. In addition, the electronic device can predict when the degradation of the accuracy of the first ML model is below a predefined threshold, which further triggers the retraining of the first ML model. In addition, the electronic device can predict the time required for data extraction and transformation for training a specific ML model retraining. In addition, the electronic device can predict the time and resources required for retraining the first ML model over time. The retraining data storage of the electronic device can maintain historical information about previous training / retraining of the first ML model. In addition, the method proposed for the example embodiment can allow the electronic device to be used for intelligent automated transfer learning of network slices by detecting slice similarity and detecting the best ML model to be used for transfer learning.

[0046] Unlike some existing methods and systems, for an already trained ML model running in the system, the active retraining engine can, for example, use an artificial intelligence-based approach to actively predict when the trained ML model will degrade and cross a predefined threshold, and thus trigger retraining of the trained ML model before the degradation crosses the predefined threshold. Thus, the active retraining engine can be at least one step ahead and predict when to retrain the trained ML model, and can retrain before the ML model degradation crosses a limit. Thus, automation to prevent or reduce ML service degradation can save time.

[0047] Unlike some existing methods and systems, when a new service request is received, the active retraining engine can find out the super model for the new service request and can determine the layers of the ML model that can be reused and the layers that need to be retrained to provide the service. In addition, the active retraining engine can proactively identify the most suitable training model for retraining. Therefore, the prediction accuracy can be further enhanced, which in turn can help optimize network services.

[0048] Referring now to the drawings, and more particularly to Figures 1 to 9 , various example embodiments are shown. Each embodiment herein can be used in combination with any other embodiment described herein.

[0049] Figure 1 is a block diagram of an electronic device (100) for automated ML model retraining according to an example embodiment. Examples of the electronic device (100) include, but are not limited to, smartphones, tablet computers, personal digital assistants (PDAs), desktop computers, Internet of Things (IoT), wearable devices, and the like. In an embodiment, the electronic device (100) includes an active retraining engine (110) including circuitry, a memory (120), at least one processor (130) including processing circuitry, a communicator (140) including communication circuitry, and a main display (150), wherein the main display is a physical hardware component that can be used to display to a user. Examples of the main display include, but are not limited to, a light emitting diode display, a liquid crystal display, a projector, and the like. The active retraining engine (110) may be implemented by processing circuitry such as one or more of the following: logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, and the like, and may optionally be driven by firmware. The circuitry may, for example, be embodied in one or more semiconductor chips, or on a substrate support such as a printed circuit board.

[0050] A first ML model and a second ML model are run in an electronic device (100). An active retraining engine (110) predicts accuracy degradation of the first ML model using the second ML model. The accuracy degradation is due to an unplanned event that occurs in the first ML model. In addition, the active retraining engine (110) determines whether the predicted accuracy degradation meets a predefined threshold. In addition, when the predicted accuracy degradation meets the predefined threshold, the active retraining engine (110) retrains the first ML model.

[0051] In an embodiment, the active retraining engine (110) receives data on the accuracy of the first ML model, including model type, parameters and hyperparameters, network nodes, cell models, slice / cell configuration information, existing models that can be used for transfer learning, model training time, model prediction accuracy, resources used for model training, extraction time, time window for data extraction, data generation mode, model accuracy data, and execution time of each training pipeline. In addition, the active retraining engine (110) stores the data on the accuracy of the first ML model in the retraining data storage (111) (refer to Figure 2 ). In addition, the active retraining engine (110) predicts the accuracy degradation of the first ML model by analyzing the data of the accuracy of the first ML model using the second ML model.

[0052] In an embodiment, the active retraining engine (110) estimates an expected time to complete model retraining and data extraction based on the data of accuracy stored in the retraining data store (111). In addition, the active retraining engine (110) extracts the data of accuracy of the first ML model from the retraining data store (111). In addition, the active retraining engine (110) predicts incoming requests for the first ML model. In addition, the active retraining engine (110) estimates resources and resource constraints. In addition, the active retraining engine (110) creates a plan for retraining the first ML model based on the predicted incoming requests, the expected time to complete retraining, the estimated resources and resource constraints. In addition, the active retraining engine (110) scales resources up / down and in / out based on the created plan. In addition, the active retraining engine (110) triggers retraining of the first ML model based on the created plan.

[0053] In another embodiment, the active retraining engine (110) receives a request to configure an ML service with a new network slice. In addition, the active retraining engine (110) determines a network slice with a first ML model that is similar to the new network slice and capable of transfer learning. In addition, the active retraining engine (110) predicts a super model of the first ML model for transfer learning based on input from a model registry, and remaining layers of the first ML model to be retrained. In addition, the active retraining engine (110) triggers retraining of the remaining layers of the first ML model.

[0054] The memory (120) stores the first ML model and the second ML model. The memory (120) stores instructions to be executed by the processor (130). The memory (120) may include a non-volatile storage element. Examples of such non-volatile storage elements may include a magnetic hard disk, an optical disk, a floppy disk, a flash memory, or a form of an electrically programmable memory (EPROM) or an electrically erasable programmable (EEPROM) memory. In addition, in some examples, the memory (120) may be considered a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or a propagating signal. However, the term "non-transitory" should not be interpreted as the memory (120) being non-removable. In some examples, the memory (120) may be configured to store a larger amount of information than its storage space. In some examples, the non-transitory storage medium may store data that may change over time (e.g., in a random access memory (RAM) or a cache memory). The memory (120) may be an internal storage unit, or it may be an external storage unit of the electronic device (100), cloud storage, or any other type of external storage.

[0055] The processor (130) is configured to execute instructions stored in the memory (120). The processor (130) may be a general-purpose processor such as a central processing unit (CPU), an application processor (AP), etc., a graphics processing unit only such as a graphics processing unit (GPU), a visual processing unit (VPU), etc. The processor (130) may include multiple cores to execute instructions. The communicator (140) is configured to perform internal communication between hardware components in the electronic device (100). In addition, the communicator (140) is configured to facilitate communication between the electronic device (100) and other devices via one or more networks (e.g., radio technologies). The communicator (140) includes electronic circuits specific to standards for implementing wired or wireless communication.

[0056] Functions associated with the ML model may be performed by a non-volatile / volatile memory (120) and a processor (130). One or more processors (130) control the processing of input data according to a predefined operating rule or ML model stored in the non-volatile / volatile memory (120). The predefined operating rule or ML model is provided by training or learning. Here, providing by learning means forming a predefined operating rule or an ML model of desired characteristics by applying a learning method to a plurality of learning data. Learning may be performed in the electronic device (100) itself in which the ML model according to the embodiment is executed, and / or may be implemented by a separate server / system. The ML model may be or include a plurality of neural network layers. Each layer has a plurality of weight values, and the layer operation is performed by the calculation of the previous layer and the operation of the plurality of weights. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks. A learning method is a method for training a predetermined target device (e.g., a robot) using a plurality of learning data to enable, allow, or control the target device to make a determination or prediction. Examples of learning methods include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0057] although Figure 1 The hardware components of the electronic device (100) are shown, but it should be understood that other embodiments are not limited thereto. In other embodiments, the electronic device (100) may include fewer or greater numbers of components. In addition, the labels or names of the components are for illustrative purposes only and do not limit the scope. One or more components may be combined together to perform the same or substantially similar functions for automated ML model retraining.

[0058] Figure 2 is a block diagram of a system (1000) according to an example embodiment, the system (1000) including an active retraining engine (110) for retraining a first ML model with accuracy degradation. In an embodiment, the system (1000) may be an electronic device (100). The system (1000) includes an active retraining engine (110), a data lake (201), a data extractor (202), an artificial intelligence (AI) server or a machine learning model operation platform (300), a performance monitoring engine (204), a trigger engine (203), and a prediction service / model (205).

[0059] In an embodiment, the active retraining engine (110) includes a retraining data store (111), an intelligent resource optimization scheduling engine (IROSE) (112), a transfer learning manager (TLM) (113) and a policy controller (114). The retraining data store (111), IROSE (112), TLM (113) and the policy controller (114) are implemented by processing circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc., and can be optionally driven by firmware. The circuits can be embodied, for example, in one or more semiconductor chips, or on a substrate support such as a printed circuit board. Each processor herein includes a processing circuit.

[0060] The active retraining engine (110) is directly or indirectly connected to the data lake / storage device / storage (201), the data extractor (202), the AI / ML OPS platform (300), and the performance monitoring engine (204), wherein the AI ​​server and / or ML OPS platform (300) is directly or indirectly connected to the trigger engine (203) and the prediction service / model (205). The AI ​​server and / or ML OPS platform (300) includes a training pipeline manager (310), a feature store (320), a data extractor (330), a cloud resource manager (340), a request queue (350), a model registry (360), a trained ML model (370), and a model server (380). The training pipeline manager (310) sequentially performs data extraction, data validation, data preparation, model training, model evaluation, and model validation. On Prem, K8S, GCP, AWS are examples of cloud resource managers (340).

[0061] At 401-405, the retraining data store (111) collects information about the accuracy of the current ML model from different platforms, including the data lake (201), the data extractor (202), the training pipeline manager (310), the cloud resource manager (340), and the performance monitoring engine (204). The retraining data store (111) has adapters to collect information about the performance of the first ML model for monitoring. The retraining data store (111) collects real data and prediction information and provides feedback based on the prediction accuracy. The retraining data store (111) stores the collected data, which is a continuous process. The retraining data store (111) passes the data to the IROSE (112), where the IROSE (112) further predicts the need for retraining or transfer learning of the first ML model.

[0062] IROSE (112) predicts ML model retraining for an already running ML service or a new network service request. At 406, IROSE (112) predicts data extraction time for each ML model based on data in the retraining data store (111). IROSE (112) uses reinforcement learning to predict the training time for each request and each resource for each request based on the data in the retraining data store (111). IROSE (112) triggers the created plan as an action. After deploying the updated ML model, IROSE (112) decides whether to keep the new ML model in the electronic device (100) or roll back to the previous ML model based on the performance monitoring results after obtaining feedback from the newly deployed model. At 407, IROSE (112) triggers data extraction for a specific use case of expected retraining. At 408, IROSE (112) creates an execution plan (i.e., requests for serial and / or parallel training) based on the predicted incoming requests, the predicted time to complete training, the estimated resources, and the resource constraints so as to reduce the total training time. At 409, IROSE (112) scales resources up / down and in / out based on the predicted training plan. At 410, IROSE (112) triggers retraining in the training pipeline manager (310) based on the actions performed in steps 407-409 to create a trained ML model (370).

[0063] At 411, a new network slice is to be deployed in the network, i.e., an ML service needs to be configured in the new slice. IROSE (112) compares and finds the nearest slice with the same ML service. At 412, IROSE (112) contacts TLM (113), where TLM (113) predicts a super model for transfer learning based on input from the model registry (360) and finds the number of remaining layers to be retrained. At 413, IROSE (112) then triggers TLM (113) using the super model and using the layers to be retrained as input to create a trained ML model (370).

[0064] The TLM (113) predicts those ML models that are most suitable for transfer learning to the new trained ML model, which may also be a parameter in the training request. The TLM (113) uses deep dynamic learning to identify the number of layers of the first ML model to retrain.

[0065] The policy controller (114) may include a YAML file that may be used to define scheduling parameters, policies, and / or maximum possible scalable load.

[0066] although Figure 2The hardware components of the active retraining engine (110) are shown, but it should be understood that other embodiments are not limited thereto. In other embodiments, the active retraining engine (110) may include fewer or greater numbers of components. In addition, the labels or names of the components are for illustrative purposes only and do not limit the scope. One or more components may be combined together to perform the same or substantially similar functions for retraining a first ML model with degraded accuracy.

[0067] Figure 3 is a flowchart (S300) illustrating a method for automated ML model retraining according to an example embodiment. In an embodiment, the method allows an active retraining engine (110) to perform steps S301-S304 of the flowchart (S300). At step S301, the method includes running a first ML model and a second ML model. At step S302, the method includes predicting an accuracy degradation of the first ML model using the second ML model. At step S303, the method includes determining whether the predicted accuracy degradation meets a predefined threshold. At step S304, the method includes retraining the first ML model when the predicted accuracy degradation meets the predefined threshold.

[0068] The proposed method considers various different scenarios that may lead to ML model performance degradation. In addition, a detailed architecture is proposed in this disclosure to detect ML model performance degradation and explain how it will coexist and cooperate with the AI ​​server.

[0069] The various actions, actions, blocks, steps, etc. in the flowchart (S300) may be performed in the order presented, in a different order, or simultaneously. In addition, in some embodiments, some of the actions, actions, blocks, steps, etc. may be omitted, added, modified, skipped, etc. without departing from the scope.

[0070] Figure 4is a flow chart (S400) illustrating the internal workings of an IROSE (112) for automatic accuracy degradation prediction of a first ML model in the telecommunications domain according to an example embodiment. Closed-loop corrective / preventive actions taken over time for prediction problems may change time series data. At S401, a new trained model (e.g., a first ML model) is created at an electronic device (100), where no data is available for model degradation at this stage. At S402, the IROSE (112) collects data related to repeated ML model accuracy degradation, including detection of data pattern changes or model prediction errors, and slice-level or cell-level configuration snapshots. At S403, the IROSE (112) creates a time series of ML model accuracy degradation when ML model accuracy degradation is detected. Alternatively, at S404, the IROSE (112) determines feature importance of different slice-level or cell-level configuration parameters and a list of top N parameters upon receiving a service request. At S405, IROSE (112) triggers periodic prediction of ML model accuracy degradation and notifies changes to the top N parameters. At S406, if ML model accuracy degradation is predicted or if any configuration is changed for the registered parameters, IROSE (112) triggers retraining. Therefore, IROSE (112) predicts ML model accuracy degradation in advance and triggers retraining of the ML model early, so that the ML model is updated in a shorter time.

[0071] Figure 5 is a flow chart (S500) illustrating the internal working of the TLM (113) for intelligent automated migration of learning of network slices according to an example embodiment. At S501, a new network slice (e.g., a first ML model) is created at an electronic device (100), where no KPI data is available for the newly created network slice. Therefore, the ML model cannot be trained and deployed. At S502, the TLM (113) detects slice similarity by comparing slice configuration parameters based on a threshold configuration, and provides a list of the top N similar slice IDs. At S503, the TLM (113) may, for example, collect minimum KPI data (e.g., 7 days of KPI data) for the newly created slice. At S504, the TLM (113) may perform test predictions using the KPI data of the new slice and the trained ML models of the detected similar slices. At S505, the TLM (113) may select the best ML model based on the prediction of the test results. At S506, the TLM (113) may deploy the selected ML model. At S507 , the TLM ( 113 ) triggers retraining based on the availability of new data for the slice.

[0072] Figures 6A to 6C A comparison of existing methods of ML model retraining and the proposed method of automated ML model retraining according to example embodiments is shown. Fig. 6A and 6B Three stages of an existing method of ML model retraining are shown. In the first stage of the existing method as shown in 610, at 602, the training pipeline manager (310) of the AI ​​server (300) analyzes the network KPI data (601) and trains the ML model (603) and predicts anomalies at the network (604) and takes preventive or mitigation measures.

[0073] In the second stage of the existing method as shown in 620, new network KPI data (605) is generated after the preventive / reducing measures. In addition, the AI ​​server (300) detects ML model performance degradation, and the ML model performance degradation further causes degradation of the network (604).

[0074] In the third stage of the prior art method, as shown in 630, upon detecting degradation in the network (604), the performance monitoring engine (204) of the AI ​​server (300) retrieves the ML model and predicts the network KPI based on the changes in the network KPI data. In addition, the performance monitoring engine (204) checks the drift of the prediction relative to the previous prediction and retrains the ML model based on the drift (e.g., see Fig. 6A and 6B 606 in the above table).

[0075] Figure 6C Two phases of the proposed automated ML model retraining method are shown. In the first phase of the proposed method as shown in 640, the training pipeline manager (310) of the AI ​​server (300) analyzes the network KPI data (601) and trains the ML model (603), predicts anomalies (604) at the network and takes preventive / reducing actions based on the anomalies. In parallel, the active retraining engine (110) receives accuracy data from all components (201, 310, 204) of the AI ​​server (300), as well as feedback from the network (604), and detects checks for changes in KPI data due to network updates, configurations, or changes in traffic patterns, and predicts drift in the ML model. The active retraining engine (110) actively detects and triggers the AI ​​server (300) to retrain the ML model so that when new KPI data is available, ML model degradation does not occur.

[0076] In the second phase of the proposed method as shown in 650, after the prevention / reduction actions and training have been actively completed according to the retrained model, new network KPI data (605) is generated. Therefore, no ML model accuracy degradation occurs at the AI ​​server (300) and no accuracy degradation occurs in the network (604).

[0077] Figure 7Different network architecture implementations with an active retraining engine (110) according to example embodiments are shown. The network architecture implementation shown in (710) includes a standalone server, a self-organizing network (SON) server, and an eNodeB / gNnodeB (eNB / gNB). The standalone server is directly or indirectly connected to the SON via an interface N (ITF-N). The SON is directly or indirectly connected to the eNB / gNB. The standalone server includes an AI server (300A) with an active retraining engine (110), multiple applications (App1-AppN), and a policy controller (114).

[0078] Another network architecture implementation shown in (720) includes a SON / SON server connected to an eNB / gNB. The SON server includes an AI server (300A) with an active retraining engine (110), multiple applications (App1-AppN) and a policy controller (114).

[0079] Another network architecture implementation shown in (730) is an open radio access network (O-RAN) compatible architecture, including service management and orchestration (SMO), a non-real-time radio intelligent controller (non-RT RIC), a near-real-time radio intelligent controller (near-RT RIC), and an eNB / gNB. The SMO is directly or indirectly connected to the non-RT RIC. The non-RT RIC is directly or indirectly connected to the near-RT RIC. The near-RT RIC is directly or indirectly connected to the eNB / gNB through at least an E2 interface. The SMO includes an AI server (300A) with an active retraining engine (110). The non-RT RIC includes an application (App 1), congestion prediction and mitigation (CPM), and a policy controller (114).

[0080] Figure 8 An example scenario of automated ML model retraining in response to degradation in accuracy of a first ML model according to an example embodiment is shown. AI server 300A is an example of system 1000. Consider that AI server 300A is providing virtual reality (VR) services to at least VR device 840 via cellular network 850. Later, another VR device (830) begins accessing the VR service via cellular network (850), which creates congestion in the cellular network for providing VR services to multiple VR devices (830, 840). The main cause of the congestion is due to the degradation in performance of the ML model of AI server 300A in providing VR services to multiple VR devices 830, 840. Upon predicting congestion in the cellular network, AI server (300A) triggers retraining of the ML model to process network resources for providing VR services to multiple VR devices (830, 840) without creating congestion (e.g., see 820).

[0081] To trigger the retraining of the ML model, the retraining data store (111) collects and stores accuracy data, including data generation patterns (801) from the data lake (201), extraction time (802) from the data extractor (202), execution time (803) for training each pipeline for DL ​​PRB utilization KPIs for detecting congestion and used cloud resources from the training pipeline manager (310), and neural network model accuracy data (804) for DL ​​PRB utilization prediction for congestion from the performance monitoring engine (204). At 805, IROSE (112) predicts DL PRB utilization KPI prediction model degradation due to changes in KPI data patterns in the field based on the accuracy data. In addition, at 806, IROSE (112) scales up cloud resource scheduling for training congestion-related KPIs and prioritizes model training requests over other training requests in the training pipeline manager (310). Furthermore, at 807 , IROSE ( 112 ) starts retraining of the DL PRB KPI prediction model.

[0082] Fig. 9 An example scenario of automated ML model retraining due in response to receiving a new service request according to an example embodiment is shown. AI server 300A is an example of system 1000. Consider that AI server 300A is running a car network slice of car 930 via cellular network 950. Later, VR device (940) joins cellular network (950) and requests access to VR services via cellular network (950). Based on the request, AI server (300A) creates a VR slice with a new ML model for providing VR services to VR device (940). A new ML model is created using the proposed method.

[0083] Initially, the retraining data store (111) collects and stores accuracy data, including data generation patterns (901) from the data lake (201), extraction time (902) from the data extractor (202), execution time (903) for training each pipeline for the VR slice KPI data used and the corresponding cloud resources being used for the pipeline from the training pipeline manager (310), and neural network model accuracy data (904) for VR slice data prediction from the performance monitoring engine (204). At 905, IROSE (112) detects new service requests for VR slices in the network field based on the accuracy data, which VR slice is similar to the vehicular network slice in the cellular network (950). In addition, at 906, IROSE (112) uses the vehicular network slice training model as a super model and retrains the last two layers of the vehicular network slice ML model on top of it for VR slices. In addition, at 907, IROSE (112) obtains a vehicular network model that can be used for transfer learning of VR slices.

[0084] Fig.10 A wireless communication system according to an example embodiment is shown.

[0085] refer to Fig.10 , which shows a base station 1010 and a terminal 1020 as parts of nodes using a wireless channel in a wireless communication system. Fig.10 Only one base station is shown, but the wireless communication system may further include another base station that is the same as or similar to the base station 1010 .

[0086] The base station 1010 is a network infrastructure that provides wireless access to the terminal 1020. The base station 1010 may have coverage defined based on the distance at which a signal can be transmitted. In addition to the term "base station", the base station 1010 may also be referred to as an "access point (AP)", "eNode B (eNB)", "fifth generation node", "next generation node B (gNB)", "wireless point", "transmission / reception" or other terms having the same or equivalent meanings.

[0087] The terminal 1020, which is a device used by a user, communicates with the base station 1010 through a wireless channel. The link from the base station 1010 to the terminal 1020 is called a downlink (DL), and the link from the terminal 1020 to the base station 1010 is called an uplink (UL). Fig.10, but the terminal 1020 and other terminals can communicate with each other through a wireless channel. In this context, the link (device-to-device link, D2D) between the terminal 1020 and another terminal is called a side link, and the side link can be mixed with the PC5 interface. In some other embodiments of the present disclosure, the terminal 1020 can be operated without any user participation. According to an embodiment of the present disclosure, the terminal 1020 is a device that performs machine type communication (MTC) and may not be carried by a user. In addition, according to an embodiment of the present disclosure, the terminal 1020 may be a narrowband (NB)-Internet of Things (IoT) device.

[0088] Terminal 1020 may be referred to as “user equipment (UE)”, “customer client equipment (CPE)”, “mobile station”, “subscriber station”, “remote terminal”, “wireless terminal”, “electronic device”, “user device” or any other term having the same or equivalent technical meaning.

[0089] The base station 1010 may perform beamforming with the terminal 1020. The base station 1010 and the terminal 1020 may transmit and receive radio signals in a relatively low frequency band (e.g., FR 1 (frequency range 1) of NR). In addition, the base station 1010 and the terminal 1020 may transmit and receive radio signals in a relatively high frequency band (e.g., FR 2 (or FR 2-1, FR 2-2, FR 2-3), FR 3, or a millimeter wave (mmWave) band (e.g., 28 GHz, 30 GHz, 38 GHz, 60 GHz) of NR). In order to improve channel gain, the base station 1010 and the terminal 1020 may perform beamforming. In this context, beamforming may include transmit beamforming and receive beamforming. The base station 1010 and the terminal 1020 may assign directivity to a transmit signal or a receive signal. To this end, the base station 1010 and the terminal 1020 may select a serving beam through a beam search or beam management process. After selecting the serving beam, subsequent communication may be performed through resources having a quasi co-location (QCL) relationship with resources that have transmitted the serving beam.

[0090] The first antenna port and the second antenna port may be evaluated as being in such a QCL relationship if a wide-scale characteristic of a channel carrying symbols on a first antenna port can be estimated from a channel carrying symbols on a second antenna port. For example, the wide-scale characteristic may include at least one of delay spread, Doppler spread, Doppler shift, average gain, average delay, and spatial receiver parameters.

[0091] Despite Fig.10In the embodiment, both the base station 1010 and the terminal 1020 are described as performing beamforming, but the embodiments of the present disclosure are not necessarily limited thereto. In some embodiments of the present disclosure, the terminal may or may not perform beamforming. Similarly, the base station may or may not perform beamforming. That is, only one of the base station and the terminal may perform beamforming, or both the base station and the terminal may not perform beamforming.

[0092] In the present disclosure, a beam means a spatial stream of a signal in a radio channel, and may be formed by one or more antennas (or antenna elements), and the formation process thereof may be referred to as beamforming. Beamforming may include at least one of analog beamforming and digital beamforming (e.g., precoding). Reference signals transmitted based on beamforming may include, for example, a demodulation reference signal (DM-RS), a channel state information reference signal (CSI-RS), a synchronization signal / physical broadcast channel (SS / PBCH), or a sounding reference signal (SRS). In addition, for each reference signal configuration, IEs such as CSI-RS resources, SRS resources, etc. may be used, and the configuration may include information associated with the beam. The beam association information may refer to whether the corresponding configuration (e.g., CSI-RS resource) uses the same spatial domain filter as other configurations (e.g., another CSI-RS resource within the same CSI-RS resource set) or uses a different spatial domain filter, or which reference signal is QCL, or if it is QCL, what type it has (e.g., QCL type A, B, C, or D).

[0093] According to the related art, in a communication system in which the cell radius of a base station is relatively large, each base station is installed so that each base station includes the functions of a digital processing unit (or distributed unit (DU)) and a radio frequency (RF) processing unit (or radio unit (RU)). However, since high frequency bands are used for the fourth generation (4G) system and / or its subsequent communication systems (e.g., the fifth generation (5G), and the cell coverage of the base station is reduced, the number of base stations covering a specific area is increased). Therefore, this results in a greater burden on the initial installation cost of installing more base stations for the communication provider. In order to reduce the installation cost of the base station, a structure has been proposed in which the DU and RU of the base station are separated so that one or more RUs are connected to one DU via a wired network and one or more RUs distributed geographically are arranged to cover a specific area.

[0094] For example, a method for automated machine learning (ML) model training performed by an electronic device including at least one processor includes running a first ML model and a second ML model, identifying information about accuracy degradation of the first ML model for a network system using the second ML model, identifying, by the electronic device based on the information about the accuracy degradation of the first ML model, that a predicted accuracy degradation corresponds to a predefined threshold, and training the first ML model based on identifying that the predicted accuracy degradation corresponds to the predefined threshold.

[0095] For example, the accuracy degradation is due to an unplanned event that occurred in the first ML model.

[0096] For example, identifying information about accuracy degradation of a first ML model using a second ML model includes receiving, by an electronic device, data about the accuracy of the first ML model, the data including at least one of the following: model type, parameters and hyperparameters, network nodes, cell models, slice / cell configuration information, existing models that can be used for transfer learning, model training time, model prediction accuracy, resources used for model training, extraction time, time window for data extraction, data generation mode, model accuracy data, and execution time of each training pipeline, storing, by the electronic device, the data about the accuracy of the first ML model to a database, and identifying information about the accuracy degradation of the first ML model based on analyzing, by the electronic device, the data about the accuracy degradation of the first ML model using the second ML model.

[0097] For example, training the first ML model includes identifying, by the electronic device, an expected time to complete model training and data extraction based on accuracy data stored in a database, obtaining, by the electronic device, accuracy data of the first ML model from the database, identifying, by the electronic device, an incoming request for the first ML model, identifying, by the electronic device, resources and resource constraints, creating, by the electronic device, a plan for training the first ML model based on the identified incoming request, the expected time to complete training, the identified resources and resource constraints, scaling, by the electronic device, up / down and / or inward / outward, resources based on the created plan, and triggering, by the electronic device, training of the first ML model based on the created plan.

[0098] For example, training the first ML model includes receiving, by the electronic device, a request to configure an ML service using a new network slice, identifying, by the electronic device, a network slice having the first ML model, the network slice having the first ML model being similar to the new network slice and capable of transfer learning, identifying, by the electronic device, a super model of the first ML model for transfer learning and remaining layers of the first ML model to be trained based on input from a model registry, and triggering, by the electronic device, training of the remaining layers of the first ML model.

[0099] For example, the method includes comparing configuration parameters of a new network slice with configuration parameters of multiple network slices based on a threshold configuration, and based on comparing the configuration parameters of the new network slice with the configuration parameters of multiple network slices, identifying a network slice having a first ML model, which network slice having the first ML model is similar to the new network slice and capable of transfer learning.

[0100] For example, the method includes obtaining information about accuracy degradation of a first ML model from a database, identifying a first time interval for training the first ML model and a second time interval for obtaining data for training the first ML model, and identifying a third time interval for training the first ML model based on the first time interval and the second time interval, and training the first ML model during the third time interval.

[0101] For example, the method includes scaling up resources for training a first ML model, and training the first ML model based on the scaled-up resources.

[0102] For example, the method includes identifying that a traffic pattern is changed, and identifying that information about a key performance indicator (KPI) is to be changed based on identifying that the traffic pattern is changed.

[0103] For example, the method includes identifying at least one configuration parameter of a network slice associated with a first ML model, and training the first ML model based on the at least one configuration parameter being changed.

[0104] For example, an electronic device for automated machine learning (ML) model training includes a memory, at least one processor, and an active training engine coupled to the memory and the at least one processor, the active training engine including a circuit. The active training engine is configured to run a first ML model and a second ML model, use the second ML model to identify information about accuracy degradation of the first ML model for a network system, identify that the predicted accuracy degradation meets a predefined threshold based on the information about the accuracy degradation of the first ML model, and train the first ML model based on identifying that the predicted accuracy degradation corresponds to the predefined threshold.

[0105] For example, the accuracy degradation is due to an unplanned event that occurred in the first ML model.

[0106] For example, the engine is configured to identify information about accuracy degradation of the first ML model using the second ML by at least the following operations: receiving data about accuracy degradation of the first ML model, the data including at least one of the following: model type, parameters and hyperparameters, network nodes, cell models, slice / cell configuration information, existing models that can be used for transfer learning, model training time, model prediction accuracy, resources used for model training, extraction time, time window for data extraction, data generation mode, model accuracy data, and execution time of each training pipeline, storing data about the accuracy of the first ML model to a database, and identifying information about the accuracy degradation of the first ML model based on analyzing the data about the accuracy of the first ML model using the second ML model.

[0107] For example, the engine is configured to train the first ML model by at least the following operations: identifying an expected time to complete model training and data extraction based on accuracy data stored in a database, obtaining accuracy data of the first ML model from the database, identifying incoming requests for the first ML model, identifying resources and resource constraints, creating a plan for training the first ML model based on the identified incoming requests, the expected time to complete training, the identified resources and resource constraints, scaling resources up / down and / or in / out based on the created plan, and triggering training of the first ML model based on the created plan.

[0108] For example, the engine is configured to train a first ML model by at least the following operations: receiving a request to configure an ML service with a new network slice, identifying a network slice having a first ML model, the network slice having the first ML model being similar to the new network slice and capable of transfer learning, identifying a super model of the first ML model for transfer learning and remaining layers of the first ML model to be trained based on input from a model registry, and triggering training of the remaining layers of the first ML model.

[0109] For example, the engine is configured to compare configuration parameters of a new network slice with configuration parameters of multiple network slices based on a threshold configuration, and identify a network slice having a first ML model based on comparing the configuration parameters of the new network slice with the configuration parameters of multiple network slices, wherein the network slice having the first ML model is similar to the new network slice and capable of transfer learning.

[0110] For example, the engine is further configured to obtain information about accuracy degradation of the first ML model from the database, identify a first time interval for training the first ML model and a second time interval for obtaining data for training the first ML model, identify a third time interval for training the first ML model based on the first time interval and the second time interval, and train the first ML model during the third time interval.

[0111] For example, the engine is configured to scale up resources used to train the first ML model and train the first ML model based on the scaled-up resources.

[0112] For example, the engine is configured to identify that a traffic pattern is changed, and based on identifying that the traffic pattern is changed, identify that information about a key performance indicator (KPI) is to be changed.

[0113] For example, the engine is configured to identify at least one configuration parameter of a network slice associated with a first ML model, and train the first ML model based on the at least one configuration parameter being changed.

[0114] For example, a method for automated machine learning (ML) model retraining by an electronic device including at least one processor includes running a first ML model and a second ML model, predicting accuracy degradation of the first ML model using the second ML model, determining whether the predicted accuracy degradation satisfies a predefined threshold, and retraining the first ML model based on determining that the predicted accuracy degradation satisfies the predefined threshold.

[0115] For example, the accuracy degradation is due to an unplanned event that occurred in the first ML model.

[0116] For example, predicting accuracy degradation of a first ML model using a second ML model includes receiving, by an electronic device, data about the accuracy of the first ML model, the data including at least one of: model type, parameters and hyperparameters, network nodes, cell models, slice / cell configuration information, existing models that can be used for transfer learning, model training time, model prediction accuracy, resources used for model training, extraction time, time window for data extraction, data generation mode, model accuracy data, and execution time of each training pipeline, storing, by the electronic device, the data on the accuracy of the first ML model to a retraining data storage, and predicting accuracy degradation of the first ML model based on analyzing the data on the accuracy of the first ML model using the second ML model.

[0117] For example, retraining the first ML model includes estimating, by the electronic device, an expected time to complete model retraining and data extraction based on accuracy data stored in a retraining data store, extracting, by the electronic device, accuracy data of the first ML model from the retraining data store, predicting, by the electronic device, incoming requests for the first ML model, estimating, by the electronic device, resources and resource constraints, creating, by the electronic device, a plan for retraining the first ML model based on the predicted incoming requests, the expected time to complete retraining, the estimated resources and resource constraints, scaling, by the electronic device, up / down and / or inward / outward, resources based on the created plan, and triggering, by the electronic device, retraining of the first ML model based on the created plan.

[0118] For example, retraining the first ML model includes receiving, by the electronic device, a request to configure an ML service using a new network slice, determining, by the electronic device, a network slice having the first ML model, the network slice having the first ML model being similar to the new network slice and capable of transfer learning, predicting, by the electronic device, a super model of the first ML model for transfer learning and remaining layers of the first ML model to be retrained based on input from a model registry, and triggering, by the electronic device, retraining of the remaining layers of the first ML model.

[0119] For example, an electronic device for automated machine learning (ML) model retraining, wherein the electronic device includes a memory, at least one processor, and an active retraining engine, the active retraining engine coupled to the memory and the at least one processor, the active retraining engine including a circuit. The active retraining engine is configured to run a first ML model and a second ML model, predict accuracy degradation of the first ML model using the second ML model, determine whether the predicted accuracy degradation meets a predefined threshold, and retrain the first ML model when the predicted accuracy degradation meets the predefined threshold.

[0120] For example, the accuracy degradation is due to an unplanned event that occurred in the first ML model.

[0121] For example, the engine is configured to predict the accuracy degradation of the first ML model using the second ML by at least the following operations: receiving data on the accuracy of the first ML model, the data including at least one of the following: model type, parameters and hyperparameters, network nodes, cell models, slice / cell configuration information, existing models that can be used for transfer learning, model training time, model prediction accuracy, resources used for model training, extraction time, time window for data extraction, data generation mode, model accuracy data, and execution time of each training pipeline, storing the data on the accuracy of the first ML model to a retraining data storage, and predicting the accuracy degradation of the first ML model by analyzing the data on the accuracy of the first ML model using the second ML model.

[0122] For example, the engine is configured to retrain the first ML model by at least the following operations: estimating an expected time to complete model retraining and data extraction based on the accuracy data stored in the retraining data store, extracting the accuracy data of the first ML model from the retraining data store, predicting incoming requests for the first ML model, estimating resources and resource constraints, creating a plan for retraining the first ML model based on the predicted incoming requests, the expected time to complete retraining, the estimated resources and resource constraints, scaling resources up / down and in / out based on the created plan, and triggering retraining of the first ML model based on the created plan.

[0123] For example, the engine is configured to retrain the first ML model by at least the following operations: receiving a request to configure an ML service with a new network slice, determining a network slice having the first ML model, the network slice having the first ML model being similar to the new network slice and capable of transfer learning, predicting a super model of the first ML model for transfer learning and remaining layers of the first ML model to be retrained based on input from a model registry, and triggering retraining of the remaining layers of the first ML model.

[0124] As used herein, "based on" encompasses at least based on.

[0125] The embodiments disclosed herein may be implemented using at least one hardware device to control elements.

[0126] The foregoing description of the example embodiments will so fully reveal the general nature of the example embodiments herein that others can easily modify and / or adapt such specific embodiments for various applications by applying current knowledge without departing from the general concept, and therefore, such adjustments and modifications should and are intended to be understood as being within the meaning and scope of equivalents of the disclosed embodiments. It should be understood that the wording or terminology employed herein is for the purpose of description rather than limitation. Therefore, although the example embodiments herein have been described according to preferred embodiments, those skilled in the art will recognize that the example embodiments herein can be practiced by modification within the scope of the embodiments as described herein. Although the present disclosure has been shown and described with reference to various embodiments, it should be understood that the various embodiments are intended to be illustrative rather than restrictive. Those skilled in the art will further understand that various changes can be made in form and detail without departing from the true spirit and full scope of the present disclosure (including the appended claims and their equivalents). It will also be understood that any embodiment described herein may be used in combination with any other embodiment described herein.

[0127] For one or more embodiments, at least one of the components described in one or more of the foregoing figures may be configured to perform one or more operations, techniques, processes and / or methods as described herein. For example, a processor (e.g., a baseband processor) as described herein in conjunction with one or more of the foregoing figures may be configured to operate according to one or more of the examples described herein. For another example, a circuit associated with a UE, a base station, a network element, etc., as described above in conjunction with one or more of the foregoing figures may be configured to operate according to one or more of the examples described herein.

[0128] Unless explicitly stated otherwise, any of the above embodiments may be combined with any other embodiment (or combination of embodiments). The foregoing description of one or more embodiments provides illustration and description, but is not intended to be exhaustive or to limit the scope of the embodiments to the precise forms disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practice of the various embodiments.

[0129] The methods according to various embodiments described in the claims and / or the specification of the present disclosure may be implemented in hardware, software, or a combination of hardware and software.

[0130] When implemented by software, a computer-readable storage medium storing one or more programs (software modules) may be provided. One or more programs stored in such a computer-readable storage medium (e.g., a non-transitory storage medium) are configured to be executed by one or more processors in an electronic device. One or more programs include instructions that cause the electronic device to perform the method according to the embodiments described in the claims or description of the present disclosure.

[0131] Such a program (e.g., software module, software) may be stored in a random access memory, a non-volatile memory including flash memory, a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic disk storage device, a compact disk-ROM (CD-ROM), a digital versatile disk (DVD), other types of optical storage devices, or a magnetic tape cassette. Alternatively, it may be stored in a memory configured with a combination of some or all of the above. In addition, the corresponding constituent memories may be provided in multiple quantities.

[0132] In addition, the program can be stored in an attachable storage device that can be accessed via a communication network, such as, for example, the Internet, an intranet, a local area network (LAN), a wide area network (WAN), or a storage area network (SAN), or a communication network configured with a combination thereof. Such a storage device can access the apparatus for executing the embodiments of the present disclosure through an external port. In addition, the apparatus for executing the embodiments of the present disclosure can access a separate storage device on a communication network.

[0133] In the above-mentioned specific embodiments of the present disclosure, the components included therein may be expressed in singular or plural form, depending on the specific embodiments proposed. However, for ease of description, such singular or plural expressions may be appropriately selected for the context presented, and the present disclosure is not limited to singular forms or plural elements. Therefore, an element expressed in plural form may be formed by a singular element, or an element expressed in singular form may be formed by a plurality of elements.

[0134] While specific embodiments have been described in the detailed description of the present disclosure, it goes without saying that various modifications are possible without departing from the scope of the present disclosure.

Claims

1. A method for automated machine learning (ML) model training by an electronic device comprising at least one processor, the method comprising: Run the first ML model and the second ML model; identifying, for a network system, information regarding degradation in accuracy of the first ML model using the second ML model; identifying, by the electronic device, based on the information regarding the accuracy degradation of the first ML model, that a predicted accuracy degradation corresponds to a predefined threshold; and The first ML model is trained based on identifying that the degradation in accuracy of the prediction corresponds to the predefined threshold.

2. The method according to claim 1, wherein: The accuracy degradation is due to unintended events occurring in the first ML model.

3. The method according to claim 1, wherein: Identifying the information about the accuracy degradation of the first ML model using the second ML model includes: Receiving, by the electronic device, data about the accuracy of the first ML model, the data comprising at least one of: model type, parameters and hyperparameters, network nodes, cell models, slice / cell configuration information, existing models that can be used for transfer learning, model training time, model prediction accuracy, resources used for model training, extraction time, time window for data extraction, data generation mode, model accuracy data, and execution time of each training pipeline; storing, by the electronic device, the data regarding the accuracy of the first ML model in a database; and The information about the degradation of the accuracy of the first ML model is identified by the electronic device based on analyzing the data about the accuracy of the first ML model using the second ML model.

4. The method according to claim 1, wherein: Training the first ML model includes: identifying, by the electronic device, an expected time to complete the model training and data extraction based on the accuracy data stored in the database; obtaining, by the electronic device, data of the accuracy of the first ML model from the database; recognizing, by the electronic device, an incoming request for the first ML model; identifying, by the electronic device, resources and resource constraints; creating, by the electronic device, a plan for training the first ML model based on the identified incoming request, the expected time to complete the training, the identified resources, and the resource constraints; scaling, by the electronic device, the resources up / down and / or in / out based on the created plan; and Training of the first ML model is triggered by the electronic device based on the created plan.

5. The method according to claim 1, wherein: Training the first ML model includes: Receiving, by the electronic device, a request to configure an ML service using a new network slice; identifying, by the electronic device, a network slice having the first ML model, wherein the network slice having the first ML model is similar to the new network slice and capable of transfer learning; identifying, by the electronic device, a super model of the first ML model for transfer learning, and remaining layers of the first ML model to be trained based on input from a model registry; and Training of the remaining layers of the first ML model is triggered by the electronic device.

6. The method according to claim 5, wherein: The method further comprises: comparing configuration parameters of the new network slice with configuration parameters of the plurality of network slices based on a threshold configuration; and Based on comparing the configuration parameters of the new network slice with the configuration parameters of the multiple network slices, a network slice having the first ML model is identified, wherein the network slice having the first ML model is similar to the new network slice and capable of transfer learning.

7. The method according to claim 1, wherein: The method further comprises: obtaining the information about the accuracy degradation of the first ML model from a database; identifying a first time interval for training the first ML model and a second time interval for obtaining data for training the first ML model; Based on the first time interval and the second time interval, identifying a third time interval for training the first ML model; and During the third time interval, the first ML model is trained.

8. The method according to claim 1, wherein: The method further comprises: scaling up resources for training the first ML model; and The first ML model is trained based on the scaled-up resources.

9. The method according to claim 1, wherein: The method further comprises: Identify that traffic patterns have changed; and Information about key performance data (KPI) is identified to be changed based on identifying that the traffic pattern is changed.

10. The method according to claim 1, wherein: The method further comprises: identifying at least one configuration parameter of a network slice associated with the first ML model; and The first ML model is trained based on the at least one configuration parameter being changed.

11. An electronic device for automated machine learning (ML) model training, wherein: The electronic device comprises: Memory; at least one processor; and an active training engine coupled to the memory and the at least one processor, the active training engine comprising circuitry, the active training engine being configured to: Run the first ML model and the second ML model, identifying information regarding accuracy degradation of the first ML model for the network system using the second ML model, Based on the information about the accuracy degradation of the first ML model, identifying that a predicted accuracy degradation satisfies a predefined threshold, and The first ML model is trained based on identifying that the degradation in accuracy of the prediction corresponds to the predefined threshold.

12. The electronic device according to claim 11, wherein: The accuracy degradation is due to unintended events occurring in the first ML model.

13. The electronic device according to claim 11, wherein: The engine is configured to identify the information about the accuracy degradation of the first ML model using the second ML by at least: receiving data about the accuracy of the first ML model, the data comprising at least one of: model type, parameters and hyperparameters, network nodes, cell models, slice / cell configuration information, existing models that can be used for transfer learning, model training time, model prediction accuracy, resources used for model training, extraction time, time window for data extraction, data generation mode, model accuracy data, and execution time of each training pipeline; storing the data regarding the accuracy of the first ML model to a database; and Based on analyzing the data regarding the accuracy of the first ML model using the second ML model, the information regarding degradation of the accuracy of the first ML model is identified.

14. The electronic device of claim 11, wherein the engine is configured to train the first ML model by at least the following operations: identifying an expected time to complete said model training and data extraction based on the accuracy data stored in the database; obtaining data of the accuracy of the first ML model from the database; identifying an incoming request for the first ML model; Identify resources and resource constraints; creating a plan for training the first ML model based on the identified incoming requests, the expected time to complete the training, the identified resources, and the resource constraints; scaling said resources up / down and / or in / out based on the created plan; and Training of the first ML model is triggered based on the created plan.

15. The electronic device of claim 11, wherein the engine is configured to train the first ML model by at least the following operations: receiving a request to configure an ML service using a new network slice; Identifying a network slice having the first ML model, wherein the network slice having the first ML model is similar to the new network slice and capable of transfer learning; Based on input from a model registry, identifying a super model of the first ML model for transfer learning, and remaining layers of the first ML model to train; and Triggering training of the remaining layers of the first ML model.

Images