Automated Incremental Tuning of Machine Learning Models for Fault Localization in Overhead Power Lines

Abstract

Conventional methods for reducing the size of machine learning models generally involve expensive and slow exploration of new architectures or pruning model parameters to generate sparse machine learning models. However, in some environments, such as when a regression task is embedded in hardware to run in real time on time-series data, the pruned machine learning model must be dense. Therefore, an automated incremental tuning method is disclosed for simultaneously adjusting both the architecture and parameters of a machine learning model initially trained for a first environment for deployment in a second environment. In one embodiment, during tuning, constraints are imposed on the dimensionality of the machine learning model to prune it into a smaller, denser machine learning model suitable for embedding in hardware and other potential environments.

Description

[Technical Field]

[0001] background FIELD OF THE INVENTION FIELD Embodiments described herein relate generally to machine learning, and more particularly to automated, incremental tuning of machine learning models used, for example, for fault location in overhead power lines. [Background technology]

[0002] 2. Description of Related Art Artificial neural networks, such as convolutional neural networks (CNNs) and deep neural networks (DNNs), can achieve satisfactory levels of performance in a variety of applications. However, the best-performing neural networks are often large and complex. This size can hinder implementation in applications that require strict real-time execution with limited computing power, such as when the neural network must be embedded in hardware.

[0003] One such application is power system protection, where faulted portions of a power system must be disconnected in real time to limit physical damage to the rest of the power system. For example, a line protection system may be used to protect overhead power lines by tripping a circuit breaker as quickly as possible when a fault is detected up to a predetermined range setting. One task of such a line protection system is to estimate the location of the fault. Machine learning regression models can be trained to perform this fault location. However, the trained model must be installed on relays, which have limited computational power and therefore cannot accommodate large-scale machine learning models.

[0004] Retraining or "tuning" a machine learning model may be used to quickly adapt the machine learning model to a new dataset. For example, in transfer learning, an existing machine learning model that was previously trained on one task is retrained on a different, but often related, task. Tuning a pre-trained model, as opposed to training from scratch, has several advantages. These advantages include faster learning, improved performance, and preventing overfitting to smaller datasets. However, reusing the same architecture as an existing machine learning model limits the machine learning model's ability to retrain.

[0005] Such limitations also exist in conventional pruning methods. For example, Pytortch™ and TensorFlow™ provide model pruning libraries that perform local (i.e., layer-by-layer) and global pruning of machine learning models by setting and fixing weights and biases to zero. The weights and biases are chosen randomly or based on importance, as determined according to some metric (e.g., using L1 regularization). The resulting machine learning model is sparse, but without significantly compromising performance. However, while such pruning can reduce the computational requirements for running the machine learning model, it does not reduce the dimensionality of the pruned portion of the machine learning model.

[0006]

number

[0007] Han et al., "Once-for-All: Train One Network and Specialize It for Efficient Deployment," ICLR 2020, incorporated herein by reference as if fully set forth herein, proposed training a large-scale machine learning model and then selecting subnetworks. In this method, the training phase is decoupled from the specialization phase. The training phase focuses on improving the accuracy of all subnetworks, which is derived by selecting different parts of the once-for-all network. In the specialization phase, a subset of subnetworks is sampled to train accuracy and latency predictors. A predictor-guided architecture search is performed to obtain specialized subnetworks based on the target hardware and constraints.

[0008] Yihui et al., "AMC: AutoML for Model Compression and Acceleration on Mobile Devices," arXiv 2019, incorporated herein by reference as if fully set forth herein, proposed a model compression method that leverages reinforcement learning to provide a model compression policy. First, an iterative pruning method is proposed in which several configurations are selected from a candidate set of configurations and then scheduled for training. Comparisons between estimated confidence bounds associated with various configurations are used to progressively prune the candidate set of configurations by removing low-performing configurations. Second, for a given layer of a machine learning model, compression parameters are calculated based on the pruning losses of each hidden layer in the machine learning model. Third, the pruning method clusters input weights according to their patterns, and each cluster is pruned to achieve a predetermined sparsity. Fourth, a spare distillation framework (SDF) distills knowledge from a computationally intensive teacher model while simultaneously pruning a student model in a single training pass.

[0009] Broadly, these publications and others can be categorized into several classes of refinements: (i) selection of the best submodel by a supermodel using expensive and slow processes (e.g., neural architecture search, reinforcement learning, etc.), (ii) sparsification of machine learning models, or (iii) scheduling of best candidate configurations for pruning based on performance estimates.

[0010] What is needed is a training process that can obtain a reduced-dimensionality dense machine learning model that can be run in real time on devices with limited computing power, such as relays in a line protection system, for example, to perform regression tasks using time series data. Summary of the Invention [Problem to be solved by the invention]

[0011] overview Therefore, a system, method, and non-transitory computer-readable medium for automated incremental tuning of machine learning models are disclosed. A goal of certain embodiments is to rapidly and simultaneously adjust both the architecture and parameters of a machine learning model to changes in the environment (e.g., with respect to input / output data distribution and / or computational resources). A further goal of certain embodiments is to iteratively retrain layers of a layered machine learning model, such as a neural network, in a defined order until a stopping condition is met. A further goal of certain embodiments is to prune the machine learning model until a specific reduction measure is achieved to generate a smaller, denser machine learning model. A further goal of certain embodiments is to vary the learning rate of different layers, for example, by decaying the learning rate after each iteration in the case of pruning. [Means for solving the problem]

[0012] In one embodiment, a method includes using at least one hardware processor to receive a machine learning model having an architecture and parameters trained for operation in a first environment, simultaneously tuning both the architecture and parameters of the machine learning model using a training dataset for a second environment different from the first environment, and deploying the tuned machine learning model to the second environment.

[0013] The machine learning model may comprise a neural network, for example, the neural network may be a convolutional neural network.

[0014] The simultaneous adjustment of both the architecture and parameters of the machine learning model may include retraining each of one or more layers of the neural network until a stopping condition is met. The one or more layers may be multiple layers. The multiple layers may be trained in order from a layer of the multiple layers that is closest to the output of the machine learning model to another layer of the multiple layers that is closest to the input of the machine learning model.

[0015] Co-tuning both the architecture and parameters of the machine learning model may include, in each of one or more iterations, selecting a layer from among a plurality of layers in the neural network that has not been retrained in any of the one or more previous iterations, retraining the selected layer, determining whether a stopping condition is met, stopping the co-tuning when the stopping condition is met, and adding a next iteration to the one or more iterations when the stopping condition is not met. The one or more iterations may be multiple iterations, and each selected layer is retrained according to a learning rate, and co-tuning both the architecture and parameters of the machine learning model may further include, in at least one of the multiple iterations, changing the learning rate before the next iteration when the stopping condition is not met.

[0016] Retraining each selected layer may include pruning the selected layer. The stopping condition may include a threshold indicating a measure of reduction in size of the machine learning model. Each selected layer may be retrained according to a learning rate, and simultaneously adjusting both the architecture and parameters of the machine learning model further includes reducing the learning rate before the next iteration when the stopping condition is not met in each of one or more iterations.

[0017] Retraining the selected layer may include selecting multiple alternative layers, each having a different size from the selected layer; training the multiple alternative layers using the training dataset to minimize a cross-entropy loss of the machine learning model; and selecting the alternative layer from the multiple alternative layers with the lowest error measure as the retrained layer. The cross-entropy loss may include a negative log-likelihood loss, and the error measure may include a mean squared error. Each of the multiple alternative layers may have a size smaller than the selected layer. Each of the multiple alternative layers may be selected to have a size within a limited search space around the size of the selected layer that is different from the size of the selected layer and any other layer from the multiple alternative layers. Each of the multiple alternative layers may have a size smaller than the selected layer, and the neural network is a convolutional neural network, and the limited search space is defined based on the number of filters to be pruned from the selected layer. The method may further include using at least one hardware processor to determine the number of filters to be pruned using principal component analysis.

[0018] The tuned machine learning model may estimate a location of a fault on a power line based on one or more measured parameters, the training dataset including labeled feature vectors, each of the labeled feature vectors including a value for each of the one or more measured parameters and labeled with the fault location, the concurrent tuning including pruning the machine learning model, the second environment being a relay configured to trip a circuit breaker on the power line, and deploying the tuned machine learning model including installing the tuned machine learning model in a controller of the relay.

[0019] It should be understood that any of the features in the above-described methods may be implemented individually or with any subset of other features in any combination. Thus, to the extent that the appended claims suggest particular dependencies between features, the disclosed embodiments are not limited to those particular dependencies. Rather, any feature described herein may be combined with any other feature described herein, or may be implemented in any combination of features without any one or more other features described herein. Furthermore, any of the methods described above and elsewhere herein may be embodied individually or in any combination in executable software modules of a processor-based system, such as a server, and / or in executable instructions stored on a non-transitory computer-readable medium.

[0020] BRIEF DESCRIPTION OF THE DRAWINGS The details of the present invention, both as to its structure and operation, can be gleaned in part by studying the accompanying drawings, in which like reference numerals refer to like parts, and in which: [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 illustrates an exemplary infrastructure upon which one or more of the processes described herein may be implemented, according to one embodiment. [Figure 2]FIG. 1 illustrates an exemplary processing system in which one or more of the processes described herein may be performed, according to one embodiment. [Figure 3] FIG. 1 illustrates an environment for a line protection system, according to one embodiment. [Figure 4] FIG. 1 illustrates a process for automated incremental adjustment, according to one embodiment. [Figure 5] FIG. 1 illustrates an example of a tuning process for a layered machine learning model, according to one embodiment. [Figure 6] FIG. 1 illustrates an example of a retraining process, according to one embodiment. [Figure 7] FIG. 1 illustrates the operation of stopping conditions and gradually decreasing learning rates during automated progressive pruning, according to one embodiment. [Figure 8] FIG. 1 illustrates a process for making a trip decision, according to one embodiment. [Figure 9A] FIG. 1 illustrates the dead zones achieved by various examples of machine learning models for fault localization according to experiments. [Figure 9B] FIG. 1 illustrates the dead zones achieved by various examples of machine learning models for fault localization according to experiments. [Figure 9C] FIG. 1 illustrates the dead zones achieved by various examples of machine learning models for fault localization according to experiments. [Figure 9D] FIG. 1 illustrates the dead zones achieved by various examples of machine learning models for fault localization according to experiments. [Figure 10] 10 is a graph of convergence times for an embodiment of an automated progressive pruning and brute force method according to an experiment. DETAILED DESCRIPTION OF THE INVENTION

[0022] Detailed Description In one embodiment, a system, method, and non-transitory computer-readable medium for automated incremental tuning of machine learning models are disclosed. After reading this description, it will be clear to those skilled in the art how to implement the present invention in various alternative embodiments and applications. However, although various embodiments of the present invention are described herein, it is understood that these embodiments are presented for the purposes of example and illustration only, and are not limiting. Therefore, this detailed description of various embodiments should not be interpreted as limiting the scope or breadth of the present invention, which is set forth in the appended claims.

[0023] 1. Exemplary Foundations FIG. 1 illustrates an exemplary infrastructure upon which one or more of the disclosed processes may be implemented, according to one embodiment. The infrastructure may comprise a platform 110 (e.g., one or more servers) that hosts and / or executes one or more of the various functions, processes, methods, and / or software modules described herein. The platform 110 may comprise a dedicated server or, alternatively, may be implemented in a computing cloud in which resources of one or more servers are dynamically and elastically allocated to multiple tenants based on demand. In either case, the servers may be co-located and / or geographically distributed. The platform 110 may also comprise or be communicatively connected to software 112 and / or one or more databases 114. Additionally, the platform 110 may be communicatively connected to one or more user systems 130 via one or more networks 120. The platform 110 may also be communicatively connected to one or more external systems 140 via one or more networks 120.

[0024] Network 120 may include the Internet, and platform 110 may communicate with user systems 130 over the Internet using standard transmission protocols such as HyperText Transfer Protocol (HTTP), HTTP Secure (HTTPS), File Transfer Protocol (FTP), FTP Secure (FTPS), and Secure Shell FTP (SFTP), as well as proprietary protocols. While platform 110 is shown as connected to various systems over a single set of networks 120, it should be understood that platform 110 may be connected to various systems over a different set of one or more networks. For example, platform 110 may be connected to a subset of user systems 130 and / or external systems 140 over the Internet, but may also be connected to one or more other user systems 130 and / or external systems 140 over an intranet. Furthermore, although only a few user systems 130 and external systems 140, one set of software 112, and one set of databases 114 are shown, it should be understood that the infrastructure may comprise any number of user systems, external systems, software applications, and databases.

[0025] User systems 130 may comprise any type of computing device capable of wired and / or wireless communication, including, but not limited to, desktop computers, laptop computers, tablet computers, smartphones or other mobile phones, servers, game consoles, televisions, set-top boxes, electronic kiosks, point-of-sale terminals, etc. However, it is generally contemplated that user systems 130 comprise personal computers or workstations of agents of organizations responsible for designing machine learning models, such as electric power system operators or developers of line protection systems for electric power systems. Each user system 130 may comprise or be communicatively connected to a client application 132 and / or one or more local databases 134.

[0026] In one embodiment, external system 140 may comprise one or more devices on which machine learning models, such as neural networks (e.g., CNNs, DNNs, etc.), are deployed. For example, external system 130 may be a line protection system comprising relays for overhead power lines, or other protection systems. While such systems may be any type of system performing any type of task, the disclosed embodiments may be particularly beneficial for real-time systems having limited computing capabilities, including, for example, limited processing resources, limited volatile and / or non-volatile memory, etc.

[0027] Platform 110 may comprise a web server that hosts one or more websites and / or web services. In embodiments in which a website is provided, the website may comprise a graphical user interface including one or more screens (e.g., web pages) generated, for example, in HyperText Markup Language (HTML) or other language. Platform 110 transmits or serves one or more screens of the graphical user interface in response to requests from user system 130. In some embodiments, these screens may be provided in the form of a wizard, in which case two or more screens may be served sequentially, with one or more of the sequential screens depending on the interaction of a user or user system 130 with one or more preceding screens. Requests to and responses from platform 110, including screens of the graphical user interface, may both be communicated over network 120, which may include the Internet, using standard communication protocols (e.g., HTTP, HTTPS, etc.). These screens (e.g., web pages) may comprise a combination of content and elements such as text, images, video, animations, references (e.g., hyperlinks), frames, inputs (e.g., text boxes, text areas, check boxes, radio buttons, drop-down menus, buttons, forms, etc.), scripts (e.g., JavaScript), etc., which may contain or be derived from data stored in one or more databases (e.g., database 114) locally and / or remotely accessible to platform 110. It should be understood that platform 110 may also respond to other requests from user system 130.

[0028] Platform 110 may include, be communicatively coupled to, or otherwise have access to, one or more databases 114. For example, platform 110 may include one or more database servers managing one or more databases 114. Software 112 executing on platform 110 and / or client applications 132 executing on user systems 130 may submit data (e.g., user data, form data, etc.) stored in database 114 and / or request access to data stored in database 114. Any suitable database may be utilized, including, but not limited to, MySQL®, Oracle®, IBM®, Microsoft SQL™, Access™, PostgreSQL™, MongoDB®, etc., including cloud-based databases and proprietary databases. Data may be sent to platform 110, for example, using the well-known POST request supported by HTTP, via FTP, etc. This data and other requests may be processed by server-side web technologies, for example, servlets or other software modules executed by platform 110 (e.g., included in software 112).

[0029] In embodiments in which web services are provided, platform 110 may receive requests from external systems 140 and provide responses in Extensible Markup Language (XML), JavaScript® Object Notation (JSON), and / or any other appropriate or desired format. In such embodiments, platform 110 may provide an application programming interface (API) that defines how user systems 130 and / or external systems 140 may interact with the web services. Thus, user systems 130 and / or external systems 140 (which may themselves be servers) can define their own user interfaces and rely on web services to implement or otherwise provide back-end processes, methods, functions, storage, etc. described herein. For example, in such embodiments, client applications 132 running on one or more user systems 130 may interact with software 112 running on platform 110 to perform one or more or portions of one or more of the various functions, processes, methods, and / or software modules described herein.

[0030] Client application 132 may be "thin," in which case processing is primarily performed on the server side by software 112 on platform 110. A basic example of a thin client application 132 is a browser application, which simply requests, receives, and renders web pages on user system 130, while software 112 on platform 110 is responsible for generating the web pages and managing database functions. Alternatively, client application 132 may be "thick," in which case processing is primarily performed on the client side by user system 130. It should be understood that client application 132 may perform some amount of processing relative to software 112 on platform 110 at any point along this spectrum between "thin" and "thick," depending on the design goals of a particular implementation. In any case, the software described herein, which may reside entirely on either the platform 110 (e.g., in which case the software 112 performs all processing) or the user system 130 (e.g., in which case the client application 132 performs all processing), or which may be distributed between the platform 110 and the user system 130 (e.g., in which case both the software 112 and the client application 132 perform processing), may comprise one or more executable software modules that contain instructions to implement one or more of the processes, methods, or functions described herein.

[0031] 2. Exemplary Processing Device 2 is a block diagram illustrating an exemplary wired or wireless system 200 that may be used in connection with various embodiments described herein. For example, system 200 may be used as or in conjunction with one or more of the functions, processes, or methods (e.g., for storing and / or executing software) described herein and may represent components of platform 110, user system 130, external system 140 (e.g., a line protection system), and / or other processing devices described herein. System 200 may be a server or any conventional personal computer, or any other processor-enabled device capable of wired or wireless data communication. Other computer systems and / or architectures may also be used, as will be apparent to those skilled in the art.

[0032] System 200 preferably includes one or more processors 210. Processor 210 may comprise a central processing unit (CPU). Additional processors may be provided, such as a graphics processing unit (GPU), auxiliary processors for managing input / output, auxiliary processors for performing floating-point operations, dedicated microprocessors (e.g., digital signal processors) with architectures suitable for fast execution of signal processing algorithms, subordinate slave processors (e.g., back-end processors) of a main processing system, additional microprocessors or controllers for dual or multiprocessor systems, and / or coprocessors. Such auxiliary processors may be separate processors or may be integrated with processor 210. Examples of processors that may be used with system 200 include, but are not limited to, any of the processors available from Intel Corporation of Santa Clara, California (e.g., Pentium™, Core i7™, Xeon™, etc.), any of the processors available from Advanced Micro Devices, Incorporated of Santa Clara, California, any of the processors available from Apple Inc. of Cupertino (e.g., A-series, M-series, etc.), any of the processors available from Samsung Electronics Company of Seoul, South Korea (e.g., Exynos™), any of the processors available from NXP Semiconductors NV of Eindhoven, The Netherlands, etc.

[0033] Processor 210 is preferably connected to communication bus 205. Communication bus 205 may include a data channel for facilitating information transfer between storage and other peripheral components of system 200. Additionally, communication bus 205 may provide a set of signals used for communication with processor 210, including a data bus, an address bus, and / or a control bus (not shown). Communication bus 205 may include any standard or non-standard bus architecture, such as, for example, an industry standard architecture (ISA), an extended industry standard architecture (EISA), a Micro Channel Architecture (MCA), a peripheral component interconnect (PCI) local bus, a bus architecture conforming to standards promulgated by the Institute of Electrical and Electronics Engineers (IEEE), including the IEEE 488 general-purpose interface bus (GPIB), IEEE 696 / S-100, etc.

[0034] System 200 preferably includes main memory 215 and may also include secondary memory 220. Main memory 215 provides storage of instructions and data for programs executing on processor 210, such as any of the software described herein. It should be understood that the programs stored in memory and executed by processor 210 may be written and / or compiled according to any suitable language, including, but not limited to, C / C++, Java, JavaScript, Perl, Visual Basic, .NET, etc. Main memory 215 is typically a semiconductor-based memory such as dynamic random access memory (DRAM) and / or static random access memory (SRAM). Other semiconductor-based memory types include, for example, synchronous dynamic random access memory (SDRAM), Rambus dynamic random access memory (RDRAM), ferroelectric random access memory (FRAM), etc., including read only memory (ROM).

[0035] Secondary memory 220 is a non-transitory computer-readable medium on which computer-executable code (e.g., any of the software disclosed herein) and / or other data is stored. Computer software or data stored in secondary memory 220 is loaded into main memory 215 for execution by processor 210. Secondary memory 220 may include, for example, semiconductor-based memory such as programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), and flash memory (a block-oriented memory similar to EEPROM).

[0036] Secondary memory 220 may optionally include internal media 225 and / or removable media 230. Removable media 230 may be read from and written to in any known manner. Removable storage media 230 may be, for example, a magnetic tape drive, a compact disc (CD) drive, a digital versatile disc (DVD) drive, other optical drive, a flash memory drive, etc.

[0037] In alternative embodiments, secondary memory 220 may include other similar means for allowing computer programs or other data or instructions to be loaded into system 200. Such means may include, for example, a communications interface 240 that allows software and data to be transferred to system 200 from an external storage medium 245. Examples of external storage medium 245 include an external hard disk drive, an external optical drive, an external magneto-optical drive, etc.

[0038] As mentioned above, system 200 may include a communications interface 240. Communications interface 240 allows software and data to be transferred between system 200 and an external device (e.g., a printer), a network, or other information source. For example, computer software or executable code may be transferred to system 200 from a network server (e.g., platform 110) via communications interface 240. Examples of communications interface 240 include an internal network adapter, a network interface card (NIC), a Personal Computer Memory Card International Association (PCMCIA) network card, a Card Bus network adapter, a wireless network adapter, a Universal Serial Bus (USB) network adapter, a modem, a wireless data card, a communications port, an infrared interface, an IEEE 1394 Firewire, and any other device capable of interfacing system 200 with a network (e.g., network 120) or another computing device.Communications interface 240 preferably implements industry published protocol standards such as Ethernet (IEEE 802 standard), Fibre Channel, digital subscriber line (DSL), asynchronous digital subscriber line (ADSL), Frame Relay, asynchronous transfer mode (ATM), integrated digital services network (ISDN), personal communications service (PCS), transmission control protocol / Internet protocol (TCP / IP), serial line Internet protocol / point to point protocol (SLIP / PPP), etc., although customized or non-standard interface protocols may also be implemented.

[0039] The software and data transferred via communications interface 240 are typically in the form of electrical communications signals 255. These signals 255 may be provided to communications interface 240 via communications channel 250. In one embodiment, communications channel 250 may be a wired or wireless network (e.g., network 120) or any of a variety of other communications links. Communications channel 250 carries signals 255 and may be implemented using a variety of wired or wireless communications means, including wire or cable, optical fiber, conventional telephone line, cellular phone link, wireless data, communications link, radio frequency (“RF”) link, or infrared link, to name just a few.

[0040] Computer-executable code (e.g., computer programs such as the disclosed software) is stored in main memory 215 and / or secondary memory 220. Computer-executable code can also be received via communications interface 240 and stored in main memory 215 and / or secondary memory 220. Such computer programs, when executed, enable system 200 to perform various functions of the disclosed embodiments described elsewhere herein.

[0041] As used herein, the term "computer-readable medium" refers to any non-transitory computer-readable storage medium used to provide computer-executable code and / or other data to or within system 200. Examples of such media include main memory 215, secondary memory 220 (including internal memory 225 and / or removable media 230), external storage media 245, and any peripheral devices (including network information servers or other network appliances) communicatively coupled to communication interface 240. These non-transitory computer-readable media are means for providing software and / or other data to system 200.

[0042] In embodiments implemented using software, the software may be stored on a computer-readable medium and loaded into system 200 via removable medium 230, I / O interface 235, or communication interface 240. In such embodiments, the software is loaded into system 200 in the form of electrical communication signals 255. The software, when executed by processor 210, preferably causes processor 210 to perform one or more of the processes and functions described elsewhere herein.

[0043] In one embodiment, I / O interface 235 provides an interface between one or more components of system 200 and one or more input and / or output devices. Examples of input devices include, but are not limited to, sensors, keyboards, touchscreens or other touch-sensing devices, cameras, biometric sensing devices, computer mice, trackballs, pen-based pointing devices, etc. Examples of output devices include, but are not limited to, other processing devices, cathode ray tubes (CRTs), plasma displays, light-emitting diode (LED) displays, liquid crystal displays (LCDs), printers, vacuum fluorescent displays (VFDs), surface-conduction electron-emitter displays (SEDs), field emission displays (FEDs), etc. In some cases, input and output devices may be combined, such as in the case of touch-sensitive displays (e.g., smartphones, tablets, or other mobile devices).

[0044] System 200 may also include optional wireless communication components that facilitate wireless communication over a voice network and / or (e.g., in the case of user system 130) a data network. The wireless communication components comprise an antenna system 270, a radio system 265, and a baseband system 260. In system 200, radio frequency (RF) signals are transmitted and received wirelessly by antenna system 270 under the control of radio system 265.

[0045] In one embodiment, antenna system 270 may include one or more antennas and one or more multiplexers (not shown) that perform switching functions to provide transmit and receive signal paths for antenna system 270. In the receive path, the received RF signal may be coupled from the multiplexer to a low noise amplifier (not shown) that amplifies the received RF signal and transmits the amplified signal to radio system 265.

[0046] In alternative embodiments, the radio system 265 may comprise one or more radios configured to communicate over various frequencies. In one embodiment, the radio system 265 may combine a demodulator (not shown) and a modulator (not shown) into a single integrated circuit (IC). The demodulator and modulator may also be separate components. In the incoming path, the demodulator removes the RF carrier signal, leaving a baseband received audio signal that is transmitted from the radio system 265 to the baseband system 260.

[0047] If the received signal contains audio information, the baseband system 260 decodes the signal and converts it to an analog signal. The signal is then amplified and sent to a speaker. The baseband system 260 also receives analog audio signals from a microphone. These analog audio signals are converted to digital signals and encoded by the baseband system 260. The baseband system 260 also encodes the digital signals for transmission and generates baseband transmit audio signals that are routed to a modulator portion of the radio system 265. The modulator mixes the baseband transmit audio signal with an RF carrier signal to generate an RF transmit signal that can be routed to the antenna system 270 and passed through a power amplifier (not shown). The power amplifier amplifies the RF transmit signal and routes it to the antenna system 270, where the signal is switched to an antenna port for transmission.

[0048] The baseband system 260 is also communicatively coupled to the processor 210. The processor 210 may have access to data storage areas 215 and 220. The processor 210 is preferably configured to execute instructions (i.e., computer programs, such as the disclosed software), which may be stored in the main memory 215 or the secondary memory 220. Computer programs may also be received from the baseband processor 260 and stored in the main memory 210 or the secondary memory 220, or executed upon receipt. Such computer programs, when executed, may enable the system 200 to perform various functions of the disclosed embodiments.

[0049] 3. Example track protection system 3 illustrates the operation of a line protection system 330, according to one embodiment. The line protection system 330 is an example of an external system 140 on which machine learning models developed on the platform 110 may be deployed. A power line 310 may be provided between two substations 320A and 320B. One or more voltage measurement units 312 may measure the voltage on the power line 310 and output the voltage measurements to the line protection system 330. Additionally, one or more current measurement units 314 may measure the current on the power line 310 and output the current measurements to the line protection system 330.

[0050] Line protection system 330 may include a controller 332 that may implement system 200 or some subset thereof. For example, controller 332 may include one or more processors 210 and store machine learning model 334 in memory (e.g., main memory 215 and / or secondary memory 220). Controller 332 may receive voltage measurements from voltage measurement unit 312, current measurements from current measurement unit 314, and / or other measurements from one or more other sensors or devices associated with power line 310. Controller 332 may also derive one or more additional measurements from the received measurements.

[0051] The controller 332 may apply a machine learning model 334 to the measurements as part of a time-domain protection scheme. For example, the measurements may be input to the machine learning model 334 to estimate the location of the fault 340 or output a decision on whether to trip (i.e., open) the circuit breaker 316 on the power line 310. If the location of the fault 340 is within a protection zone defined by the range setting 336 of the line protection system 330, the controller 332 may determine to trip the circuit breaker 316 (i.e., switch the circuit breaker 316 to an open state). Otherwise, the controller 332 may determine not to trip the circuit breaker 316 (i.e., keep the circuit breaker 316 in a closed state). Once the controller 332 determines to trip the circuit breaker 316, the controller 332 may send a control signal to the circuit breaker 316 (e.g., directly or via a relay control circuit breaker 316) to open the circuit breaker 316, thereby isolating the fault 340 from the substation 320A. It should be understood that there may be a similar line protection system 330 at the other end of the power line 110 that performs the same function to isolate the fault 340 from the substation 320B if the location of the fault 340 is within its respective protection zone.

[0052] 4. Automated incremental adjustments Next, a method for automated progressive tuning (APT) is described in detail. In one embodiment, the automated progressive tuning trains a machine learning model using simultaneous learning of the architecture and parameters of the machine learning model. For layered machine learning models, such as neural networks, this simultaneous training may iterate across individual layers of the machine learning model until a stopping condition is met. The term "simultaneous" in the context of "simultaneous tuning," "simultaneous learning," "simultaneous training," etc., should be understood to refer to adjusting, learning, training, etc., the architecture of a machine learning model simultaneously with the machine learning model's parameters. This is in contrast to, for example, learning parameters while the architecture is fixed, or learning an architecture while the parameters are fixed. One goal of automated progressive tuning is to address the limited architecture limitations faced by conventional tuning methods. The disclosed automated progressive tuning can be used for environment-specific tuning, searching for optimal architectures for general tasks, and is cheaper and faster than conventional tuning methods, such as neural architecture search (NAS), which are often known to be very expensive and slow.

[0053] 4 illustrates a process 400 for automated incremental adjustment, according to one embodiment. It should be understood that process 400 may be embodied in one or more software modules executed by one or more hardware processors (e.g., processor 210), for example, as a software application (e.g., software 112, client application 132, and / or a distributed application including both software 112 and client application 132), which may be executed entirely by a processor of platform 110, entirely by a processor of user system 130, or distributed across platform 110 and user system 130, such that some portions or modules of the software application are executed by platform 110 and other portions or modules of the software application are executed by user system 130. The described process may be implemented as instructions expressed in source code, object code, and / or machine code. These instructions may be executed directly by hardware processor 210 or may be executed by a virtual machine operating between the object code and hardware processor 210. Additionally, the disclosed software may be built on or interfaced with one or more existing systems.

[0054] Alternatively, the described processes may be implemented as hardware components (e.g., general-purpose processors, integrated circuits (ICs), application-specific integrated circuits (ASICs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, etc.), a combination of hardware components, or a combination of hardware and software components. To clearly illustrate the interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps are described herein generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the invention. Furthermore, the grouping of functions within a component, block, module, circuit, or step is for ease of description. Particular functions or steps may be moved from one component, block, module, circuit, or step to another without departing from the invention.

[0055] Additionally, although process 400 is shown with a particular arrangement and order of sub-processes, process 400 may be performed with fewer, more, or different sub-processes, as well as with a different arrangement and / or order of the sub-processes. Additionally, even if the sub-processes are described or illustrated in a particular order, it should be understood that any sub-process that is not dependent on the completion of another sub-process may be performed before, after, or in parallel with the other independent sub-process.

[0056] Initially, in subprocess 410, a machine learning model having an architecture and parameters trained for operation in a first environment is received. The machine learning model may be an artificial neural network, which may be referred to herein simply as a "neural network." A neural network is any collection of connected units or nodes called "neurons" that loosely model neurons in a biological brain in that they send signals to other neurons through connections called "edges." By way of example, the neural network may be a convolutional neural network (CNN) and / or a deep neural network (DNN). A convolutional neural network is a special type of neural network that includes an input layer, one or more hidden layers, and an output layer, and that uses convolution instead of typical matrix multiplication in at least one of the layers. A deep neural network is any neural network with multiple hidden layers. Thus, a convolutional neural network may be a deep neural network, and vice versa. However, it should be understood that the machine learning model may include other types of neural networks, such as recurrent neural networks (RNNs), and / or other types of models, such as generalized linear models, spline interpolation models, etc.

[0057] In subprocess 420, both the architecture and parameters of the machine learning model received in subprocess 410 are simultaneously tuned using a training dataset for a second environment. While not required, the second environment is generally considered to be different from the first environment. Even though the environments are different, the task performed in each environment may be the same. For example, the first environment may have a first input-output data distribution for the task, and the second environment may have a second input-output data distribution for the same task that is different from the first input-output data distribution. As another example, the first environment may be a relatively computationally intensive environment (e.g., a server-based platform such as platform 110), while the second environment may be a relatively computationally light environment with significantly less computational power than the first environment (e.g., an embedded system such as track protection system 330, a mobile system such as a smartphone, an Internet of Things (IoT) device, etc.). In this case, subprocess 420 may include pruning to reduce the size of the machine learning model, as described in more detail elsewhere herein. Differences in computing power may include differences in processing resources, memory resources, available power (eg, plugged in vs. battery powered), network bandwidth, and the like.

[0058] In sub-process 430, the tuned machine learning model is deployed to a second environment. Sub-process 430 may include installing the tuned machine learning model in one or more external systems 140. For example, the tuned machine learning model may be trained to determine a fault location on a power line based on power line measurements (e.g., voltage and / or current), and the external systems 140 may include one or more line protection systems 330. In this case, the tuned, preferably pruned, machine learning model may be installed in memory (e.g., main memory 215 and / or secondary memory 220) of controller 332 of line protection system 330. It should be understood that machine learning models may be trained and tuned for any other task and / or deployed to any other type of external system 140 in a similar manner.

[0059] 5 illustrates an example of a sub-process 420 for a layered machine learning model, according to one embodiment. In the illustrated example, the simultaneous tuning of both the architecture and parameters of the machine learning model is incremental in that the machine learning model is adjusted layer by layer until a stopping condition is met. Again, examples of layered machine learning models include, but are not limited to, convolutional neural networks, deep neural networks, recurrent neural networks, and many other types of neural networks.

[0060] In sub-process 422, it is determined whether a stopping condition is met. The stopping condition may include one or more stopping criteria that may be defined according to the purpose of the particular tuning process. For example, if the tuning includes pruning, the stopping condition may be satisfaction of a threshold value measured by a reduction in the size of the machine learning model. Alternatively, the stopping condition may include another metric of the tuned machine learning model meeting a threshold value, the number of iterations exceeding a predetermined number of epochs, receiving a user action (e.g., terminating the tuning via a graphical user interface provided by software 112), the expiration of a timer, etc. In any case, if the stopping condition is not met (i.e., "No" in sub-process 422), sub-process 420 proceeds to sub-process 424. Otherwise, if the stopping condition is met (i.e., "Yes" in sub-process 422), sub-process 420 may terminate (e.g., and proceed to sub-process 430).

[0061] In subprocess 424, a new layer is selected for adjustment. In particular, a layer within a machine learning model (e.g., a neural network) that has not been retrained in any previous iteration of subprocess 426 may be selected. In one embodiment, the layers may be trained in a defined order, from the layer closest to the machine learning model's output (i.e., furthest from the input), which may be referred to as the "top" layer, to the layer closest to the machine learning model's input (i.e., furthest from the output), which may be referred to as the "bottom" layer. In an alternative embodiment, the layers may be trained in order from the bottom to the top. In an alternative embodiment, the layers may be trained from a higher layer (e.g., other than the top layer) to a lower layer (e.g., other than the bottom layer), or from a lower layer to a higher layer. In yet another alternative, the layers may be trained in another order (e.g., from a higher layer to a lower layer, or from a lower layer to a higher layer, according to a priority assigned to the layers, etc.), or in a random order. As yet another alternative, layers may be selected according to one or more selection criteria in each iteration of sub-process 424. In this case, the selection criteria may include a measure of layer importance or other attribute that is minimum or maximum relative to all layers that have not been retrained in the previous iteration of sub-process 426.

[0062] In sub-process 426, the layer selected in sub-process 424 is retrained. The retraining in sub-process 426 involves simultaneously retraining both the architecture and parameters of the layer. In other words, both the architecture and parameters of the layer may change during retraining. In one embodiment, this retraining may include pruning, in which case the size of the layer is reduced in at least one dimension. Once the layer has been retrained, sub-process 420 returns to sub-process 422 to determine whether a stopping condition is met after retraining.

[0063] Each layer may be retrained in sub-process 426 according to the learning rate. In one embodiment, after a layer is retrained in sub-process 426 and a stopping condition has not yet been met in sub-process 422, the learning rate may be changed before the next iteration of sub-process 424 or 426. For example, if the retraining includes pruning, the learning rate may be reduced before the next iteration according to a decay function. Alternatively, the learning rate may be increased before the next iteration. As another alternative, the learning rate may be kept constant across all iterations. As yet another alternative, the learning rate may vary up or down or remain the same from iteration to iteration depending on one or more criteria or patterns.

[0064] The following algorithm represents exemplary pseudocode for one embodiment of the sub-process 420 for automated incremental adjustment:

[0065] [Table 1]

[0066] 6 illustrates an example of retraining in sub-process 426, according to one embodiment. In the illustrated example, layer i of neural network 600 is being retrained during an iteration of sub-process 426. While neural network 600 is shown as having at least a certain number of layers, it should be understood that neural network 600 may have any number of layers, including fewer or more than those shown. Furthermore, the layer being retrained may be any layer of neural network 600. For example, a particular implementation may retrain a convolutional layer of a convolutional neural network, but the same retraining may be performed on other layers of the convolutional neural network, or other neural networks, such as gated recurrent units (GRUs) (e.g., in recurrent neural networks), fully connected (FC) layers, etc.

[0067]

number

[0068]

number

[0069] In sub-process 426, multiple replacement layers may be trained using the training dataset for the second environment to minimize the cross-entropy loss of the machine learning model (e.g., neural network 600). For simultaneous tuning of the architecture and parameters of the machine learning model, the cross-entropy loss may be summed with the mean squared error (MSE) loss of the entire machine learning model for the task (e.g., a regression task for fault localization). To this end, the following joint optimization problem may be solved: Formula (1):

[0070]

number

[0071] The training loss can be augmented as follows: Formula (2):

[0072]

number

[0073]

number

[0074] Target y for NLL classifier class may be obtained as follows: Formula (3):

[0075]

number

[0076]

number

[0077] The learning rate may be varied or held constant across iterations of sub-process 426. In one embodiment, retraining in sub-process 420 progresses from near-top layers (e.g., the top layer, last convolutional layer, or other upper layers of a convolutional neural network) to near-bottom layers (e.g., the bottom layer, the first convolutional layer in a convolutional neural network, or other lower layers), with the learning rate gradually decreasing. In other words, higher layers are adjusted more aggressively than lower layers. The learning rate may decay linearly with each iteration. Alternatively, the learning rate may decay according to other suitable profiles, including nonlinear profiles. As another alternative, the learning rate may gradually increase, either linearly or nonlinearly. As yet another alternative, the learning rate may remain constant across all iterations of sub-process 426. As yet another alternative, the learning rate may decrease, increase, or remain constant with each iteration depending on one or more criteria.

[0078] In some applications (e.g., when the existing machine learning model is very large), it may be sufficient to stop the adjustment before reaching the bottom layer of the sequence of layers being trained, for example, if the size of the adjusted machine learning model at that point meets some pruning objective. In this case, continuing to retrain layers is futile. Therefore, sub-process 422 may be used to define stopping conditions that stop the adjustment when one or more stopping criteria are met, even if only a portion of the layers have been retrained.

[0079] 5. Automated Progressive Pruning A simple way to generate smaller, denser machine learning models using conventional methods is to define the architecture as hyperparameters and tune the architecture using methods for tuning hyperparameters. However, when significantly reducing the size of a machine learning model (e.g., from a very large existing machine learning model to a very small one), such methods are costly and wasteful because they require retraining every layer for every candidate architecture.

[0080] Thus, in one embodiment, the automated incremental tuning of process 400 is used to prune an existing machine learning model by adding a constraint to at least one dimension of the existing machine learning model. In particular, at least one dimension of the adjusted machine learning model is constrained to be strictly smaller than that dimension of the existing machine learning model. As an example of a convolutional neural network, the dimension may be the kernel and filter size or the number of filters in a layer in the convolutional neural network. Advantageously, pruning by process 400 can produce a smaller, denser machine learning model without the costly and wasteful retraining of the simple methods described above. Pruning by the automated incremental tuning of process 400 is referred to herein as automated progressive pruning (APP).

[0081] In one embodiment of automated incremental pruning, the stopping condition in sub-process 422 may be that a measure of size reduction in at least one dimension from the existing machine learning model to the adjusted machine learning model satisfies a threshold. The measure of reduction may be a measure of overall size reduction, such as the ratio of the number of parameters in the adjusted machine learning model to the number of parameters in the existing machine learning model, in which case the stopping condition may be met when the ratio falls below a threshold. Alternatively, the measure of size reduction may be the radius of a circular neighborhood around a desired number of parameters in the final adjusted machine learning model, in which case the stopping condition may be met when the radius falls below a threshold.

[0082] The following algorithm represents exemplary pseudocode for one embodiment of automated progressive pruning:

[0083] [Table 2-1] [Table 2-2]

[0084] In contrast to conventional methods, one embodiment of automated incremental pruning starts with an existing machine learning model and co-learns an architecture along with the model parameters in an iterative or incremental manner. Instead of decoupling architecture learning from parameter optimization and employing architecture search, which can be costly and slow, automated incremental pruning modifies at least one dimension of the existing architecture while optimizing parameters. Thus, automated incremental pruning addresses the limitations of conventional methods of reusing the same architecture of an existing machine learning model.

[0085] In one embodiment, the adjustment in subprocess 420 starts from the upper layers (i.e., closer to the output of the machine learning model) and progresses toward the lower layers (i.e., closer to the input of the machine learning model) with each iteration. After each iteration, the learning rate may be reduced before the next iteration. The learning rate may be gradually reduced according to some algorithm. This gradual reduction in the learning rate is due to the inventors' recognition that the upper layers (i.e., closer to the output of the machine learning model) are more likely to be redundant than the lower layers (i.e., closer to the input of the machine learning model) to accomplish the task of the machine learning model. Therefore, the upper layers can be pruned more aggressively, while the role of the lower layers is primarily to embed the input data, which requires dense connectivity for effective embedding.

[0086]

number

[0087] Formula (4):

[0088]

number

[0089] Formula (5):

[0090]

number

[0091]

number

[0092] The overall training can be viewed as a bilevel optimization given by:

[0093]

number

[0094] where:

[0095]

number

[0096] 6. Exemplary Use Cases In one embodiment, the tuned machine learning model generated by subprocess 420 may be deployed as machine learning model 334 in controller 332 of track protection system 330. In this case, machine learning model 334 may be a convolutional neural network, which is generally sufficient for fault localization. Convolutional neural networks can efficiently embed time-series data to correctly learn labels such as fault location (e.g., fault distance) through filtering operations performed by one-dimensional convolutional layers, improving interpretability compared to other machine learning models. Compared to the NAS method, process 400 is much faster and more transparent.

[0097] FIG. 8 shows a process 800 for making a trip decision according to an embodiment in which machine learning models for a more computationally rich environment are pruned by process 400 to machine learning model 334 and deployed to a computationally poor environment of line protection system 330 to estimate the location of a fault. In subprocess 810, controller 332 of line protection system 330, which functions as a protection device, may receive measurement signals from multiple sensors connected to power line 310. For example, these measurement signals may include or consist of values ​​of voltage measured by voltage measurement unit 312 and current measured by current measurement unit 314. In subprocess 820, controller 332 may apply the tuned machine learning model 334 to the measurement signals to estimate the location of a fault 340 on power line 310. Then, in subprocess 830, controller 332 may compare the estimated fault location with a predetermined threshold representing range setting 336. For example, controller 332 may determine whether the estimated fault location is equal to or less than range setting 336. When the estimated fault location meets the threshold (i.e., "Yes" in sub-process 830), indicating that the fault 340 is within the protection zone defined by the range setting 336, the controller 332 may trip the circuit breaker 316 in sub-process 840 to electrically isolate the fault location on the power line 310. Otherwise, when the controller 332 determines that the estimated fault location does not meet the threshold (i.e., "No" in sub-process 830), the controller 332 determines not to trip the circuit breaker 316 in sub-process 850.

[0098] In fault-location tasks, the need for appropriate metrics is essential when evaluating the performance of a machine learning model 334. The metrics must reliably capture the machine learning model's 334 ability to make the correct decision as quickly as possible. Because the problem is causal and temporal in nature, the machine learning model 334 can only use past trajectories (e.g., with a single threshold crossing at the fault location) to make a trip decision, resulting in asymmetries in decision-making at each time sample. The accuracy of the trip decision depends on the time sample considered, which affects the degree of operational errors (i.e., false negatives) and misoperations (i.e., false positives) that creep into the resulting decision. Furthermore, the rate of true positives is a separate measure of performance. The trade-off between decision accuracy and decision speed makes quantifying performance a nontrivial task.

[0099] Since the machine learning model 334 outputs the fault location, the idea is to compare the fault location with thresholds that define the protection zone (e.g., subprocess 830). This allows testing of the machine learning model 334 against a set of different range settings 336. For each range setting 336, the machine learning model 334 was tested for a specific safety counter. It was expected that if the actual fault location is close to the range setting 336, the decision may not be perfect. Therefore, the machine learning model 334 was evaluated by calculating how far away from the range setting 336 the machine learning model 334 made a perfect decision. This can be represented by a dead zone, where at least one decision fails as either an operation error or a misoperation for a particular range setting 336. The dead zone captures the full width of the region encompassing erroneous estimates generated from false positive (Type I) and false negative (Type II) errors. A machine learning model 334 with a good estimate will result in a narrow dead zone, while a machine learning model 334 with a poor estimate will result in a wide dead zone. A dead zone equal to zero means that the machine learning model 334 is making perfect decisions for all cases.

[0100] The machine learning model 334 for estimating pruned fault locations using automated progressive pruning as disclosed herein was tested on a transmission system in the presence of converter-interfaced renewable generation. The transmission system had two lines, with one line optionally open.

[0101] 9A-9D illustrate dead zones achieved by various examples of machine learning models for fault location during testing. In each of FIGS. 9A-9D, the x-axis represents different range settings 336 corresponding to thresholds applied (e.g., in subprocess 830) to the fault location estimates output by the respective machine learning model 334 (e.g., in subprocess 820). The y-axis represents the fault location and corresponding dead zone range. The solid center line represents the case where the fault location exactly equals the range setting 336. Cases above the center line are cases where the fault location is greater than the range setting 336 and therefore should determine to not trip (i.e., inhibit) the line protection system 330, as represented by subprocess 850. Cases below the center line are cases where the fault location is less than the range setting 336 and therefore should determine to trip the line protection system 330, as represented by subprocess 840. The dashed lines represent user-specified boundaries for erroneous estimation. The further the dead zone is outside the dashed line, the better the machine learning model. The hatched areas collectively represent the dead zones, with the hatched areas above the centerline representing areas containing at least one false negative (i.e., incorrect suppression) for a given fault location and the hatched areas below the centerline representing areas containing at least one false positive (i.e., incorrect trip) for a given fault location. For testing, the average speed was calculated for all range settings 336 for all correct trip decisions, including safety counter delays. A 70% range setting 336 is typically used in track protection systems 330. Therefore, the width of the dead zone for the 70% range setting 336, represented as 0.7 on the x-axis, is highlighted to demonstrate the performance of each machine learning model for this range setting 336. The safety and reliability of the machine learning models at the 70% range setting 336 were also calculated.

[0102] Figure 9A shows the dead zone of the existing machine learning model before pruning. At the 70% range setting 336, the dead zone was 17.9%, the calculated safety was 99.3%, the calculated reliability was 98.4%, and the average speed was 24.8 milliseconds.

[0103] 9B shows the dead zone of a machine learning model after pruning the existing machine learning model using a brute force method where a given layer of the existing machine learning model was manually created. At a 70% range setting 336, the dead zone was 19.4%, the calculated safety was 98.7%, the calculated reliability was 98.8%, and the average speed was 14.8 milliseconds.

[0104] 9C shows the dead zone of the machine learning model after automated incremental pruning according to the first embodiment, where all layers of the new architecture remain trainable so that fine-tuning affects all layers. At the 70% range setting 336, the dead zone was 18.4%, the calculated safety was 98.8%, the calculated reliability was 98.3%, and the average speed was 14.8 milliseconds.

[0105] 9D shows the dead zone of a machine learning model after automated incremental pruning according to a second embodiment, in which all layers except two being trained are frozen during the iterations of sub-process 426, and two learnable layers are trained using several epochs of Adam training. In this second embodiment, the entire architecture is fine-tuned after the two learnable layers are trained in each iteration of sub-process 426. At the 70% range setting 336, the dead zone was 18.8%, the calculated safety was 98.7%, the calculated reliability was 98.5%, and the average speed was 14.8 milliseconds. This second embodiment resulted in faster pruning, but with only a small performance degradation compared to the first embodiment.

[0106] In standard practice, a dead zone of 20% or less at a 70% range setting 336 is considered satisfactory. Thus, both the first and second embodiments of automated incremental pruning achieved satisfactory results. Furthermore, both embodiments of automated incremental pruning produced smaller dead zones than the brute force method. Furthermore, both embodiments of automated incremental pruning achieved smaller pruned machine learning models compared to the brute force method, as shown in the table below:

[0107] [Table 3]

[0108] In particular, compared to the time required by the brute force method to prune and fine-tune a machine learning model at one given layer (i.e., one iteration), there was a less-than-linear increase in the time required for pruning and fine-tuning with increasing numbers of iterations in automated incremental pruning. Figure 10 is a graph of convergence times for both the automated incremental pruning and brute force method embodiments. The time corresponding to the automated incremental pruning embodiment is for an architecture with two to four alternate layers and two iterations of pruning, while the time corresponding to the brute force method corresponds to one alternate layer and one iteration of pruning.

[0109] 7. Exemplary Embodiments The disclosed embodiments simultaneously tune both the architecture and parameters of a machine learning model, such as a neural network. The neural network may be a convolutional neural network, a deep neural network, a recurrent neural network, or another type of artificial neural network. In one embodiment, tuning is performed using constraints on the dimensionality of the machine learning model to prune the machine learning model. Pruning reduces the overall size of the machine learning model, making it suitable for execution (e.g., in real time) on systems with limited computational power, such as a line protection system 330 for power lines or other embedded systems. Unlike traditional pruning methods that result in sparse machine learning models, the disclosed pruning produces dense machine learning models with reduced dimensionality that perform well for regression tasks (e.g., fault localization) on time-series data. Dense machine learning models are particularly beneficial for deployment on hardware that cannot accommodate the computational complexity associated with sparse matrix operations. The disclosed automated incremental tuning is also less costly and more interpretable than traditional methods.

[0110] The disclosed automated incremental tuning may be used to tune a machine learning model for any application requiring adaptation of the machine learning model from a first environment to a second environment. The first and second environments may differ in some characteristics. The difference may be any change in the data, such as a change in the input / output distribution of the data. For example, the first and second environments may both require performance of the same task (e.g., fault location) but have different distributions of source impedance ratios or line impedances. More generally, the first and second environments may include the same task performed with different data distributions. Alternatively, the first and second environments may include performance of different but related tasks.

[0111] In one embodiment, the disclosed automated incremental adjustment may be automated incremental pruning. The disclosed automated incremental adjustment is particularly useful in environments where pruned dense machine learning models are needed. One such environment is hardware embedded, where functions are written in a low-level programming language (e.g., C) for computational efficiency and sparse matrix calculations are expensive. More generally, automated incremental pruning may be used to reduce the size of a machine learning model trained in a first, computationally richer environment (e.g., where more computing resources are available) and deployed in a second, computationally poorer environment (e.g., where fewer computing resources are available).

[0112] One example described and tested for the second computationally poor environment is a line protection system 330 for protecting overhead lines. In particular, the task of fault location in time-domain protection was tested to demonstrate the performance of the machine learning model generated by automated progressive pruning. However, it should be understood that this is merely an example, and the first and / or second environments may include other tasks for other applications. Examples of such tasks include, but are not limited to, power system state estimation, topology estimation, parameter estimation (e.g., grid inertia), power flow estimation, load forecasting, etc. Notably, all of these specific examples benefit from a final dense machine learning model that is small enough to be incorporated into hardware. Thus, an overall model may be trained to achieve a satisfactory performance level in a first computationally richer environment and then pruned using the disclosed automated progressive pruning to obtain a smaller, denser model for deployment in a second computationally poorer environment without significantly compromising performance.

[0113] Embodiment 1: A method including: receiving, using at least one hardware processor, a machine learning model having an architecture and parameters trained for operation in a first environment; co-tuning both the architecture and parameters of the machine learning model using a training dataset for a second environment different from the first environment; and deploying the tuned machine learning model to the second environment.

[0114] Embodiment 2: The method of embodiment 1, wherein the machine learning model comprises a neural network.

[0115] Embodiment 3: The method of embodiment 2, wherein the neural network is a convolutional neural network.

[0116] Embodiment 4: The method of embodiment 2 or 3, wherein simultaneously adjusting both the architecture and parameters of the machine learning model includes, for each of one or more layers of the neural network, retraining the layer until a stopping condition is met.

[0117] Embodiment 5: The method of embodiment 4, wherein the one or more layers is a plurality of layers. Embodiment 6: The method of embodiment 5, wherein the multiple layers are trained in order from a layer of the multiple layers that is closest to the output of the machine learning model to another layer of the multiple layers that is closest to the input of the machine learning model.

[0118] Embodiment 7: The method according to any one of embodiments 2 to 6, wherein the simultaneous adjustment of both the architecture and parameters of the machine learning model includes, in each of one or more iterations, selecting a layer from among a plurality of layers in the neural network that has not been retrained in any of the one or more iterations prior to the iteration; retraining the selected layer; determining whether a stopping condition is met; stopping the simultaneous adjustment when the stopping condition is met; and adding a next iteration to the one or more iterations when the stopping condition is not met.

[0119] Embodiment 8: The method of embodiment 7, wherein the one or more iterations are multiple iterations, each selected layer is retrained according to a learning rate, and the simultaneous adjustment of both the architecture and parameters of the machine learning model further includes, when a stopping condition is not met in at least one of the multiple iterations, changing the learning rate before the next iteration.

[0120] Embodiment 9: The method of embodiment 7 or 8, wherein retraining each selected layer includes pruning the selected layer.

[0121] Embodiment 10: The method of embodiment 9, wherein the stopping condition includes a threshold indicating a measure of reduction in the size of the machine learning model.

[0122] Embodiment 11: The method described in embodiment 9 or 10, wherein each selected layer is retrained according to a learning rate, and the simultaneous adjustment of both the architecture and parameters of the machine learning model further includes, when a stopping condition is not met in each of one or more iterations, reducing the learning rate before the next iteration.

[0123] Embodiment 12: A method according to any one of embodiments 7 to 11, wherein retraining the selected layer includes selecting multiple alternative layers, each having a size different from the selected layer, training the multiple alternative layers using the training dataset to minimize the cross-entropy loss of the machine learning model, and selecting the alternative layer among the multiple alternative layers having the lowest error measure as the retrained layer.

[0124] Embodiment 13: The method of embodiment 12, wherein the cross-entropy loss comprises a negative log-likelihood loss and the error measure comprises a mean squared error.

[0125] Embodiment 14: The method of embodiment 12 or 13, wherein each of the plurality of alternative layers has a smaller size than the selected layer.

[0126] Embodiment 15: A method as described in any one of embodiments 12 to 14, wherein each of the plurality of alternative layers is selected to have a size within a limited search space around the size of the selected layer that is different from the size of the selected layer and any other layer among the plurality of alternative layers.

[0127] Embodiment 16: The method of embodiment 15, wherein each of the multiple alternative layers has a size smaller than the selected layer, the neural network is a convolutional neural network, and the limited search space is defined based on the number of filters pruned from the selected layer.

[0128] Embodiment 17: The method of embodiment 16, wherein the method further includes using at least one hardware processor to determine the number of filters to be pruned using principal component analysis.

[0129] Embodiment 18: The method of any one of the preceding embodiments, wherein the tuned machine learning model estimates a location of a fault on a power line based on one or more measured parameters, the training dataset includes labeled feature vectors, each of the labeled feature vectors including a value for each of the one or more measured parameters and labeled with the fault location, the concurrent tuning includes pruning the machine learning model, the second environment is a relay configured to trip a circuit breaker on the power line, and deploying the tuned machine learning model includes installing the tuned machine learning model in a controller of the relay.

[0130] Embodiment 19: A system comprising at least one hardware processor and software, which, when executed by the at least one hardware processor, performs the method described in any one of embodiments 1 to 18.

[0131] Embodiment 20: A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by a processor, causing the processor to perform a method according to any one of embodiments 1 to 18.

[0132] The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles described herein may be applied to other embodiments without departing from the spirit or scope of the present invention. It should therefore be understood that the description and drawings presented herein represent presently preferred embodiments of the present invention and, therefore, represent the subject matter broadly contemplated by the present invention. It is further understood that the scope of the present invention fully encompasses other embodiments that may become apparent to those skilled in the art, and therefore, the scope of the present invention is not limited.

[0133] As used herein, the terms "comprising," "comprise," and "comprises" are open-ended. For example, "A comprises B" means that A can include either: (i) B alone, or (ii) B in combination with one or more, and possibly any number of, other components. In contrast, the terms "consisting of," "consist of," and "consists of" are closed-ended. For example, "A consists of B" means that A includes only B and does not include other components in the same context.

[0134] Combinations described herein, such as "at least one of A, B, or C," "one or more of A, B, or C," "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, C, or any combination thereof," include any combination of A, B, and / or C, and may include multiple As, multiple Bs, or multiple Cs. Specifically, combinations such as "at least one of A, B, or C," "one or more of A, B, or C," "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, C, or any combination thereof," may be A only, B only, C only, A and B, A and C, B and C, or A, B, and C, and any such combination may include one or more members of its components A, B, and / or C. For example, a combination of A and B may include one A and multiple Bs, multiple A and one B, or multiple A and multiple Bs.

Claims

1. Using at least one hardware processor, receiving a machine learning model having an architecture and parameters trained for operation in a first environment; co-tuning both the architecture and the parameters of the machine learning model using a training dataset for a second environment different from the first environment; Deploying the tuned machine learning model to the second environment. The method includes:

2. The method of claim 1 , wherein the machine learning model comprises a neural network.

3. The method of claim 2 , wherein the neural network is a convolutional neural network.

4. 3. The method of claim 2, wherein simultaneously adjusting both the architecture and the parameters of the machine learning model comprises, for each of one or more layers of the neural network, retraining the layer until a stopping condition is met.

5. The method of claim 4 , wherein the one or more layers is a plurality of layers.

6. 6. The method of claim 5, wherein the plurality of layers are trained in order from a layer of the plurality of layers that is closest to an output of the machine learning model to another layer of the plurality of layers that is closest to an input of the machine learning model.

7. The simultaneous tuning of both the architecture and the parameters of the machine learning model includes, in each of one or more iterations, selecting a layer from among a plurality of layers in the neural network that has not been retrained in an iteration prior to any of the one or more iterations; retraining the selected layers; and determining whether a stopping condition is met; stopping the simultaneous adjustment when the stopping condition is met; adding a next iteration to the one or more iterations when the stopping condition is not met; The method of claim 2 , comprising:

8. 8. The method of claim 7, wherein the one or more iterations are a plurality of iterations, each selected layer is retrained according to a learning rate, and wherein jointly adjusting both the architecture and the parameters of the machine learning model further comprises, when the stopping condition is not satisfied in at least one of the plurality of iterations, changing the learning rate before the next iteration.

9. The method of claim 7 , wherein retraining each selected layer comprises pruning the selected layer.

10. The method of claim 9 , wherein the stopping condition comprises a threshold indicating a measure of reduction in size of the machine learning model.

11. 10. The method of claim 9 , wherein each selected layer is retrained according to a learning rate, and wherein jointly adjusting both the architecture and the parameters of the machine learning model further comprises, when the stopping condition is not met in each of the one or more iterations, reducing the learning rate before the next iteration.

12. retraining the selected layers selecting a plurality of alternative layers each having a different size from the selected layer; training the plurality of substitution layers using the training dataset to minimize cross-entropy loss of the machine learning model; selecting the alternative layer having the lowest error measure from among the plurality of alternative layers as the retrained layer; The method of claim 7, comprising:

13. The method of claim 12 , wherein the cross-entropy loss comprises a negative log-likelihood loss and the error measure comprises a mean squared error.

14. The method of claim 12 , wherein each of the plurality of replacement layers has a size smaller than the selected layer.

15. 13. The method of claim 12, wherein each of the plurality of alternative layers is selected to have a size within a limited search space around the size of the selected layer that is different from the size of the selected layer and any other layer of the plurality of alternative layers.

16. 16. The method of claim 15, wherein each of the plurality of alternative layers has a size smaller than the selected layer, the neural network is a convolutional neural network, and the limited search space is defined based on a number of filters pruned from the selected layer.

17. The method of claim 16 , wherein the method further comprises using the at least one hardware processor to determine the number of filters to be pruned using principal component analysis.

18. 2. The method of claim 1 , wherein the tuned machine learning model estimates a location of a fault on a power line based on one or more measured parameters, the training dataset includes labeled feature vectors, each of the labeled feature vectors including a value for each of the one or more measured parameters and labeled with a fault location, the co-tuning includes pruning the machine learning model, the second environment is a relay configured to trip a circuit breaker on the power line, and deploying the tuned machine learning model includes installing the tuned machine learning model in a controller of the relay.

19. at least one hardware processor; Software and wherein the software, when executed by the at least one hardware processor, receiving a machine learning model having an architecture and parameters trained for operation in a first environment; co-tuning both the architecture and the parameters of the machine learning model using a training dataset for a second environment different from the first environment; Deploying the tuned machine learning model to the second environment. The system is configured as follows:

20. A non-transitory computer-readable medium having stored thereon instructions that, when executed by a processor, cause the processor to: receiving a machine learning model having an architecture and parameters trained for operation in a first environment; co-tuning both the architecture and the parameters of the machine learning model using a training dataset for a second environment different from the first environment; deploying the tuned machine learning model in the second environment; Non-transitory computer-readable medium.

Images