Global optimization for neural network training

Abstract

The dataset is received to train a machine learning model to perform a recognition task. Optimization is performed during the training of the machine learning model. The optimization includes at least the steps of: searching for a minimum value of the loss function; adding an additional term to the loss function in response to finding a local minimum; continuing to check for other local minimums until a criterion is met; and identifying the global minimum having the lowest minimum value among the found local minimums. The machine learning model may be updated using the parameters identified at the global minimum.

Description

[Background technology]

[0001] This application relates generally to computers and computer applications, more specifically to machine learning, machine learning model training, neural network training optimization, global optimization in machine learning training, and descent-based optimization for training machine learning models. [Overview of the Initiative]

[0002] This summary of the disclosure is provided to aid in understanding computer systems and methods for optimizing machine learning training, and is not intended to limit the disclosure or the invention. It should be understood that various aspects and features of the disclosure may, in some examples, be used separately to their advantage, or in other examples, be used in combination with other aspects and features of the disclosure to their advantage. Therefore, modifications and alterations may be made to computer systems and / or methods of their operation to achieve different effects.

[0003] In at least some embodiments, the computer implementation method includes the step of receiving a dataset for training a machine learning model to perform a recognition task. The method also includes the step of performing optimization during the training of the machine learning model, wherein the optimization includes at least: searching for a minimum of a loss function; adding an additional term to the loss function in response to finding a local minimum and continuing to find another local minimum until a criterion is met; and identifying a global minimum having the lowest minimum of the found local minimums. The method also includes the step of updating the machine learning model with the parameters identified at the global minimum.

[0004] In at least some embodiments, the system includes at least one processor. The system also includes at least one memory device coupled to the at least one processor. The at least one processor is configured to receive a dataset for training a machine learning model to perform a recognition task. The at least one processor also includes a step of performing optimization during the training of the machine learning model, wherein the optimization is configured to include at least: a step of searching for a minimum of a loss function; a step of adding an additional term to the loss function in response to finding a local minimum and continuing to find another local minimum until a criterion is met; and a step of identifying a global minimum having the lowest minimum of the found local minimums. The at least one processor is also configured to update the machine learning model with the parameters identified at the global minimum.

[0005] Computer-readable storage media are also provided that store programs of machine-executable instructions for performing one or more of the methods described herein.

[0006] The following sections will describe in detail further features and the structure and operation of various embodiments, with reference to the attached drawings. In these drawings, the same reference numerals indicate the same or functionally similar elements. [Brief explanation of the drawing]

[0007] [Figure 1] An example of a computing environment in which machine learning training optimization can be implemented in at least some embodiments is shown.

[0008] [Figure 2] Figures 2A to 2D show snippets of loss surfaces in which a global minimum can be observed using neural network training optimization in one embodiment.

[0009] [Figure 3] Figures 3A to 3C show the results of experiments on neural network training optimization in at least some embodiments.

[0010] [Figure 4] This flowchart illustrates an optimization method using local minimum filling for training machine learning models in at least some embodiments.

[0011] [Figure 5] This is another flowchart illustrating an alternative optimization method using local minimum filling for training machine learning models in at least some embodiments.

[0012] [Figure 6] This is a diagram showing the components of a system capable of performing machine learning, such as training and optimization, in one embodiment. [Modes for carrying out the invention]

[0013] In at least some embodiments, the computer implementation method includes the step of receiving a dataset for training a machine learning model to perform a recognition task. The method also includes the step of performing optimization during the training of the machine learning model, wherein the optimization includes at least: searching for a minimum of a loss function; adding an additional term to the loss function in response to finding a local minimum and continuing to find another local minimum until a criterion is met; and identifying a global minimum having the lowest minimum of the found local minimums. The method also includes the step of updating the machine learning model with the parameters identified at the global minimum.

[0014] Advantageously, optimization can be continued to verify more precise parameters of the trained model, which leads to improved performance of the trained model in recognition systems such as medical applications, but is not limited to such applications. By leveraging metadynamic principles, system optimization can be achieved more efficiently and quickly with fewer resources, and global minimums can be identified to be used as the basis for optimization of machine learning training. Optimization can be performed on any network architecture to perform optimization of machine learning models.

[0015] One or more of the following features are separable from each other or are optional. For example, in one embodiment, the machine learning model includes a deep neural network, and the optimization includes descent-based optimization. Thus, for example, deep neural network training can be improved, which can lead to the execution of a more powerful neural network, for example, in performing recognition tasks.

[0016] In another embodiment, the additional term is a Gaussian bias centered on the local minimum. Thus, for example, the loss surface around the local minimum can be filled.

[0017] In another embodiment, the criterion further includes a threshold number for local minimums. Thus, by providing, for example, a predetermined threshold number for local minimums, the training process in machine learning can be performed efficiently.

[0018] In another embodiment, the additional term is further added to the loss function until the local minimum is filled. For example, the local landscape may be modified so that the search does not return to the same location on the loss surface.

[0019] In yet another aspect, the method can further comprise storing the additional terms. In yet another aspect, the method further comprises accessing the stored additional terms and reconstructing the original landscape of the loss function by subtracting the added additional terms. The ability to restore the original loss surface can be useful in applications that refer to the original loss again.

[0020] In another aspect, a plurality of instances of the optimization are executed in parallel at different initialization points of the loss surface of the loss function. Thus, for example, the search for the global minimum can be accelerated.

[0021] In yet another aspect, the method also further comprises using the updated machine learning model in performing a recognition task. For example, by using an updated machine learning model that has been trained to have better execution parameters, an improvement in performance in the recognition task can be provided.

[0022] A system is provided that includes at least one computer processor and at least one memory device coupled to the at least one computer processor, where the at least one computer processor is configured to execute one or more of the methods described herein. A computer program product is also provided that includes a computer-readable storage medium having program instructions embodied therein, where the program instructions are readable by a computer to cause the computer to execute one or more of the methods described herein.

[0023] Various aspects of this disclosure are described by explanatory text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of computer program products (CPPs). With respect to any flowchart, depending on the technology involved, operations may be performed in a different order than those shown in a given flowchart. For example, also depending on the technology involved, two operations shown in consecutive blocks of a flowchart may be performed in reverse order, as a single integrated step, simultaneously, or in a manner that overlaps at least partially in time.

[0024] Embodiments of a computer program product ("CPP Embodiment" or "CPP") are terms used in this disclosure to describe any set of one or more storage media ("mediums") that are collectively comprised of one or more sets of storage devices that collectively contain machine-readable code corresponding to instructions and / or data for performing computer operations specified in a given CPP claim. A "storage device" is any tangible device capable of holding and storing instructions for use by a computer processor. Computer-readable storage media may be, but are not limited to, electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, mechanical storage media, or any suitable combination thereof. Some known types of storage devices, including these media, include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices (such as pits / lands formed on the main surface of a punch card or disk), or any suitable combination of the foregoing. When the term "computer-readable storage medium" is used in this disclosure, it shall not be interpreted as storage in the form of a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, optical pulses passing through optical fiber cables, electrical signals communicated through wires, and / or other transmission media.As those skilled in the art will understand, data typically moves at several intermittent points during the normal operation of a storage device, such as during access, defragmentation, or garbage collection. However, this does not mean that the storage device is temporary, because data is not temporary while it is stored in it.

[0025] The computing environment 100 includes an example of an environment for executing at least some of the computer code involved in performing the method of the invention, such as the machine learning training optimization algorithm code 200. In addition to the machine learning training optimization algorithm code 200, the computing environment 100 includes, for example, a computer 101, a wide area network (WAN) 102, an end user device (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, the computer 101 includes a processor set 110 (including processing circuits 120 and a cache 121), a communication fabric 111, volatile memory 112, persistent storage 113 (including the operating system 122 and the machine learning training optimization algorithm code 200 as shown above), a peripheral device set 114 (including a user interface (UI) device set 123, storage 124, and an Internet of Things (IoT) sensor set 125), and a network module 115. The remote server 104 includes a remote database 130. The public cloud 105 includes a gateway 140, a cloud orchestration module 141, a host physical machine set 142, a virtual machine set 143, and a container set 144.

[0026] Computer 101 may take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device currently known or to be developed in the future that is capable of running programs, accessing networks, or querying databases such as remote database 130. As is well understood in the field of computer technology, and depending on the technology, the execution of a computer implementation method may be distributed among multiple computers and / or across multiple locations. On the other hand, in this presentation of the computing environment 100, in order to keep the presentation as concise as possible, the detailed discussion focuses on a single computer, specifically computer 101. Computer 101 is not shown in the cloud in Figure 1, but it may be located in the cloud. On the other hand, computer 101 is not required to be located in the cloud, except to any extent that can be definitively shown.

[0027] The processor set 110 includes one or more computer processors of any kind currently known or to be developed in the future. The processing circuitry 120 may be distributed across multiple packages, for example, multiple coordinated integrated circuit chips. The processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. The cache 121 is memory located within the processor chip package and is typically used for data or code that should be available for high-speed access by threads or cores running on the processor set 110. The cache memory is typically organized into multiple levels depending on its relative proximity to the processing circuitry. Alternatively, some or all of the cache for the processor set may be located "off-chip". In some computing environments, the processor set 110 may be designed to operate with qubits and perform quantum computing.

[0028] Computer-readable program instructions are typically loaded onto computer 101 and cause the processor set 110 of computer 101 to execute a series of operational steps, thereby realizing the computer implementation method. As a result, the instructions thus executed instantiate the methods specified in the flowcharts and / or descriptions of the computer implementation methods contained herein (collectively referred to as the "Methods of the Invention"). These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 121 and other storage media discussed below. The program instructions and associated data are accessed by the processor set 110 to control and direct the execution of the Methods of the Invention. In the computing environment 100, at least some of the instructions for executing the Methods of the Invention may be stored in machine learning training optimization algorithm code 200 in persistent storage 113.

[0029] The communication fabric 111 is a signal conduction path that enables various components of the computer 101 to communicate with one another. Typically, this fabric is made up of switches and conductive paths, such as buses, bridges, physical input / output ports, and similar components. Other types of signal communication paths, such as optical fiber communication paths and / or wireless communication paths, may be used.

[0030] The volatile memory 112 is any type of volatile memory currently known or to be developed in the future. Examples include dynamic type random access memory (RAM) or static type RAM. Typically, the volatile memory 112 is characterized by random access, but this is not required unless explicitly stated. In computer 101, the volatile memory 112 is located in a single package and is internal to computer 101, but alternatively or additionally, the volatile memory may be distributed across multiple packages and / or located externally to computer 101.

[0031] The persistent storage 113 is any form of non-volatile storage for a computer, currently known or to be developed in the future. The non-volatility of this storage means that the stored data is maintained whether or not power is directly supplied to the computer 101 and / or the persistent storage 113. The persistent storage 113 may be read-only memory (ROM), but typically, at least a portion of the persistent storage allows for writing, deleting, and rewriting of data. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. The operating system 122 may take multiple forms, such as various known proprietary operating systems or open-source Portable Operating System Interface type operating systems, which utilize kernels. The code included in the machine learning training optimization algorithm code 200 typically includes at least some of the computer code included in performing the method of the invention.

[0032] The peripheral device set 114 includes a set of peripheral devices for the computer 101. Data communication connections between the peripheral devices and other components of the computer 101 can be implemented in various ways, such as Bluetooth connections, Near-Field Communication (NFC) connections, connections made by cables (such as Universal Serial Bus (USB) type cables), insertable connections (e.g., Secure Digital (SD) cards), connections made via local area communication networks, and even connections made via wide area networks such as the Internet. In various embodiments, the UI device set 123 may include components such as a display screen, speaker, microphone, wearable devices (such as goggles and smartwatches), keyboard, mouse, printer, touchpad, game controller, and haptic device. Storage 124 is external storage such as an external hard drive, or insertable storage such as an SD card. Storage 124 may be persistent and / or volatile. In some embodiments, storage 124 may take the form of a quantum computing memory device that stores data in the form of qubits. In embodiments where computer 101 is required to have a large amount of storage (for example, computer 101 locally stores and manages a large database), this storage may be provided by peripheral storage devices designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor set 125 consists of sensors that can be used in Internet of Things applications. For example, one sensor may be a thermometer and another may be a motion detector.

[0033] The network module 115 is a collection of computer software, hardware, and firmware that enables computer 101 to communicate with other computers via the WAN 102. The network module 115 may include hardware such as a modem or Wi-Fi® signal transceiver, software for packetizing and / or depackaging data for communication network transmission, and / or web browser software for exchanging data over the internet. In some embodiments, the network control function and network forwarding function of the network module 115 are executed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing software-defined networking (SDN)), the control function and forwarding function of the network module 115 are executed on physically separate devices, such that the control function manages several different network hardware devices. Computer-readable program instructions for performing the method of the present invention can typically be downloaded to computer 101 from an external computer or external storage device via a network adapter card or network interface included in the network module 115.

[0034] WAN102 is any wide area network (e.g., the Internet) capable of transmitting computer data over non-local distances using any currently known or future-developed technology for transmitting computer data. In some embodiments, WAN102 may be replaced and / or supplemented by a local area network (LAN), such as a Wi-Fi network, designed to transmit data between devices located in a local area. WANs and / or LANs typically include computer hardware such as copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0035] The end-user device (EUD) 103 is any computer system used and controlled by an end-user (e.g., a customer of the company operating computer 101), and may take any of the forms considered above in relation to computer 101. Typically, EUD 103 receives useful and valuable data from the operation of computer 101. For example, in a hypothetical case where computer 101 is designed to provide recommendations to the end-user, these recommendations would typically be transmitted from computer 101's network module 115 to EUD 103 via WAN 102. In this way, EUD 103 can display or otherwise present the recommendations to the end-user. In some embodiments, EUD 103 may be a client device such as a thin client, heavy client, mainframe computer, or desktop computer.

[0036] The remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. The remote server 104 may be controlled and used by the same entity that operates computer 101. The remote server 104 represents a machine that collects and stores useful and valuable data for use by other computers, such as computer 101. For example, in a hypothetical case where computer 101 is designed and programmed to provide recommendations based on historical data, this historical data may be provided to computer 101 from the remote database 130 of the remote server 104.

[0037] The public cloud 105 is any computer system available for use by multiple entities, providing on-demand availability of computer system resources and / or other computing capabilities, particularly data storage (cloud storage) and computing power, without requiring direct and active management by the user. Cloud computing typically leverages resource sharing to achieve coherence and economies of scale. Direct and active management of the computing resources of the public cloud 105 is performed by the computer hardware and / or software of the cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by virtual computing environments running on various computers that make up the computers of the host physical machine set 142, which is the universe of physical computers available in and / or to the public cloud 105. The virtual computing environment (VCE) typically takes the form of virtual machines from the virtual machine set 143 and / or containers from the container set 144. These VCEs may be stored as images and are understood to be transportable either as images or after instantiation of the VCEs, in and between various physical machine hosts. The cloud orchestration module 141 manages the transfer and storage of images, deploys new instances of VCE, and manages active instances of VCE deployments. The gateway 140 is a collection of computer software, hardware, and firmware that enables the public cloud 105 to communicate via the WAN 102.

[0038] Here, we will provide some further explanation about virtualized computing environments (VCEs). A VCE can be stored as an "image." From this image, a new active instance of the VCE can be instantiated. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to an operating system feature in which the kernel allows for the existence of multiple isolated user-space instances called containers. These isolated user-space instances typically behave as actual computers from the perspective of the programs running within them. Computer programs running on a normal operating system can utilize all of that computer's resources, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, programs running inside a container can only use the contents of the container and the devices allocated to the container; this feature is known as containerization.

[0039] The private cloud 106 is similar to the public cloud 105, except that its computing resources are available only for use by a single enterprise. While the private cloud 106 is shown as communicating with the WAN 102, in other embodiments, the private cloud may be completely isolated from the internet and accessible only via a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types), often implemented by different vendors. Each of the multiple clouds remains a separate discrete entity, but the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the multiple configuration clouds. In this embodiment, both the public cloud 105 and the private cloud 106 are part of a larger hybrid cloud.

[0040] Deep neural networks are a class of machine learning algorithms based on artificial neural networks, which include multiple hidden layers between the input and output layers. Artificial neural networks are also referred to as neural networks in this specification. While training a deep neural network or neural network using a descent-based algorithm, the goal is to determine the minimum point at which training can stop and to provide a well-trained model that can accurately provide recognition and / or predictions for their respective applications. Descent-based algorithms aim to minimize or optimize the loss function, which indicates the cost or error between the predictions and predicted values ​​of a machine learning model. For example, the loss function indicates how well or poorly the machine learning model performed its task.

[0041] Gradient descent is an example of a descent-based algorithm, an iterative method for locating the minimum of a function. This descent-based algorithm is an optimization technique for locating the parameters or coefficients of a function that have a minimum value, for example, here the function has a minimum value. However, the algorithm does not always find the global minimum and may become confined to a local minimum. The global minimum is the lowest value of the function, while a local minimum is the lowest value of the function in a particular neighborhood.

[0042] However, identifying the global minimum among the local minima on the loss surface can be difficult. For example, there may be tasks where the global optimal solution is far from a local minimum, and identifying it requires a detailed search of the loss surface. For instance, in the past, the only guaranteed way to identify the global minimum of an optimization problem was a complete search of the parameter space.

[0043] Disclosed are systems, methods, and / or techniques that can provide protocols for enabling optimization algorithms, such as descent-based algorithms, in machine learning, such as neural network training, to explore a loss surface to find a global optimal solution and / or an approximation thereof based on a predetermined threshold, in one or more embodiments. In at least some embodiments, a descent-based optimization process for training a recognition system can be enhanced by a Gaussian filling technique of the loss surface. A computer implementation method can add a penalty term (e.g., a Gaussian bias) to the loss when a local minimum is reached, for example, to encourage the optimization process to explore other regions of the parameter space. For example, if the optimization procedure reaches a local minimum, a Gaussian bias can be added to the loss function. The Gaussian bias can be added until a local minimum is filled, and the optimization procedure can continue. Thus, for example, a descent-based algorithm can be prevented from returning to previous points or local minimums, which can enable improved identification of global minimums. In some embodiments, this may be beneficial for recognition systems used in rare event identification, medical applications, risk analysis, and / or other technical applications. For example, improved recognition of global minimums results in a trained neural network model that can accurately perform those functions, e.g., recognition and / or prediction, thereby improving the overall performance of the neural network model. Another benefit of the neural network training optimization disclosed herein may be that the optimization techniques enable training to reach global minimums more quickly, and therefore speed up the neural network training process. In this way, the execution time and memory space of the computer processor can be utilized more efficiently.

[0044] Figures 2A to 2D show snippets of loss surfaces in which a global minimum can be found using neural network training optimization in one embodiment. In the figures, the y-axis represents the loss function, the x-axis represents the parameter space, and the curve represents the loss space. The snippet terminology refers to loss surfaces that extend further to the right and / or left outside the figure of the illustrated graph, but the snippet portion is shown to illustrate the concept of navigating around various minimums identified in accordance with this disclosure. In Figure 2A, a descent-based algorithm during neural network training identifies a local minimum of 202. For example, a descent-based algorithm finds a minimum loss without realizing, for example, that the identified minimum loss may be a local minimum, and that a global minimum may exist elsewhere in the loss space greater than other local minimums. If no other processes are performed, the algorithm may become confined at this local minimum. In Figure 2B, the neural network training optimization described herein in one or more embodiments can fill the loss space with a penalty term, e.g., a Gaussian bias 204, so that the descent-based algorithm no longer recognizes this region as a minimum and continues its search to find another minimum that may be the global minimum. In Figure 2C, the descent-based algorithm first performs an epoch in which it finds another minimum and adds a first Gaussian at the newly found minimum. After adding the first Gaussian, the curve or loss space generates a first modified minimum 206. However, this first modified minimum still represents a local minimum relative to the overall curve, as indicated by the continuing concave area above the first modified minimum 206. Thus, the optimization, upon detecting that a minimum and a concave area still exist, performs one or more additional epochs adding one or more additional Gaussians until the entire concave area is filled as shown in Figure 2D.In Figure 2D, in one or more embodiments, the neural network training optimization can fill the loss space with a sum of multiple Gaussian biases 208 in order to find the global optimization or global minimum in this example.

[0045] In some embodiments, the objective function f, i.e., the function to be minimized, can be a function of the difference between observed data values ​​(e.g., actual values) and a predictive model with a constant peak (predicted values). For example, this can be as simple as the sum of absolute differences (or, for example, the mean absolute error (MAE), which is the average absolute error between the actual and predicted values). Another example is the mean squared error (MSE), which is the mean squared error between the actual and predicted values.

[0046] In some embodiments, for example, during gradient descent, the algorithm determines the current position p n The gradient in is used to iteratively calculate the following point, (where the learning speed is

number

number

number

[0047] To enhance system sampling and enable global optimization by suppressing the re-examination of sampled states (and local minimums), neural network training optimization in at least some embodiments uses minimal algorithms.

number

number

[0048] In some embodiments, for a one-dimensional problem, the added bias takes the form of a one-dimensional Gaussian, where,

number

number

number

number

[0049] In at least some embodiments, the kernel can be a multidimensional function when a high-dimensional surface is the target of optimization. The Gaussian is a local minimum.

number

[0050] Figures 3A to 3C show experimental results of neural network training optimization in at least some embodiments. Figure 3A shows the loss surface of a non-convex function commonly used as a performance test problem for optimization algorithms. The three-dimensional surface shows peaks and valleys where multiple local and global minimums exist. Figures 3B and 3C show two-dimensional top views of the loss function shown in Figure 3A, where unshaded areas represent valleys (minims) of various depths, and shaded areas represent peaks of various heights. For example, the global minimum (e.g., the steepest valley on the loss surface) is assumed to be at the × mark indicated at 302. In the experiment, after 5,000 iterations, it was observed that the gradient descent was confined at the first local minimum 304. For example, in Figure 3B, the optimization phase of the gradient descent, starting from (3.5, -3.5), ends at the diamond mark indicated at 304 in order to search for the global minimum at (0, 0). However, using the methods described herein, for example, an optimization step of gradient descent with an added penalty term (e.g., Gaussian), it was observed that the optimization reached the global minimum of 302 in 11 iterations. Referring to Figure 3C, an optimization step of gradient descent with an added bias (indicating a filled valley) starting from (3.5,-3.5) to search for the global minimum at 0,0) reaches the global minimum of 302. The figure also shows the change in the loss landscape due to the added bias or Gaussian (e.g., filled valley). In the experiment, the learning speed was set to 0.001. Other learning speeds may be used.

[0051] The algorithms disclosed herein enable, for example, a processor to efficiently explore the loss landscape. For example, adding biases improves the spatial exploration. In at least some embodiments, a stopping criterion may be provided. For example, after N minimums have been identified, the deepest of N may be selected. N can be pre-configured or given. In at least some embodiments, the landscape can be recovered at any time by storing the added biases and removing or subtracting them from the original loss function.

[0052] Figure 4 is a flowchart illustrating a method for optimization using local minimum filling according to at least some embodiments. The method enables optimizations such as descent-based algorithms in machine learning, such as neural network training, to search a loss surface for global minimum retrieval. For example, a protocol for global optimization in neural network training of a recognition system may be provided. While this description refers to neural network training, the method can also be used to train other machine learning architectures in which optimizations such as descent-based algorithms may be used. One example is a regression model. Furthermore, while this description refers to gradient descent-based optimization algorithms, other variations of optimization algorithms such as alternating minimization, stochastic gradient descent, block coordinate descent, backpropagation, and quasi-Newton's method can be used. In addition, filling the loss surface with an additional term (e.g., Gaussian bias) can be applied to other machine learning optimizations in which decisions are made based on parameter values ​​other than gradient.

[0053] In 402, the method includes the step of receiving a dataset for training a machine learning model to perform a recognition task. The dataset may include input / output pairs so that supervised learning can be performed. For example, output data corresponding to input data may be used as actual or predicted values ​​when calculating a loss function during training optimization, where predicted values ​​generated by the machine learning model are compared to actual values. In some embodiments, the machine learning model may be a deep neural network. In some embodiments, the recognition task may include tasks that identify images such as medical images, tasks that recognize cell structures, and / or others.

[0054] In 404, the method also includes a step of performing optimizations such as descent-based optimization during the training of a machine learning model. An example of descent-based optimization includes the gradient descent algorithm and / or its variations. Other optimizations may also be applied. The optimization includes a step of searching for the minimum value of the loss function used in the optimization. During training, in response to finding a local minimum, the method includes a step of adding an additional term to the loss function and continuing the optimization process to find another local minimum until a criterion is met. For example, the method includes a step of searching for more local minimums, continuing until a criterion is met. In at least some embodiments, the additional term may be a Gaussian bias centered on a local minimum. In at least some embodiments, the additional term is added to the loss function until a local minimum is filled. Thus, the optimization algorithm may move on with its optimization and not reconsider this local minimum. Data associated with the local minimum may be stored for later reference. In at least some embodiments, the criterion may be a threshold number of local minimums, e.g., confirmation that a predetermined threshold number of local minimums has been identified. Among the multiple local minima identified during this optimization, the one with the lowest minimum value can be identified as the global minimum.

[0055] In 406, the method includes a step of updating the machine learning model using the parameters identified at the global minimum. For example, the parameters associated with the lowest loss function value may be used as machine learning model parameters.

[0056] Beneficially, the method may enable efficient exploration of the loss surface or landscape, and by identifying the global minimum among the local minima of the loss function on the loss surface, it may lead to a trained machine learning model with improved performance. In some embodiments, the method does not require prior knowledge of the landscape of the loss function when identifying the global minimum.

[0057] In at least some embodiments, the method may also include a step of storing additional terms. For example, information about how many additional terms were used to fill the local minimum space and / or the values ​​of those additional terms may be stored. Thus, for example, the original landscape of the loss function can be reconstructed by removing the added additional terms, for example, by subtraction. The ability to recover the original landscape may be useful for applications that require information about the original structure, for example.

[0058] In at least some embodiments, multiple instances of optimization can be performed in parallel at different initialization points of the loss surface of the loss function. For example, different parts of the loss surface can be explored simultaneously or in parallel, which can accelerate the training process to find the global minimum, thereby accelerating the confirmation of the optimal solution. Thus, for example, training can be accelerated, and this can lead to building a more powerful model for performing its recognition task more quickly.

[0059] In 408, the method also includes a step of using an updated machine learning model in performing a recognition task. For example, a machine learning model, such as a neural network, performs the recognition task. Beneficially, a trained machine learning model may be capable of performing complex recognition tasks or solving problems.

[0060] Figure 5 is another flowchart illustrating a method for filling in local minima to perform optimization according to at least some embodiments. The method demonstrates neural network training for a recognition system, for example, using a protocol that adds a bias term to the loss function to search for a global optimal solution so that the optimal parameters for the neural network can be identified. Here, the optimization strategy is based on gradient descent. However, it should be understood that any optimization method may be used. The method may be run or implemented on one or more computer processors.

[0061] In 502, the processor collects or receives a dataset of input / output pairs, where the output is a recognition label associated with the recognition task.

[0062] In step 504, the processor constructs a network. For example, an appropriate machine learning architecture may be selected for a recognition task. One example is a neural network or a deep neural network.

[0063] At 506, the processor is at a local minimum.

number

number

number

[0064] In 508, the processor initiates the optimization process using a loss function. In at least some embodiments, a bias is added only when a local minimum is identified. For example, a loss function with a bias may have a zero scale unless a local minimum is reached. At each step of the optimization process, the processor may calculate the gradient of the loss function. For example, for each parameter in the loss function, the gradient of the loss function may be calculated with respect to the parameter.

[0065] In step 510, if the gradient crosses a threshold, for example, if it falls below it, the processor identifies the current position as a local minimum and adds this minimum identified current position to the minimum list in step 512. The minimum list contains all local minimums identified during optimization.

[0066] In 514, based on the presence of a minimum, the processor updates the loss function and calculates a new gradient. For example, the loss function at the locations identified in 510 and 512 may be updated by adding an additional term (also called a penalty term or bias). The additional term (penalty term or bias) may be a Gaussian bias. For example, if a minimum is identified, the processor may make the scale (omega (ω)) of the loss term non-zero. In this way, a bias may be added. In at least some embodiments, the bias is added to the loss function and the location p is updated based on the gradient of the loss. In some embodiments, this changes the loss landscape. Note that the bias added to the loss function is isolated from the neural network bias.

[0067] Optimization continues until the loss space is explored in a satisfactory manner, for example, if among all the identified minimums there exists one that provides a relatively lower loss value than the others, this will be designated as the global minimum. For example, in 516, it is determined whether the criteria have been met. One example of a criterion is that a threshold number of local minima have been confirmed. For example, the loss space has been explored sufficiently so that a predetermined N local minima have been confirmed, in which case the global minimum can be identified from the confirmed local minima. If the criteria are met, the method proceeds to 518. Otherwise, the method continues to 510, where it is determined whether the new gradient calculated in 514 falls below the threshold, and as a result, the location of that new gradient may be identified as another local minima.

[0068] In step 518, the processor updates the neural network's weights and biases based on the identified global minimum.

[0069] In 520, the trained neural network is used for the recognition task.

[0070] In another embodiment, the processor performs an optimization search to "jump" away from the minimum in step p n+1 by automatically adding a direct additional Gaussian bias (having different scaling parameters) directly to step p. For example, step p n+1 (e.g.,

Number

[0071] In at least some embodiments, the processor automatically performs multiple optimizations at different starting points, e.g., simultaneously or in parallel. This technique helps to achieve efficient parallel training.

[0072] The methods disclosed herein can accelerate the training of machine learning models.

[0073] In at least some embodiments, in the presence of a minimum, the processor adds a Gaussian (additional term) in an area of small gradient to continue the loss surface search. Adding a Gaussian to the neighborhood with a small gradient enables an optimization process that addresses not only saddle points but also minima. It also provides a way to easily reconstruct the discovered loss surface. For example, the added additional term can be removed, e.g., subtracted, at the position to reconstruct the original loss surface.

[0074] The system and method may enable descent-based algorithms, such as neural network training or other machine learning training, to explore the loss surface to find a global optimal solution. Additional terms (also referred herein as penalty terms, biases, Gaussians, and Gaussian biases) may be added to the loss when a local minimum is reached, to encourage the optimization process to explore other regions of the parameter space. For example, if the optimization procedure reaches a local minimum, a Gaussian bias may be added to the loss function. The Gaussian may be added until the local minimum is filled, and the optimization procedure can continue. In the additional term, the Gaussian may be centered on the local minimum, and the standard deviation or spread (σ) may be a function of the learning rate. In another embodiment, the standard deviation or spread in the additional term may be determined using a Hessian estimate of the loss.

[0075] In some instances, the loss landscape may change during optimization, which can prevent re-examining known local minima. In at least some embodiments, the loss landscape can be recovered at any point as the opposite of adding all Gaussians, e.g., the sum of all Gaussians, by removing, e.g., subtracting, the added terms. Multiple independent optimizations (e.g., starting from different points on the loss surface) may be initiated to discover parts of the loss landscape.

[0076] Trained machine learning models, such as neural network models trained using minimal value filling as described herein, can be used in recognition systems or tasks such as, but not limited to, medical applications and risk analysis. Another example of use cases might be observing low-energy configurations when designing raw materials.

[0077] Figure 6 is a diagram showing the components of a system capable of performing machine learning, such as training and optimizing a neural network, in one embodiment. One or more hardware processors 602, such as a central processing unit (CPU), a graphics process unit (GPU), and / or a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and / or another processor, are coupled to a memory device 604 and can train a neural network to perform a recognition task. The memory device 604 may include random access memory (RAM), read-only memory (ROM), or another memory device, and may store data or processor instructions, or both, for implementing various functions associated with the method or system described herein or both. One or more processors 602 may execute computer instructions stored in the memory 604 or received from another computer device or medium. The memory device 604 may store instructions and / or data for the functions of one or more hardware processors 602, and may include an operating system, other programs of instructions and / or data. One or more hardware processors 602 may receive a dataset for training a machine learning model to perform a recognition task. One or more hardware processors 602 may perform optimizations such as descent-based optimization during training of the machine learning model. The optimization may include at least a step of finding a minimum of a loss function, for example, where the loss function is the one used in the optimization, where an additional term is added to the loss function in response to finding a local minimum and the search for another local minimum continues until a criterion is met, and a step of identifying a global minimum having the lowest minimum of the found local minimums.One or more hardware processors 602 may update a machine learning model using parameters identified at a global minimum. In one embodiment, a dataset may be stored in a memory device 606 or received from a remote device via a network interface 608 and temporarily loaded into a memory device 604 for training the machine learning model. The trained machine learning model, e.g., a neural network, may be stored in the memory device 604 for execution or use in inference by one or more hardware processors 602, for example. One or more hardware processors 602 may be coupled with interface devices, e.g., a network interface 608 for communicating with a remote system via a network, and an input / output interface 610 for communicating with input devices and / or output devices such as a keyboard, mouse, display, and / or other.

[0078] The technical terms used herein are intended to describe only specific embodiments and are not intended to limit the invention. Where used herein, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context otherwise explicitly indicates. Where used herein, the term "or" is an inclusive operator and may mean "and / or" unless the context otherwise explicitly or explicitly indicates. Where used herein, the terms "comprise," "comprises," "comprising," "include," "includes," "including," or "having," or any combination thereof, may specify the presence of a described feature, integer, step, operation, element, or component or combination thereof, but it will be further understood that this does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components or groups or combinations thereof. Where used herein, the phrase "in at least some embodiments" may, but not necessarily, refer to the same embodiment. As used herein, the phrase "in one embodiment" may, but not necessarily, refer to the same embodiment. As used herein, the phrase "in another embodiment" may, but not necessarily, refer to a different embodiment. Furthermore, embodiments, components of embodiments, or both may be freely combined with each other, provided they are not mutually exclusive.

[0079] All means-plus-function elements or step-plus-function elements (if any) in the following claims are intended to include any structures, materials, or actions for performing a function in combination with other claimed elements, as specifically claimed. Although the description of the present invention is expressed for illustrative and explanatory purposes, it is not intended to be exhaustive or limited to the invention in the disclosed forms. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. These embodiments have been selected and described to best illustrate the principles and practical applications of the invention and to enable other those skilled in the art to understand the invention in terms of various embodiments with various modifications suitable for the particular use intended.

Claims

1. The stage of receiving a dataset to train a machine learning model in order to perform a recognition task; During the training of the machine learning model, an optimization is performed, wherein the optimization includes at least: The stage of searching for the minimum value of the loss function; A step in which an additional term is added to the loss function in response to the confirmation of a local minimum, and the confirmation of another local minimum continues until the criterion is met; and The step of identifying the global minimum value that has the lowest minimum value among the confirmed local minimums. Including; and The step of updating the machine learning model using the parameters identified at the global minimum. A computer implementation method comprising the following features.

2. The computer implementation method according to claim 1, wherein the machine learning model includes a deep neural network, and the optimization includes descent-based optimization.

3. The computer implementation method according to claim 1, wherein the aforementioned additional term is a Gaussian bias centered on the local minimum.

4. The computer implementation method according to claim 1, wherein the aforementioned criteria include a threshold number of minimum values.

5. The computer implementation method according to claim 1, wherein the additional term is added to the loss function until the local minimum is filled.

6. The computer implementation method according to claim 1, further comprising the step of storing the additional items.

7. The computer implementation method according to claim 6, further comprising the step of reconstructing the original landscape of the loss function by accessing the stored additional terms and subtracting the added additional terms.

8. The computer implementation method according to claim 1, wherein multiple instances of the optimization are executed in parallel at different initialization points of the loss surface of the loss function.

9. The computer implementation method according to claim 1, further comprising the step of using the updated machine learning model in performing a recognition task.

10. A computer program product comprising a computer-readable storage medium in which program instructions are embodied, wherein the program instructions are: Procedure for receiving a dataset to train a machine learning model to perform a recognition task; A procedure for performing optimization during the training of the machine learning model, wherein the optimization comprises at least: Procedure for finding the minimum value of the loss function; A procedure for adding an additional term to the loss function in response to the confirmation of a local minimum, and continuing to check for another local minimum until the criterion is met; and Procedure for identifying the global minimum having the lowest minimum value among the confirmed local minimums. Including; and A procedure for updating the machine learning model using the parameters identified at the global minimum. A computer program product that is readable by the computer in order to execute it.

11. The computer program product according to claim 10, wherein the machine learning model includes a deep neural network, and the optimization includes descent-based optimization.

12. The computer program product according to claim 10, wherein the aforementioned additional term is a Gaussian bias centered on the local minimum.

13. The computer program product according to claim 10, wherein the aforementioned criteria include a minimum threshold number.

14. The computer program product according to claim 10, wherein the additional term is added to the loss function until the minimum value is filled.

15. The computer program product according to claim 10, wherein the computer is further capable of storing the additional items.

16. The computer program product according to claim 10, wherein multiple instances of the optimization are executed in parallel at different initialization points of the loss surface of the loss function.

17. The computer program product according to claim 10, wherein the computer is further made to use the updated machine learning model in performing a recognition task.

18. at least one processor; and At least one memory device connected to the aforementioned at least one processor; The aforementioned at least one processor is at least: Receiving a dataset for training a machine learning model to perform a recognition task; Performing optimizations during the training of the machine learning model, wherein the optimizations include at least: Finding the minimum value of the loss function; In response to the confirmation of a local minimum, an additional term is added to the loss function, and the confirmation of another local minimum is continued until the criterion is met; and Identify the global minimum value that has the lowest minimum value among the confirmed local minimums. Including; and Updating the machine learning model using the parameters identified at the global minimum. It is configured to do so. A system equipped with these features.

19. The system according to claim 18, wherein the machine learning model includes a deep neural network, and the optimization includes descent-based optimization.

20. The system according to claim 18, wherein the aforementioned additional term is a Gaussian bias centered on the minimum value.

Images