Decentralized Distributed Deep Learning

Abstract

Various embodiments for decentralized distributed deep learning by one or more processors in a computing system are provided. Asynchronous distributed training of one or more machine learning models can be performed by generating a list of neighbor nodes for each of a plurality of nodes, and creating a first thread for continuous communication according to a weight management operation, and creating a second thread for continuous calculation of gradients for each node. One or more variables are shared between the first thread and the second thread.

Description

Technical Field

[0001] The present invention relates generally to computing systems, and more particularly to various embodiments for decentralized distributed deep learning in computing systems using computing processors. Background Art

[0002] Computing systems can be found in the workplace, at home, or in school. Due to recent advances in information technology and the growing popularity of the Internet, a wide variety of computer systems have been used in machine learning. Machine learning is a form of artificial intelligence that allows computers to evolve behaviors based on empirical data. Machine learning can use training examples to capture interesting properties of their unknown underlying probability distributions. Training data can be viewed as examples showing the relationship between observed variables. The main focus of machine learning research is to automatically learn to recognize complex patterns and make intelligent decisions based on data. SUMMARY OF THE INVENTION

[0004] Different embodiments are provided for decentralized distributed deep learning by one or more processors in a computing system. In one embodiment, by way of example only, a method is provided for decentralized distributed deep learning by a processor in a computing system. Asynchronous distributed training of one or more machine learning models may be performed by generating a list of neighbor nodes for each of a plurality of nodes, and creating a first thread for continuous communication based on a weight management operation, and creating a second thread for continuous calculation of gradients for each node. One or more variables are shared between the first thread and the second thread. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In order to readily understand the advantages of the present invention, a more particular description of the present invention briefly described above will be presented by reference to specific embodiments shown in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the present invention and are not therefore to be considered limiting of its scope, the present invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0006] Figure 1 is a block diagram illustrating an exemplary cloud computing node according to an embodiment of the present invention;

[0007] Figure 2 is an additional block diagram illustrating an exemplary cloud computing environment according to an embodiment of the present invention;

[0008] Figure 3 is an additional block diagram describing an abstract model layer according to an embodiment of the present invention;

[0009] Figure 4 is another block diagram depicting a deep learning system according to another embodiment of the present invention;

[0010] Figure 5 is an additional block diagram depicting various user hardware and computing components according to aspects of the present invention;

[0011] Figure 6 is another block diagram describing a neighbor list for decentralized distributed deep learning according to an embodiment of the present invention;

[0012] Figure 7A-7B is a diagram illustrating operations for generating a neighbor list for decentralized distributed deep learning according to an embodiment of the present invention;

[0013] Figures 8A-8B is a graph depicting operational results from decentralized distributed deep learning according to an embodiment of the present invention; and

[0014] Fig. 9 is a flowchart depicting an exemplary method for performing decentralized distributed deep learning by a processor in a computing environment according to an embodiment of the present invention in which aspects of the present invention may be implemented. DETAILED DESCRIPTION

[0015] The present invention relates generally to the field of deep learning, and more specifically to performing multiple reductions in deep learning implementations. Deep learning refers to a class of machine learning algorithms based on learning multiple levels of features or representations of a data set. Current deep learning methods include using a cascade of multiple layers of nonlinear processing units to perform feature extraction and transformation. Feature extraction refers to the process of receiving an initial set of measurement data and constructing derived values ​​(or features) to facilitate subsequent learning and generalization steps. In many cases, higher-level features are derived from lower-level features to generate a hierarchical representation of the measurement data and derived features.

[0016] In addition, deep learning algorithms are based on distributed representations. Distributed representations operate under the assumption that observed (or measured) data are the result of the interaction of one or more factors organized into one or more layers. Conceptually, deep learning introduces an additional assumption: the layers of factors that interact to provide measured data represent levels of abstraction or arrangement. Under this assumption, multiple layers and layer sizes correspond to different amounts of abstraction.

[0017] Any or all data utilized and created in a deep learning system may be transmitted over one or more networks and may subsequently be subject to any limitations of the one or more networks. Specifically, for large-scale deep learning systems, any network communication may suffer from bottlenecks due to the large number of learners (e.g., workers), the frequency of data exchange across the network, and the amount of data exchanged. It should be noted that a learner / worker can be an application (e.g., a piece of code that takes input, performs one or more necessary calculations and communications to iteratively generate a more accurate and efficient solution). In the context of the present patent application, both the weight management thread and the gradient calculation thread are part of the learner / worker process. In addition, communication across multi-layer networks may be largely inefficient because the weakest link or node in the network will largely determine how the network will perform as a whole.

[0018] In addition, one or more challenges may arise when training deep learning systems. In a distributed cloud computing environment, computing devices may vary in processing speed (e.g., slow / fast computing), uneven network connectivity (e.g., fast / slow network links, different network differences, etc.), and even different levels of system noise, for example due to a heterogeneous environment. In addition, due to the elasticity of the distributed cloud computing environment, computing devices become available dynamically. In neural network ("NN") training, synchronous training is rigid and uncontrolled asynchrony often leads to inefficient or inaccurate training results. Thus, there is a need for a distributed training system with regular actions to improve the accuracy of training results while adapting to heterogeneous and elastic computing environments so as to utilize all available computing resources.

[0019] Thus, the present invention provides decentralized distributed deep learning in a computing system. Asynchronous distributed training of one or more machine learning models can be performed by generating a list of neighbor nodes for each of a plurality of nodes, and creating a first thread for continuous communication based on a weight management operation, and creating a second thread for continuous calculation of the gradient of each node. One or more variables are shared between the first thread ("thread 1") and the second thread ("thread 2"). It should be noted that machine learning and deep learning models have many parameters. The training process is to determine the best / optimal value assignment to the parameters so that the model matches the data as closely as possible. These model parameters are also referred to as weights (e.g., weight parameters). For example, an example management / update equation for weights is:

[0020] new weight =old weight +learning rate * gradient, (1),

[0021] where "gradient" is what is produced by the gradient computation thread (e.g., thread 2), and learning rate is a selected / defined value that may be less than the gradient. In one aspect, the weight management operations of a first thread (e.g., thread 1) may communicate with one or more application programming interfaces ("APIs") having downlink framework capabilities. For example, the following equation may be used for thread 1 (weight management):

[0022] W′=pull weights (2)

[0023] W=weight apply (W′, g) (3),

[0024] And for thread 2 (gradient computation):

[0025] g=calGrad(W) (4),

[0026] push gradient (g) (5),

[0027] Where W' is equal to pull weights , where W is equal to weight apply (W', g), and g is the previously calculated gradient (e.g., the gradient calculated last time / previous round). The gradient calculation can be gradient ("g") equal to calGad (W) and push gradient (g) situation.

[0028] By further explanation, thread 2 is equal to the equation "g = calGrad (W)", which indicates that based on the current weights (model parameters) W, thread 2 will use the function calGrad () to calculate the gradient. Then, push gradient (g) Action / operation Send gradient g to thread 1. Thread 1 (e.g., W = weight apply (W′, g)) Use the above equation (1) to perform weight apply Action / operation. Thread 1 also executes W′=pull weights Action / operation to receive a weight from the next neighbor in its neighbor list. Thread 1 performs an average of its weight and the received neighbor weight to generate a new weight. This new weight is sent to Thread 2 for its calculation.

[0029] In an additional aspect, the present invention provides asynchronous distributed training of machine learning models by performing calculations and communications on each learner (trainer) respectively. In one aspect, a neighbor list can be constructed for each learner. The neighbor list satisfies connectivity, propagation, and deadlock-free properties. A first thread (e.g., "thread 1") is created for continuous communication, and weight updates satisfy data consistency properties. A second thread (e.g., "thread 2") is created for continuous calculation. Shared variables, local gradients, and weight factors can be shared to communicate between thread 1 and thread 2. Fault detection operations can be used to detect remote node / link failures to maintain progress. In one aspect, the fault detection operation can be a timeout operation, wherein if the communication with the neighbor does not return after the timeout value, the operation continues to the next operation. In addition, in an additional aspect, the data consistency operation can be an atomic data consistency operation. That is, the data consistency operation maintains data consistency by limiting (e.g., not allowing) any weight update operation at the local learner when the weight is sent and averaged.

[0030] It should be noted that one or more calculations may be performed using different mathematical operations or functions that may involve one or more mathematical operations (e.g., performing rate of change / calculation operations, analytically or computationally solving differential equations or partial differential equations, using addition, subtraction, division, multiplication, standard deviation, mean, average, percentages, statistical modeling using statistical distributions, by finding minimum, maximum or similar thresholds of combined variables, etc.).

[0031] In general, "optimization" may refer to and / or be defined as "maximizing," "minimizing," or achieving one or more specific goals, objectives, goals, or intentions. Optimization may also refer to maximizing benefit to a user (e.g., maximizing machine learning model benefit). Optimization may also refer to making the most efficient or functional use of a situation, opportunity, or resource.

[0032] Additionally, "optimize" need not refer to the best solution or result, but may refer to a solution or result that is "good enough," for example, for a particular application. In some implementations, the goal is to suggest the "best" combination of machine learning models, but there may be various factors that may lead to alternative suggestions for combinations of machine learning models that produce better results. Here, the term "optimize" may refer to such results based on a minimum (or maximum, depending on what parameters are considered in the optimization problem). In additional aspects, the terms "optimize" and / or "optimization" may refer to operations performed in order to achieve improved results (such as reduced execution costs or increased resource utilization), regardless of whether the optimal result is actually achieved. Similarly, the term "optimize" may refer to components used to perform such improvement operations, and the term "optimize" may be used to describe the results of such improvement operations.

[0033] It is understood in advance that although the present disclosure includes detailed descriptions about cloud computing, the implementation of the teachings cited herein is not limited to cloud computing environments. Instead, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0034] Cloud computing is a model for enabling service delivery of convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a service provider. The cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0035] Features are as follows:

[0036] On-demand self-service: Cloud consumers can unilaterally provision computing capabilities, such as server time and network storage, automatically on demand without requiring human interaction with the provider of the service.

[0037] Broad Network Access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0038] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically assigned and reassigned as needed. There is a sense of location independence, as consumers typically have no control or knowledge of the exact location of the provided resources, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0039] Rapid elasticity: Capacity can be quickly and elastically provided (in some cases, automatically) to quickly scale down and quickly released to quickly scale up. To the consumer, the capacity available for provisioning generally appears unlimited and can be purchased at any time and in any quantity.

[0040] Measured services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the utilized services.

[0041] The business model is as follows:

[0042] Software as a Service (SaaS): The capability provided to the consumer is to use the provider's applications running on the cloud infrastructure. The applications are accessible from different client devices through a thin client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0043] Platform as a Service (PaaS): The capability provided to consumers is to deploy consumer-created or acquired applications on the cloud infrastructure, which are created using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure including networks, servers, operating systems, or storage, but have control over the deployed applications and possibly the configuration of the application hosting environment.

[0044] Infrastructure as a Service (IaaS): The capabilities provided to consumers are processing, storage, networking, and other basic computing resources on which consumers can deploy and run arbitrary software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but have control over the operating system, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0045] The deployment model is as follows:

[0046] Private Cloud: Cloud infrastructure is operated solely for an organization. It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0047] Community Cloud: The cloud infrastructure is shared by several organizations and supports a specific community with shared concerns (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0048] Public cloud: Cloud infrastructure is made available to the public or large industry groups and is owned by an organization that sells cloud services.

[0049] Hybrid cloud: A cloud infrastructure that is a composition of two or more clouds (private, community, or public) that remain unique entities but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0050] Cloud computing environments are service-oriented, focusing on state, low coupling, modularity, and semantic interoperability. The core of cloud computing is the infrastructure that consists of a network of interconnected nodes.

[0051] See now Figure 1, a schematic diagram of an example of a cloud computing node is shown. Cloud computing node 10 is only one example of a suitable cloud computing node and is not intended to impose any limitations on the scope of use or functionality of the embodiments of the present invention described herein. Regardless, cloud computing node 10 can be implemented and / or perform any of the functions set forth above.

[0052] In cloud computing node 10 there is a computer system / server 12, which may operate with numerous other general or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems or devices, and the like.

[0053] Computer system / server 12 may be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer system / server 12 may be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0054] like Figure 1 As shown, the computer system / server 12 in the cloud computing node 10 is shown in the form of a general-purpose computing device. The components of the computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples different system components including the system memory 28 to the processor 16.

[0055] Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0056] Computer system / server 12 typically includes a variety of computer system readable media. Such media can be any available media that can be accessed by computer system / server 12, and it includes volatile and nonvolatile media, removable and non-removable media.

[0057] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer system / server 12 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, and generally referred to as a "hard drive"). Although not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media may be provided. In such instances, each may be connected to the bus 18 via one or more data media interfaces. As will be further depicted and described below, the system memory 28 may include at least one program product having a collection (e.g., at least one) of program modules configured to perform the functions of an embodiment of the present invention.

[0058] A program / utility 40 having a set (at least one) of program modules 42, as well as an operating system, one or more application programs, other program modules, and program data, can be stored in system memory 28 by way of example and not limitation. Each or some combination of the operating system, one or more application programs, other program modules, and program data can include an embodiment of a networking environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0059] The computer system / server 12 may also communicate with one or more external devices 14, such as a keyboard, a pointing device, a display 24, etc.; one or more devices that enable a user to interact with the computer system / server 12; and / or any device that enables the computer system / server 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may occur via an input / output (I / O) interface 22. Furthermore, the computer system / server 12 may communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other components of the computer system / server 12 via a bus 18. It should be understood that, although not shown, other hardware and / or software components may be used in conjunction with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archiving storage systems, etc.

[0060] See now Figure 2 , depicts an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10, and local computing devices used by cloud consumers (such as personal digital assistants (PDAs) or mobile phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N) can communicate with the cloud computing nodes 10. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as the private clouds, community clouds, public clouds, or hybrid clouds described above, or a combination thereof. This allows the cloud computing environment 50 to provide infrastructure, platforms, and / or software as services without the cloud consumer having to maintain resources on local computing devices. It should be understood that Figure 2 The types of computing devices 54A-N shown in are intended to be illustrative only, and computing node 10 and cloud computing environment 50 may communicate with any type of computerized device via any type of network and / or network addressable connection (eg, using a web browser).

[0061] See now Figure 3 , showing the cloud computing environment 50 ( Figure 2 ) provides a set of functional abstraction layers. It should be understood in advance that Figure 3 The components, layers, and functions shown in are intended to be illustrative only, and embodiments of the present invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0062] The device layer 55 includes physical and / or virtual devices that are embedded with and / or independent electronic devices, sensors, actuators, and other objects to perform different tasks in the cloud computing environment 50. Each device in the device layer 55 combines networking capabilities with other functional abstraction layers so that information obtained from the device can be provided to other functional abstraction layers, and / or information from other abstraction layers can be provided to the device. In one embodiment, the different devices comprising the device layer 55 can be incorporated into a physical network collectively referred to as the "Internet of Things" (IoT). As will be understood by those of ordinary skill in the art, such a physical network allows the intercommunication, collection, and dissemination of data to achieve a variety of purposes.

[0063] The device layer 55 as shown includes sensors 52, actuators 53, a "learning" thermostat with integrated processing 56, sensors and networked electronics, cameras 57, controllable home outlets / receptacles 58, and as shown a controllable electrical switch 59. Other possible devices may include, but are not limited to, various additional sensor devices, networked devices, electronic devices (such as remote control devices), additional actuator devices, so-called "smart" appliances (such as refrigerators or washing / drying machines), and a wide variety of other possible interconnected objects.

[0064] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: host 61; server 62 based on RISC (Reduced Instruction Set Computer) architecture; server 63; blade server 64; storage device 65; and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0065] Virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 71 ; virtual storage 72 ; virtual networks 73 , including virtual private networks; virtual applications and operating systems 74 ; and virtual clients 75 .

[0066] In one instance, the management layer 80 may provide the functionality described below. Resource provisioning 81 provides dynamic acquisition of computing resources and other resources for performing tasks within a cloud computing environment. Metering and pricing 82 provides cost tracking when resources are utilized within a cloud computing environment, as well as billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection of data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management so that the required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides pre-scheduling and procurement of cloud computing resources in anticipation of future requirements for the cloud computing resources according to the SLA.

[0067] The workload layer 90 provides examples of functions that can utilize a cloud computing environment. Examples of workloads and functions that can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analysis processing 94; transaction processing 95; and, in the context of the illustrated embodiment of the present invention, different workloads and functions 96 are used for decentralized distributed deep learning. In addition, the workloads and functions 96 for decentralized distributed deep learning may include operations such as analysis, entity and obligation analysis, and as will be further described, may include user and device management functions. Those of ordinary skill in the art will recognize that the workloads and functions 96 for decentralized distributed deep learning can also work in conjunction with other parts of different abstraction layers, such as those in the hardware and software layer 60, the virtual layer 70, the management layer 80, and other workloads 90 (such as data analysis processing 94) to accomplish the different purposes of the illustrated embodiment of the present invention.

[0068] As previously described, the mechanisms of the illustrated embodiments provide novel methods for decentralized distributed deep learning in a computing environment. In one aspect, each learner may have its own input data stream and a predefined neighbor list, wherein one or more rules are used to create the neighbor list. It should be noted that a learner is a process running on a node. A learner process may have multiple threads running simultaneously. A first thread (e.g., "thread 1") and a second thread (e.g., "thread 2") may be created. The first thread may perform weight management operations. The weight management operations may include obtaining local gradients to obtain updated local weights (of a neural network "NN"). If more than one local gradient is received from the second thread, all local gradients are received. The local gradient may be set equal to zero to avoid applying the local gradient more than once, especially if the weight exchange is too fast (e.g., greater than a weight exchange threshold). The next neighbor from the list neighbor may be selected. Weights may be exchanged between the local node and the selected neighbor and weight to form a new weight vector.

[0069] The second thread may perform gradient calculations. The gradient calculation operation may continuously calculate the local gradients of each node based on each node input data (e.g., mini-batch) and its current weight (e.g., asynchronous training). The gradient calculation operation may send the calculated local gradients to the first thread. If a new weight vector from the first thread (e.g., thread 1) is available, the new weight vector may be assigned to the current weight, otherwise, the current weight is maintained. The weight vector may be updated from the local result by applying an update equation. The weight vector may also be updated to the average of the current weight and the weight from the remote neighbor. As a requirement of the operation of the present invention, weight updates are not allowed at the local learner when the weights are transmitted and averaged. In the case where the first thread (e.g., thread 1) is busy exchanging weights with other nodes, the local gradients in the first thread may be placed in its queue for the next round. The current weights used by the second thread (e.g., thread 2) always originate from the weight exchange in the first thread (e.g., thread 1).

[0070] Figure 4 4 is a functional block diagram depicting a deep learning system 400 according to some embodiments of the present invention. As shown, the deep learning system 400 includes computing systems 410A, 410B, 410C, and 410D (e.g., Figure 1 The computing system 12) and multiple networks 420A, 420B, 420C and 420D. The deep learning system 400 can be a multi-dimensional reduction system that enables improved communication efficiency within a deep learning environment.

[0071] Computing systems 410A, 410B, 410C, and 410D may be desktop computers, laptop computers, dedicated computer servers, or any other computer systems known in the art. In some embodiments, computing systems 410A, 410B, 410C, and 410D represent computer systems that utilize cluster computers and components to act as a single seamless resource pool. Typically, computing systems 410A, 410B, 410C, and 410D represent any electronic device or combination of electronic devices capable of executing machine-readable program instructions.

[0072] As depicted, each computing system 410A, 410B, 410C, and 410D includes a graphics processing unit (GPU) 412A, 412B, 412C, and 412D, respectively. Each GPU 412A, 412B, 412C, and 412D is a circuit designed to manipulate and alter memory to facilitate the creation of images for display. In at least one embodiment, GPUs 412A, 412B, 412C, and 412D are configured to perform a multidimensional reduction algorithm according to a multidimensional reduction method.

[0073] Each of networks 420A, 420B, 420C, and 420D can be, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of both, and include wired, wireless, or fiber optic connections. In general, according to embodiments of the present invention, networks 420A, 420B, 420C, and 420D can be any combination of connections and protocols that will support communications between computing systems 410A, 410B, 410C, and 410D. In at least one embodiment of the present invention, networks 420A, 420B, 420C, and 420D facilitate data transfer between computing systems 410 within deep learning system 400.

[0074] Now go to Figure 5 , a block diagram showing exemplary functional components 500 of different mechanisms according to the illustrated embodiment. In one aspect, Figures 1 to 4 One or more of the components, modules, services, applications, and / or functions described in the Figure 5 5. A decentralized distributed deep learning service 510 is shown according to various aspects of the present invention, which is combined with a processing unit ("processor") 520 to perform various computations, data processing and other functions. The decentralized distributed deep learning service 510 can be composed of Figure 1 Computer system / server 12 and / or Figure 4 The processing unit 520 may communicate with the memory 530. The decentralized distributed deep learning service 510 may include a machine learning component 540 (e.g., a deep learning component), a neighbor node component 550, and / or a weight management component 560, and a gradient calculation component 570, each of which may communicate with each other.

[0075] As one of ordinary skill in the art will appreciate, the depiction of different functional units in the decentralized distributed deep learning service 510 is for illustrative purposes, as the functional units may be located elsewhere within the decentralized distributed deep learning service 510 or within and / or between distributed computing components. In one embodiment, by way of example only, the decentralized distributed deep learning service 510 may modularly build a neural network ("NN") for a deep learning problem.

[0076] The machine learning component 540, together with the neighbor node component 550, the weight management component 560, and the gradient calculation component 570, can perform asynchronous distributed training of one or more machine learning models by generating a neighbor node list for each of the plurality of nodes, and creating a first thread for continuous communication according to the weight management operation, and creating a second thread for continuous calculation of the gradient of each node by sharing one or more variables between the first thread and the second thread in a computing environment (e.g., a heterogeneous and / or elastic computing environment). The machine learning component 540 can perform a fault detection operation to detect one or more faults of one or more of the plurality of nodes or links in the neighbor node list.

[0077] More specifically, the neighbor node component 550 can identify a list of neighbor nodes of a local node in a deep learning network. The neighbor node component 550 can generate a list of neighbor nodes of a local node.

[0078] The weight management component 560 can create a first thread and apply the gradient of the local node of the neighbor node list obtained from the second thread to 1) obtain the updated weight of the selected node; 2) set the gradient equal to zero value; 3) select a neighbor node from the neighbor node list; and / or 4) exchange weights with the local neighbor node; and average the exchanged weights to generate a weight vector. When one or more weights are transmitted and averaged to maintain data consistency, the weight management component 560 can limit the weight to be updated at the selected node in the neighbor node list.

[0079] The gradient calculation may define continuous calculation of the gradient of the second thread for each node to include: continuously determining the gradient of the selected node from a neighbor node list based on input data of the selected node, sending the determined gradient of the selected node to the first thread, and / or assigning the weight vector received from the first thread to the current weight of the selected node.

[0080] The machine learning component 540 can maintain a plurality of desired properties for each node included in the neighbor node list. The plurality of desired properties may include a connection property in which each node included in the neighbor node list is connected, a propagation property, a deadlock-free property that ensures continuous progress of each operation, and a data consistency property.

[0081] The machine learning component 540 can skip a neighbor node in a neighbor node list (for performing one or more operations) when a communication response is not received from the neighbor node before expiration of a time period.

[0082] Will Figure 6 Consider the following example shown in Figure 1-5 The realization of the aforementioned functions. Figure 6An example of a neighbor list 600 for decentralized distributed deep learning is depicted. Each learner (e.g., Figure 6 Learner 1 or "node 1" in has a list of its neighbors. That is, by way of example only, node 1 may be traversed in one direction (e.g., looking to its left) and may include one or more nodes, and may then be traversed in another direction (e.g., looking to its right) and select a few additional nodes, then return to the first direction (e.g., left) and skip a few nodes, then switch back to the alternate direction (e.g., right) and skip a few nodes. As such, node 1's neighbor list may be nodes 1-4, 8, node n-6, node n-2, node n-1, and node n. That is, node 1's neighbor list may be N(1) = (2, n, 4, n-2, 8, n-6, ..., n+2^i, -(n-2+2^i), ... until i = floor(log_2(n)).

[0083] The selected neighbor node list can have one or more required attributes, and the attributes include 1) connection attribute, 2) propagation attribute, 3) deadlock-free attribute, and 4) data consistency attribute. That is, multiple required attributes can include the connection attribute, propagation attribute, deadlock-free attribute and data consistency attribute that each node included in the neighbor node list is connected to. By skipping the nodes of the number of definitions / selections, node 1 can reach a farther distance with the step distance of the selected number. When skipping multiple nodes, reaching a further distance with the step distance of the selected number is what the "propagation" attribute refers to. It should be noted that one or more replacement operations can be used to build the neighbor list. However, the present invention can select one of the alternative operations to reach the farthest number of neighbor nodes in the step of the selected number.

[0084] use Figure 6 In the example where N(neighbors) is equal to 8, the neighbor list of node 1 is: N(1) = (2, 8, 4, 6), and the neighbor list of node 2 is: N(2) = (3, 9, 5, 7) = (3, 1, 5, 7). When N is equal to 9, node 1 and node 9 are the same. The neighbor list of node 3 is: N(3) = (4, 2, 6, 8). In addition, the general expression can be N(j) = j-1 + N(1).

[0085] Therefore, as an additional example, when j is an odd number (e.g., connection and propagation), it is considered that the neighbor list of node "j" and node "j" can be j-1+N(1). It should be noted that even numbers do not initiate communication (e.g., no deadlock). In one aspect, a deadlock situation is that node A is waiting to hear back from node B, and at the same time node B is waiting to hear back from node A, but neither node A nor node B knows the situation, so they both wait. If both node A and node B initiate communication at the same time, this may occur. Here, the present invention only allows nodes with odd indexes (or "even" indexes based on user preferences) to initiate communication to nodes with even indexes (or "odd" indexes based on user preferences). This avoids deadlock situations. When weights are transmitted and averaged to each node, weight updates are not allowed at the local learner (e.g., atomic data consistency).

[0086] Now go to Figure 7A-7B , depicting a diagram of operations 700, 755 for generating a neighbor list for decentralized distributed deep learning using 8-nodes. Decentralized distributed deep learning can be performed by Figure 1 Computer system / server 12 and / or Figure 4 and 5 The deep learning system provided by

[0087] In one aspect, neighbor lists for 8 nodes (e.g., nodes 1-8) can be generated by traversing in only one direction, resulting in neighbor list N(1)=:(2,4,8). It should be noted that even-indexed nodes (e.g., nodes 2, 4, 6, and 8) do not initiate communication, so only the neighbor lists of odd-indexed nodes communicate (e.g., synchronous communication), where N(3)=:(4,6,2), N(5)=:(6,8,4), and N(7):=(8,2,6). These neighbor lists are built from N(1) by shifting clockwise by 2 steps each time. In one aspect, steps as used herein represent operational steps for selecting neighbors to be included in this list (e.g., looking both clockwise and counterclockwise, and Figure 7A-7B (clockwise only).

[0088] Now suppose each node holds a number, and node i holds a number x i Now assume that all nodes have just completed their computations and are ready to proceed to the communication step exchanging x i For example, node 1 sends x i.given node x1., node 2 sends x1 to node 1. Node 1 calculates the average of x1 and x2, and node 2 also calculates the average of x1 and x2. At the same time, node 3 and node 4 exchange x3 and x3 and calculate the average. Node 5 and node 6 exchange x5 and x6 and calculate the average. Node 7 and node 8 exchange x7 and x8 and calculate the average, which is using the following equation:

[0089]

[0090] If all communications (e.g., receiving weights from neighbors and sending local weights to neighbors) occur simultaneously and take substantially the same amount of time, then, after each node completes, the present invention provides nodes 1 and 2 with (x1+x2) / 2, nodes 3 and 4 with (x3+x4) / 2, nodes 5 and 6 with (x5+x6) / 2, and nodes 7 and 8 with (x7+x8) / 2, which mathematically corresponds to: [x1_new1, x2_new1, x3_new1, x4_new1, x5_new1, x6_new1, x7_new1, x8_new1] = A {12，34，56，78} *[x1, x2, x3, x4, x5, x6, x7, x8], where A is a symmetric, random matrix, as shown by matrix 710 and circle 712.

[0091] In an additional step, as shown in circle 722, the neighbor items are (1, 4), (3, 6), (5, 8), (7, 2). Then, after communication between these neighbors, circle 722 shows [x 1_ new2,x 2- new2,x 3- new2,x 4_ new2,x 5_ new2,x 6_ new2,x 7_ new2,x 8_ new1], which is equal to (=) matrix 720, where A {14，36，58，72} [x 1_ new1,x 2_ new1,x 3_ new1,x 4_ new1,x 5_ new1,x 6_ new1,x 7_ new1,x 8_ new1]=A {14，36，58，72} *A {12，34，56，78} *[x1, x2, x3, x4, x5, x6, x7, x8], as shown in matrix 720.

[0092] A is depicted in matrix 730. {14，36，58，72} *A {12，34，56，78} The matrix shows that nodes 1 and 4 receive average weights of x1, x2, x3, x4. Nodes 5 and 8 receive average weights of x5, x6, x7, x8. Nodes 3 and 6 receive average weights of x3, x4, x5, x6. Nodes 7 and 2 receive average weights of x7, x8, x1, x2, where the node pairs are shown in the ellipse of the node pair list circle 732: node 1 and node 4, node 5 and node 8, node 3 and node 6, and node 7 and node 2.

[0093] Now turning to the additional step, as shown in Figure B, the neighbors can be selected as (1, 8), (3, 2), (5, 4), (7, 6). Then, after this step, all nodes have the average value of x1,,...,x8, as shown in the product of the three matrices 750, 760 and 780. That is, A {18，32，54，76} is equal to matrix 750. Matrix 760 describes A {14，36，58，72} *A [12，34，56，78} The matrix 780 depicts A {18，32，54，76} *A {14，36，58，72} *A {12，34，56，78} This means that if all communications take roughly the same time, then after 3 steps, a perfect mix of x1, ..., x8 can be achieved. If the nodes and links have different speeds, then these communications may not occur in the sync manager. A mix of nodes can produce different results. However, as long as similar restricted / regulated operations are performed to maintain connectivity, propagation, deadlock-free, and data consistency properties, similar and / or more efficient / optimized results (e.g., training results) are produced.

[0094] In one aspect, a fault detection operation may be performed to determine whether one or more of the multiple nodes or links in the neighbor node list are experiencing a fault tolerance problem. For example, a fault detection operation for node anomalies may be performed for slow nodes, faulty nodes, and also assist in node recovery.

[0095] A slow node may be a slow local node where the computation time exceeds a defined threshold or is longer than other computation times, such that fewer local gradient computations and / or weight updates are performed, but the deep learning system is still progressing.

[0096] A failed node may be a remote failed node where the weights of neighbors have not been received, and the failure detection operation will detect the failure and skip this detected failed node.

[0097] For node recovery, when a local node recovers from a failure, the recovered node may request the current weight from one of the plurality of neighbor nodes to start calculation (eg, gradient g calculation).

[0098] Similarly, fault detection operations for node anomalies can be performed for slow links, faulty links, and can also assist in link recovery.

[0099] A slow link between local node behavior and remote node behavior can include weight exchange times exceeding a defined time threshold (e.g., long exchange time), which can produce multiple prominent local gradients, but the deep learning system still proceeds.

[0100] The faulty link of the local node behavior may include that the calculation of the local node fails to contribute globally to the neighbor node list. Thus, it is necessary to repair the faulty link to achieve sharing of calculations (e.g., weights and gradients) with the neighbor node list.

[0101] A faulty link to a remote node's behavior may include a remote node's failure to receive contributions from a local node. Thus, a faulty link to a remote node may need to be repaired to enable the local node to share computations (e.g., weights and gradients) with the remote node from a neighbor node list. In one aspect, a link recovery operation may be performed. When the link is repaired, the local node continues to operate normally and the deep learning system may continue to progress.

[0102] In an additional aspect, a timeout operation for the fault detection operation can be performed when no response is received from a neighbor after a defined time period expires. Thus, the neighbor node is unable to respond and continue to use the fault detection operation to prevent the local node from permanently waiting for a faulty node / link.

[0103] Figures 8A-8B is a graph depicting operational results from decentralized distributed deep learning. In one aspect, Figure 1-7A -One or more of the components, modules, services, applications and / or functions described in -7B may be used to illustrate the operational results shown in Figure 8.

[0104] Right now, Figures 8A-8B Shows the speedup of single-node performance of different algorithms when using multiple nodes. The diagonal line represents perfect linear speedup. The algorithm of AD-PSGD performs most efficiently / accurately (e.g., best) among the other algorithms. It should be noted that the algorithms used are Stochastic Gradient Descent ("SGD"), AllReduce from High Performance Computing ("HPC") for calculating global averages, Decentralized Parallel SGD ("D-PSGD"), Elastic Asynchronous SGD ("EAMSGD"), and Asynchronous Decentralized Stochastic Gradient Descent ("AD-PSGD") algorithms.

[0105] like Fig. 8A As shown, graph 810 depicts an increase in communication speed for concentrated jobs on a fast network, and graph 810 depicts an increase in communication speed for concentrated jobs on a slow network by using communication operations (e.g., thread 1) to perform weight update operations and communications and using a computing device (e.g., thread 2) to calculate gradients.

[0106] like Figure 8B As shown, diagrams 830 and 840 depict asynchronous operations of using communication operations (e.g., thread 1) to perform weight update operations and communication and using computing devices (e.g., thread 2) to calculate gradients for slow computing process devices for assisting and locating the effects of slow learners. That is, in a synchronous scheme such as the AllReduce algorithm, fast learners must wait for the slowest learners, thus slowing down the entire training process. In the asynchronous scheme of the present invention, Figure 8B It is shown that even with a very slow learner, the training process using the illustrated embodiment of the invention does not slow down that much compared to other schemes.

[0107] in addition, Figure 8B Diagrams 850 and 860 illustrate asynchrony and decentralization that utilizes communication operations (e.g., thread 1) to help locate slow network links within a pair of learners to perform weight update operations as well as communicate and use computational devices (e.g., thread 2) to compute gradients.

[0108] Now go to Fig. 9 , depicts a method 900 for performing decentralized distributed deep learning in a computing environment by a processor, in which aspects of the illustrated embodiments may be implemented. That is, Fig. 9 900 is a flow chart of an additional example method 900 for pipelined multi-directional reduction in a computing environment according to an example of the present technology. Function 900 may be implemented as a method executed as instructions on a machine, wherein the instructions are included on at least one computer-readable medium or a non-transitory machine-readable storage medium. Function 900 may start in block 902.

[0109] As in block 904, a list of neighbor nodes may be generated for each of the plurality of nodes. As in block 906, a first thread may be created for continuous communication according to the weight management operation. As in block 908, a second thread may be created for continuous calculation of the gradient of each node. As in block 910, there are one or more variables between the first thread and the second thread. Function 900 may end, as in block 912.

[0110] In one aspect, in combination Fig. 9 At least one box and / or as Fig. 9The operation of 900 may include each of the following. Operation 900 may define a weight management operation as including (e.g., performing a weight management operation by performing each of the following): 1) applying a gradient of a node selected from the obtained neighbor node list of the second thread to obtain an updated weight of the selected node, 2) setting the gradient equal to a zero value, 3) selecting a neighbor node from the neighbor node list, and / or 4) exchanging weights with the selected neighbor node and averaging the exchanged weights to generate a weight vector.

[0111] The operation of 900 can define a continuous calculation of the gradient of the second thread for each node in the node to include (e.g., performing the continuous calculation by performing one or more of the following): 1) continuously determining the gradient of the selected node from a neighbor node list based on the input data of the selected node; 2) sending the determined gradient of the selected node to the first thread; and / or 3) assigning the weight vector received from the first thread to the current weight of the selected node.

[0112] Operation 900 can maintain multiple required attributes of each node included in the neighbor node list, wherein the multiple required attributes include connection attributes, propagation attributes, deadlock-free attributes, and data consistency attributes. That is, the multiple required attributes can include the connection attributes, propagation attributes, deadlock-free attributes for ensuring continuous progress of each operation, and data consistency attributes to which each node included in the neighbor node list is connected. The operation in 900 can detect one or more failures of one or more nodes in multiple nodes or links in the neighbor node list.

[0113] When no communication response is received from a neighbor node before a time period expires, the operation of 900 may skip a neighbor node in the neighbor node list, and / or limit weights to be updated at selected nodes in the neighbor node list when one or more weights are transmitted and averaged to maintain data consistency.

[0114] The present invention may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon, the computer-readable program instructions being used to cause a processor to execute aspects of the present invention.

[0115] A computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device (such as a punch card) or a raised structure in a groove with instructions recorded thereon), and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0116] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0117] The computer-readable program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, an electronic circuit (including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA)) may execute computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit so as to perform aspects of the present invention.

[0118] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0119] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine that, when executed by a processor of a computer or other programmable data processing apparatus, creates a device for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, a programmable data processing apparatus, and / or other device to function in a particular manner, such that a computer-readable storage medium having instructions stored therein includes an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0120] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other apparatus that causes a series of operating steps to be performed on a computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0121] The flow chart and block diagram in the accompanying drawings show the architecture, function and operation of the possible implementation of the system, method and computer program product according to different embodiments of the present invention. To this end, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the function marked in the frame may not occur in the order marked in the figure. For example, depending on the function involved, the two frames shown in succession can actually be performed substantially at the same time, or these frames can sometimes be performed in reverse order. It will also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented by a system based on special-purpose hardware, and the system based on special-purpose hardware performs a specified function or action or performs a combination of special-purpose hardware and computer instructions.

Claims

1. A method for performing decentralized distributed deep learning in a computing environment by one or more processors, comprising: performing asynchronous distributed training of one or more machine learning models by generating a list of neighbor nodes for each node of a plurality of nodes and creating a first thread for continuous communication according to a weight management operation and creating a second thread for continuous calculation of gradients for each node, wherein one or more variables are shared between the first thread and the second thread; In conjunction with performing the asynchronous distributed training, when a communication response is not received from the neighbor node before expiration of a time period, a neighbor node in the neighbor node list is skipped.

2. The method of claim 1 , further comprising defining the weight management operation to include: applying the gradient of a node selected from the obtained neighbor node list of the second thread to obtain an updated weight of the selected node; setting said gradient equal to a value of zero; Selecting a neighbor node from the neighbor node list; or The weights are exchanged with the selected neighbor nodes and the exchanged weights are averaged to generate a weighted vector.

3. The method according to claim 1, further comprising: Defining the continuous computation of the gradient of the second thread for each of the nodes to include: continuously determining a gradient of the selected node from the list of neighbor nodes based on input data of the selected node; sending the determined gradient of the selected node to the first thread; The weight vector received from the first thread is assigned to the current weight of the selected node.

4. The method according to claim 1, further comprising: A plurality of desired attributes are maintained by each node included in the neighbor node list, wherein the plurality of desired attributes include a connection attribute, a propagation attribute, a deadlock-free attribute, and a data consistency attribute.

5. The method according to claim 1, further comprising: One or more failures of one or more nodes or links in the plurality of nodes or links in the neighbor node list are detected.

6. The method of claim 1, further comprising: When one or more weights are sent and averaged to maintain data consistency, the constraint weights are updated at selected nodes in the neighbor node list.

7. A system for decentralized distributed deep learning in a computing environment, comprising: One or more computers having executable instructions that, when executed, cause the system to: performing asynchronous distributed training of one or more machine learning models by generating a list of neighbor nodes for each node of a plurality of nodes and creating a first thread for continuous communication according to a weight management operation and creating a second thread for continuous calculation of gradients for each node, wherein one or more variables are shared between the first thread and the second thread; as well as In conjunction with performing the asynchronous distributed training, when a communication response is not received from the neighbor node before expiration of a time period, a neighbor node in the neighbor node list is skipped.

8. The system according to claim 7, wherein: The executable instructions further define the weight management operation to include: applying the gradient of a node selected from the obtained neighbor node list of the second thread to obtain an updated weight of the selected node; setting said gradient equal to a value of zero; Select a neighbor node from the neighbor node list; or The weights are exchanged with the selected neighbor nodes and the exchanged weights are averaged to generate a weighted vector.

9. The system according to claim 7, wherein: The executable instructions further define the continuous calculation of the gradient of the second thread for each of the nodes to include: continuously determining a gradient of the selected node from the list of neighbor nodes based on input data of the selected node; sending the determined gradient of the selected node to the first thread; The weight vector received from the first thread is assigned to the current weight of the selected node.

10. The system according to claim 7, wherein: The executable instructions further maintain a plurality of desired attributes for each node included in the neighbor node list, wherein the plurality of desired attributes include a connectivity attribute, a propagation attribute, a deadlock-free attribute, and a data consistency attribute.

11. The system according to claim 7, wherein: The executable instructions further detect one or more failures of one or more nodes or links in the plurality of nodes or links in the neighbor node list.

12. The system according to claim 7, wherein: The executable instructions restrict weights to be updated at selected nodes in the neighbor node list when one or more weights are sent and averaged to maintain data consistency.

13. A computer program product for performing decentralized distributed deep learning by a processor in a computing environment, the computer program product comprising a non-transitory computer-readable storage medium having a computer-readable program code portion stored therein, the computer-readable program code portion comprising: an executable portion that performs asynchronous distributed training of one or more machine learning models by generating a neighbor node list for each of a plurality of nodes and creating a first thread for continuous communication according to a weight management operation and creating a second thread for continuous calculation of gradients for each node, wherein one or more variables are shared between the first thread and the second thread; And in conjunction with performing the asynchronous distributed training, when a communication response is not received from the neighbor node before expiration of a time period, skipping the neighbor node in the neighbor node list.

14. The computer program product of claim 13, further comprising an executable portion that defines the weight management operation to include: applying the gradient of a node selected from the obtained neighbor node list of the second thread to obtain an updated weight of the selected node; setting said gradient equal to a value of zero; Selecting a neighbor node from the neighbor node list; or The weights are exchanged with the selected neighbor nodes and the exchanged weights are averaged to generate a weighted vector.

15. The computer program product of claim 13, further comprising an executable portion that defines, for each of the nodes, the continuous computation of the gradient by the second thread to include: continuously determining a gradient of the selected node from the list of neighbor nodes based on input data of the selected node; sending the determined gradient of the selected node to the first thread; The weight vector received from the first thread is assigned to the current weight of the selected node.

16. The computer program product of claim 13, further comprising an executable portion for maintaining a plurality of desired attributes by each node included in the neighbor node list, wherein: The plurality of desired properties include a connection property, a propagation property, a deadlock-free property, and a data consistency property.

17. The computer program product of claim 13, further comprising an executable portion, the executable portion: One or more failures of one or more nodes or links in the plurality of nodes or links in the neighbor node list are detected.

18. The computer program product of claim 13, further comprising an executable portion that constrains weights to be updated at selected nodes in the neighbor node list when one or more weights are transmitted and averaged to maintain data consistency.

Images