Data processing method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a data processing method, device, electronic device, and storage medium, which relate to the field of artificial intelligence technology. The data processing method includes: obtaining a training sample set and determining at least two nodes for training a neural network; dividing the training sample set into multiple sample subsets according to the number of nodes, and sending each sample subset to each node respectively, and training the neural network through each node; controlling the interactive communication between each GPU through the controller in each node to update the parameters of the neural network; the controller of each node is used to perform the following steps: determining the first communication indication information and the second communication indication information according to the number of nodes and the number of GPUs; sending the first communication indication information and the second communication indication information to the GPU corresponding to the node, so that the GPU of each node communicates according to the first communication indication information and the second communication indication information. Dynamic expansion and contraction of the training equipment can be achieved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to a data processing method, device, electronic device, and storage medium. Background Art

[0002] In recent years, artificial intelligence, particularly deep learning, has achieved significant breakthroughs in areas such as image recognition and language processing, and has begun to be widely applied in commercial activities. One of the key reasons for such significant breakthroughs in deep learning is that it trains neural networks, processing a large number of samples and learning numerous features from them.

[0003] For training neural network models, a data-parallel model can be used. This involves generating multiple copies of the model and feeding samples into each copy for training. Typically, neural network models, such as deep neural networks, are trained on multiple machines, or clusters consisting of multiple nodes. Each machine can include multiple local GPUs (graphics processing units). However, in current scenarios, dynamic scaling of training devices is not supported when the size of the training cluster changes. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a data processing method, device, and electronic device. The technical solutions provided by the embodiments of the present application are as follows:

[0005] In one aspect, an embodiment of the present application provides a method for data processing, the method comprising:

[0006] Obtaining a training sample set and determining at least two nodes for training a neural network; wherein each node includes a controller and at least two graphics processing units (GPUs);

[0007] The training sample set is divided into multiple sample subsets according to the number of nodes, and each sample subset is sent to each node respectively, and the neural network is trained through each node;

[0008] The controller in each node controls the interactive communication between the GPUs to update the parameters of the neural network;

[0009] The controller of each node is used to perform the following steps:

[0010] Determine first communication indication information and second communication indication information based on the number of nodes and the number of GPUs; wherein the first communication indication information is used to instruct GPUs of different nodes to communicate with each other; and the second communication indication information is used to instruct different GPUs within the same node to communicate with each other;

[0011] The first communication instruction information and the second communication instruction information are sent to the GPU corresponding to the node, so that the GPU of each node communicates according to the first communication instruction information and the second communication instruction information.

[0012] In some possible implementations, the model parameters of the neural network include weight parameters of multiple parts;

[0013] The method also includes:

[0014] A negotiation request is sent to each node respectively; the negotiation request is used to cause each GPU of each node to generate an initial gradient respectively, and determine the target gradient to be interacted with by each GPU based on the generated initial gradients; wherein the initial gradient includes the gradient generated by each GPU for the weight parameters of each part of the neural network.

[0015] The GPU of each node communicates based on the gradient of the communication function according to the first communication indication information and the second communication indication information, including:

[0016] Each GPU of each node establishes communication based on the first communication indication information and the second communication indication information, and performs interactive transmission for the target gradient.

[0017] In some possible implementations, the method further includes:

[0018] Determine the gradient fusion method and gradient fusion threshold;

[0019] Send the gradient fusion method and gradient fusion threshold to each GPU of each node;

[0020] Determining the target gradient to be interacted by each GPU based on the generated initial gradients includes:

[0021] For each node, the initial gradient is fused according to the gradient fusion method. If the information amount of the fused gradient reaches the gradient fusion threshold, the fused gradient is used as the target gradient.

[0022] In some possible implementations, determining a gradient fusion method includes:

[0023] Constructing a static graph of the communication framework based on preset fusion rule information; wherein the fusion rule information includes information for indicating different gradients to be fused;

[0024] The gradient fusion method is determined based on the static graph of the communication framework and the initial gradients to be fused.

[0025] In some possible implementations, determining the gradient fusion threshold includes:

[0026] Determine the bandwidth utilization rate of each node for test data with different data volumes through a preset script;

[0027] The amount of test data whose bandwidth utilization meets the preset conditions is set as the gradient fusion threshold.

[0028] In some possible implementations, each GPU of each node establishes communication based on the first communication indication information and the second communication indication information, and performs interactive transmission for the target gradient, including:

[0029] Each node establishes communication between the GPUs within each node based on the second communication instruction information, and performs interactive transmission for the target gradient to obtain the first updated gradient of each GPU after the interaction;

[0030] Each node establishes communication between GPUs of different nodes based on the first communication indication information, and performs interactive transmission based on the first updated gradient to obtain the second updated gradient of each GPU after the interaction;

[0031] Each node establishes communication between the GPUs within each node based on the second communication indication information, and performs interactive transmission based on the second updated gradient to obtain the third updated gradient of each GPU after the interaction.

[0032] In some possible implementations, each GPU contains target gradients corresponding to model parameters of each part;

[0033] Perform interactive transmission for the target gradient and obtain the first updated gradient of each GPU after the interaction, including:

[0034] The GPUs in each node gradually exchange and fuse each other's target gradients until each GPU contains the corresponding first fused gradient, thereby obtaining the GPU's first updated gradient. The first fused gradient corresponding to each GPU is obtained by fusing the target gradients of each GPU corresponding to each part of the model parameters, and the first fused gradient of each GPU corresponds to a different part of the model parameters.

[0035] In some possible implementations, performing interactive transmission based on the first updated gradient to obtain the second updated gradient of each GPU after the interaction includes:

[0036] The GPUs of different nodes gradually exchange each other's first updated gradients and fuse them until all interacting GPUs contain the same second fused gradient, thus obtaining the second updated gradients of each GPU after interaction. The second fused gradients are obtained by fusing the first updated gradients corresponding to each interacting GPU.

[0037] In some possible implementations, performing interactive transmission based on the second updated gradient to obtain the third updated gradient of each GPU after the interaction includes:

[0038] The GPUs in each node gradually exchange their second updated gradients until each GPU contains a third fused gradient, thereby obtaining a third updated gradient for each GPU. The third fused gradient is obtained by fusing the second updated gradients of the GPUs in each node.

[0039] On the other hand, an embodiment of the present application provides a data processing device, the device comprising:

[0040] An acquisition module is used to acquire a training sample set and determine at least two nodes for training a neural network; wherein each node includes a controller and at least two graphics processing units (GPUs);

[0041] The sending module is used to divide the training sample set into multiple sample subsets according to the number of nodes, and send each sample subset to each node respectively, so as to train the neural network through each node;

[0042] The communication module is used to control the interactive communication between the GPUs through the controllers in each node to update the parameters of the neural network;

[0043] The controller of each node is used to perform the following steps:

[0044] Determine first communication indication information and second communication indication information based on the number of nodes and the number of GPUs; wherein the first communication indication information is used to instruct GPUs of different nodes to communicate with each other; and the second communication indication information is used to instruct different GPUs within the same node to communicate with each other;

[0045] The first communication instruction information and the second communication instruction information are sent to the GPU corresponding to the node, so that the GPU of each node communicates according to the first communication instruction information and the second communication instruction information.

[0046] In some possible implementations, the model parameters of the neural network include weight parameters of multiple parts;

[0047] The apparatus further includes a negotiation module, configured to:

[0048] A negotiation request is sent to each node respectively; the negotiation request is used to cause each GPU of each node to generate an initial gradient respectively, and determine the target gradient to be interacted with by each GPU based on the generated initial gradients; wherein the initial gradient includes the gradient generated by each GPU for the weight parameters of each part of the neural network.

[0049] The GPU of each node communicates based on the gradient of the communication function according to the first communication indication information and the second communication indication information, including:

[0050] Each GPU of each node establishes communication based on the first communication indication information and the second communication indication information, and performs interactive transmission for the target gradient.

[0051] In some possible implementations, the apparatus further includes a determining module configured to:

[0052] Determine the gradient fusion method and gradient fusion threshold;

[0053] Send the gradient fusion method and gradient fusion threshold to each GPU of each node;

[0054] When determining the target gradient to be interacted with for each GPU based on the generated initial gradients, the negotiation module is specifically configured to:

[0055] For each node, the initial gradient is fused according to the gradient fusion method. If the information amount of the fused gradient reaches the gradient fusion threshold, the fused gradient is used as the target gradient.

[0056] In some possible implementations, when determining the gradient fusion mode, the determination module is specifically configured to:

[0057] Constructing a static graph of the communication framework based on preset fusion rule information; wherein the fusion rule information includes information for indicating different gradients to be fused;

[0058] The gradient fusion method is determined based on the static graph of the communication framework and the initial gradients to be fused.

[0059] In some possible implementations, when determining the gradient fusion threshold, the determination module is specifically configured to:

[0060] Determine the bandwidth utilization rate of each node for test data with different data volumes through a preset script;

[0061] The amount of test data whose bandwidth utilization meets the preset conditions is set as the gradient fusion threshold.

[0062] In some possible implementations, each GPU of each node establishes communication based on the first communication indication information and the second communication indication information, and performs interactive transmission for the target gradient, including:

[0063] Each node establishes communication between the GPUs within each node based on the second communication instruction information, and performs interactive transmission for the target gradient to obtain the first updated gradient of each GPU after the interaction;

[0064] Each node establishes communication between GPUs of different nodes based on the first communication indication information, and performs interactive transmission based on the first updated gradient to obtain the second updated gradient of each GPU after the interaction;

[0065] Each node establishes communication between the GPUs within each node based on the second communication indication information, and performs interactive transmission based on the second updated gradient to obtain the third updated gradient of each GPU after the interaction.

[0066] In some possible implementations, each GPU contains target gradients corresponding to model parameters of each part;

[0067] Perform interactive transmission for the target gradient and obtain the first updated gradient of each GPU after the interaction, including:

[0068] The GPUs in each node gradually exchange and fuse each other's target gradients until each GPU contains the corresponding first fused gradient, thereby obtaining the GPU's first updated gradient. The first fused gradient corresponding to each GPU is obtained by fusing the target gradients of each GPU corresponding to each part of the model parameters, and the first fused gradient of each GPU corresponds to a different part of the model parameters.

[0069] In some possible implementations, performing interactive transmission based on the first updated gradient to obtain the second updated gradient of each GPU after the interaction includes:

[0070] The GPUs of different nodes gradually exchange each other's first updated gradients and fuse them until all interacting GPUs contain the same second fused gradient, thus obtaining the second updated gradients of each GPU after interaction. The second fused gradients are obtained by fusing the first updated gradients corresponding to each interacting GPU.

[0071] In some possible implementations, performing interactive transmission based on the second updated gradient to obtain the third updated gradient of each GPU after the interaction includes:

[0072] The GPUs in each node gradually exchange their second updated gradients until each GPU contains a third fused gradient, thereby obtaining a third updated gradient for each GPU. The third fused gradient is obtained by fusing the second updated gradients of the GPUs in each node.

[0073] On the other hand, an embodiment of the present application further provides an electronic device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the computer program to implement the method provided in any optional embodiment of the present application.

[0074] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method provided in any optional embodiment of the present application.

[0075] On the other hand, an embodiment of the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the method provided in any optional embodiment of the present application.

[0076] The beneficial effects of the technical solution provided by the embodiments of the present application are as follows:

[0077] The controller maintained by each node determines the first communication indication information and the second communication indication information according to the number of nodes and the number of GPUs, and sends the first communication indication information and the second communication indication information to the GPU corresponding to the node, so that the GPU of each node communicates according to the first communication indication information and the second communication indication information. When the number of nodes or GPUs changes, the first communication indication information and the second communication indication information can be re-determined at the beginning of the next iteration, thereby automatically increasing or decreasing the number of nodes during training. When resources are limited, the GPU can be used to start training on a small number of processes first; when resources are abundant, the capacity can be automatically expanded dynamically, and the expanded GPU can be used to train on the process at the beginning of the next iteration, thereby realizing dynamic expansion and contraction of the training equipment.

[0078] When a node fails, the controllers of the remaining available nodes can reconstruct new first communication indication information and second communication indication information, so as to continue training without interrupting the training task.

[0079] In addition, the fusion method is determined by the static graph of the communication framework. Since the communication framework itself is part of the static graph, the fusion efficiency can be effectively improved compared to the dynamic gradient fusion process. By fusing the initial gradients and then reaching the gradient fusion threshold before performing target gradient communication, communication at a larger granularity can be achieved, thereby improving communication efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0081] Figure 1 A schematic diagram of an application environment of a data processing method provided in an example;

[0082] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application;

[0083] Figure 3This is a schematic diagram of a solution for performing gradient negotiation among nodes in an example of this application;

[0084] Figure 4 A schematic diagram of a solution for determining a gradient fusion threshold in an example of this application;

[0085] Figure 5 This is a schematic diagram of communication between GPUs within a node in an example of this application;

[0086] Figure 6 A schematic diagram of the gradients of each GPU when communicating between GPUs within a node in an example of this application;

[0087] Figure 7 A schematic diagram of the gradients of each GPU when communicating between GPUs within a node in an example of this application;

[0088] Figure 8 A schematic diagram of the gradients of each GPU when communicating between GPUs within a node in an example of this application;

[0089] Figure 9 A schematic diagram of the gradients of each GPU when communicating between GPUs within a node in an example of this application;

[0090] Figure 10 A schematic diagram of the gradients of each GPU when communicating between GPUs within a node in an example of this application;

[0091] Figure 11 A schematic diagram of GPU communication between nodes in an example of this application;

[0092] Figure 12 This is a schematic diagram of communication between GPUs within a node in an example of this application;

[0093] Figure 13 A schematic diagram of a communication scheme between GPUs of various nodes in an example of this application;

[0094] Figure 14 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0095] Figure 15 A schematic structural diagram of an electronic device applicable to an embodiment of the present application. DETAILED DESCRIPTION

[0096] The following describes the embodiments of the present application in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0097] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a", "an", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements, and / or components, but do not exclude implementation as other features, information, data, steps, operations, elements, components, and / or combinations thereof supported by the present technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can refer to the element and the other element establishing a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here can include wireless connection or wireless coupling. The term "and / or" used here indicates at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B". When describing multiple (two or more) items, if the relationship between the multiple items is not clearly defined, the multiple items may refer to one, multiple or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" can be implemented as parameter A including A1 or A2 or A3, and can also be implemented as parameter A including at least two of the three items A1, A2, and A3.

[0098] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0099] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0100] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0101] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0102] Deep learning often requires multi-GPU parallel training. One issue to consider with multi-GPU parallel training is inter-GPU communication performance, as gradient reduction is required between GPUs in data parallel mode. Data parallelism (DP) is the most widely used parallelization strategy. Its characteristics are as follows: the model is replicated across multiple nodes (also called machines or workers), and each GPU maintains a complete copy of the model. The input dataset can be partitioned across multiple GPUs, and each batch of input training data is divided among the data-parallel nodes. Each node processes a subset of the data. Weights are regularly synchronized with other GPUs using collective communication primitives or parameter servers. After backpropagation, gradients must be communicated and reduced to ensure that the optimizer performs the same updates on all nodes. In other words, weight updates calculated by individual nodes are aggregated to obtain a final weight update that reflects all input updates.

[0103] The following describes several optional embodiments to illustrate the technical solutions provided by this application and the technical effects produced by the technical solutions of this application. It should be noted that the following embodiments can refer to, draw on, or combine with each other, and the same terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0104] The data processing method provided in the embodiment of the present application can be executed by any computer device, and optionally, can be executed by a server, wherein the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0105] Figure 1 Schematic diagram of the application environment of the data processing method provided in the embodiment of the present application. The application environment may include a control server 101 and a working server 102, wherein the working server 102 may include multiple machines, i.e., multiple nodes 102a, and each node 102a maintains a controller 102b and at least two GPUs. Specifically, the control server 101 obtains a training sample set and determines at least two nodes for training a neural network; wherein each node includes a controller and at least two graphics processing units (GPUs); the training sample set is divided into multiple sample subsets according to the number of nodes, and each sample subset is sent to each node respectively, and the neural network is trained through each node; the control server 101 controls the interactive communication between each GPU through the controller in each node to update the parameters of the neural network. The controller 102b of each node is used to perform the following steps: determine the first communication indication information and the second communication indication information based on the number of nodes and the number of GPUs; wherein the first communication indication information is used to instruct the GPUs of different nodes to communicate; the second communication indication information is used to instruct the different GPUs within the same node to communicate; the first communication indication information and the second communication indication information are sent to the GPUs corresponding to the nodes, so that the GPUs of each node communicate according to the first communication indication information and the second communication indication information.

[0106] Technical personnel in this technical field can understand that a server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0107] In some possible implementations, a data processing method is provided, which can be executed by a control server.

[0108] Figure 2 The following is a flow chart of a data processing method provided by an embodiment of the present application. Taking the execution subject as a control server as an example, the data processing method provided by the present application may include the following steps:

[0109] Step S201: Obtain a training sample set and determine at least two nodes for training a neural network.

[0110] Each node includes a controller and at least two graphics processing units (GPUs).

[0111] Specifically, each node can maintain a controller, which can be a Gloo controller. Gloo is a Kubernetes-native ingress controller based on Envoy; Envoy is an open source edge and service proxy for cloud-native applications; Kubernetes is an open source framework for managing containerized applications across multiple hosts in a cloud platform.

[0112] Step S202: Divide the training sample set into multiple sample subsets according to the number of nodes, and send each sample subset to each node respectively, so as to train the neural network through each node.

[0113] Specifically, multiple copies of the neural network model are generated for the neural network, each GPU maintains a complete copy of the model, the training data can be partitioned across multiple GPUs, and each batch of input training data is divided among data-parallel nodes, with each node processing the corresponding sample subset.

[0114] Step S203: Control the GPUs to communicate with each other through the controllers in each node to update the parameters of the neural network.

[0115] Specifically, each GPU processes the weight parameters of multiple parts of the neural network to obtain the gradient for each weight parameter. Then, the GPUs communicate with each other regarding the gradient of each weight parameter. After communication, the gradients are synchronized between the GPUs, thereby updating the parameters of the neural network.

[0116] The controller of each node is used to perform the following steps:

[0117] (1) Determine the first communication indication information and the second communication indication information based on the number of nodes and the number of GPUs.

[0118] The first communication indication information is used to instruct GPUs of different nodes to communicate with each other; and the second communication indication information is used to instruct different GPUs within the same node to communicate with each other.

[0119] Specifically, the first communication instruction may be in the form of grouping GPUs of different nodes by GPU identifiers, and the second communication instruction may be in the form of grouping GPUs within the same node by GPU identifiers.

[0120] For example, if node 1 includes GPU1, GPU2, GPU3, and GPU4, and node 2 includes GPU5, GPU6, GPU7, and GPU8, the first communication indication may be {1,5}, {2,6}, etc., and the second communication indication may be {1,2,3,4} and {5,6,7,8}.

[0121] (2) The first communication instruction information and the second communication instruction information are sent to the GPU corresponding to the node, so that the GPU of each node communicates according to the first communication instruction information and the second communication instruction information.

[0122] In a specific implementation process, a controller is maintained inside each node, and each controller sends the first communication indication information and the second communication indication information to the GPU inside the node where it is located, and the controllers between the nodes can communicate with each other.

[0123] Specifically, GPUs in different nodes may communicate with each other through the first communication indication information, and different GPUs in the same node may communicate with each other through the second communication indication information.

[0124] In the above embodiment, the controller maintained by each node determines the first communication indication information and the second communication indication information based on the number of nodes and the number of GPUs, and sends the first communication indication information and the second communication indication information to the GPU corresponding to the node, so that the GPU of each node communicates according to the first communication indication information and the second communication indication information. When the number of nodes or GPUs changes, the first communication indication information and the second communication indication information can be re-determined at the beginning of the next iteration, thereby automatically increasing or decreasing the number of nodes during the training process. When resources are limited, the GPU can be used to start training on a small number of processes first; when resources are abundant, the capacity can be automatically expanded dynamically, and the expanded GPU can be used to perform training on the process at the beginning of the next iteration, thereby realizing dynamic expansion and contraction of the training equipment.

[0125] When a node fails, the controllers of the remaining available nodes can reconstruct new first communication indication information and second communication indication information, so as to continue training without interrupting the training task.

[0126] In some possible implementations, the model parameters of the neural network include weight parameters of multiple parts.

[0127] The data processing method may further include:

[0128] Send negotiation requests to each node separately.

[0129] The negotiation request is used to enable each GPU of each node to generate an initial gradient respectively, and determine the target gradient to be interacted by each GPU based on the generated initial gradients; the initial gradients include the gradients generated by each GPU for the weight parameters of each part of the neural network.

[0130] Specifically, a coordinator can be set up in the control server, and the coordinator sends a negotiation request to each node. After each node receives the negotiation request, if an initial gradient is generated for the same weight parameter of the neural network, it means that the negotiation is completed. The neural network includes weight parameters of multiple parts, and each GPU can generate multiple initial gradients for the weight parameters of multiple parts accordingly.

[0131] In the specific implementation process, if the negotiation between the nodes is completed, each GPU determines the target gradient with interaction based on the initial extraction, that is, determines which gradients can be used for transmission.

[0132] Specifically, the initial gradients in each GPU can be fused to obtain the target gradient. The specific process of obtaining the target gradient will be further explained in detail below.

[0133] like Figure 3 As shown, the coordinator sends a negotiation request to each training process, that is, each node. If the GPUs in each node generate initial gradients for each weight parameter in response to the negotiation request, the negotiation between the nodes is completed. Then, the target gradients to be interacted by each GPU are generated based on the initial gradients, and communication is carried out based on the target gradients.

[0134] In some possible implementations, the GPUs of each node communicate based on the first communication indication information and the second communication indication information by performing gradient communication based on the communication function, which may include:

[0135] Each GPU of each node establishes communication based on the first communication indication information and the second communication indication information, and performs interactive transmission for the target gradient.

[0136] Specifically, communication between GPUs of different nodes can be established according to the first communication indication information, and communication between different GPUs within each node can be established according to the second communication indication. Then, interactive transmission can be performed for the target gradient. The specific process of communication based on the target gradient will be further elaborated below.

[0137] Specifically, each GPU can communicate based on the NCCL communication function, where the NCCL communication function is a collective communication library that can implement collective communication and point-to-point communication, focusing on communication between GPUs.

[0138] The process of determining the target gradient will be further described below with reference to embodiments.

[0139] In some possible implementations, the data processing method of the present application may further include:

[0140] (1) Determine the gradient fusion method and gradient fusion threshold.

[0141] The gradient fusion threshold is used to indicate that if the fusion between the initial gradients reaches the set threshold, the fused gradient can be transmitted.

[0142] Among them, the gradient fusion method can be used to indicate which gradients can be fused.

[0143] In some possible implementations, determining the gradient fusion threshold may include:

[0144] a. Determine the bandwidth utilization rate of each node for test data with different data volumes using a preset script;

[0145] b. The amount of test data whose bandwidth utilization meets the preset conditions is set as the gradient fusion threshold.

[0146] Specifically, test data may be used to test the transmission between nodes, and a suitable gradient fusion threshold may be determined through different bandwidth utilizations.

[0147] The preset condition may be that the bandwidth utilization is greater than a preset threshold.

[0148] In some possible implementations, determining the gradient fusion method may include:

[0149] Constructing a static graph of the communication framework based on preset fusion rule information; wherein the fusion rule information includes information for indicating different gradients to be fused;

[0150] The gradient fusion method is determined based on the static graph of the communication framework and the initial gradients to be fused.

[0151] The communication framework exists as an operation within the TensorFlow (TensorFlow symbolic math system) static graph. A static graph is a program that generates the neural network structure during compilation and execution, and then performs the corresponding operations. Static computation is defined first and then executed. Subsequent runs do not require rebuilding the computational graph, resulting in faster speed compared to dynamic graphs.

[0152] During communication, a for loop iterates over all gradients, counting the number of generated gradient elements. The order in which gradients are generated is determined by TensorFlow itself, and communication occurs when the gradient fusion threshold is exceeded. Because the communication framework is part of the TensorFlow static graph, it is naturally more efficient than dynamic gradient fusion.

[0153] (2) Send the gradient fusion method and gradient fusion threshold to each GPU of each node.

[0154] Specifically, the gradient fusion mode and the gradient fusion threshold may be sent to the controller of each node respectively, and then distributed by the controller of each node to each GPU in the respective node.

[0155] In some possible implementations, determining a target gradient for each GPU to interact with based on the generated initial gradients may include:

[0156] For each node, the initial gradient is fused according to the gradient fusion method. If the information amount of the fused gradient reaches the gradient fusion threshold, the fused gradient is used as the target gradient.

[0157] After gradient negotiation, all nodes know the initial gradient that can be communicated at the current moment. Single communication of a single gradient is relatively inefficient. Based on the gradient fusion method, multiple initial gradients can be fused, and then the fused target gradient can be communicated at a larger granularity. That is, gradient communication can be performed when the amount of information in the fused gradient reaches the gradient fusion threshold.

[0158] like Figure 4 As shown, in one example, given the communication parameters and traffic model, the starting size of the data packet, that is, the data volume of the given test data, the real-time bandwidth utilization can be output through a preset script. For example, the autotune script is started by mpirun to output the bandwidth utilization in real time. Among them, mpirun is a fast execution command of the MPI (a cross-language communication protocol) program, and autotune is an automatic tuning script. A gradient fusion threshold is determined according to the bandwidth utilization. In the reverse calculation process, the data is first accumulated to the fusion threshold, that is, the initial gradient is fused according to the fusion method. If the amount of fused data is greater than the gradient fusion threshold, that is, gradient fusion communication 1, gradient fusion communication 2 and gradient fusion communication 3 shown in the figure, the fused gradient is used as the target gradient for communication.

[0159] In the above embodiment, the fusion mode is determined by the static graph of the communication framework. Since the communication framework itself is part of the Tensorflow static graph, the fusion efficiency can be effectively improved compared to the dynamic gradient fusion process. By fusing the initial gradients and then reaching the gradient fusion threshold before performing target gradient communication, communication at a larger granularity can be achieved, thereby improving communication efficiency.

[0160] The above embodiment describes the specific process of determining the target gradient. The following will further describe the process of communication between GPUs based on the target gradient in conjunction with specific implementation methods.

[0161] In some possible implementations, each GPU of each node establishes communication based on the first communication indication information and the second communication indication information, and performs interactive transmission for the target gradient, which may include:

[0162] (1) Each node establishes communication between the GPUs within each node based on the second communication indication information, and performs interactive transmission for the target gradient to obtain the first updated gradient of each GPU after the interaction.

[0163] like Figure 5 As shown, the target gradients between GPU0-GPU4 in node 1 are interactively transmitted, and the target gradients between GPU0-GPU4 in node 2 are interactively transmitted.

[0164] Specifically, each GPU contains the target gradient corresponding to the model parameters of each part.

[0165] Perform interactive transmission for the target gradient and obtain the first updated gradient of each GPU after the interaction, including:

[0166] The GPUs in each node gradually exchange and fuse each other's target gradients until each GPU contains the corresponding first fused gradient, thereby obtaining the GPU's first updated gradient. The first fused gradient corresponding to each GPU is obtained by fusing the target gradients of each GPU corresponding to each part of the model parameters, and the first fused gradient of each GPU corresponds to a different part of the model parameters.

[0167] In the specific implementation process, a scatter-reduce operation is performed on each GPU in each node. That is, in each round of gradient transfer, each GPU sends one of its own gradients to its right neighbor and receives the gradients sent by its left neighbor and accumulates them.

[0168] like Figure 6 As shown in the figure, the communication of GPUs in a node is explained. The node includes five nodes, GPU0-GPU4, and each node contains five gradients. The target gradient a0 of GPU0 is passed to the target gradient a1 of GPU1, the target gradient b1 of GPU1 is passed to the target gradient b2 of GPU2, the target gradient c2 of GPU2 is passed to the target gradient c3 of GPU3, the target gradient d3 of GPU3 is passed to the target gradient d4 of GPU4, and the target gradient e4 of GPU4 is passed to the target gradient e0 of GPU0. Figure 7 The target gradient of each GPU is transferred in turn. The target gradient e0+e4 of GPU0 is transferred to the target gradient e1 of GPU1. The target gradient a1+a0 of GPU1 is transferred to the target gradient a2 of GPU2. The target gradient b2+b1 of GPU2 is transferred to the target gradient b3 of GPU3. The target gradient c3+c2 of GPU3 is transferred to the target gradient c4 of GPU4. The target gradient d4+d3 of GPU4 is transferred to the target gradient d0 of GPU0. Figure 8The gradients of each GPU shown are then transferred in sequence to obtain Figure 9 The gradients of each GPU shown are passed on until we get Figure 10 Each GPU contains a corresponding first fused gradient, that is, the first fused gradient contained in GPU0 is b2+b1+b3+b4+b0, the first fused gradient contained in GPU1 is c3+c2+c4+c0+c1, the first fused gradient contained in GPU2 is d4+d3+d0+d1+d2, the first fused gradient contained in GPU3 is e0+e4+e1+e2+e3, and the first fused gradient contained in GPU4 is a1+a0+a2+a3+a4.

[0169] (2) Each node establishes communication between GPUs of different nodes based on the first communication indication information, and performs interactive transmission based on the first updated gradient to obtain the second updated gradient of each GPU after the interaction.

[0170] Specifically, performing interactive transmission based on the first updated gradient to obtain the second updated gradient of each GPU after the interaction may include:

[0171] The GPUs of different nodes gradually exchange each other's gradients, update the gradients, and fuse them until all interacting GPUs contain the same second fused gradient, thus obtaining the second updated gradients of each GPU after interaction.

[0172] The second fused gradient is obtained by fusing the first update gradients corresponding to the interacting GPUs. That is, the second update gradients ultimately obtained by the interacting GPUs of different node pairs are the same, which are all second fused gradients obtained by fusing the first update gradients.

[0173] like Figure 11 As shown, each GPU on each node splits the data into two parts. This allows all GPUs to simultaneously receive and send gradients across nodes, improving bandwidth utilization. All-reduce operations can be used between nodes. For example, all-reduce operations can be performed between GPU0 on node 1 and GPU0 on node 2, between GPU1 on node 1 and GPU1 on node 2, between GPU3 on node 1 and GPU3 on node 2, and between GPU4 on node 1 and GPU4 on node 2.

[0174] Taking GPU0 of node 1 and GPU0 of node 2 as an example, if the gradients contained in GPU0 of node 1 are a0, b0, c0, d0, and the gradients contained in GPU0 of node 2 are A0, B0, C0, D0, then after the all-reduce operation, GPU0 of both node 1 and node 2 contain a0+A0, b0+B0, c0+C0, d0+D0.

[0175] (3) Each node establishes communication between the GPUs within each node based on the second communication indication information, and performs interactive transmission based on the second updated gradient to obtain the third updated gradient of each GPU after the interaction.

[0176] Specifically, performing interactive transmission based on the second updated gradient to obtain the third updated gradient of each GPU after the interaction may include:

[0177] The GPUs in each node gradually exchange the second updated gradients with each other until each GPU contains the third fused gradient, thereby obtaining the third updated gradient of each GPU.

[0178] The third fused gradient is obtained by fusing the second updated gradients of each GPU in each node. That is, the third updated gradients finally obtained by different GPUs in the same node are the same, which are all the third fused gradients obtained by fusing the second updated gradients.

[0179] like Figure 12 As shown in the figure, the target gradients between GPU0 and GPU4 in node 1 are exchanged and transmitted, and the target gradients between GPU0 and GPU4 in node 2 are exchanged and transmitted. An all-gather operation is performed on each GPU in each node. That is, in each round of gradient transmission, each GPU sends one of its own gradients to its right neighbor and receives and replaces the gradients sent by its left neighbor.

[0180] After executing the intra-node scatter-reduce operation, the inter-node all-reduce operation, and the intra-node all-gather operation in sequence, the third fused gradients between each GPU of each communicating node are the same and contain all the target gradients in each GPU of each node before communication.

[0181] In the above embodiment, by sequentially executing the intra-node scatter-reduce operation, the inter-node all-reduce operation, and the intra-node all-gather operation, during the inter-node all-reduce operation, all GPUs can simultaneously receive and send gradients across nodes, thereby improving bandwidth utilization.

[0182] Afterwards, the third fused gradients between each GPU of each node in communication are the same and include all target gradients in each GPU of each node before communication.

[0183] In order to more clearly illustrate the data processing method of the present application, further detailed description will be given below with reference to examples.

[0184] like Figure 13 As shown, in one example, the data processing method of the present application may include the following steps:

[0185] The control server obtains a training sample set and determines at least two nodes for training a neural network; wherein each node includes a controller and at least two graphics processing units (GPUs);

[0186] The control server divides the training sample set into multiple sample subsets according to the number of nodes, and sends each sample subset to each node to train the neural network through each node;

[0187] The background process of each node is started, and each node maintains a Gloo controller;

[0188] The Gloo controller detects the startup script and determines the first communication instruction information and the second communication instruction information; wherein the first communication instruction information is used to instruct the GPUs of different nodes to communicate; the second communication instruction information is used to instruct the GPUs within the same node to communicate;

[0189] The control server determines the bandwidth utilization rate of each node for test data of different data volumes through a preset script; the data volume of the test data whose bandwidth utilization rate meets the preset conditions is set as the gradient fusion threshold;

[0190] The control server constructs a static graph of the communication framework based on preset fusion rule information; wherein the fusion rule information includes information for indicating different gradients to be fused; and determines a gradient fusion method based on the static graph of the communication framework and each initial gradient to be fused;

[0191] The coordinator in the control server sends a negotiation request to each node. Each node negotiates based on the gradient fusion method and gradient fusion threshold to determine the target gradient to be transmitted in this communication.

[0192] Gradient communication between GPUs, including:

[0193] Each node establishes communication internally through the second communication indication information and performs a scatter-reduce operation to obtain the first updated gradient of each GPU after the interaction;

[0194] The nodes establish communication with each other through the first communication indication information, perform an all-reduce operation based on the first updated gradient, and obtain the second updated gradient of each GPU after the interaction;

[0195] Each node establishes communication internally through the second communication indication information, performs an all-gather operation based on the second updated gradient, and obtains the third updated gradient of each GPU after the interaction.

[0196] The above-mentioned data processing method determines the first communication indication information and the second communication indication information according to the number of nodes and the number of GPUs through the controller maintained by each node, and sends the first communication indication information and the second communication indication information to the GPU corresponding to the node, so that the GPU of each node communicates according to the first communication indication information and the second communication indication information. When the number of nodes or GPUs changes, the first communication indication information and the second communication indication information can be re-determined at the beginning of the next iteration, thereby automatically increasing or decreasing the number of nodes during training. When resources are limited, the GPU can be used to start training on a small number of processes first; when resources are abundant, the capacity can be automatically expanded dynamically, and the expanded GPU can be used to train on the process at the beginning of the next iteration, thereby realizing dynamic expansion and contraction of the training equipment.

[0197] When a node fails, the controllers of the remaining available nodes can reconstruct new first communication indication information and second communication indication information, so as to continue training without interrupting the training task.

[0198] In addition, the fusion method is determined by the static graph of the communication framework. Since the communication framework itself is part of the static graph, the fusion efficiency can be effectively improved compared to the dynamic gradient fusion process. By fusing the initial gradients and then reaching the gradient fusion threshold before performing target gradient communication, communication at a larger granularity can be achieved, thereby improving communication efficiency.

[0199] like Figure 14 As shown, in some possible implementations, a data processing device is provided, including:

[0200] An acquisition module 1401 is configured to acquire a training sample set and determine at least two nodes for training a neural network; wherein each node includes a controller and at least two graphics processing units (GPUs);

[0201] The sending module 1402 is used to divide the training sample set into multiple sample subsets according to the number of nodes, and send each sample subset to each node respectively, so as to train the neural network through each node;

[0202] Communication module 1403, used to control the interactive communication between the GPUs through the controllers in each node to update the parameters of the neural network;

[0203] The controller of each node is used to perform the following steps:

[0204] Determine first communication indication information and second communication indication information based on the number of nodes and the number of GPUs; wherein the first communication indication information is used to instruct GPUs of different nodes to communicate with each other; and the second communication indication information is used to instruct different GPUs within the same node to communicate with each other;

[0205] The first communication instruction information and the second communication instruction information are sent to the GPU corresponding to the node, so that the GPU of each node communicates according to the first communication instruction information and the second communication instruction information.

[0206] In some possible implementations, the model parameters of the neural network include weight parameters of multiple parts;

[0207] The apparatus further includes a negotiation module, configured to:

[0208] A negotiation request is sent to each node respectively; the negotiation request is used to cause each GPU of each node to generate an initial gradient respectively, and determine the target gradient to be interacted with by each GPU based on the generated initial gradients; wherein the initial gradient includes the gradient generated by each GPU for the weight parameters of each part of the neural network.

[0209] The GPU of each node communicates based on the gradient of the communication function according to the first communication indication information and the second communication indication information, including:

[0210] Each GPU of each node establishes communication based on the first communication indication information and the second communication indication information, and performs interactive transmission for the target gradient.

[0211] In some possible implementations, the apparatus further includes a determining module configured to:

[0212] Determine the gradient fusion method and gradient fusion threshold;

[0213] Send the gradient fusion method and gradient fusion threshold to each GPU of each node;

[0214] When determining the target gradient to be interacted with for each GPU based on the generated initial gradients, the negotiation module is specifically configured to:

[0215] For each node, the initial gradient is fused according to the gradient fusion method. If the information amount of the fused gradient reaches the gradient fusion threshold, the fused gradient is used as the target gradient.

[0216] In some possible implementations, when determining the gradient fusion mode, the determination module is specifically configured to:

[0217] Constructing a static graph of the communication framework based on preset fusion rule information; wherein the fusion rule information includes information for indicating different gradients to be fused;

[0218] The gradient fusion method is determined based on the static graph of the communication framework and the initial gradients to be fused.

[0219] In some possible implementations, when determining the gradient fusion threshold, the determination module is specifically configured to:

[0220] Determine the bandwidth utilization rate of each node for test data with different data volumes through a preset script;

[0221] The amount of test data whose bandwidth utilization meets the preset conditions is set as the gradient fusion threshold.

[0222] In some possible implementations, each GPU of each node establishes communication based on the first communication indication information and the second communication indication information, and performs interactive transmission for the target gradient, including:

[0223] Each node establishes communication between the GPUs within each node based on the second communication instruction information, and performs interactive transmission for the target gradient to obtain the first updated gradient of each GPU after the interaction;

[0224] Each node establishes communication between GPUs of different nodes based on the first communication indication information, and performs interactive transmission based on the first updated gradient to obtain the second updated gradient of each GPU after the interaction;

[0225] Each node establishes communication between the GPUs within each node based on the second communication indication information, and performs interactive transmission based on the second updated gradient to obtain the third updated gradient of each GPU after the interaction.

[0226] In some possible implementations, each GPU contains target gradients corresponding to model parameters of each part;

[0227] Perform interactive transmission for the target gradient and obtain the first updated gradient of each GPU after the interaction, including:

[0228] The GPUs in each node gradually exchange and fuse each other's target gradients until each GPU contains the corresponding first fused gradient, thereby obtaining the GPU's first updated gradient. The first fused gradient corresponding to each GPU is obtained by fusing the target gradients of each GPU corresponding to each part of the model parameters, and the first fused gradient of each GPU corresponds to a different part of the model parameters.

[0229] In some possible implementations, performing interactive transmission based on the first updated gradient to obtain the second updated gradient of each GPU after the interaction includes:

[0230] The GPUs of different nodes gradually exchange each other's first updated gradients and fuse them until all interacting GPUs contain the same second fused gradient, thus obtaining the second updated gradients of each GPU after interaction. The second fused gradients are obtained by fusing the first updated gradients corresponding to each interacting GPU.

[0231] In some possible implementations, performing interactive transmission based on the second updated gradient to obtain the third updated gradient of each GPU after the interaction includes:

[0232] The GPUs in each node gradually exchange their second updated gradients until each GPU contains a third fused gradient, thereby obtaining a third updated gradient for each GPU. The third fused gradient is obtained by fusing the second updated gradients of the GPUs in each node.

[0233] The data processing device of the present application determines the first communication indication information and the second communication indication information according to the number of nodes and the number of GPUs through the controller maintained by each node, and sends the first communication indication information and the second communication indication information to the GPU corresponding to the node, so that the GPU of each node communicates according to the first communication indication information and the second communication indication information. When the number of nodes or GPUs changes, the first communication indication information and the second communication indication information can be re-determined at the beginning of the next iteration, thereby automatically increasing or decreasing the number of nodes during training. When resources are limited, the GPU can be used to start training on a small number of processes first; when resources are abundant, the capacity can be automatically expanded dynamically, and the expanded GPU can be used to train on the process at the beginning of the next iteration, thereby realizing dynamic expansion and contraction of the training equipment.

[0234] When a node fails, the controllers of the remaining available nodes can reconstruct new first communication indication information and second communication indication information, so as to continue training without interrupting the training task.

[0235] In addition, the fusion method is determined by the static graph of the communication framework. Since the communication framework itself is part of the static graph, the fusion efficiency can be effectively improved compared to the dynamic gradient fusion process. By fusing the initial gradients and then reaching the gradient fusion threshold before performing target gradient communication, communication at a larger granularity can be achieved, thereby improving communication efficiency.

[0236] The device of the embodiment of the present application can execute the method provided by the embodiment of the present application, and its implementation principle is similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, and will not be repeated here.

[0237] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program stored in the memory, the method in any optional embodiment of the present application can be implemented.

[0238] Figure 15 FIG. 1 shows a schematic structural diagram of an electronic device to which an embodiment of the present invention is applicable. Figure 15 As shown, the electronic device may be a server or a user terminal, and the electronic device may be used to implement the method provided in any embodiment of the present invention.

[0239] like Figure 15 As shown in FIG, the electronic device 1500 may mainly include at least one processor 1501 ( Figure 15 1502, a communication module 1503 and an input / output interface 1504 and other components are shown in FIG. Optionally, each component can be connected and communicated through a bus 1505. It should be noted that, Figure 15 The structure of the electronic device 1500 shown is merely illustrative and does not constitute a limitation on the electronic devices to which the methods provided in the embodiments of the present application are applicable.

[0240] Memory 1502 may be used to store an operating system and application programs, etc. Application programs may include computer programs that implement the methods described in the embodiments of the present invention when called by processor 1501, and may also include programs for implementing other functions or services. Memory 1502 may be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and computer programs, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0241] The processor 1501 is connected to the memory 1502 via a bus 1505 and implements corresponding functions by calling the application program stored in the memory 1502. Among them, the processor 1501 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof, which can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of the present invention. The processor 1501 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0242] The electronic device 1500 can be connected to a network via a communication module 1503 (which may include, but is not limited to, components such as a network interface) to communicate with other devices (such as a user terminal or a server) via the network to implement data interaction, such as sending data to or receiving data from other devices. The communication module 1503 may include a wired network interface and / or a wireless network interface, etc., that is, the communication module may include at least one of a wired communication module and a wireless communication module.

[0243] The electronic device 1500 can be connected to the required input / output devices, such as a keyboard, a display device, etc., through the input / output interface 1504. The electronic device 1500 itself can have a display device, and can also be connected to other external display devices through the interface 1504. Optionally, a storage device, such as a hard disk, can also be connected through the interface 1504, so that data in the electronic device 1500 can be stored in the storage device, or data in the storage device can be read, and data in the storage device can also be stored in the memory 1502. It can be understood that the input / output interface 1504 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 1504 can be a component of the electronic device 1500, or it can be an external device connected to the electronic device 1500 when needed.

[0244] Bus 1505, used to connect the various components, may include a path for transmitting information between the components. Bus 1505 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, for example. Depending on their function, bus 1505 may be classified as an address bus, a data bus, a control bus, or the like.

[0245] Optionally, for the solution provided in the embodiment of the present invention, the memory 1502 can be used to store a computer program for executing the solution of the present invention, and be run by the processor 1501. When the processor 1501 runs the computer program, the actions of the method or device provided in the embodiment of the present invention are implemented.

[0246] Based on the same principle as the method provided in the embodiment of the present application, the embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the corresponding content of the aforementioned method embodiment can be implemented.

[0247] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the corresponding content of the aforementioned method embodiment can be implemented.

[0248] It should be noted that the terms "first," "second," "third," "fourth," "1," "2," etc. (if any) in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described in the drawings.

[0249] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the order of implementation of these steps is not limited to the order indicated by the arrows. Unless otherwise clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times respectively. Under different scenarios at the execution time, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0250] The above description is only an optional implementation method for some implementation scenarios of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of this application, the use of other similar implementation methods based on the technical ideas of this application also falls within the protection scope of the embodiments of this application.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining a training sample set and determining at least two nodes for training a neural network; wherein each of the nodes includes a controller and at least two graphics processing units (GPUs); Dividing the training sample set into a plurality of sample subsets according to the number of nodes, and sending each of the sample subsets to each of the nodes respectively, so as to train the neural network through each of the nodes; Constructing a static graph of the communication framework based on preset fusion rule information; wherein the fusion rule information is used to indicate different gradients to be fused; Determining a gradient fusion method based on the communication framework static graph and each initial gradient to be fused; Sending the gradient fusion mode and gradient fusion threshold to the controller of each node; Controlling the interactive communication between the GPUs through the controller in each of the nodes to update the parameters of the neural network; The controller of each node is used to perform the following steps: Sending the gradient fusion method and the gradient fusion threshold to each GPU of the node; receiving a negotiation request; the negotiation request is used to cause each GPU of the node to generate an initial gradient, and fuse the initial gradients according to the gradient fusion method, and if the information amount of the fused gradient reaches the gradient fusion threshold, use the fused gradient as the target gradient; wherein the initial gradient includes the gradient generated by each GPU for the weight parameters of each part of the neural network; Determining first communication indication information and second communication indication information based on the number of nodes and the number of GPUs; wherein the first communication indication information is used to instruct GPUs of different nodes to communicate with each other; and the second communication indication information is used to instruct different GPUs within the same node to communicate with each other; The first communication indication information and the second communication indication information are sent to each GPU of the node, so that each GPU of each node establishes communication based on the first communication indication information and the second communication indication information, and performs interactive transmission for the target gradient.

2. The method according to claim 1, characterized in that The gradient fusion threshold is determined by: Determine the bandwidth utilization rate corresponding to each node for test data with different data volumes using a preset script; The data volume of the test data whose bandwidth utilization meets the preset conditions is set as the gradient fusion threshold.

3. The method according to claim 1, characterized in that The GPUs of the nodes establish communication based on the first communication indication information and the second communication indication information, and perform interactive transmission for the target gradient, including: Each of the nodes establishes communication between the GPUs within each node based on the second communication indication information, and performs interactive transmission for the target gradient, and obtains a first updated gradient of each of the GPUs after the interaction; Each of the nodes establishes communication between the GPUs of different nodes based on the first communication indication information, and performs interactive transmission based on the first updated gradient to obtain the second updated gradient of each of the GPUs after the interaction; Each of the nodes establishes communication between the GPUs within each node based on the second communication indication information, and performs interactive transmission based on the second updated gradient to obtain a third updated gradient of each of the GPUs after the interaction.

4. The method according to claim 3, characterized in that Each GPU contains the target gradient corresponding to the model parameters of each part; The interactive transmission of the target gradient to obtain the first updated gradient of each GPU after the interaction includes: The GPUs in each node gradually exchange and fuse each other's target gradients until each GPU contains a corresponding first fused gradient, thereby obtaining a first updated gradient of the GPU; wherein the first fused gradient corresponding to each GPU is obtained by fusing the target gradients of each GPU corresponding to each part of the model parameters, and the first fused gradient of each GPU corresponds to a different part of the model parameters.

5. The method according to claim 3, characterized in that The interactive transmission based on the first updated gradient to obtain the second updated gradient of each of the GPUs after the interaction includes: The GPUs of different nodes gradually exchange each other's first updated gradients and fuse them until all the interacting GPUs contain the same second fused gradient, thereby obtaining the second updated gradients of the interacting GPUs; wherein the second fused gradients are obtained by fusing the first updated gradients corresponding to the interacting GPUs.

6. The method according to claim 3, characterized in that The interactive transmission based on the second updated gradient to obtain the third updated gradient of each of the GPUs after the interaction includes: The GPUs in each node gradually exchange the second updated gradients with each other until each GPU contains a third fused gradient, thereby obtaining a third updated gradient for each GPU; wherein the third fused gradient is obtained by fusing the second updated gradients of each GPU in each node.

7. A data processing device, characterized in that: The device comprises: An acquisition module is used to acquire a training sample set and determine at least two nodes for training a neural network; wherein each of the nodes includes a controller and at least two graphics processing units (GPUs); A sending module, configured to divide the training sample set into a plurality of sample subsets according to the number of nodes, and send each of the sample subsets to each of the nodes, so as to train the neural network through each of the nodes; a determination module configured to construct a communication framework static graph based on preset fusion rule information; wherein the fusion rule information is used to indicate different gradients to be fused; determine a gradient fusion method based on the communication framework static graph and each initial gradient to be fused; and send the gradient fusion method and gradient fusion threshold to the controller of each node; a communication module, configured to control the interactive communication between the GPUs through the controller in each of the nodes, so as to update the parameters of the neural network; The controller of each node is used to perform the following steps: Sending the gradient fusion method and the gradient fusion threshold to each GPU of the node; receiving a negotiation request; the negotiation request is used to cause each GPU of the node to generate an initial gradient, and fuse the initial gradients according to the gradient fusion method, and if the information amount of the fused gradient reaches the gradient fusion threshold, use the fused gradient as the target gradient; wherein the initial gradient includes the gradient generated by each GPU for the weight parameters of each part of the neural network; Determining first communication indication information and second communication indication information based on the number of nodes and the number of GPUs; wherein the first communication indication information is used to instruct GPUs of different nodes to communicate with each other; and the second communication indication information is used to instruct different GPUs within the same node to communicate with each other; The first communication indication information and the second communication indication information are sent to each GPU of the node, so that each GPU of each node establishes communication based on the first communication indication information and the second communication indication information, and performs interactive transmission for the target gradient.

8. The device according to claim 7, characterized in that The determining module is used for: Determine the bandwidth utilization rate corresponding to each node for test data with different data volumes using a preset script; The data volume of the test data whose bandwidth utilization meets the preset conditions is set as the gradient fusion threshold.

9. The device according to claim 7, characterized in that The GPUs of the nodes establish communication based on the first communication indication information and the second communication indication information, and perform interactive transmission for the target gradient, including: Each of the nodes establishes communication between the GPUs within each node based on the second communication indication information, and performs interactive transmission for the target gradient, and obtains a first updated gradient of each of the GPUs after the interaction; Each of the nodes establishes communication between the GPUs of different nodes based on the first communication indication information, and performs interactive transmission based on the first updated gradient to obtain the second updated gradient of each of the GPUs after the interaction; Each of the nodes establishes communication between the GPUs within each node based on the second communication indication information, and performs interactive transmission based on the second updated gradient to obtain a third updated gradient of each of the GPUs after the interaction.

10. The device according to claim 9, characterized in that Each GPU contains the target gradient corresponding to the model parameters of each part; The interactive transmission of the target gradient to obtain the first updated gradient of each GPU after the interaction includes: The GPUs in each node gradually exchange and fuse each other's target gradients until each GPU contains a corresponding first fused gradient, thereby obtaining a first updated gradient of the GPU; wherein the first fused gradient corresponding to each GPU is obtained by fusing the target gradients of each GPU corresponding to each part of the model parameters, and the first fused gradient of each GPU corresponds to a different part of the model parameters.

11. The device according to claim 9, characterized in that The interactive transmission based on the first updated gradient to obtain the second updated gradient of each GPU after the interaction includes: The GPUs of different nodes gradually exchange each other's first updated gradients and fuse them until all the interacting GPUs contain the same second fused gradient, thereby obtaining the second updated gradients of the interacting GPUs; wherein the second fused gradients are obtained by fusing the first updated gradients corresponding to the interacting GPUs.

12. The device according to claim 9, characterized in that The interactive transmission based on the second updated gradient to obtain the third updated gradient of each of the GPUs after the interaction includes: The GPUs in each node gradually exchange the second updated gradients with each other until each GPU contains a third fused gradient, thereby obtaining a third updated gradient for each GPU; wherein the third fused gradient is obtained by fusing the second updated gradients of each GPU in each node.

13. An electronic device, characterized in that: The electronic device includes a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the computer program to implement the method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which implements the method according to any one of claims 1 to 6 when executed by a processor.

15. A computer program product, characterized in that The computer product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images