Optical interconnection system based on parameter server architecture and data interaction method thereof
By adopting a fiber-direct optical interconnection system in the parameter server architecture, the problem of high cost of multi-computer communication in the existing technology is solved, and low-cost and high-efficiency large-model training is achieved.
Patent Information
- Application Number
- CN202411700620.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-26
- Publication Date
- 2025-02-11
AI Technical Summary
The existing training method based on parameter server architecture adopts a common switch scheme for communication between multiple machines, which leads to high costs.
An optical interconnection system based on fiber connection is adopted, and the service node and the working node are directly connected through optical fiber, eliminating switch forwarding and achieving low-cost data interaction.
Reduces the economic and power consumption cost of the system, improves the efficiency of large-scale model training, and reduces network latency.
Smart Images
Figure CN119420423A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of communication network technology, and in particular to an optical interconnection system based on a parameter server architecture and a data interaction method thereof. Background Art
[0002] The rise of large models has continuously increased the requirements for computing power infrastructure. With the substantial increase in model size and training data, running large models on a single machine will be extremely slow. Distributed training can greatly improve training efficiency by training models on multiple machines.
[0003] The parameter server (PS) architecture is a mainstream method in distributed training, such as Figure 1 As shown in the figure, the parameter server architecture includes at least one server node (the server node is the parameter server) and multiple worker nodes. Training based on the parameter server is divided into three steps: (1) Each worker copies the complete model parameters from the server; (2) Use the data on each worker to calculate the gradient on the model parameters of this copy; (3) Each worker passes the calculated gradient to the server, and the server updates the parameters.
[0004] Among them, server nodes do not communicate with each other, and each server is responsible for the model parameters assigned to it; worker nodes do not communicate with each other, and only communicate with their corresponding servers.
[0005] It can be seen that the PS architecture involves communication between multiple machines. In actual applications, communication between multiple machines adopts a universal switch solution, that is, all machines are connected to the switch with cables / cables, and routing addressing and signal distribution are completed through the switch. Considering that the transmission rate of optical fiber is higher than that of cable, most of the current high-speed switches use optical cable interfaces. The higher the port transmission rate of the switch, the more expensive the switch price.
[0006] It can be seen that the existing parameter server-based training method is still costly. Summary of the invention
[0007] The present invention provides an optical interconnection system based on a parameter server architecture and a data interaction method thereof, so as to solve the defect of high cost caused by adopting a general switch solution for communication between multiple machines in the prior art, realize the low cost of the parameter server architecture, and improve the efficiency of large model training.
[0008] The present invention provides an optical interconnection system based on a parameter server architecture, comprising at least one service node and a plurality of working nodes; wherein: The optical transmitting end of any service node is connected to the optical receiving end of any working node through an optical fiber; The optical transmitting end of any working node is connected to the optical receiving end of any service node through an optical fiber; Any of the working nodes is used to pull model parameters from any of the service nodes connected thereto; Any of the working nodes is further used to obtain training data, perform training tasks using the training data, and obtain task execution results; and send the task execution results to any of the service nodes through the optical transmitting end of any of the working nodes; Any one of the service nodes is used to update the model parameters based on the task execution result.
[0009] According to an optical interconnection system based on a parameter server architecture provided by the present invention, a service-end optical transceiver module is provided in any service node; the service-end optical transceiver module is provided with a first optical transmitting chip, an optical beam splitter and an optical transmitting end of the service node; The optical output end of the first optical emission chip of any service node is connected to the optical beam splitter, and the output end of the optical beam splitter is connected to the optical emission end of the service node through an optical fiber.
[0010] According to an optical interconnection system based on a parameter server architecture provided by the present invention, when there are multiple optical beam splitters, the multiple optical beam splitters are connected in a cascade manner.
[0011] According to an optical interconnection system based on a parameter server architecture provided by the present invention, any working node is provided with a working end optical transceiver module, and the working end optical transceiver module is provided with a second optical transmitting chip, a second optical receiving chip and an optical transmitting end and an optical receiving end of the working node; The second optical receiving chip is used to receive the model parameters from the optical receiving end of any working node via an optical fiber, convert the model parameters into electrical signals, and output the electrical signals to a processor on any working node, so that the processor performs the training task based on the electrical signals; The second optical transmission chip is used to receive the electrical signal obtained after the task execution result of any working node is modulated and transformed, convert the electrical signal into an optical signal, and transmit the optical signal to the optical output end of any working node through an optical fiber.
[0012] According to an optical interconnection system based on a parameter server architecture provided by the present invention, a second driving circuit board is also provided in any one of the working nodes; The second driving circuit board is used to receive electric energy from an external power source, and use the electric energy to supply power to the second light emitting chip and the second light receiving chip respectively.
[0013] According to an optical interconnection system based on a parameter server architecture provided by the present invention, a first optical transmitting chip and a first optical receiving chip are provided in any service node; The first optical receiving chip is used to receive the model parameters from the optical input port via the optical fiber, convert the model parameters into electrical signals, and output the electrical signals to the processor on the working node, so that the processor performs the training task based on the electrical signals; The first optical transmission chip is used to receive the electrical signal obtained after the task execution result on the working node is modulated and transformed, convert the electrical signal into an optical signal, and transmit the optical signal to the optical output port of the working node through the optical fiber.
[0014] The present invention also provides a data interaction method, which is applied to any service node in any of the optical interconnection systems based on the parameter server architecture, and comprises the following steps: Copying the model parameters of the artificial intelligence model to be trained into multiple copies of the model parameters; broadcasting the optical signal modulated based on the multiple copies of the model parameters to the at least one working node; Any working node is used to obtain data to be trained, perform a training task on the received model parameters based on the data to be trained, and obtain the gradient of the model parameters; Receive the gradient of the model parameters, and update the model parameters of the artificial intelligence model to be trained according to the gradient of the model parameters.
[0015] According to a data interaction method provided by the present invention, the step of copying the model parameters of the artificial intelligence model to be trained into multiple copies of the model parameters; and broadcasting the optical signals modulated based on the multiple copies of the model parameters to the at least one working node includes: In the case where there are multiple service nodes, a model parameter subset input by a user is received; wherein the model parameter subset is obtained by equally dividing the model parameters of the artificial intelligence model to be trained based on the total number of service nodes; Broadcasting an optical signal obtained by modulating the model parameters in the model parameter subset to any working node; any working node is used to pull the model parameter subsets on each service node from each service node respectively to obtain all model parameters of the artificial intelligence model to be trained; any working node is also used to obtain data to be trained, perform a training task on all the received model parameters based on the data to be trained, obtain the gradients of all the model parameters, and divide the gradients of all the model parameters into gradient subsets corresponding to each model parameter subset; The receiving the gradient of the model parameter and updating the model parameter of the artificial intelligence model to be trained according to the gradient of the model parameter includes: A gradient subset corresponding to a locally stored model parameter subset is received, and the locally stored model parameter subset is updated based on the gradient subset; and the updated model parameter subsets of all service nodes are merged as all model parameters after training.
[0016] The present invention also provides a data interaction method, which is applied to any working node in any of the optical interconnection systems based on the parameter server architecture, and comprises the following steps: Receive model parameters of an artificial intelligence model to be trained, obtain data to be trained, perform a training task on the received model parameters based on the data to be trained, and obtain the gradient of the model parameters; wherein the model parameters of the artificial intelligence model to be trained are obtained by the service node replicating multiple copies of all model parameters of the artificial intelligence model to be trained, modulating the replicated multiple copies of the model parameters into optical signals, and broadcasting the model parameters through optical fibers; The gradient of the model parameters is transmitted to the service node via optical fiber, so that the service node updates the model parameters of the artificial intelligence model to be trained according to the gradient of the model parameters.
[0017] According to a data interaction method provided by the present invention, the receiving of model parameters of an artificial intelligence model to be trained, obtaining data to be trained, performing a training task on the received model parameters based on the data to be trained, and obtaining the gradient of the model parameters includes: In the case where there are multiple service nodes, a subset of model parameters on each service node is pulled from each service node to obtain all model parameters of the artificial intelligence model to be trained; wherein the subset of model parameters is obtained by pre-dividing all model parameters of the artificial intelligence model to be trained based on the total number of service nodes; Acquire data to be trained, perform a training task on all the received model parameters based on the data to be trained, obtain gradients of all the model parameters, and divide the gradients of all the model parameters into gradient subsets corresponding to each model parameter subset; The step of transmitting the gradient of the model parameters to the service node via an optical fiber so that the service node updates the model parameters of the artificial intelligence model to be trained according to the gradient of the model parameters comprises: Each gradient subset is sent to the service node to which the model parameter subset corresponding to each gradient subset belongs, so that the service node updates the model parameter subset stored in the service node based on the gradient subset; the model parameter subsets after all service nodes are updated are merged as all model parameter subsets after training is completed. The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned data interaction methods when executing the computer program.
[0018] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the data interaction method described in any one of the above methods is implemented.
[0019] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned data interaction methods.
[0020] The optical interconnection system based on parameter server architecture and its data interaction method provided by the present invention, the system includes at least one service node and multiple working nodes; wherein the optical transmitting end of any service node is connected to the optical receiving end of any working node through optical fiber; the optical transmitting end of any working node is connected to the optical receiving end of any service node through optical fiber; any working node is used to pull model parameters from any service node connected thereto, and receive the model parameters through the optical receiving end of any working node; any working node is also used to obtain training data, perform training tasks using the training data, and obtain task execution results; the task execution results are sent to any service node through the optical transmitting end of any working node; any service node is used to update the model parameters based on the task execution results. The present application directly connects the server node and the worker node with optical fiber, does not need to be forwarded through a switch, saves one photoelectric-electrical-optical conversion, greatly saves economic and power consumption costs, and reduces network latency. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0022] Figure 1 It is a schematic diagram of the parameter server architecture provided by the prior art.
[0023] FIG. 2( a ) is one of the structural schematic diagrams of the optical interconnection system based on the parameter server architecture provided by the present invention.
[0024] FIG2( b ) is a second schematic diagram of the structure of the optical interconnection system based on the parameter server architecture provided by the present invention.
[0025] Figure 3 It is a structural schematic diagram of the service end optical transceiver module provided by the present invention.
[0026] Figure 4 It is a structural schematic diagram of the working end optical transceiver module provided by the present invention.
[0027] Figure 5 It is a flow chart of a data interaction method provided by the present invention and applied to any service node in an optical interconnection system based on a parameter server architecture.
[0028] Figure 6 It is a flow chart of a data interaction method provided by the present invention and applied to any working node in an optical interconnection system based on a parameter server architecture.
[0029] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0031] Below is combined with Figure 2 (a) - Figure 7 Specific embodiments of the present invention are described.
[0032] Figure 2 (a) is one of the structural schematic diagrams of the optical interconnection system based on the parameter server architecture provided by the present invention, and Figure 2 (b) is the second structural schematic diagram of the optical interconnection system based on the parameter server architecture provided by the present invention. Combined with Figure 2 (a) and Figure 2 (b) for explanation, the system includes at least one service node (such as server1 and server2 in Figure 2 (a) and Figure 2 (b)) and multiple working nodes (such as worker1, worker2, worker3, worker4 in Figure 2 (a) and Figure 2 (b)).
[0033] Wherein, the optical transmitting end of any service node is connected to the optical receiving end of any working node via an optical fiber; The optical transmitting end of any working node is connected to the optical receiving end of any service node through an optical fiber; Any one of the working nodes is used to pull the model parameters from any one of the service nodes connected thereto, and receive the model parameters through the optical receiving end of any one of the working nodes; Any of the working nodes is further used to obtain training data, perform training tasks using the training data, and obtain task execution results; and send the task execution results to any of the service nodes through the optical transmitting end of any of the working nodes; Any one of the service nodes is used to update the model parameters based on the task execution result.
[0034] Specifically, each service node (hereinafter referred to as server) is provided with a server-side optical transceiver module, and each working node (hereinafter referred to as worker) is provided with a worker-side optical transceiver module; the server and the worker are connected through optical fiber and realize data exchange. There is no need for communication between workers. In order to save equipment space and facilitate installation and maintenance, the above-mentioned optical fibers can be bundled and packaged in the optical fiber backplane, and each optical fiber is respectively connected to the above-mentioned server-side optical transceiver module and the worker-side optical transceiver module. Among them, the optical transceiver module, also known as the optical transceiver integrated module (referred to as the optical module), is the core component of optical communication, which can complete the optical-electrical / electrical-optical conversion of optical signals, and is mainly composed of two parts: the receiving part (i.e., the optical receiving end) and the transmitting part (i.e., the optical transmitting end), the optical receiving end is used to realize the optical-electrical conversion, and the optical transmitting end is used to realize the electrical-optical conversion.
[0035] Figure 2 (b) shows a schematic diagram of an optical interconnection system based on a parameter server architecture with multiple servers. Assuming that there are M optical transceiver modules on the server side, M is the number of server nodes (M is an integer greater than or equal to 1), that is, one optical transceiver module is equipped for each server node; assuming that there are N optical transceiver modules on the worker side, N is the number of worker nodes (N=2n, n is a natural number), that is, one optical transceiver module is equipped for each worker node. Each server-side optical transceiver module has N optical output ports and N optical input ports, which are connected to one side of the optical fiber backplane through a ribbon parallel optical fiber; each worker-side optical transceiver module has M optical output ports and M optical input ports, which are connected to the other side of the optical fiber backplane through a ribbon parallel optical fiber. There are 2N·M ports on both sides of the optical fiber backplane, with 2N·M optical fibers built in. Through the optical fiber on the optical fiber backplane, the N optical output ports of a single server are connected to one optical input port of each worker, and the N optical input ports of a single server are connected to one optical output port of each worker, respectively, to achieve full connection of the ports on both sides.
[0036] During the model training process, the above service node (i.e., server) loads the model parameters, and the worker node loads the data to be trained locally. This step does not involve data interaction between the server and the worker.
[0037] Each worker pulls the model parameters from the server. Specifically, the server first needs to convert the model parameters into a bit stream, and then further convert them into high and low level signals, which are fed into the server's optical transceiver module. The server's optical transceiver module converts the high and low level signals into optical signals after electrical-optical conversion, and transmits them to the worker via optical fiber. The worker uses the worker's optical transceiver module to perform optical-electrical conversion and read the model parameters. The worker retrieves the local training data, performs forward and backward propagation of the model on the model parameters, and calculates the gradient of the parameters. This step does not involve data interaction between the server and the worker. Each worker passes the calculated gradient to the server, and the server updates the parameters.
[0038] Repeat the above steps until the training is completed.
[0039] The above-mentioned embodiment constructs an optical interconnection system based on a parameter server architecture, which includes at least one service node and multiple working nodes; wherein the optical transmitting end of any service node is connected to the optical receiving end of any working node through an optical fiber; the optical transmitting end of any working node is connected to the optical receiving end of any service node through an optical fiber; any working node is used to pull model parameters from any service node connected thereto, and receive the model parameters through the optical receiving end of any working node; any working node is also used to obtain training data, perform training tasks using the training data, and obtain task execution results; the task execution results are sent to any service node through the optical transmitting end of any working node; any service node is used to update the model parameters based on the task execution results. The present application directly connects the server node and the worker node with optical fiber, and does not need to be forwarded through a switch, which saves one photoelectric-electrical-optical conversion, greatly saves economic and power consumption costs, and reduces network latency.
[0040] In one embodiment, any of the above service nodes is provided with a service-side optical transceiver module, and the structural diagram of the service-side optical transceiver module is as follows: Figure 3 As shown, the service-side optical transceiver module mainly includes a driver circuit board, a first optical transmitter chip, a first optical receiver chip, several optical beam splitters, several optical fibers, etc. The specific configuration details vary slightly according to the number N of worker nodes in the system. Among them, the optical output end of the first optical transmitter chip is connected to the optical beam splitter, and the output end of the optical beam splitter is connected to the optical transmitter end of the service node through an optical fiber; the first optical receiver chip on the service-side optical transceiver module is directly connected to the optical receiver end through an optical fiber; The driver circuit board is mainly used to supply power to the first optical transmitting chip and the first optical receiving chip, and to connect the input and output electrical signals. It integrates 1 transmitting end electrical amplifier and N receiving end electrical amplifiers for amplifying the electrical signals of the input and output service end optical transceiver module.
[0041] The first optical transmission chip has only one optical transmission unit, which is used to convert one electrical signal received by the service node into an optical signal. The passive optical beam splitters are divided into N parts (the number of optical beam splitters in each group is 1, 2, 4, etc.), and output to the N optical output ports of the module. Among them, the optical emission unit can adopt mainstream electro-optical conversion technologies such as vertical cavity surface emitting laser (VECSEL, Vertical-External-Cavity Surface-Emitting Laser), EML (Electro-Absorption Modulated Laser, electro-absorption modulated laser), DML (Direct Modulation Laser, direct modulation laser), silicon-based optical modulator, etc. The passive optical beam splitters are connected in cascade mode, such as Figure 3 shown.
[0042] The first optical receiving chip has N optical detection units for receiving optical signals transmitted from the N optical receiving ports of the server optical transceiver module (connected via optical fibers) and converting the optical signals into electrical signals (i.e., data streams recognizable by the server node).
[0043] The above-mentioned embodiment makes full use of the broadcast mode characteristics of the server node to the worker node in the parameter server architecture. The data propagation from one server node to multiple worker nodes uses a single optical emitting unit + multiple groups of passive optical beam splitters cascaded to achieve low-cost broadcast mode. By using passive optical beam splitters to replace the original configuration of multiple optical emitting units, there is no need to configure optical emitting units and electrical amplifiers that are consistent with the number of worker nodes, which saves the cost of (N-1)·M optical emitting units and electrical amplifiers. The cost of the added passive optical splitter is negligible compared to high-frequency active devices such as optical emitting units and electrical amplifiers, thereby reducing the cost of the optical module.
[0044] In one embodiment, any of the above working nodes (ie, workers) is provided with a working end optical transceiver module, such as Figure 4 As shown, Figure 4 A schematic diagram of the structure of a working end optical transceiver module located on a working node is shown, wherein the working end optical transceiver module is provided with a second optical transmitting chip, a second optical receiving chip, and an optical transmitting end and an optical receiving end of the working node; The second optical receiving chip is used to receive the model parameters from the optical receiving end of any working node via an optical fiber, convert the model parameters into electrical signals, and output the electrical signals to a processor on any working node, so that the processor performs the training task based on the electrical signals; The second optical transmission chip is used to receive an electrical signal obtained after the task execution result of any working node is modulated and transformed, convert the electrical signal into an optical signal, and transmit the optical signal to the optical output end of any working node through an optical fiber; Specifically, Figure 4 As shown, the working end optical transceiver module mainly includes a second driving circuit board, a second optical transmitting chip, a second optical receiving chip, several optical fibers, etc. The specific configuration details vary slightly according to the number M of servers. The driving circuit board is mainly used to power the second optical transmitting chip and the second optical receiving chip, and connect the input and output electrical signals. M transmitting end electrical amplifiers and M receiving end electrical amplifiers are integrated thereon to amplify the input and output electrical signals of the working end optical transceiver module. The above-mentioned input electrical signal carries the data stream to be sent by the worker node, and the output electrical signal carries the external incoming data stream received by the worker node; The second driving circuit board is used to receive electric energy from an external power source, and use the electric energy to power the second light emitting chip and the second light receiving chip respectively.
[0045] exist Figure 4 In the embodiment, the second optical transmission chip has M optical transmission units, which are used to convert M electrical signals input to the working node into optical signals and output them to M optical output ends of the optical transceiver module at the working end through optical fibers; The second optical receiving chip has M optical detection units, which are used to receive optical signals transmitted from the M optical input ports of the optical transceiver module at the working end (the optical input ports are connected to the second optical receiving chip through optical fibers), and convert the optical signals into electrical signals (i.e., into data streams recognizable by the worker node). The optical detection units can use technologies such as photodiodes (PIN) made of various materials and avalanche photodiodes (APD, Avalanche Photodiode).
[0046] In one embodiment, a first optical transmitting chip and a first optical receiving chip are provided in any of the service nodes, wherein the first optical receiving chip is used to receive the model parameters from the optical input port via the optical fiber, convert the model parameters into electrical signals, and output the electrical signals to the processor on the working node, so that the processor performs the training task based on the electrical signals; The first optical transmission chip is used to receive the electrical signal obtained after the task execution result on the working node is modulated and transformed, convert the electrical signal into an optical signal, and transmit the optical signal to the optical output port of the working node through the optical fiber.
[0047] In the above-mentioned embodiment, by setting up optical transceiver modules on the service node and the working node, the data interaction between the two can be transmitted through optical fiber, which improves the data interaction speed in model training and is conducive to improving the efficiency of model training.
[0048] In one embodiment, if Figure 5 As shown, the present application also provides a data interaction method, which is applied to any service node in the above-mentioned optical interconnection system based on the parameter server architecture, and the method includes: Step 501, copy the model parameters of the artificial intelligence model to be trained into multiple copies of model parameters; broadcast the optical signal modulated based on the multiple copies of model parameters to at least one working node; any working node is used to obtain the data to be trained, perform a training task on the received model parameters based on the data to be trained, and obtain the gradient of the model parameters.
[0049] Specifically, the optical interconnection system based on the parameter server architecture with a single server and four worker nodes shown in FIG. 2 (a) is taken as an example for explanation. The method is applied to any service node in the system. The detailed steps are as follows.
[0050] S1 and server1 load all model parameters, and workers 1-4 load the data to be trained locally respectively; this step does not involve communication between the server and the worker.
[0051] S2. Each worker copies the complete model parameters from the server. The specific implementation is as follows.
[0052] S2.1, server1 converts the model parameters to be sent into a bit stream, which is further converted into high and low level signals and fed into the server-side optical transceiver module of server1.
[0053] S2.2. The server-side optical transceiver module of the server1 mentioned above includes at least one optical splitter. The server-side optical transceiver module of server1 amplifies an input electrical signal, converts it into an optical signal through a first optical transmitting chip, and divides it into four parts through two groups of passive optical splitters, and outputs it to the four optical output ports of the server-side optical transceiver module.
[0054] S2.3. Four identical optical signals are transmitted to the optical fiber backplane through the optical fiber ribbon and further distributed to the input ports of the optical transceiver modules at the working ends of the four worker nodes.
[0055] S2.4. For each worker, the optical transceiver module at the working end converts the input optical signal into an electrical signal through an optical detector, amplifies the signal and outputs it to the worker, further converts it into a data bit stream, and further converts it into a model parameter that can be read by the worker.
[0056] S3. Each worker retrieves a batch of training data and performs forward and backward propagation of the model on the copied model parameters, and calculates the gradient of the parameters; this step does not involve communication between the server and the worker.
[0057] Step 502: Receive the gradient of the model parameters, and update the model parameters of the artificial intelligence model to be trained according to the gradient of the model parameters.
[0058] In detail, this includes the following execution process.
[0059] S4. Each worker passes the calculated gradient to the server, and the server updates the parameters. The specific implementation is as follows.
[0060] S4.1. Each worker converts the gradient data to be sent into a bit stream, which is further converted into high and low level signals and fed into the optical transceiver module of each worker.
[0061] S4.2. Each worker amplifies the input electrical signal and converts it into an optical signal through the optical transmission chip, and outputs it to the optical output port of the module through the optical fiber.
[0062] S4.3. The four different optical signals output by the four workers are transmitted to the optical fiber backplane through the optical fiber ribbon and further converged to the input port of the optical transceiver module of the server1 node.
[0063] S4.4, the optical transceiver module of the server converts the 4 input optical signals into 4 electrical signals through 4 optical detectors, amplifies and outputs them to the server, further converts them into data bit streams, and further converts them into gradient data readable by the server. The server processes the 4 sets of gradient data to obtain updated model parameters.
[0064] S5. Repeat the above S2-S4 until the training is completed.
[0065] The above embodiment, by designing an optical interconnection system based on a parameter server architecture, converts the model parameters on the service node into optical signals and distributes them to different working nodes in a broadcast manner to complete model training, thereby improving the efficiency of model training through distributed training. In particular, in the data transmission from one server node to multiple worker nodes, a single optical transmission unit + multiple groups of passive optical beam splitters are used in cascade, and the broadcast mode is realized at a low cost without the need to configure optical transmission units and electrical amplifiers that are consistent with the number of worker nodes, thereby reducing the cost of optical modules.
[0066] In one embodiment, the above-mentioned step 501 includes: when there are multiple service nodes, receiving a model parameter subset input by a user; wherein the model parameter subset is obtained by equally dividing the model parameters of the artificial intelligence model to be trained based on the total number of service nodes; broadcasting the optical signal obtained after modulating the model parameters in the model parameter subset to any working node; the any working node is used to pull the model parameter subsets on each service node from each service node respectively to obtain all model parameters of the artificial intelligence model to be trained; the any working node is also used to obtain the data to be trained, perform the training task on the received all model parameters based on the data to be trained, obtain the gradients of all model parameters, and divide the gradients of all model parameters into gradient subsets corresponding to each model parameter subset.
[0067] The above step 502 includes: receiving a gradient subset corresponding to a locally stored model parameter subset, and updating the locally stored model parameter subset based on the gradient subset; and merging the updated model parameter subsets of all service nodes as all model parameters after training.
[0068] Specifically, the optical interconnection system based on the parameter server architecture with 2 servers and 4 worker nodes shown in FIG. 2 (b) is taken as an example for explanation. The method is applied to any service node in the system. The detailed steps are as follows.
[0069] Considering that the parameters of the large model are very large, one server may not be able to complete the update calculation of all parameters. Now two servers are used to save and update the model parameters.
[0070] Step 1: Divide the model parameters into two parts. Server 1 loads half of the model parameters, server 2 loads the other half of the model parameters, and workers 1-4 load the data to be trained locally. This step does not involve communication between the server and the worker.
[0071] Step 2: Each worker copies the complete model parameters from server1 and server2. The specific implementation is as follows.
[0072] Step 2.1, server1 converts half of the model parameters to be sent into a bit stream, further converts it into high and low level signals, and feeds it into the server-side optical transceiver module of server1. The operation of server2 is the same as above.
[0073] Step 2.2: It should be emphasized that the server optical transceiver module of each service node contains at least one optical beam splitter. The server optical transceiver module of server1 amplifies the input electrical signal and converts it into an optical signal through the first optical transmitter chip, and divides it into 4 parts through 2 sets of passive optical beam splitters, and outputs it to the 4 optical output ports of the module. The operation of server2 is the same as above.
[0074] In step 2.3, the four identical optical signals output by server1 and server2 are transmitted to the optical fiber backplane through optical fibers, and are further distributed to the input ports of the optical transceiver modules at the working ends of the four worker nodes.
[0075] Step 2.4: For each worker, the optical transceiver module at the working end converts the two input optical signals into two electrical signals through two optical detectors, amplifies and outputs them to the worker, further converts them into two data bit streams, and further merges and converts them into complete full-volume model parameters.
[0076] Step 3: Each worker retrieves a batch of data and performs forward and backward propagation of the model on the parameters of this copy, calculates the gradient of the parameters, and splits the gradient into two parts, A and B, according to the way server1 and server2 split the original parameters. This step does not involve communication between the server and the worker.
[0077] Step 4: Each worker transmits the calculated gradients A and B to server1 and server2 respectively, and server1 and server2 update the parameters. The specific implementation is as follows.
[0078] Step 4.1: Each worker converts the two gradient data A and B to be sent into a bit stream, which is further converted into two high and low level signals and fed into the optical transceiver module of each worker.
[0079] Step 4.2: Each worker amplifies the two input electrical signals and converts them into two optical signals through the optical transmission chip, and outputs them to the two optical output ports of the module through optical fibers.
[0080] In step 4.3, the eight different optical signals output by the four workers are transmitted to the optical fiber backplane through the optical fiber ribbon, and further converged to the input ports of the optical transceiver modules corresponding to the server1 and server2 nodes.
[0081] Step 4.4, the optical transceiver module of server1 converts the 4 input optical signals into 4 electrical signals through 4 optical detectors, amplifies and outputs them to server1, further converts them into data bit streams, and further converts them into gradient data readable by server1. Server1 processes the 4 sets of gradient data to obtain half of the updated model parameters, and the operation of server2 is the same as above.
[0082] Step 5: Repeat steps 2-4 until the training is completed.
[0083] The above embodiment utilizes multiple service nodes to train a part of the model parameters respectively, so as to realize synchronous training of multiple service nodes, which can improve the efficiency of model training. In addition, the service-side optical transceiver module of each service node includes at least one optical splitter, so that the server node and the worker node can be fully connected by optical fiber without the need for forwarding through a switch, thus eliminating one optoelectronic-electrical-optical conversion, greatly saving economic and power consumption costs, and reducing network latency.
[0084] In one embodiment, a data interaction method is also provided, such as Figure 6 As shown, the method is applied to any working node in the optical interconnection system based on the parameter server architecture, and specifically includes: step 601, receiving the model parameters of the artificial intelligence model to be trained, obtaining the data to be trained, performing the training task on the received model parameters based on the data to be trained, and obtaining the gradient of the model parameters; wherein the model parameters of the artificial intelligence model to be trained are the model parameters after the service node copies all the model parameters of the artificial intelligence model to be trained in multiple copies, and modulating the copied multiple model parameters into optical signals and broadcasting them through optical fibers; Step 602: Transmit the gradient of the model parameters to the service node via optical fiber, so that the service node updates the model parameters of the artificial intelligence model to be trained according to the gradient of the model parameters.
[0085] In one embodiment, when there are multiple service nodes, the above step 601 includes: when there are multiple service nodes, respectively pulling a subset of model parameters on each service node from each service node to obtain all model parameters of the artificial intelligence model to be trained; wherein the model parameter subset is obtained by pre-dividing all model parameters of the artificial intelligence model to be trained based on the total number of service nodes; obtaining data to be trained, performing a training task on all received model parameters based on the data to be trained, obtaining gradients of all model parameters, and dividing the gradients of all model parameters into gradient subsets corresponding to each model parameter subset; The above step 602 includes: sending each gradient subset to the service node to which the model parameter subset corresponding to each gradient subset belongs, so that the service node updates the model parameter subset stored in the service node based on the gradient subset; merging the model parameter subsets after all service nodes are updated as all model parameters after training.
[0086] The detailed steps of this embodiment can be compared with the detailed steps of any of the above service nodes, and will not be repeated here.
[0087] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute a data interaction method applied to any service node in an optical interconnection system based on a parameter server architecture, the method comprising: copying the model parameters of the artificial intelligence model to be trained into multiple copies of the model parameters; broadcasting the optical signal modulated based on the multiple copies of the model parameters to the at least one working node; any working node, for obtaining the data to be trained, performing the training task on the received model parameters based on the data to be trained, and obtaining the gradient of the model parameters; receiving the gradient of the model parameters, and updating the model parameters of the artificial intelligence model to be trained according to the gradient of the model parameters. Alternatively, a data interaction method is executed for any working node in an optical interconnection system based on a parameter server architecture, the method comprising: receiving model parameters of an artificial intelligence model to be trained, acquiring data to be trained, performing a training task on the received model parameters based on the data to be trained, and obtaining the gradient of the model parameters; wherein the model parameters of the artificial intelligence model to be trained are obtained by the service node replicating multiple copies of all model parameters of the artificial intelligence model to be trained, modulating the replicated multiple copies of the model parameters into optical signals, and broadcasting the model parameters via optical fibers; the gradient of the model parameters is transmitted to the service node via optical fibers, so that the service node updates the model parameters of the artificial intelligence model to be trained according to the gradient of the model parameters.
[0088] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0089] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data interaction method provided by the above methods and applied to any service node in an optical interconnection system based on a parameter server architecture, the method including: copying the model parameters of the artificial intelligence model to be trained into multiple copies of model parameters; broadcasting the optical signal modulated based on the multiple copies of the model parameters to at least one working node; any working node, for obtaining data to be trained, performing a training task on the received model parameters based on the data to be trained, and obtaining the gradient of the model parameters; receiving the gradient of the model parameters, and updating the model parameters of the artificial intelligence model to be trained according to the gradient of the model parameters. Alternatively, a data interaction method is executed for any working node in an optical interconnection system based on a parameter server architecture, the method comprising: receiving model parameters of an artificial intelligence model to be trained, acquiring data to be trained, performing a training task on the received model parameters based on the data to be trained, and obtaining the gradient of the model parameters; wherein the model parameters of the artificial intelligence model to be trained are obtained by the service node replicating multiple copies of all model parameters of the artificial intelligence model to be trained, modulating the replicated multiple copies of the model parameters into optical signals, and broadcasting the model parameters via optical fibers; the gradient of the model parameters is transmitted to the service node via optical fibers, so that the service node updates the model parameters of the artificial intelligence model to be trained according to the gradient of the model parameters.
[0090] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the data interaction method provided by the above-mentioned methods and applied to any service node in an optical interconnection system based on a parameter server architecture, the method comprising: copying the model parameters of the artificial intelligence model to be trained into multiple copies of the model parameters; broadcasting the optical signal modulated based on the multiple copies of the model parameters to the at least one working node; any working node, for obtaining the data to be trained, performing a training task on the received model parameters based on the data to be trained, and obtaining the gradient of the model parameters; receiving the gradient of the model parameters, and updating the model parameters of the artificial intelligence model to be trained according to the gradient of the model parameters. Alternatively, a data interaction method is executed for any working node in an optical interconnection system based on a parameter server architecture, the method comprising: receiving model parameters of an artificial intelligence model to be trained, acquiring data to be trained, performing a training task on the received model parameters based on the data to be trained, and obtaining the gradient of the model parameters; wherein the model parameters of the artificial intelligence model to be trained are obtained by the service node replicating multiple copies of all model parameters of the artificial intelligence model to be trained, modulating the replicated multiple copies of the model parameters into optical signals, and broadcasting the model parameters via optical fibers; the gradient of the model parameters is transmitted to the service node via optical fibers, so that the service node updates the model parameters of the artificial intelligence model to be trained according to the gradient of the model parameters.
[0091] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. An optical interconnection system based on a parameter server architecture, characterized in that: It includes at least one service node and multiple working nodes; wherein, The optical transmitting end of any service node is connected to the optical receiving end of any working node through an optical fiber; The optical transmitting end of any working node is connected to the optical receiving end of any service node through an optical fiber; Any of the working nodes is used to pull model parameters from any of the service nodes connected thereto; Any of the working nodes is further used to obtain training data, perform training tasks using the training data, and obtain task execution results; and send the task execution results to any of the service nodes through the optical transmitting end of any of the working nodes; Any one of the service nodes is used to update the model parameters based on the task execution result.
2. The optical interconnection system based on parameter server architecture according to claim 1, characterized in that: Any of the service nodes is provided with a service-end optical transceiver module; the service-end optical transceiver module is provided with a first optical transmission chip, an optical beam splitter and an optical transmission end of the service node; The optical output end of the first optical emission chip of any service node is connected to the optical beam splitter, and the output end of the optical beam splitter is connected to the optical emission end of the service node through an optical fiber.
3. The optical interconnection system based on parameter server architecture according to claim 2, characterized in that: When there are multiple optical beam splitters, the multiple optical beam splitters are connected in cascade.
4. The optical interconnection system based on parameter server architecture according to claim 1, characterized in that: Any of the working nodes is provided with a working end optical transceiver module, and the working end optical transceiver module is provided with a second optical transmitting chip, a second optical receiving chip, and an optical transmitting end and an optical receiving end of the working node; The second optical receiving chip is used to receive the model parameters from the optical receiving end of any working node via an optical fiber, convert the model parameters into electrical signals, and output the electrical signals to a processor on any working node, so that the processor performs the training task based on the electrical signals; The second optical transmission chip is used to receive the electrical signal obtained after the task execution result of any working node is modulated and transformed, convert the electrical signal into an optical signal, and transmit the optical signal to the optical output end of any working node through an optical fiber.
5. The optical interconnection system based on parameter server architecture according to claim 4, characterized in that: A second driving circuit board is also provided in any one of the working nodes; The second driving circuit board is used to receive electric energy from an external power source, and use the electric energy to supply power to the second light emitting chip and the second light receiving chip respectively.
6. The optical interconnection system based on parameter server architecture according to claim 1, characterized in that: A first optical transmitting chip and a first optical receiving chip are provided in any of the service nodes; The first optical receiving chip is used to receive the model parameters from the optical input port via the optical fiber, convert the model parameters into electrical signals, and output the electrical signals to the processor on the working node, so that the processor performs the training task based on the electrical signals; The first optical transmission chip is used to receive the electrical signal obtained after the task execution result on the working node is modulated and transformed, convert the electrical signal into an optical signal, and transmit the optical signal to the optical output port of the working node through the optical fiber.
7. A data interaction method, applied to any service node in the optical interconnection system based on the parameter server architecture as described in any one of claims 1 to 6, characterized in that: include: Copying the model parameters of the artificial intelligence model to be trained into multiple copies of the model parameters; broadcasting an optical signal modulated based on the multiple copies of the model parameters to the at least one working node; Any working node is used to obtain data to be trained, perform a training task on the received model parameters based on the data to be trained, and obtain the gradient of the model parameters; Receive the gradient of the model parameters, and update the model parameters of the artificial intelligence model to be trained according to the gradient of the model parameters.
8. The data interaction method according to claim 7, characterized in that: The step of copying the model parameters of the artificial intelligence model to be trained into multiple copies of the model parameters; Broadcasting the optical signal modulated based on the multiple copies of the model parameters to the at least one working node includes: In the case where there are multiple service nodes, a model parameter subset input by a user is received; wherein the model parameter subset is obtained by equally dividing the model parameters of the artificial intelligence model to be trained based on the total number of service nodes; Broadcasting an optical signal obtained by modulating the model parameters in the model parameter subset to any working node; any working node is used to pull the model parameter subsets on each service node from each service node respectively to obtain all model parameters of the artificial intelligence model to be trained; any working node is also used to obtain data to be trained, perform a training task on all the received model parameters based on the data to be trained, obtain the gradients of all the model parameters, and divide the gradients of all the model parameters into gradient subsets corresponding to each model parameter subset; The receiving the gradient of the model parameter and updating the model parameter of the artificial intelligence model to be trained according to the gradient of the model parameter includes: A gradient subset corresponding to a locally stored model parameter subset is received, and the locally stored model parameter subset is updated based on the gradient subset; and the updated model parameter subsets of all service nodes are merged as all model parameters after training.
9. A data interaction method, applied to any working node in the optical interconnection system based on the parameter server architecture as described in any one of claims 1 to 6, characterized in that: include: Receive model parameters of an artificial intelligence model to be trained, obtain data to be trained, perform a training task on the received model parameters based on the data to be trained, and obtain the gradient of the model parameters; wherein the model parameters of the artificial intelligence model to be trained are obtained by the service node replicating multiple copies of all model parameters of the artificial intelligence model to be trained, modulating the replicated multiple copies of the model parameters into optical signals, and broadcasting the replicated multiple copies of the model parameters through optical fibers; The gradient of the model parameters is transmitted to the service node via optical fiber, so that the service node updates the model parameters of the artificial intelligence model to be trained according to the gradient of the model parameters.
10. The data interaction method according to claim 9, characterized in that: The receiving model parameters of the artificial intelligence model to be trained, acquiring the data to be trained, performing a training task on the received model parameters based on the data to be trained, and obtaining the gradient of the model parameters includes: In the case where there are multiple service nodes, a subset of model parameters on each service node is pulled from each service node to obtain all model parameters of the artificial intelligence model to be trained; wherein the subset of model parameters is obtained by pre-dividing all model parameters of the artificial intelligence model to be trained based on the total number of service nodes; Acquire data to be trained, perform a training task on all the received model parameters based on the data to be trained, obtain gradients of all the model parameters, and divide the gradients of all the model parameters into gradient subsets corresponding to each model parameter subset; The step of transmitting the gradient of the model parameters to the service node via an optical fiber so that the service node updates the model parameters of the artificial intelligence model to be trained according to the gradient of the model parameters comprises: Each gradient subset is sent to a service node to which the model parameter subset corresponding to each gradient subset belongs, so that the service node updates the model parameter subset stored in the service node based on the gradient subset; and the model parameter subsets after all service nodes have been updated are merged and used as all model parameters after training.