Training system and training method of neural network model

Abstract

The application provides a neural network model training system and method, wherein the training system comprises a control and data plane, a training interconnection plane, the control and data plane comprises a master processor and at least one preprocessing processor, the training interconnection plane comprises at least one training processor, the master processor, the at least one preprocessing processor and the at least one training processor are connected through an optical communication link, in each training stage of a neural network model training process performed by the training system, the master processor, the at least one preprocessing processor and the at least one training processor perform data interaction based on a target wavelength of the optical communication link corresponding to each training stage, and the target wavelength is obtained based on bandwidth demand of each training stage and a preset communication performance optimization target. The application can improve the training efficiency of the neural network model.

Description

Technical Field

[0001] This application relates to the fields of computer technology and optical communication technology, and in particular to a training system and training method for a neural network model. Background Technology

[0002] In related technologies, multiprocessor interconnect training systems used for neural network model training suffer from drawbacks such as limited bandwidth, high transmission loss, high energy consumption, large equipment cabling overhead, and limited port expansion. Processor interconnect solutions such as PCIe (Peripheral Component Interconnect Express) and Ethernet exhibit poor bandwidth and latency performance. Optical interconnect high-performance computing systems offer advantages such as high bandwidth, low overhead, and low energy consumption, but lack specific optimization for neural network model training scenarios. Summary of the Invention

[0003] In a first aspect, this application proposes a training system for a neural network model, the training system comprising a main control processor, at least one preprocessor, and at least one training processor; the main control processor, the at least one preprocessor, and the at least one training processor are connected via an optical communication link; in each training stage of the neural network model training process executed by the training system, the main control processor, the at least one preprocessor, and the at least one training processor interact with each other based on the target wavelength of the optical communication link corresponding to each training stage; the target wavelength is obtained based on the bandwidth requirements of each training stage and a preset communication performance optimization target.

[0004] Secondly, this application proposes a training method for a neural network model. The method is applied to a main control processor in the training system described in the first aspect. The main control processor, at least one preprocessor, and at least one training processor are connected via an optical communication link. Data interaction is performed based on the target wavelength of the optical communication link corresponding to each training stage in the entire neural network model training process. The target wavelength is obtained based on the bandwidth requirements of each training stage and a preset communication performance optimization target. The method includes: the main control processor sending control data to the preprocessor and the training processor; the main control processor and the preprocessor sending graph data to the training processor based on the control data; each training processor training the neural network model based on the control data and the graph data to obtain gradient data, and sending the gradient data to the main control processor; the main control processor performing gradient aggregation and model parameter updates based on the gradient data, and sending the updated model parameters to each training processor, to iteratively perform gradient aggregation and parameter updates until a preset condition is met.

[0005] Thirdly, this application proposes an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the wavelength allocation method as described in the second aspect.

[0006] Fourthly, this application proposes a storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the second aspect.

[0007] Fifthly, this application proposes a program product comprising at least one of a program and instructions, wherein when the program or instructions are executed by an electronic device, they implement the steps of the method described in the second aspect.

[0008] The neural network model training system and method provided in this application can, during the task allocation stage of the entire neural network model training process, obtain the bandwidth requirements between the main control processor and each preprocessor and each training processor. Based on the bandwidth requirements and preset communication performance optimization objectives, a multi-objective optimization function is established, and the multi-objective optimization function is solved under preset constraints to obtain the target wavelengths between the main control processor and each preprocessor and each training processor. Wavelength allocation is then performed at each training stage based on these target wavelengths. This improves the data transmission efficiency of the training system, thereby increasing the training efficiency of the neural network model.

[0009] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0010] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a schematic diagram of the structure of a neural network model training system provided in an embodiment of this application; Figure 2 This is a schematic flowchart of a wavelength allocation method provided in an embodiment of this application; Figure 3 This is a schematic flowchart of another wavelength allocation method provided in an embodiment of this application; Figure 4 This is a flowchart illustrating another wavelength allocation method provided in the embodiments of this application; Figure 5 This is a flowchart illustrating another wavelength allocation method provided in the embodiments of this application; Figure 6 This is a flowchart illustrating another wavelength allocation method provided in the embodiments of this application; Figure 7 This is a flowchart illustrating a training method for a neural network model provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0011] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0012] The training system and training method for the neural network model of this application are described below with reference to the accompanying drawings.

[0013] Figure 1 This is a schematic diagram of the structure of a neural network model training system provided in an embodiment of this application. For example... Figure 1 As shown, the training system includes a main control processor, at least one preprocessor, and at least one training processor. The main control processor is connected to the at least one preprocessor and the at least one training processor via an optical communication link. During each training stage of the entire neural network model training process, the main control processor, the at least one preprocessor, and the at least one training processor interact with each other based on the target wavelength of the optical communication link corresponding to each training stage. The target wavelength is obtained based on the bandwidth requirements of each training stage and the preset communication performance optimization target.

[0014] It should be noted that the embodiments of this application do not limit the number of preprocessing processors and training processors. Figure 1 Taking an example consisting of M preprocessors and N training processors, where M and N are positive integers.

[0015] It should be noted that, in the embodiments of this application, the main control processor, preprocessor and training processor mentioned above can be FPGA (Field Programmable Gate Array) accelerator cards.

[0016] For example, such as Figure 1As shown, the main control processor and the preprocessor form the control and data plane, which can be used to perform master scheduling, data distribution, gradient aggregation, and parameter synchronization during the training process of the neural network. The various training processors form the training interconnect plane, which can handle point-to-point communication between training processors. The two planes are physically independent but logically coordinated, avoiding mutual interference between different communication modes. The topology of the control and data plane is a modified star structure, but wavelength allocation is optimized for different communication stages. The topology of the training interconnect plane is a reconfigurable optical ring, containing a main ring and a backup ring. The main ring connects to each training processor in a clockwise direction; the backup ring connects to each training processor in a counter-clockwise direction.

[0017] As an example, the master processor and preprocessor are used to send training data to the training processor. The master processor is also used to send control data to the preprocessor and training processor, as well as to aggregate the gradients generated by the training processor and synchronize the updated global model parameters to the training processor.

[0018] As an example, please refer to Table 1, which is an example table of communication modes and traffic characteristics at different training stages provided in the embodiments of this application.

[0019] Table 1. Examples of communication modes and traffic characteristics at different training stages.

[0020] Based on the neural network model shown in Table 1, the data senders, data receivers, and corresponding traffic characteristics at different stages of the entire process can be trained to determine the bandwidth requirements corresponding to the data transmission volume between different data senders and data receivers, and then the corresponding wavelengths can be allocated according to the bandwidth requirements between different data senders and data receivers.

[0021] In some embodiments, during the task allocation phase of the training system's entire neural network model training process, the main control processor can be used to send control data to the preprocessing processor and the training processor, and can configure corresponding wavelengths based on the bandwidth requirements between the main control processor and each preprocessing processor and each training processor. As an example, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating a wavelength allocation method provided in an embodiment of this application. This method can be applied to the training system provided in any embodiment of this application. Figure 2 As shown, the method may include, but is not limited to, the following steps: S201: Get the duration of the first phase of the task allocation phase.

[0022] For example, the duration of the first stage of the task allocation phase in the entire process of training a neural network model is obtained.

[0023] It should be noted that the task allocation phase is the initial training phase. In this phase, the main control processor is primarily responsible for sending the data storage location and training tasks to the preprocessor and training processor.

[0024] S202: Obtain the first amount of control data that the main control processor needs to transmit to each preprocessor, and obtain the second amount of control data that the main control processor needs to transmit to each training processor.

[0025] For example, the first amount of control data that the main control processor needs to send to each preprocessor during the task allocation phase is obtained, and the second amount of control data that the main control processor needs to send to each training processor during the task allocation phase is obtained.

[0026] In the embodiments of this application, the aforementioned control information may include, but is not limited to, storage location information of training data and training task information used to train the neural network model.

[0027] For example, the neural network model described above can be a GNN (Graph Neural Network).

[0028] S203: Based on the first data volume and first stage duration corresponding to each preprocessor, obtain the bandwidth requirements between the main control processor and each preprocessor.

[0029] For example, for each preprocessor, the bandwidth requirement between the main control processor and the preprocessor is obtained based on the first data volume and the first stage duration corresponding to the preprocessor, and the bandwidth requirement is sufficient to transmit the first data volume within the first stage duration.

[0030] As an example, the bandwidth requirement mentioned above can be represented as follows:

[0031] in, This refers to the bandwidth requirement between the main control processor and the j-th preprocessor. The duration of the first phase of the task allocation phase, This refers to the amount of data that the main control processor needs to transmit to the j-th preprocessor. This is the amount of data that the main control processor needs to transmit to the j-th training processor. The total number of main control processors and preprocessors. The total number of training processors, The duration of data transmission from the main control processor to the j-th preprocessor. It is the transmission time of the main control processor transmitting data to the j-th training processor.

[0032] S204: Based on the second data volume and the first stage duration corresponding to each training processor, obtain the bandwidth requirements between the main control processor and each training processor.

[0033] For example, for each training processor, based on the second data volume corresponding to the training processor and the first stage duration, the bandwidth requirement between the main control processor and the training processor is obtained, and the bandwidth requirement is sufficient to transmit the second data volume within the first stage duration.

[0034] S205: Obtain the target wavelength based on bandwidth requirements and preset communication performance optimization objectives.

[0035] For example, the single-wavelength carrying capacity and number of wavelengths are determined based on bandwidth requirements. Combined with preset communication performance optimization targets, wavelengths that meet transmission indicators are selected within the available bands and channel allocation is completed to obtain the target wavelength.

[0036] By implementing the embodiments of this application, during the task allocation stage of the entire neural network model training process in the training system, the bandwidth requirements between the main control processor and each preprocessor and each training processor can be obtained. A multi-objective optimization function is then established based on the bandwidth requirements and preset communication performance optimization objectives. This function is solved under preset constraints to obtain the target wavelengths between the main control processor and each preprocessor and each training processor. Wavelength allocation is then performed during the task allocation stage based on these target wavelengths. This improves the data transmission efficiency of the training system during the model training stage, thereby increasing the training efficiency of the neural network model.

[0037] In some embodiments, during the data distribution phase of the entire neural network model training process executed by the training system, the main control processor and the preprocessor are used to send graph data to the training processors. The corresponding wavelengths can be configured based on the bandwidth requirements between the main control processor and each preprocessor, and among each training processor. As an example, please refer to [link to example]. Figure 3 , Figure 3 This is a flowchart illustrating another wavelength allocation method provided in an embodiment of this application, which can be applied to the training system provided in this application. Figure 3 As shown, the method may include, but is not limited to, the following steps: S301: Obtain the duration of the second phase of the data distribution phase.

[0038] For example, the duration of the second phase of the data distribution stage in the entire process of training a neural network model is obtained.

[0039] It should be noted that, in the embodiments of this application, the data transmitted during the data distribution stage is training data used to train the neural network model.

[0040] For example, the data transmitted during the data distribution phase is graph data used to train the graph neural network, and the main control processor and each preprocessor are equipped with at least one storage device, each storage device storing a portion of the graph data.

[0041] Understandably, during the data distribution phase, the data senders are the main control processor and the preprocessor, and the data receivers are the training processors. In this phase, the main control processor and the preprocessor execute graph partitioning algorithms in parallel on the graph data stored on their mounted storage devices, dividing it into multiple subgraph data. The number of subgraph data is the same as the number of training processors. Then, each subgraph data is sent to the corresponding training processor. In this phase, the main control processor also needs to send model parameters to each training processor.

[0042] S302: For the main controller and each preprocessor, obtain the storage overhead of the topology of the partial graph data stored in each storage device of the main controller or preprocessor.

[0043] For example, for the main control processor and each preprocessor, the storage overhead of the topology and vertex features of the partial graph data stored in the respective storage devices of the processor is obtained.

[0044] S303: For the main control processor and each preprocessor, based on the storage overhead and vertex features corresponding to the main control processor or preprocessor, combined with the preset topology compression rate, preset feature compression rate, second stage duration and the number of processors in the training system, obtain the bandwidth requirements between the main control processor or preprocessor and each training processor.

[0045] For example, the main control processor and each preprocessor first calculate the total amount of data of the topology and vertex features in the graph data stored by themselves after compression at the corresponding compression rate, and divide it equally according to the number of training processors. After adding the amount of data of model parameters, the bandwidth requirement between the main control processor and each preprocessor and each training processor can be obtained.

[0046] As an example, the bandwidth requirement mentioned above can be represented as follows:

[0047] in, This refers to the bandwidth requirement between the main control processor or the i-th preprocessor and the j-th training processor. This indicates the number of bytes stored on the main control processor or preprocessor. Partial graph data on a storage device This refers to the space occupied by the topological structure of a portion of the graph data. For topology compression ratio, Vertex features representing the topological structure of partial graph data. For feature compression ratio, The number of processors in the training system. These are the model parameters for the neural network model. This refers to the duration of the data distribution phase.

[0048] In some embodiments, the above graph data is compressed using gradient compression.

[0049] For example, the above graph data was subjected to gradient compression using the TopK sparsity method. Let the original gradient vector be defined as... The TopK sparsity method can then be expressed as:

[0050] Where |*| represents the absolute value operation. Represents the largest absolute value A collection of indices for each element. (Introduction) The sparsity ratio characterizes the degree of sparsity of the gradient after sparsification, and can be expressed as:

[0051] in, The sparsity ratio described above represents the proportion of non-zero elements in a vector or matrix. The vertex feature dimension.

[0052] S304: Obtain the target wavelength based on bandwidth requirements and preset communication performance optimization goals.

[0053] In the embodiments of this application, step S304 can be implemented in any of the embodiments of this application. The embodiments of this application do not limit this and will not be described in detail.

[0054] By implementing the embodiments of this application, during the data distribution phase of the entire neural network model training process in the training system, the bandwidth requirements between the main control processor, each preprocessor, and each training processor can be obtained. A multi-objective optimization function is then established based on the bandwidth requirements and preset communication performance optimization objectives. This function is solved under preset constraints to obtain the target wavelengths between the main control processor and each preprocessor and training processor. Wavelength allocation is then performed during the data distribution phase based on these target wavelengths. This improves the data transmission efficiency of the training system during the data distribution phase, thereby enhancing the training efficiency of the neural network model.

[0055] In some embodiments, during the parallel training phase of the entire neural network model training process executed by the training system, each training processor is used for training the neural network model and exchanging intermediate data. The corresponding wavelength can be configured based on the bandwidth requirements between different training processors. For an example, please refer to [link to example]. Figure 4 , Figure 4 This is a flowchart illustrating another wavelength allocation method provided in an embodiment of this application, which can be applied to the training system provided in this application. Figure 4 As shown, the method may include, but is not limited to, the following steps: S401: Get the duration of the third stage of the parallel training phase.

[0056] It should be noted that the parallel training phase is the stage in which each training processor exchanges intermediate data across subgraphs point by point after obtaining training data and before gradient aggregation in order to complete forward or backward propagation.

[0057] S402: Obtain the third amount of intermediate data required by each training processor.

[0058] It should be noted that during the parallel training phase, intermediate data exchange (e.g., intermediate activations, feature exchange, message exchange) across subgraphs is required between different training processors to complete forward or backward propagation. This phase is characterized by many-to-many and point-to-point communication, with data not passing through the main control processor or preprocessor. Furthermore, different training processors typically need to communicate multiple times according to the training iterations.

[0059] S403: Obtain the bandwidth requirement for each training processor based on the third data volume and the third stage duration for each training processor.

[0060] As an example, the bandwidth requirement mentioned above can be represented as follows:

[0061] in, For the bandwidth requirement corresponding to the i-th training processor, The amount of data that the i-th training processor needs to send. This refers to the duration of the third stage of the parallel training phase.

[0062] As an example, the total amount of data sent from the i-th training processor to the j-th training processor in one training iteration is denoted as . For each layer of the GNN network Cross-subgraph cut edges from training the i-th training processor to the j-th training processor Suppose that the edge needs to transmit data at this layer. Each element bytes, and this layer may occur in one iteration. Such a transmission, then It can be calculated using the following formula:

[0063] Assuming the edges are undirected and the communication pattern is measured in units of undirected edges, the total number of bytes that the i-th training processor needs to send or receive with all other training processors during the parallel training phase can be represented as follows:

[0064] S404: Obtain the target wavelength based on bandwidth requirements and preset communication performance optimization goals.

[0065] By implementing the embodiments of this application, the bandwidth requirements between different training processors can be obtained during the parallel training phase of the entire neural network model training process in the training system. A multi-objective optimization function is then established based on the bandwidth requirements and preset communication performance optimization objectives. This multi-objective optimization function is solved under preset constraints to obtain the target wavelengths between different training processors. Wavelength allocation is then performed during the parallel training phase based on these target wavelengths. This improves the data transmission efficiency of the training system during the parallel training phase, thereby enhancing the training efficiency of the neural network model.

[0066] In some embodiments, during the gradient aggregation phase of the entire neural network model training process executed by the training system, the training processor can be used to send gradient data to the main control processor, and the corresponding wavelength can be configured based on the bandwidth requirements between the main control processor and each training processor. As an example, please refer to [link to example]. Figure 5 , Figure 5 This is a flowchart illustrating another wavelength allocation method provided in an embodiment of this application, which can be applied to the training system provided in this application. Figure 5 As shown, the method may include, but is not limited to, the following steps: S501: Obtain the duration of the fourth stage of the gradient aggregation phase.

[0067] It should be noted that during the gradient aggregation phase, each training processor needs to aggregate the calculated gradients to the master processor for global optimization. In this phase, the data senders are the individual training processors, and the data receivers are the master processor.

[0068] S502: Obtain the quantized compression ratio and sparsity ratio of gradient compression.

[0069] S503: Obtain the fourth data volume of gradient data that each training processor needs to transmit to the main control processor.

[0070] S504: Based on the fourth stage duration, quantization compression rate, sparsity rate, and the fourth data volume corresponding to each training processor, obtain the bandwidth requirements corresponding to each training processor.

[0071] For example, the bandwidth requirement for each training processor can be obtained by multiplying the fourth data volume corresponding to each training processor by the quantization compression rate and the sparsity rate, and then dividing by the duration of the fourth stage.

[0072] As an example, the bandwidth requirement mentioned above can be represented as follows:

[0073] in, The bandwidth requirement between the j-th pre-trained processor and the main processor is... Let be the amount of gradient information data that the j-th pre-trained processor needs to transmit to the main processor. The quantized compression ratio for gradient compression. The sparsity rate of gradient compression, This refers to the duration of the fourth stage of the gradient aggregation phase.

[0074] S505: Obtain the target wavelength based on bandwidth requirements and preset communication performance optimization goals.

[0075] By implementing the embodiments of this application, the bandwidth requirements between the main control processor and each training processor can be obtained during the gradient aggregation stage of the entire neural network model training process in the training system. A multi-objective optimization function is then established based on the bandwidth requirements and preset communication performance optimization objectives. This multi-objective optimization function is solved under preset constraints to obtain the target wavelength between the main control processor and each training processor. Wavelength allocation is then performed during the gradient aggregation stage based on these target wavelengths. This improves the data transmission efficiency in the training system, thereby enhancing the training efficiency of the neural network model.

[0076] It should be noted that the entire process of training a neural network model also includes a parameter synchronization phase. During this phase, the main control processor needs to synchronize the updated model parameters to all training processors. The specific implementation method for obtaining the target wavelength between the main control processor and each training processor in this phase can be the same as in the task allocation phase.

[0077] As an example, the bandwidth requirements between the master processor and each training processor during the parameter synchronization phase can be represented as follows:

[0078] in, This represents the bandwidth requirement between the main control processor and the i-th training processor. Indicates the update frequency of model parameters. The amount of data representing the model parameters. This indicates the duration of the parameter synchronization phase.

[0079] It is understandable that different stages, such as task allocation, data distribution, parallel training, gradient aggregation, and parameter synchronization, have different communication modes and bandwidth requirements. The wavelength allocation method provided in this application can dynamically adjust optical network resources based on these periodically changing communication modes. During data distribution, it provides more downlink bandwidth to the main control processor and preprocessor; during gradient aggregation, it provides more uplink bandwidth; and during parallel training and parameter synchronization, it optimizes broadcast efficiency. This avoids communication bottlenecks and fully leverages the advantages of multi-processor parallel training.

[0080] In one implementation, a corresponding optimization objective sub-function can be established for each preset communication performance optimization objective, thereby creating a multi-objective optimization function based on multiple optimization objective sub-functions. For an example, please refer to [link to example]. Figure 6 , Figure 6 This is a flowchart illustrating another wavelength allocation method provided in the embodiments of this application, as shown below. Figure 6 As shown, the method may include, but is not limited to, the following steps: S601: Obtain the bandwidth requirements for each training stage.

[0081] S602: Establish a multi-objective optimization function based on bandwidth requirements and communication performance optimization objectives.

[0082] For example, based on the bandwidth requirements of any training stage, corresponding weight coefficients are configured for at least one preset communication performance optimization objective, and the at least one communication performance optimization objective is weighted and combined to establish the multi-objective optimization function corresponding to the training stage.

[0083] In some embodiments, a communication performance optimization objective sub-function corresponding to a preset communication performance optimization objective can be established based on bandwidth requirements, so as to establish a multi-objective optimization function based on bandwidth requirements and the communication performance optimization objective sub-function.

[0084] The communication performance optimization objective sub-function includes at least one of the following: communication delay optimization objective sub-function, wavelength utilization optimization objective sub-function, and load balancing optimization objective sub-function.

[0085] As an example, the objective function for optimizing communication latency during the parallel training phase can be expressed as follows:

[0086] in, Represents the goal of communication delay optimization. Let i be the bandwidth requirement between the i-th training processor and the j-th training processor. This represents the wavelength allocation state between the i-th and j-th training processors, where M is the total number of training servers. For the bandwidth capacity of a single wavelength, This is a smoothing factor.

[0087] As an example, the wavelength utilization optimization objective function in the parallel training phase can be expressed as follows:

[0088] in, Represents the target of wavelength utilization optimization. Represents the set of available wavelengths. It is the time difference between the i-th training processor and the j-th training processor at time i. Communication load, For wavelength allocation status, This refers to the bandwidth capacity of a single wavelength.

[0089] As an example, the load balancing optimization objective function in the parallel training phase can be represented as follows:

[0090] in, Represents the load balancing optimization goal. It represents the bandwidth requirement between the i-th training processor and the j-th training processor. The wavelength allocation state between the i-th training processor and the j-th training processor.

[0091] For example, the above multi-objective optimization function can be expressed as follows:

[0092] in, , , These are preset weight values.

[0093] It should be noted that the weight values ​​for different training stages can be the same or different.

[0094] In some embodiments, for any training phase, establishing a multi-objective optimization function based on the communication performance optimization objective sub-function and a preset weight allocation strategy may include the following steps: A1: Obtain the historical bandwidth requirements for each training stage.

[0095] A2: Match the bandwidth demand with historical bandwidth demand to determine the target historical bandwidth demand that corresponds to the current bandwidth demand.

[0096] For example, based on the bandwidth requirement, the characteristic information such as the data sender, data receiver, communication mode, and traffic volume corresponding to the bandwidth requirement is determined. The above characteristic information is compared and matched with the characteristic information of historical bandwidth requirements, and the historical bandwidth requirements with the same or the most similar characteristics are selected as the target historical bandwidth requirements.

[0097] A3: Determine the target training phase corresponding to the target historical bandwidth requirement during the training phase.

[0098] A4: Obtain the preset weight parameters corresponding to the target training phase.

[0099] For example, corresponding weight parameters can be pre-configured for different training stages to obtain the preset weight parameters corresponding to the target training stage.

[0100] A5: Establish a multi-objective optimization function based on the communication performance optimization objective sub-function and preset weight parameters.

[0101] S603: Solve the multi-objective optimization function under preset constraints to obtain the target wavelength.

[0102] For example, under preset constraints, the multi-objective optimization function corresponding to any training stage is solved to obtain the target wavelength corresponding to that training stage.

[0103] In the embodiments of this application, the above-mentioned constraints include at least one of the following: optical communication physical constraints, bandwidth requirement constraints, and processor optical interface constraints.

[0104] Among them, the physical constraint of optical communication means that a wavelength can only carry one signal at a time and cannot be allocated to multiple links at the same time.

[0105] As an example, the physical constraints of optical communication during the parallel training phase can be expressed as:

[0106] in, The wavelength allocation state between the i-th training processor and the j-th training processor.

[0107] Among them, bandwidth requirement constraint means that the bandwidth allocated to each link can meet the actual data transmission requirements.

[0108] As an example, the bandwidth requirement constraint during the parallel training phase can be expressed as:

[0109] in, The wavelength allocation state between the i-th training processor and the j-th training processor. For the bandwidth capacity of a single wavelength, This represents the bandwidth requirement between the i-th training processor and the j-th training processor.

[0110] Among them, the processor optical interface constraint means that the number of wavelength channels that each processor's optical transceiver module can process at the same time is less than or equal to the maximum number of wavelength channels that the processor can process at the same time.

[0111] As an example, the processor optical interface constraint during the parallel training phase can be expressed as:

[0112] in, Indicates the first The optical interface capacity or optical transceiver channel limit of a processor, that is, the maximum capacity of the processor at any given time. The maximum number of optical signal wavelength channels or concurrent light-emitting links that can be supported simultaneously.

[0113] In an alternative implementation, the process of solving the multi-objective optimization function under preset constraints to obtain the target wavelength may include the following steps: B1: Obtain available wavelengths.

[0114] B2: Based on bandwidth requirements, available wavelengths, and constraints, a greedy allocation strategy is used to solve the multi-objective optimization function to obtain the initial wavelength allocation scheme.

[0115] For example, based on the bandwidth requirements of different processors, available wavelength resources, and the aforementioned constraints, a greedy allocation strategy is adopted to solve the multi-objective optimization function that takes into account communication delay, wavelength utilization, and load balancing, thereby obtaining the initial wavelength allocation scheme.

[0116] B3: Optimize the initial wavelength allocation scheme to obtain the target wavelength allocation scheme.

[0117] The initial wavelength allocation scheme is first fine-tuned locally and the cost is evaluated. If reinforcement learning optimization is enabled and the optimization conditions are met, reinforcement learning is used to further optimize the scheme and select the scheme with the lower cost as the target wavelength allocation scheme.

[0118] In some embodiments, the above-mentioned optimization of the initial wavelength allocation scheme to obtain the target wavelength allocation scheme includes: performing local optimization on the initial wavelength allocation scheme to obtain a first wavelength allocation scheme and a corresponding first-generation value; performing reinforcement learning optimization on the first wavelength allocation scheme to obtain a second wavelength allocation scheme and a corresponding second-generation value; if the first-generation value is less than the second-generation value, then the first wavelength allocation scheme is taken as the target wavelength allocation scheme; if the first-generation value is greater than or equal to the second-generation value, then the second wavelength allocation scheme is taken as the target wavelength allocation scheme.

[0119] B4: Obtain the target wavelength based on the target wavelength allocation scheme.

[0120] By implementing the embodiments of this application, sub-optimization functions corresponding to different communication optimization sub-objectives can be constructed during the parallel training phase of the entire neural network model training process in the training system. Based on these sub-optimization functions, a multi-objective optimization function can be constructed and solved under preset constraints to obtain the target wavelengths for different training processors. This allows for targeted adjustment of the wavelengths at different training stages, further improving the training efficiency of the neural network model.

[0121] Please see Figure 7 , Figure 7 This is a flowchart illustrating a training method for a neural network model provided in an embodiment of this application. This method can be applied to the main control processor in the training system of the neural network model provided in any embodiment of this application. Figure 7 As shown, the method may include, but is not limited to, the following steps: S701: Sends control data to the preprocessor and training processor.

[0122] The main control processor, at least one preprocessor, and at least one training processor are connected via an optical communication link. They interact with each other based on the target wavelength of the optical communication link corresponding to each training stage in the entire neural network model training process. The target wavelength is obtained based on the bandwidth requirements of each training stage and the preset communication performance optimization target.

[0123] S702: Based on the control data and the preprocessor, the initial model parameters and training data are sent to the training processor in collaboration, so that the training processor can train the neural network model based on the training data and the initial model parameters to obtain gradient data, and then send the gradient data to the main control processor.

[0124] In some embodiments, the training data described above is graph data.

[0125] S703: Performs gradient aggregation and model parameter updates based on gradient data, and sends the updated model parameters to each training processor to perform gradient aggregation and parameter updates in a loop until preset conditions are met.

[0126] For example, the master processor collects and summarizes the gradient data calculated by each training processor, optimizes and updates the model parameters based on these gradients, and then distributes the updated model to all training processors. The training processors continue to train based on the new model parameters, and the above process is repeated until the model training reaches the expected convergence effect or accuracy requirement.

[0127] By implementing the embodiments of this application, model training can be performed based on the neural network model training system provided in the embodiments of this application. At each stage of model training, a corresponding target wavelength can be obtained based on bandwidth requirements and preset communication performance optimization objectives, enabling data interaction based on the target wavelength. This improves the training efficiency of the neural network model.

[0128] To implement the above embodiments, this application also proposes an electronic device. Please see [link to relevant documentation]. Figure 8 , Figure 8 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 8 As shown, the electronic device 800 includes: a processor 801, and a memory 802 communicatively connected to the processor 801; the memory 802 stores computer execution instructions; the processor 801 executes the computer execution instructions stored in the memory to implement the method provided in the foregoing embodiments.

[0129] In some embodiments, the electronic device described above can be any processor in the training system. For example, the electronic device can be the main control processor in the training system.

[0130] To implement the above embodiments, this application also proposes a storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the methods provided in the foregoing embodiments.

[0131] To implement the above embodiments, this application also proposes a program product, including at least one of a program and instructions, wherein when the program and instructions are executed by an electronic device, they implement the steps of the method provided in the foregoing embodiments.

[0132] It should be noted that the acquisition, transmission, storage, use, and processing of data in this application comply with the relevant provisions of national laws and regulations and do not violate public order and good morals.

[0133] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0134] It is worth noting that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used the solution.

[0135] In the description of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0136] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0137] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0138] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0139] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0140] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0141] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0142] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0143] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A training system for a neural network model, characterized in that, The training system includes a main control processor, at least one preprocessor, and at least one training processor. The main control processor, the at least one preprocessor, and the at least one training processor are connected via an optical communication link; In each training stage of the entire neural network model training process executed by the processor network, the main control processor, the at least one preprocessor, and the at least one training processor interact with each other based on the target wavelength of the optical communication link corresponding to each training stage. The target wavelength is obtained based on the bandwidth requirements of each training stage and the preset communication performance optimization target.

2. The training system according to claim 1, characterized in that, During the task allocation phase of the training system's entire neural network model training process, the main control processor sends control data to the preprocessor and the training processor. The bandwidth requirement is obtained through the following steps: Obtain the duration of the first phase of the task allocation phase; Obtain a first amount of control data that the main control processor needs to transmit to each of the preprocessing processors, and obtain a second amount of control data that the main control processor needs to transmit to each of the training processors; Based on the first data volume and the first stage duration corresponding to each of the preprocessors, the bandwidth requirement between the main control processor and each of the preprocessors is obtained. Based on the second data volume corresponding to each of the training processors and the duration of the first stage, the bandwidth requirement between the main control processor and each of the training processors is obtained.

3. The training system according to claim 1, characterized in that, The main control processor and each of the preprocessors are equipped with at least one storage device. Each storage device stores a portion of the graph data used for neural network model training. During the data distribution phase of the entire neural network model training process executed by the training system, the main control processor and the preprocessors are used to send the graph data to the training processor. The bandwidth requirement is obtained through the following steps: Obtain the duration of the second phase of the data distribution phase; For the main control processor and each of the preprocessors, obtain the storage overhead of the topology and vertex features of a portion of the graph data stored in the storage devices of the main control processor or each of the preprocessors; For the main control processor and each of the preprocessors, based on the storage overhead and vertex features corresponding to the main control processor or the preprocessor, combined with the preset topology compression rate, the preset feature compression rate, the second stage duration and the number of processors in the training system, the bandwidth requirements between the main control processor, each of the preprocessors and each of the training processors are obtained.

4. The training system according to claim 1, characterized in that, During the parallel training phase of the training system, which executes the entire neural network model training process, each training processor is used for training the neural network model and exchanging intermediate data. The bandwidth requirement is obtained through the following steps: Obtain the duration of the third stage of the parallel training phase; Obtain the third amount of intermediate data required to be sent by each of the training processors; The bandwidth requirement for each training processor is obtained based on the third data volume and the third stage duration for each training processor.

5. The training system according to claim 1, characterized in that, During the gradient aggregation phase of the entire neural network model training process in the training system, the training processor sends gradient data to the main control processor. The bandwidth requirement is obtained through the following steps: Obtain the duration of the fourth stage of the gradient aggregation phase; Obtain the quantized compression ratio and sparsity ratio of gradient compression; Obtain the fourth data volume of gradient data that each of the training processors needs to transmit to the main control processor; Based on the duration of the fourth stage, the quantization compression rate, the sparsity rate, and the fourth data volume corresponding to each training processor, the bandwidth requirement corresponding to each training processor is obtained.

6. The training system according to claim 1, characterized in that, The target wavelength is obtained based on the bandwidth requirements of each training stage and the preset communication performance optimization target, including: A multi-objective optimization function is established based on the bandwidth requirements and the communication performance optimization objectives. The target wavelength is obtained by solving the multi-objective optimization function under preset constraints.

7. The training system according to claim 6, characterized in that, The communication performance optimization objectives include at least one of communication latency, wavelength utilization, and load balancing. The establishment of a multi-objective optimization function based on the bandwidth requirement and the preset communication performance optimization objectives includes: Based on the bandwidth requirements, establish a communication performance optimization target sub-function corresponding to the preset communication performance optimization target; Based on the communication performance optimization objective sub-function and the preset weight allocation strategy, the multi-objective optimization function is established.

8. The training system according to claim 7, characterized in that, The multi-objective optimization function is established based on the communication performance optimization objective sub-function and the preset weight allocation strategy, including: Obtain the historical bandwidth requirements corresponding to each training stage; Based on the bandwidth requirement and the historical bandwidth requirement, a target historical bandwidth requirement corresponding to the bandwidth requirement is determined from the historical bandwidth requirements. Determine the target training phase corresponding to the target historical bandwidth requirement in the training phase; Obtain the preset weight parameters corresponding to the target training phase; The multi-objective optimization function is established based on the communication performance optimization objective sub-function and the preset weight parameters.

9. The training system according to claim 7, characterized in that, The training system is used for the parallel training phase of the entire neural network model training process. Based on the bandwidth requirements, a communication performance optimization objective sub-function is established corresponding to the preset communication performance optimization objective, including: Based on the bandwidth requirements, obtain the target bandwidth requirements between every two training processors; The communication performance optimization objective subfunction is established based on the bandwidth requirements between every two training processors and the bandwidth capacity of a single wavelength.

10. The training system according to claim 7, characterized in that, The step of solving the multi-objective optimization function under preset constraints to obtain the target wavelength includes: Obtain available wavelengths; Based on the bandwidth requirements, the available wavelengths, and the constraints, a greedy allocation strategy is used to solve the multi-objective optimization function to obtain an initial wavelength allocation scheme. The initial wavelength allocation scheme is optimized to obtain the target wavelength allocation scheme; The target wavelength is obtained based on the target wavelength allocation scheme.

11. A method for training a neural network model, characterized in that, The method is applied to the main control processor in the training system as described in any one of claims 1-10, wherein the main control processor, at least one preprocessor, and at least one training processor are connected via an optical communication link, and data interaction is performed based on the target wavelength of the optical communication link corresponding to each training stage in the entire neural network model training process. The target wavelength is obtained based on the bandwidth requirements of each training stage and a preset communication performance optimization target. The method includes: Send control data to the preprocessor and the training processor; Based on the control data, the preprocessor and the preprocessor work together to send initial model parameters and training data to the training processor, so that the training processor can train the neural network model based on the training data and the initial model parameters to obtain gradient data, and send the gradient data to the main control processor. Gradient aggregation and model parameter updates are performed based on the gradient data, and the updated model parameters are sent to each of the training processors to perform gradient aggregation and parameter updates cyclically until preset conditions are met.

12. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in claim 11.

Images