A data processing method and device

Abstract

This specification discloses a data processing method and device. The method includes: any computing node in the collective communication cluster receives the first data slice sent by the previous computing node, performs a computing task based on the received first data slice, and splits the task data to be sent that is independent of the computing task into multiple second data slices that have no dependencies; sends each second data slice to the next computing node in sequence, so that the next computing node receives subsequent second data slices while performing the computing task based on the received second data slice; after performing the computing task based on all the task data in the computing node, the target computing result is obtained. This solution effectively improves the overall efficiency of data transmission and calculation of the collective communication cluster.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more particularly to a data processing method and device. Background Art

[0002] As a key communication mode, collective communication allows multiple computing nodes in a cluster (such as graphics processing units (GPUs)) to work together efficiently to complete complex computing tasks. It is widely used in many fields such as distributed model training, data analysis, and scientific computing.

[0003] In the process of each computing node executing a computing task, it usually involves two parts: data transmission between nodes and data protocol calculation. For each computing node, the computing node needs to receive the data transmitted by the previous node, and then perform protocol calculation based on the data and local data.

[0004] However, when executing computing tasks, computing nodes not only need local data, but also the data transmitted by the previous node. This means that the computing nodes need to wait for the previous computing node to transmit data before they can perform reduced calculations based on the data transmitted by the previous node and the local data. The time each computing node waits for data transmission before calculation seriously affects the computing efficiency of the entire cluster. Summary of the invention

[0005] This specification provides a data processing method and device to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides a data processing method, which is applied to any computing node in a collective communication cluster, where the computing nodes in the collective communication cluster form a ring topology and a plurality of task data are stored in the computing node, and the method includes:

[0008] Receive a first data slice sent by a previous computing node, execute a computing task according to the received first data slice, and split task data to be sent that is independent of the computing task into multiple second data slices that have no dependency relationship;

[0009] Sending each second data slice in sequence to a next computing node, so that the next computing node receives subsequent second data slices while performing a computing task according to the received second data slices;

[0010] After executing the computing task according to all the task data in the computing node, the target computing result is obtained.

[0011] Optionally, before splitting the task data to be sent that is independent of the computing task into a plurality of second data slices that have no dependency relationship, the method further includes:

[0012] Determine the execution time required for a reference computing node to execute a computing task; the reference computing node is any computing node in the collective communication cluster;

[0013] The amount of slice data used for data splitting is determined based on the execution time required for the reference computing node to execute the computing task.

[0014] Optionally, determining the execution time required for the reference computing node to execute the computing task specifically includes:

[0015] Performing multiple rounds of sampling on the reference computing node to determine the sampled data slices that participate in the reference computing node executing the computing task in different sampling rounds;

[0016] Determine the execution time required for the reference computing node to execute the computing task according to each sampled data slice.

[0017] Optionally, determining the amount of slice data for data splitting according to the execution time required for the reference computing node to execute the computing task specifically includes:

[0018] Determine whether the number of sampling rounds for which the execution time required for the reference computing node to perform the computing task according to each sampled data slice is longer than the specified time exceeds a preset number;

[0019] If so, the amount of slice data currently used for data splitting is adjusted.

[0020] Optionally, determining the execution time required for the reference computing node to execute the computing task according to each sampled data slice specifically includes:

[0021] For each sampled data slice, determine the computation duration for the reference computing node to perform a computation task according to the sampled data slice, the computation scheduling duration for scheduling computation resources required to perform the computation task according to the sampled data slice, the transmission scheduling duration for a previous computing node of the reference computing node to schedule transmission resources, and the transmission duration for a previous computing node of the reference computing node to transmit the sampled data slice to the reference computing node;

[0022] Determine the execution time required for the reference computing node to perform the computing task according to the sampled data slice based on the computing time, the computing scheduling time, the transmission time and the transmission scheduling time.

[0023] Optionally, the amount of slice data currently used for data splitting is adjusted, including:

[0024] If the calculation duration when executing the calculation task based on at least part of the sampled data slices is longer than the transmission duration, the amount of slice data currently used for data splitting is adjusted with the goal of minimizing the deviation between the calculation scheduling duration, the transmission duration, and the sum of the transmission scheduling duration when executing the calculation task based on each data slice and the calculation time.

[0025] Optionally, the amount of slice data currently used for data splitting is adjusted, including:

[0026] If the calculation time when executing the calculation task based on at least part of the sampled data slices is less than the transmission time, the amount of slice data currently used for data splitting is adjusted with the goal of minimizing the deviation between the sum of the calculation scheduling time, the calculation time, and the transmission scheduling time when executing the calculation task based on each data slice and the transmission time.

[0027] This specification provides a data processing device, including:

[0028] A splitting module, used to receive a first data slice sent by a previous computing node, execute a computing task according to the received first data slice, and split the task data to be sent independent of the computing task into a plurality of second data slices that have no dependency relationship;

[0029] A sending module, used to send each second data slice in sequence to a next computing node, so that the next computing node receives subsequent second data slices while performing a computing task according to the received second data slices;

[0030] The determination module is used to obtain a target calculation result after executing the calculation task according to all the task data in the calculation node.

[0031] The present specification provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the computing task execution method applied to collective communication is implemented.

[0032] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the above-mentioned computing task execution method applied to collective communication is implemented.

[0033] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0034] In the computing task execution method applied to collective communication provided in the present specification, for any computing node in the collective communication cluster, a first data slice sent by a previous computing node is received, a computing task is executed according to the received first data slice, and the task data to be sent that is independent of the computing task is split into multiple second data slices that have no dependency relationship; each second data slice is sent to the next computing node in turn, so that the next computing node receives subsequent second data slices while executing the computing task according to the received second data slice; after executing the computing task according to all the task data in the computing node, the target computing result is obtained.

[0035] It can be seen from the above method that for any computing node in this scheme, the computing node can split a piece of data to be sent into multiple data slices, and send the data slices to the next computing node in sequence. In this way, when the next computing node receives the first data slice, it can receive the second data slice while calculating the first data slice, thereby realizing the synchronization of transmission and calculation process. Compared with the method of performing calculation after transmitting a whole piece of data, this scheme can synchronously execute the calculation and transmission process after receiving part of the task data, which greatly shortens the time used for the whole process and effectively improves the overall efficiency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0037] Figure 1 A schematic diagram of a data transmission method of an existing collective communication cluster provided in this specification;

[0038] Figure 2 A schematic diagram of the relationship between various stages in the execution process of a computing task provided in this specification;

[0039] Figure 3 A flowchart of a data processing method provided in this specification;

[0040] Figure 4 A schematic diagram of a computing task execution method based on data splitting provided in this specification;

[0041] Figure 5 This is a schematic diagram of the corresponding relationship between various task stages after data splitting provided in this specification;

[0042] Figure 6 A schematic diagram of a data processing device provided in this specification;

[0043] Figure 7 A method corresponding to the Figure 4 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0045] In the process of data transmission by the current collective communication cluster, the calculation process and the transmission process are often not performed synchronously. For ease of understanding, this specification provides a schematic diagram of the data transmission method of the existing collective communication cluster, such as Figure 1 shown.

[0046] Taking All-Reduce as an example, in the scatter-reduce stage, each GPU will transfer 1 / N of the data (GPU0, a0) to the next GPU and perform reduction calculation (a0 + b0) on the data at the corresponding position. In this process, for the two data blocks a0 and b0, there is a data dependency between the transmission and calculation.

[0047] The relationship between the various stages in the execution of the computing task is as follows: Figure 2 shown.

[0048] from Figure 2 It can be seen that the calculation can only start after the transmission is completely completed, which seriously affects the execution efficiency of the entire collective communication cluster.

[0049] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0050] Figure 3 A data processing method provided in this specification is a flowchart, comprising the following steps:

[0051] S301: Receive a first data slice sent by a previous computing node, execute a computing task according to the received first data slice, and split task data to be sent that is independent of the computing task into multiple second data slices that have no dependency relationship.

[0052] For each computing node in the collective communication set, the computing node may contain multiple task data that have no dependencies. For any of the computing nodes, the computing node may send the task data to the next computing node in each computing round, and receive the task data sent by the previous computing node, and perform reduced calculations (such as sum, product, maximum value, minimum value, etc.) based on the received task data and the local task data, until each computing node obtains the reduced calculation results of all task data in all computing nodes, and the computing task is completed.

[0053] The computing nodes in the collective communication cluster form a ring topology, and each computing node can transmit data through the ring topology. Each computing node performs a reduction calculation on the received data and its own local data and then transmits it to the next node.

[0054] Taking the All-Reduce collective communication operation as an example, data is distributed and aggregated among all computing nodes. Each process starts with a set of data elements, and the result of the All-Reduce operation is the aggregation of all these data elements, and finally each computing node obtains this aggregate result.

[0055] In actual applications, the above-mentioned task data can be task data in different distributed computing task scenarios, such as transaction data and risk control data in financial scenarios, medical data in medical scenarios, and sample data, feature data or loss values ​​in distributed training scenarios, etc. Of course, it can also be task data in other scenarios, and this manual does not make specific limitations on this.

[0056] For the first round of data transmission in the collective communication group, each computing node can split each task data into multiple data slices without dependencies based on the initial splitting strategy, wherein the splitting strategy includes the amount of slice data used for data splitting, which is used to characterize the number of slice data obtained after splitting the task data, or the data size of each slice data. Under the initial splitting strategy, the task data can be evenly split into several sub-data of equal size. For ease of understanding, this specification provides a schematic diagram of a computing task execution method based on data splitting, such as Figure 4 shown.

[0057] Assuming that the number of splits under the initial split strategy is 3, each computing node can correspond to a GPU. Taking the task data in GPU0 as an example, after a0 is split, three data slices a0`, a0``, and a0``` can be obtained. The same is true for a1~a3.

[0058] During this process, for each computing node, the computing node can receive the first data slice sent by the previous computing node after splitting the task data, and at the same time, split the task data to be sent independent of the computing task into multiple second data slices that have no dependency relationship.

[0059] The task data to be sent that is independent of the computing task is task data that is not associated with the computing task being executed. Figure 4 b1 in the figure is task data independent of the calculation task (a0+b0), c1 is task data independent of the calculation task (b1+c1)...

[0060] S302: Send each second data slice to the next computing node in sequence, so that the next computing node receives subsequent second data slices while performing a computing task according to the received second data slices.

[0061] For each computing node, the computing node can send each second data slice to the next computing node in turn. After the next computing node receives the first second data slice, it can receive other second data slices while executing the computing task based on the received second data slice and the local task data.

[0062] Since there is no dependency between each second data slice, for a task data, when the next computing node receives one of the data slices, it can perform calculations based on the data slice and transmit other data slices at the same time, so that the calculation process of the task data is executed in parallel with the transmission and scheduling process.

[0063] by Figure 4 Taking GPU0 as an example, after GPU0 transmits a0` to GPU1, GPU1 can receive a0`` sent by GPU0 while performing reduction calculations on a0`, b0`, b0``, and b0```.

[0064] S303: After executing the computing task according to all the task data in the computing node, a target computing result is obtained.

[0065] For each computing node, after the computing node executes the computing task according to all the task data therein, the target computing result is obtained, thereby completing the data processing task of the computing node.

[0066] Of course, in practical applications, in order to ensure that each computing node in the collective communication cluster can obtain complete computing results and ensure data synchronization, each computing node can also complete the data processing flow and obtain the final computing results after executing computing tasks on all task data in all computing nodes.

[0067] Based on the above steps, this solution can synchronously execute the calculation and transmission process after receiving a part of the task data, thereby shortening the time used for the entire data processing process and effectively improving the data processing efficiency.

[0068] Furthermore, in actual applications, computing devices may include not only GPUs, but also CPUs and network cards. The network card is responsible for data transmission, the CPU is responsible for computing scheduling and transmission scheduling, and the GPU is responsible for the calculation within the kernel function. These three components are independent of each other and can form a pipeline parallelism. After the application data is sliced, the corresponding relationship between the various task stages during the execution of the computing task is as follows: Figure 5 shown.

[0069] Figure 5 This is a schematic diagram of the corresponding relationship between various task stages after data splitting provided in this manual.

[0070] from Figure 5 It can be seen that the total time of the computing task is composed of the computing scheduling time (loading the kernel function) and the computing time (kernel function execution time), and the total transmission time is composed of the transmission scheduling time (the time consumed by scheduling the network card) and the data transmission time (the time it takes for the network card to send data from GPU1 to GPU2). After splitting the data, the protocol calculation, transmission scheduling, and computing scheduling can be performed while the data is being transmitted.

[0071] Figure 5 It demonstrates the theoretical perfect pipeline parallel arrangement when the transmission time is greater than the calculation time. In this case, the calculation scheduling time + transmission scheduling time + calculation time equals the data transmission time.

[0072] In order to further shorten the overall execution time of the calculation, transmission and scheduling process and further improve the task execution efficiency, the computing node can adjust the initial splitting strategy (initial slice data volume) during the execution of the computing task to ensure that the shortest execution time can be achieved after the data is split based on the adjusted slice data volume.

[0073] Specifically, the overall execution time required to execute a computing task may include: computing time, computing scheduling time, transmission scheduling time, and transmission time. The computing time is the time used to execute the specified calculation. Since the time is linearly related to the amount of data, the time consumed for the specified calculation per unit of data can be expressed as: , where S is the data size of the data slice and C is the GPU computing efficiency.

[0074] The computation scheduling time is affected by the CPU performance and the degree of competition for CPU resources, expressed as β.

[0075] Transmission scheduling time: The transmission scheduling time is the same as the calculation scheduling time. It is affected by the host CPU and the network card host channel adapter (HCA) type, transmission protocol, and is also affected by the host CPU resource competition and the network card load status. It is expressed as θ.

[0076] The data transmission time is affected by the performance of the network card (HCA) and the transmission protocol. The time consumed is basically linearly related to the data size S / B, where B is the network transmission bandwidth.

[0077] In the process of executing a computing task, the computing node can determine the execution time required for the reference computing node to execute the computing task, and then adjust the current slice data volume based on the execution time, and then re-split each task data according to the adjusted slice data volume to obtain multiple data slices that have no dependencies.

[0078] The above reference computing node may refer to any computing node in the collective communication cluster. For example, the reference computing node may be the last computing node in the collective communication cluster. Of course, it may also be a computing node in the middle. This specification does not make any specific limitation on this.

[0079] It should be added that when the task data is split based on a splitting strategy other than the initial splitting strategy, for each task data, the data size of each sub-data obtained after splitting the task data may be different, but the data size of each sub-node at the corresponding position in different task data needs to remain the same.

[0080] For example, for a0`, a0``, and a0``` after splitting a0, the sizes of the three data slices can be different. The sizes of b0`, b0``, and b0``` after splitting b0 can also be different. However, it is necessary to ensure that a0` and b0` have the same size, a0`` and b0`` have the same size, and a0``` and b0``` have the same size.

[0081] In practical applications, the adjustment of the amount of sliced ​​data can be determined by two preset parameters, namely sampling frequency and viscosity. The sampling frequency represents the adjustment of the splitting strategy based on the sampled execution time after sampling the reference computing node for Ψ rounds. The higher the sampling frequency and the smaller Ψ, the higher the accuracy of the splitting strategy, the more optimized the pipeline arrangement of calculation and transmission, and the higher the overall data processing efficiency; at the same time, the higher the sampling frequency, the greater the loss (overhead) of the system load, which to a certain extent increases the preemption of the host CPU and thus increases the absolute time and instability of scheduling.

[0082] Viscosity represents the tendency or sensitivity of the slice size to be adjusted according to the collected sampling data, which may include the specified duration and preset number for determining whether to adjust the slice data volume. If the reference computing node performs the computing task according to each sampled data slice for a duration greater than the specified duration and the number of sampling rounds exceeds the preset number, it means that the N consecutive samplings exceed the normal fluctuation threshold. At this time, the slice data volume can be recalculated and adjusted according to the real-time sampling data and applied to the subsequent collective communication. The viscosity range can be from 0 to 1, calculated according to the normal distribution of N samplings exceeding the threshold.

[0083] In the process of adjusting the amount of slice data, multiple rounds of sampling can be performed on the reference computing node according to the above sampling frequency, so as to determine the sampled data slices that participate in the reference computing node to perform computing tasks in different sampling rounds, and then determine the execution time required for the reference computing node to perform computing tasks based on each sampled data slice.

[0084] Afterwards, based on the viscosity, it can be further determined whether the number of sampling rounds for which the execution time required for the reference computing node to perform the computing task based on each sampled data slice is greater than the specified time exceeds the preset number; if so, the amount of slice data currently used for data splitting is adjusted, otherwise no adjustment is made.

[0085] For each sampled data slice, the calculation time for the reference computing node to perform the computing task based on the sampled data slice, the calculation scheduling time for scheduling the computing resources required to perform the computing task based on the sampled data slice, the transmission scheduling time for the previous computing node of the reference computing node to schedule the transmission resources, and the transmission time for the previous computing node of the reference computing node to transmit the sampled data slice to the reference computing node can be determined. Then, based on the calculation time, the calculation scheduling time, the transmission time and the transmission scheduling time, the execution time required for the reference computing node to perform the computing task based on the sampled data slice is determined.

[0086] It should be noted that for any sampled data slice, since the calculation process and the transmission process can be carried out simultaneously, there is some overlap between the calculation time, calculation scheduling time, transmission scheduling time and transmission scheduling time. Therefore, the execution time required for the reference computing node to perform the calculation task based on the sampled data slice is not a simple addition of the calculation time, calculation scheduling time, transmission scheduling time and transmission scheduling time, but refers to the actual time for completing the overall process of data transmission, calculation scheduling, transmission scheduling and protocol calculation of the sampled data slice in parallel.

[0087] For example, when the transmission duration is equal to the sum of the calculation scheduling duration, the transmission scheduling duration and the calculation duration, the scheduling and calculation processes are completed synchronously when the data transmission is executed. At this time, the execution time required for the reference computing node to perform the computing task based on the sampled data slice is equal to the transmission duration.

[0088] In actual time consumption statistics, since there is usually no direct performance counter to directly obtain the scheduling time of calculation and transmission, it is usually estimated by total time (T) - calculation time - transmission time. , so the expression of the slice data volume S can be further expressed as:

[0089]

[0090] In practical applications, there are two situations in which the execution time required is the shortest. When the transmission time is greater than the calculation time, S / B = S / C +β +θ, that is, the sum of the calculation scheduling time, the calculation time, and the transmission scheduling time is equal to the transmission time. At this time, the overall execution time is the shortest.

[0091] Therefore, when adjusting the splitting strategy, if the calculation time when executing the calculation task based on at least part of the sampled data slices is longer than the transmission time, the amount of slice data currently used for data splitting is adjusted with the goal of minimizing the deviation between the sum of the calculation scheduling time, the calculation time, and the transmission scheduling time when executing the calculation task based on each data slice, thereby obtaining the adjusted splitting strategy.

[0092] In addition, when the calculation time is greater than the transmission time, if S / C = S / B+β+θ, that is, the sum of the calculation scheduling time, calculation time, and transmission scheduling time is equal to the transmission time, the overall execution time is the shortest.

[0093] Therefore, when adjusting the splitting strategy, if the calculation time when executing the calculation task based on at least part of the sampled data slices is longer than the transmission time, the amount of slice data currently used for data splitting is adjusted with the goal of minimizing the deviation between the calculation scheduling time, the transmission time, and the sum of the transmission scheduling time when executing the calculation task based on each data slice, thereby obtaining the adjusted splitting strategy.

[0094] In the actual process of adjusting the slice data volume, there are multiple sampled data slices. Therefore, the overall relationship between the calculation duration and the transmission duration can be determined based on some of the sampled data slices.

[0095] For example, when the computation time of most sampled data slices is longer than the transmission time, it can be considered that the overall computation time is longer than the transmission time. At this time, the amount of slice data currently used for data splitting is adjusted with the goal of minimizing the deviation between the computation scheduling time, the transmission time, and the sum of the transmission scheduling time when executing the computation task for each data slice and the computation time;

[0096] When the calculation time of most sampled data slices is less than the transmission time, it can be considered that the overall calculation time is less than the transmission time. At this time, the amount of slice data currently used for data splitting is adjusted with the goal of minimizing the deviation between the sum of the calculation scheduling time, the calculation time, the transmission scheduling time and the transmission calculation time.

[0097] Based on the above two situations, the corresponding relationship between the data size S of the data slice, the calculation scheduling duration, the transmission duration, the transmission scheduling duration, and the calculation duration can be expressed as:

[0098]

[0099] From the above method, it can be seen that this solution can dynamically adjust the data slice strategy, optimize the pipeline arrangement of protocol calculation and data transmission in the collective communication process in real time, better cover the scheduling overhead and improve the utilization rate of the overall calculation and transmission components. The invention can be applied to a variety of different algorithms to implement collective communication primitives, significantly improving the performance of collective communication.

[0100] The above is one or more implementations of this specification applied to the computing task execution method of collective communication. Based on the same idea, this specification also provides a corresponding computing task execution device applied to collective communication, such as Figure 6 shown.

[0101] Figure 6 A schematic diagram of a data processing device provided for this specification includes:

[0102] The splitting module 601 is used to receive a first data slice sent by a previous computing node, execute a computing task according to the received first data slice, and split the task data to be sent independent of the computing task into a plurality of second data slices that have no dependency relationship;

[0103] A sending module 602 is used to send each second data slice in sequence to a next computing node, so that the next computing node receives subsequent second data slices while performing a computing task according to the received second data slices;

[0104] The determination module 603 is used to obtain a target calculation result after executing the calculation task according to all the task data in the calculation node.

[0105] Optionally, before splitting the task data to be sent that is independent of the computing task into multiple second data slices that have no dependencies, the device also includes: an adjustment module 604, used to determine the execution time required for the reference computing node to perform the computing task; the reference computing node is any computing node in the collective communication cluster; based on the execution time required for the reference computing node to perform the computing task, determine the amount of slice data used for data splitting.

[0106] Optionally, the adjustment module 604 is specifically used to perform multiple rounds of sampling on the reference computing node, determine the sampled data slices that participate in the reference computing node to perform computing tasks in different sampling rounds; and determine the execution time required for the reference computing node to perform computing tasks based on each sampled data slice.

[0107] Optionally, the adjustment module 604 is specifically used to determine whether the number of sampling rounds in which the execution time required for the reference computing node to perform the computing task based on each sampled data slice is greater than a specified time exceeds a preset number; if so, the amount of slice data currently used for data splitting is adjusted.

[0108] Optionally, the adjustment module 604 is specifically used to determine, for each sampled data slice, the computation time for the reference computing node to perform a computing task based on the sampled data slice, the computation scheduling time for scheduling computing resources required to perform the computing task based on the sampled data slice, the transmission scheduling time for the previous computing node of the reference computing node to schedule transmission resources, and the transmission time for the previous computing node of the reference computing node to transmit the sampled data slice to the reference computing node; determine the execution time required for the reference computing node to perform the computing task based on the sampled data slice based on the computation time, the computation scheduling time, the transmission time and the transmission scheduling time.

[0109] Optionally, the adjustment module 604 is specifically used to adjust the amount of slice data currently used for data splitting with the goal of minimizing the deviation between the calculation scheduling time, the transmission time, and the sum of the transmission scheduling time and the calculation time when the calculation task is performed according to each data slice, if the calculation time when the calculation task is performed according to at least part of the sampled data slices is longer than the transmission time.

[0110] Optionally, the adjustment module 604 is specifically used to adjust the amount of slice data currently used for data splitting with the goal of minimizing the deviation between the sum of the calculation scheduling time, the calculation time, and the transmission scheduling time when the calculation task is performed according to at least part of the sampled data slices and the transmission time if the calculation time when the calculation task is performed according to at least part of the sampled data slices is less than the transmission time.

[0111] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 3 A data processing method is provided.

[0112] This manual also provides Figure 7 The one shown corresponds to Figure 3 A schematic diagram of the electronic device. Figure 7 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 3 Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0113] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0114] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0115] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0116] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0117] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0119] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0121] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0122] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0123] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0124] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0125] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0127] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0128] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A data processing method, characterized in that: The method is applied to any computing node in a collective communication cluster, the computing nodes in the collective communication cluster form a ring topology, and the computing node stores a plurality of task data, and the method comprises: Receive a first data slice sent by a previous computing node, execute a computing task according to the received first data slice, and split the task data to be sent that is independent of the computing task into multiple second data slices that have no dependency relationship based on a predetermined slice data volume for data splitting; wherein, perform multiple rounds of sampling on the reference computing node to determine the sampled data slices that participate in the reference computing node executing the computing task in different sampling rounds, and determine the execution time required for the reference computing node to execute the computing task according to each sampled data slice; determine the slice data volume according to the execution time; the reference computing node is any computing node in the collective communication cluster; the execution time is the actual time for completing the overall process of data transmission, computing resource scheduling, transmission resource scheduling and protocol calculation for each sampled data slice in parallel; Sending each second data slice in sequence to a next computing node, so that the next computing node receives subsequent second data slices while performing a computing task according to the received second data slices; After executing the computing task according to all the task data in the computing node, the target computing result is obtained.

2. The method according to claim 1, characterized in that Determining the amount of slice data for data splitting according to the execution time required for the reference computing node to execute the computing task, specifically including: Determine whether the number of sampling rounds for which the execution time required for the reference computing node to perform the computing task according to each sampled data slice is longer than the specified time exceeds a preset number; If so, the amount of slice data currently used for data splitting is adjusted.

3. The method according to claim 1, characterized in that Determining the execution time required for the reference computing node to execute the computing task according to each sampled data slice specifically includes: For each sampled data slice, determine the computation duration for the reference computing node to perform a computation task according to the sampled data slice, the computation scheduling duration for scheduling computation resources required to perform the computation task according to the sampled data slice, the transmission scheduling duration for a previous computing node of the reference computing node to schedule transmission resources, and the transmission duration for a previous computing node of the reference computing node to transmit the sampled data slice to the reference computing node; Determine the execution time required for the reference computing node to perform the computing task according to the sampled data slice based on the computing time, the computing scheduling time, the transmission time and the transmission scheduling time.

4. The method according to claim 2, characterized in that Adjust the amount of slice data currently used for data splitting, including: If the calculation duration when executing the calculation task based on at least part of the sampled data slices is longer than the transmission duration, the amount of slice data currently used for data splitting is adjusted with the goal of minimizing the deviation between the calculation scheduling duration, the transmission duration, and the sum of the transmission scheduling duration when executing the calculation task based on each data slice and the calculation time.

5. The method according to claim 2, characterized in that Adjust the amount of slice data currently used for data splitting, including: If the calculation time when executing the calculation task based on at least part of the sampled data slices is less than the transmission time, the amount of slice data currently used for data splitting is adjusted with the goal of minimizing the deviation between the sum of the calculation scheduling time, the calculation time, and the transmission scheduling time when executing the calculation task based on each data slice and the transmission time.

6. A data processing device, characterized in that: Applied to any computing node in a collective communication cluster, the computing nodes in the collective communication cluster form a ring topology, and the computing node stores multiple task data, including: A splitting module is used to receive a first data slice sent by a previous computing node, perform a computing task according to the received first data slice, and split the task data to be sent that is independent of the computing task into multiple second data slices without dependencies based on a predetermined amount of slice data for data splitting; wherein, multiple rounds of sampling are performed on the reference computing node to determine the sampled data slices that participate in the reference computing node performing the computing task in different sampling rounds, and determine the execution time required for the reference computing node to perform the computing task according to each sampled data slice; the slice data amount is determined based on the execution time; the reference computing node is any computing node in the collective communication cluster; the execution time is the actual time for completing the overall process of data transmission, computing resource scheduling, transmission resource scheduling and protocol calculation for each sampled data slice in parallel; A sending module, used to send each second data slice in sequence to a next computing node, so that the next computing node receives subsequent second data slices while performing a computing task according to the received second data slices; The determination module is used to obtain a target calculation result after executing the calculation task according to all the task data in the calculation node.

7. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 5 is implemented.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 5 is implemented.

Images