Parallel computing method and apparatus for distributed data

Abstract

The application provides a parallel computing method and device for distributed data. The method comprises: performing data splitting on original data according to a splitting strategy to obtain a plurality of data slices; distributing the plurality of data slices to a plurality of computing units, wherein each computing unit performs computation based on the distributed data slices to obtain slice computation results; and aggregating the slice computation results, wherein the splitting strategy comprises at least one of a first splitting strategy and a second splitting strategy, wherein the first splitting strategy is based on the number of data partitions distributed to each computing unit to perform splitting, and wherein the second splitting strategy is based on the number of data rows distributed to each computing unit or the total number of the plurality of computing units to perform splitting.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data, and more particularly, to a parallel computing and apparatus of distributed data. BACKGROUND

[0002] In the machine learning and big data computing scenarios, the computing logic for accelerating data categorization is often encountered, and a common method is to limit the description of the computing processing logic to a specific language or framework, and then utilize the processing collaboration at the language and framework level to split the data and tasks in a multi-machine environment (e.g., a parallel computing environment). However, there is no good solution for the case where the computing description framework cannot be constrained (or limited). SUMMARY

[0003] The present application aims to provide a parallel computing method and apparatus of distributed data.

[0004] According to one or more aspects of the present application, a parallel computing method of distributed data is provided, the method comprising: performing data splitting on to-be-processed data according to a splitting strategy to obtain a plurality of data slices; allocating the plurality of data slices to a plurality of computing units, wherein each computing unit performs computation based on the allocated data slices to obtain slice computation results; and aggregating the slice computation results, wherein the splitting strategy comprises at least one of a first splitting strategy and a second splitting strategy, wherein the first splitting strategy is to perform splitting based on the number of data partitions allocated to each computing unit, and the second splitting strategy is to perform splitting based on the number of data rows allocated to each computing unit or the total number of the plurality of computing units.

[0005] In an example embodiment according to the inventive concept, the step of performing data splitting on to-be-processed data according to a splitting strategy to obtain a plurality of data slices can comprise: performing data splitting according to a first splitting strategy to obtain a plurality of initial slices, and then adjusting the plurality of initial slices according to a second splitting strategy to obtain the plurality of data slices.

[0006] In an example embodiment according to the inventive concept, the step of adjusting the plurality of initial slices according to the second splitting strategy can be performed by repartitioning the plurality of initial slices.

[0007] In an example embodiment according to the inventive concept, the splitting strategy can further comprise a third splitting strategy, the third splitting strategy being to perform splitting according to resource scheduling information, wherein the resource scheduling information comprises at least one of an expected total runtime of the computation and expected computing resources for performing the computation.

[0008] In example embodiments according to the inventive concept, the step of performing data splitting according to the splitting strategy to obtain a plurality of data slices can include performing data splitting according to a first splitting strategy and / or a second splitting strategy to obtain a plurality of initial slices, and then adjusting the plurality of initial slices according to a third splitting strategy to obtain the plurality of data slices.

[0009] In example embodiments according to the inventive concept, the step of performing the third splitting strategy can include obtaining the resource scheduling information, obtaining a running resource indicator and a secondary splitting indicator, wherein the running resource indicator is used to represent a usage amount of the computing resource, and the secondary splitting indicator is used to represent a time cost consumed when performing the adjustment, and performing splitting based on the running resource indicator and the secondary splitting indicator according to the resource scheduling information.

[0010] In example embodiments according to the inventive concept, the computing resource can include a first computing unit that is performing computation and a second computing unit that is not performing computation, and the first computing unit includes a plurality of computing units. The step of performing splitting based on the running resource indicator and the secondary splitting indicator according to the resource scheduling information can include obtaining a to-be-allocated computing unit in the second computing unit based on the running resource indicator, inferring an expected computation duration according to the to-be-processed data or the plurality of initial slices, the first computing unit, and the to-be-allocated computing unit, inferring a total running duration change based on at least the expected computation duration and the secondary splitting indicator, and performing splitting based on the total running duration change according to the resource scheduling information.

[0011] In example embodiments according to the inventive concept, the running resource indicator can include at least one of a CPU utilization, a memory utilization, and a bandwidth utilization of each computing unit collected.

[0012] In example embodiments according to the inventive concept, the step of inferring the total running duration change can include inferring the total running duration change based on at least one of a data slice reading consumption and a preprocessing consumption, and the expected computation duration and the secondary splitting indicator.

[0013] According to one or more aspects of the present disclosure, a parallel computing device for distributed data is provided, the device comprising: a data splitting unit configured to perform data splitting on to-be-processed data according to a splitting strategy to obtain a plurality of data slices; a plurality of computing units configured to perform computation based on the allocated data slices to obtain slice computation results; and an aggregation unit configured to aggregate the slice computation results, wherein the splitting strategy comprises at least one of a first splitting strategy and a second splitting strategy, wherein the first splitting strategy is based on a number of data partitions allocated to each computing unit, and the second splitting strategy is based on a number of data rows allocated to each computing unit or a total number of the plurality of computing units.

[0014] In example embodiments according to the inventive concept, the data splitting unit can be further configured to perform data splitting according to a first splitting policy to obtain a plurality of initial slices, and then adjust the plurality of initial slices according to a second splitting policy to obtain the plurality of data slices.

[0015] In example embodiments according to the inventive concept, the data splitting unit can be further configured to perform the step of adjusting the plurality of initial slices according to the second splitting policy by re-partitioning the plurality of initial slices.

[0016] In example embodiments according to the inventive concept, the splitting policies can further include a third splitting policy, the third splitting policy being to perform splitting according to resource scheduling information, wherein the resource scheduling information includes at least one of an expected total running time and expected computing resources for performing the computation.

[0017] In example embodiments according to the inventive concept, the data splitting unit can be configured to perform data splitting according to the first splitting policy and / or the second splitting policy to obtain a plurality of initial slices, and then adjust the plurality of initial slices according to a third splitting policy to obtain the plurality of data slices.

[0018] In example embodiments according to the inventive concept, the apparatus can further include an indicator obtaining unit configured to obtain a running resource indicator and a secondary splitting indicator, wherein the running resource indicator is used to represent an amount of usage of the computing resources, and the secondary splitting indicator is used to represent a time cost consumed when performing the adjustment. The data splitting unit, when performing data splitting according to the third splitting policy, can be configured to: obtain the resource scheduling information; obtain the running resource indicator and the secondary splitting indicator from the indicator obtaining unit; and perform splitting based on the running resource indicator and the secondary splitting indicator according to the resource scheduling information.

[0019] In example embodiments according to the inventive concept, the computing resources can include a first computing unit that is performing the computation and a second computing unit that is not performing the computation, the first computing unit including a plurality of computing units. The data splitting unit can be further configured to: obtain an unassigned computing unit in the second computing unit based on the running resource indicator; estimate an expected computation time based on the to-be-processed data or the plurality of initial slices, the first computing unit, and the unassigned computing unit; estimate a total running time change based on at least the expected computation time and the secondary splitting indicator; and perform splitting based on the total running time change according to the resource scheduling information.

[0020] In example embodiments according to the inventive concept, the running resource indicator can include at least one of a CPU utilization, a memory utilization, and a bandwidth utilization of each of the computing units that are collected.

[0021] In exemplary embodiments according to the inventive concept, the data splitting unit can be further configured to estimate the total runtime variation based on at least one of data slice reading consumption, pre-processing consumption, and a predicted computation duration and a secondary splitting indicator.

[0022] Another aspect of the present disclosure provides a computer-readable storage medium storing instructions, wherein the instructions, when executed by at least one computing device, cause the at least one computing device to perform the parallel computing method of distributed data as described above.

[0023] Another aspect of the present disclosure provides a system comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform the parallel computing method of distributed data as described above.

[0024] According to one or more aspects of the present disclosure, parallel computing is achieved by performing data splitting to obtain a plurality of data slices according to a splitting strategy, performing computation on the plurality of data slices respectively to obtain slice computation results by assigning the plurality of data slices to a plurality of computing units, and aggregating the slice computation results, thereby improving user experience and optimizing overall performance. BRIEF DESCRIPTION OF DRAWINGS

[0025] These and / or other aspects and advantages of the present disclosure will become more apparent and more readily appreciated from the following detailed description of the disclosure taken in conjunction with the accompanying drawings in which:

[0026] Figure 1 is a scenario diagram illustrating an application scenario of the parallel computing method according to an embodiment of the present disclosure;

[0027] Figure 2 is a flowchart illustrating the parallel computing method according to an embodiment of the present disclosure;

[0028] Figure 3 is a schematic diagram illustrating a splitting strategy of the parallel computing method according to an embodiment of the present disclosure;

[0029] Figure 4 is a block diagram illustrating a parallel computing device according to an embodiment of the present disclosure; and

[0030] Figure 5 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] Embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. Examples of the embodiments are illustrated in the drawings, in which like reference numerals refer to like parts throughout the various drawings. The embodiments will be described below by referring to the drawings, in order to explain the present disclosure.

[0032] Figure 1 is a scenario diagram illustrating an application scenario of a parallel computing method according to an embodiment of the present disclosure.

[0033] In the application scenario of parallel computing applied to distributed data, the to-be-processed data can be logically or physically sharded first. For example, the to-be-processed data can include 0000.parquet, 0001.parquet, 0002.parquet and 0003.parquet as shown in FIG. 1. Figure 1 Parquet is a columnar storage format, and Parquet can be bound to a data framework and can be adapted to multiple languages and components. For example, 0000.parquet, 0001.parquet, 0002.parquet and 0003.parquet can be parquet files or partitions. In the step of performing data splitting, the data 0000.parquet, 0001.parquet, 0002.parquet and 0003.parquet can be divided into data slice 1, data slice 2 and data slice 3. The data slice 1, data slice 2 and data slice 3 can be the same as or different from the partitions of the data 0000.parquet, 0001.parquet, 0002.parquet and 0003.parquet. For example, the data slice 1 can be the same as the data 0000.parquet. In another embodiment, by way of example only, the data slice 1 can include a part of the data 0000.parquet and the data 0002.parquet, and the split data slice contains neither repeated nor missed data (or data rows), that is, the split data includes all of 0000.parquet, 0001.parquet, 0002.parquet and 0003.parquet.

[0034] Then, the data slice 1, data slice 2 and data slice 3 are assigned to multiple computing units, wherein each computing unit performs a computation based on the assigned data slice to obtain a slice computation result. For example, three computing units (not shown) respectively undergo phase 1, phase 2 and phase 3, thereby obtaining computation result 1, computation result 2 and computation result 3 respectively.

[0035] Next, the computation result 1, computation result 2 and computation result 3 are aggregated to obtain a computation result. For example, as shown in FIG. 2, the computation result 1, computation result 2 and computation result 3 are aggregated to obtain the data 0004.parquet. Figure 1

[0036] ​In the batch prediction service scenario, there is low dependency between different data. Therefore, it is very suitable to use parallel computing method, or batchable computing. Next, referring to Figure 2 and Figure 3 parallel computing methods for improving the performance of batchable computing will be described. More specifically, by performing different levels of data splitting strategies, the overall performance of batchable computing is improved.

[0037] Figure 2 is a flowchart illustrating a parallel computing method according to an embodiment of the present disclosure, Figure 3 is a schematic diagram illustrating a splitting strategy of a parallel computing method according to an embodiment of the present disclosure.

[0038] As shown in Figure 2 , in step S10, data splitting is performed on the to-be-processed data according to a splitting strategy to obtain a plurality of data slices.

[0039] As shown in Figure 3 , the splitting strategy includes at least one of a first splitting strategy and a second splitting strategy.

[0040] More specifically, the first splitting strategy is to perform splitting based on the number of data partitions to be allocated to each computing unit. Wherein, the data partition can refer to logical partition / physical partition. In this case, according to the first splitting strategy, data located in the same logical partition / physical partition can be allocated to the same computing unit, thereby reducing the case of performing read / write between different partitions. In another embodiment, the data partition can be data0000.parquet, 0001.parquet, 0002.parquet and 0003.parquet as shown in Figure 1 , that is, the data partition can refer to partitioning by file. In this case, the first splitting strategy can allocate data located in the same file to the same computing unit, thereby reducing the simultaneous read / write operation of different computing units on a single file, and avoiding the transfer of a specific file between different computing units. For example only, data0000.parquet and 0001.parquet can be divided into data slice 1, data0002.parquet can be divided into data slice 2, and data0003.parquet can be divided into data slice 3.

[0041] In addition, the second splitting strategy is to perform splitting based on the number of data rows to be allocated to each computing unit or the total number of a plurality of computing units. More specifically, as shown in Figure 1The illustrated data 0000.parquet, 0001.parquet, 0002.parquet and 0003.parquet are preprocessed for operational nature. For example, a repartitioning operation can be performed on the data 0000.parquet, 0001.parquet, 0002.parquet and 0003.parquet based on the number of data rows assigned per compute unit or the total number of compute units. The repartitioning operation can refer to a secondary physical / logical partitioning. For example, by repartitioning the data 0000.parquet, 0001.parquet, 0002.parquet and 0003.parquet. The repartitioned data can be denoted as 0000'.parquet, 0001'.parquet, 0002'.parquet. As an example, the data 0000'.parquet can include all of the data 0000.parquet and a portion of the data 0001.parquet, 0001'.parquet can include the remaining portion of 0001.parquet and a portion of 0002.parquet, 0002'.parquet can include the remaining portion of 0002.parquet and all of 0003.parquet. And such that the repartitioned data 0000'.parquet, 0001'.parquet, 0002'.parquet have the same or comparable amount of data. In this case, data slice 1 corresponds to data 0000'.parquet, data slice 2 corresponds to data 0001'.parquet, and data slice 3 corresponds to data 0002'.parquet.

[0042] In another embodiment, the repartitioning operation can also mean supplementing data of a predetermined number of rows or deleting data of a predetermined number of rows for at least some of the data 0000.parquet, 0001.parquet, 0002.parquet and 0003.parquet, so that the number of data slices in each compute unit is comparable or the number of data rows in each data slice is comparable. As an example only, a portion of data 0000.parquet and 0001.parquet can be divided into data slice 1, a portion of data 0000.parquet and data 0002.parquet can be divided into data slice 2, and a portion of data 0000.parquet and data 0003.parquet can be divided into data slice 3. In this case, the supplementing and deleting operation will not change the total amount of data, for example, the supplementing and deleting operation will not change the total number of rows of data.

[0043] It should be noted that although Figure 3The splitting strategies shown in the middle include at least one of a first splitting strategy and a second splitting strategy, however, in example embodiments, only the first splitting strategy can be performed, or only the second splitting strategy can be performed.

[0044] As shown in the middle, the splitting strategies shown in the middle include at least one of a first splitting strategy and a second splitting strategy, however, in example embodiments, only the first splitting strategy can be performed, or only the second splitting strategy can be performed. Figure 3 As shown in the middle, the splitting strategies shown in the middle include at least one of a first splitting strategy and a second splitting strategy, however, in example embodiments, only the first splitting strategy can be performed, or only the second splitting strategy can be performed. As shown in the middle, the splitting strategies shown in the middle include at least one of a first splitting strategy and a second splitting strategy, however, in example embodiments, only the first splitting strategy can be performed, or only the second splitting strategy can be performed. As shown in the middle, the splitting strategies shown in the middle include at least one of a first splitting strategy and a second splitting strategy, however, in example embodiments, only the first splitting strategy can be performed, or only the second splitting strategy can be performed.

[0045] In example embodiments, the splitting strategies can further include a third splitting strategy. The third splitting strategy is to perform splitting according to resource scheduling information, wherein the resource scheduling information includes at least one of an expected total runtime of the computation and a computing resource performing the computation. For example, the expected total runtime can represent an expected total runtime or a target runtime set by a user. By way of example only, the expected total runtime can be 1 hour, 1 day, or 1 week, but is not limited thereto. The computing resource performing the computation can represent a computing unit available for use. The computing resource can include a first computing unit that is performing the computation and a second computing unit that is not performing the computation. In other words, the computing resource can refer to a plurality of computing units (i.e., the first computing unit) that are performing the computation and a computing unit (i.e., the second computing unit) that is assignable to the computation in addition to the plurality of computing units.

[0046] In another embodiment, when performing the third splitting strategy, runtime metrics and repartition metrics can also be obtained (e.g., in real time). The runtime metrics are used to represent the usage of computing resources, and the repartition metrics are used to represent the time cost consumed when performing the adjustment. For example, the runtime metrics can include at least one of the CPU utilization, memory utilization, and bandwidth utilization of each computing unit collected.

[0047] In an example embodiment, the step of performing the third splitting strategy includes obtaining resource scheduling information, obtaining runtime metrics and repartition metrics, and performing splitting based on the runtime metrics and the repartition metrics according to the resource scheduling information.

[0048] In a further example embodiment, the step of performing splitting based on the runtime metrics and the repartition metrics according to the resource scheduling information can include obtaining a to-be-allocated computing unit in the second computing unit based on the runtime metrics, predicting a predicted computation duration according to the to-be-processed data or the plurality of initial slices, the first computing unit, and the to-be-allocated computing unit, predicting a total runtime duration change based on at least the predicted computation duration and the repartition metrics, and performing splitting based on the total runtime duration change according to the resource scheduling information.

[0049] It should be noted that although the second computing unit is not allocated to the computation, its CPU utilization, memory utilization, and bandwidth utilization can still be occupied by other tasks (e.g., other computing tasks, updates, read / write, maintenance, etc.). Therefore, the runtime metrics can be used to effectively detect whether there is a resource (i.e., a to-be-allocated computing unit) in the second computing unit that can be allocated to the computation. For example, in the case where the user is sensitive to the runtime duration, if there is a to-be-allocated computing unit, then the capacity can be expanded (i.e., the number of computing units performing parallel computation can be increased). In other words, the runtime metrics can be used to balance the resource usage and the total runtime duration. In the case where the total runtime duration needs to be shortened, the resource usage is increased; in the case where the computing resources need to be saved (e.g., for other tasks to give way), the capacity is reduced and the total runtime duration is increased.

[0050] In a further example embodiment, in a case where there are computing units to be allocated in the computing resource, if allocating the computing units to be allocated to the computation (i.e., allocating more resource usage) will cause the predicted computation duration to decrease. For example, the remaining computation duration can be estimated according to the data to be processed or the number of initial slices, the first computing unit, the user-set expected total runtime is compared with the sum of the remaining computation duration and the already run duration, if the user-set expected total runtime is less than the sum of the remaining computation duration and the already run duration, then the scaling can be performed, otherwise, the scaling can not be performed.

[0051] Considering that allocating the computing units to be allocated to the computation will bring operations such as repartitioning, which will cause the secondary split index to increase. Therefore, the total runtime change is estimated based on the predicted computation duration and the secondary split index, so that whether to perform the scaling can be adaptively adjusted. Further, if the total runtime change causes the total runtime to decrease, the scaling can be performed, otherwise, the scaling can not be performed; for example, if the total runtime change causes the total runtime to decrease by more than a predetermined threshold, the scaling can be performed, otherwise, the scaling can not be performed, so that the total runtime of the computation is not caused to increase due to operations such as repartitioning.

[0052] In another embodiment, the step of estimating the total runtime change can include estimating the total runtime change based on at least one of data slice read consumption (Partitions Read), preprocessing consumption (Preprocessing), and the predicted computation duration and the secondary split index.

[0053] Although not shown, for example, the computing capacity of the A unit can be greater than that of the B unit, and the computing capacity of the B unit can be greater than that of the C unit. Alternatively, in another embodiment, although the A unit, the B unit and the C unit have the same hardware configuration, part of the computing capacity in the B unit and the C unit is still occupied, so that the computing capacity of the A unit > the computing capacity of the B unit > the computing capacity of the C unit. In this case, according to the comprehensive consideration of the running resource index and the secondary split index, the data slice 1, the data slice 2 and the data slice 3 to be respectively allocated to the A unit, the B unit and the C unit are adjusted (e.g., the repartitioning operation as described above is performed) based on the resource scheduling information, so that the number of data slices or the number of data rows allocated to each computing unit matches the corresponding computing capacity.

[0054] As Figure 3As shown, in another embodiment, the data splitting can be performed according to the first splitting strategy and / or the second splitting strategy to obtain a plurality of initial slices first, and then the plurality of initial slices are adjusted according to the third splitting strategy to obtain a plurality of data slices. Here, only an example of performing the first splitting strategy and / or the second splitting strategy first, and then performing the third splitting strategy is shown. In different embodiments, the execution of the splitting strategies can have different orders, and the combination of the splitting strategies executed in different orders will be understood to be included in the scope of the present application.

[0055] Referring back to Figure 2 In step S20, the plurality of data slices are allocated to the plurality of computing units, wherein each computing unit performs a computation based on the allocated data slice to obtain a slice computation result.

[0056] In another embodiment, during the execution of the computation by the respective computing units, the running resource indicators and / or the secondary splitting indicators of the plurality of computing units can be collected in real time, so that the running state feedback is received, and according to the running state feedback, the data slice 1, the data slice 2 and the data slice 3 that have been allocated to the computing unit 1, the computing unit 2 and the computing unit 3 are adjusted based on the running resource indicators and the secondary splitting indicators according to the resource scheduling information according to the comprehensive consideration of the consumption of the running resource indicators and the secondary splitting indicators during the computation, so as to adaptively adjust or allocate the computing performance, thereby improving the overall performance of the batchable computation. For example, although not shown, if there is a computing unit to be allocated (for example, a D unit), and the total running time changes as described above so that the total running time decreases by more than a predetermined threshold, then capacity expansion can be performed. In an example embodiment, the repartitioning operation as described above can be performed on the data slice 1, the data slice 2 and the data slice 3 to obtain the data slice 1', the data slice 2', the data slice 3' and the data slice 4', and they are respectively allocated to the A unit, the B unit, the C unit and the D unit to perform the computation.

[0057] In step S30, the slice computation results are aggregated. For example, in a batch estimation service scenario, as Figure 1 and Figure 3 As shown, by performing batch computation based on the data slice 1, the data slice 2 and the data slice 3 respectively, the computation result 1, the computation result 2 and the computation result 3 are obtained respectively, and the above computation results are aggregated to obtain the final result, i.e., the estimated value in the estimation service scenario.

[0058] As described above with reference to steps S10 to S30, since the parallel computing is implemented by performing data splitting according to different splitting strategies (at least one of the first splitting strategy, the second splitting strategy, and the third splitting strategy) to obtain a plurality of data slices, performing computation by assigning the plurality of data slices to a plurality of computing units respectively to obtain slice computation results, and aggregating the slice computation results, the use experience is improved and the overall performance can be optimized.

[0059] Figure 4 is a block diagram of the parallel computing apparatus 10 of distributed data according to the present disclosure.

[0060] According to one or more aspects of the present disclosure, the present disclosure provides a parallel computing apparatus 10 of distributed data, the apparatus 10 comprising: a data splitting unit 110, a computing unit 120, and an aggregation unit 130.

[0061] The data splitting unit 110 is configured to perform data splitting according to a splitting strategy to obtain a plurality of data slices. Wherein, the splitting strategy comprises at least one of a first splitting strategy, a second splitting strategy, and a third splitting strategy. The first splitting strategy is to perform splitting based on the number of data partitions to be assigned to each computing unit, the second splitting strategy is to perform splitting based on the number of data rows to be assigned to each computing unit or the total number of the plurality of computing units, and the third splitting strategy is to perform splitting according to resource scheduling information, wherein the resource scheduling information comprises at least one of the total runtime of computation and the computing resource for performing computation. The data splitting unit 110 is configured to perform the steps of data splitting described with reference to Figure 2 and Figure 3 Therefore, the redundant description is omitted here.

[0062] The computing unit 120 is configured to perform computation based on the assigned data slices to obtain slice computation results. The computing unit 120 can be configured to perform step S20 described with reference to Figure 2 Therefore, the redundant description is omitted here.

[0063] The aggregation unit 130 is configured to aggregate the slice computation results. The aggregation unit 130 can be configured to perform step S30 described with reference to Figure 2 Therefore, the redundant description is omitted here.

[0064] In addition, the parallel computing device 10 can further include an index obtaining unit 140. The index obtaining unit 140 is configured to obtain the running resource index and the secondary split index of the plurality of computing units, and the data split unit 110 is configured to, when performing data split according to the third split strategy: obtain the resource scheduling information; obtain the running resource index and the secondary split index from the index obtaining unit 140; perform split based on the running resource index and the secondary split index according to the resource scheduling information. Wherein, the running resource index is used to represent the time cost consumed by the plurality of computing units when performing calculation, and the secondary split index is used to represent the time cost consumed when performing adjustment. The index obtaining unit 140 can be configured to perform the method described with reference to Figure 3 , and thus the redundant description is omitted here.

[0065] As to the apparatus in the above embodiments, the specific manner in which each module / unit performs operations has been described in detail in the embodiments related to the method, and thus will not be described in detail here.

[0066] Figure 5 is a block diagram illustrating an electronic device 500 according to an example embodiment of the present disclosure.

[0067] Referring to Figure 5 , the electronic device 500 includes at least one memory 501 and at least one processor 502, the at least one memory 501 stores computer executable instructions, when the computer executable instructions are executed by the at least one processor 502, the at least one processor 502 is caused to perform the parallel computing method of distributed data according to the embodiments of the present disclosure.

[0068] As an example, the electronic device 500 can be a PC computer, a tablet device, a personal digital assistant, a smart phone, or other devices capable of executing the above instructions. Here, the electronic device 500 is not necessarily a single electronic device, but can also be any device or collection of circuits capable of executing the above instructions (or instruction sets) alone or jointly. The electronic device 500 can also be part of an integrated control system or system manager, or can be configured to interface with a local or remote (e.g., via wireless transmission) portable electronic device.

[0069] In the electronic device 500, the processor 502 can include a central processor (CPU), a graphics processor (GPU), a programmable logic device, a special-purpose processor system, a microcontroller, or a microprocessor. As an example but not limitation, the processor can also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0070] The processor 502 can execute instructions or code stored in the memory 501, which can also store data. The instructions and data can also be transmitted and received via a network using a network interface device, which can employ any known transmission protocol.

[0071] The memory 501 can be integrated with the processor 502, for example, by arranging RAM or flash memory within an integrated circuit microprocessor or the like. Additionally, the memory 501 can comprise a separate device, such as an external disk drive, memory array, or other storage device usable by any database system. The memory 501 and the processor 502 can be operatively coupled, or can communicate with each other, for example, through I / O ports, network connections, or the like, so that the processor 502 can read files stored in the memory.

[0072] Additionally, the electronic device 500 can also include a video display, such as a liquid crystal display, and a user interface, such as a keyboard, mouse, touch input device, or the like. All of the components of the electronic device 500 can be connected via a bus and / or network.

[0073] According to embodiments of the disclosure, there can also be provided a computer-readable storage medium in which instructions stored therein are executable by at least one processor to cause the at least one processor to perform the parallel computing method of distributed data according to embodiments of the disclosure. Examples of the computer-readable storage medium herein include read-only memory (ROM), programmable read-only memory (PROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk memory, a hard disk drive (HDD), a solid state drive (SSD), a card-type memory such as a multimedia card, a secure digital (SD) card, or an extreme digital (XD) card, a magnetic tape, a floppy disk, a magneto-optical data storage device, an optical data storage device, a hard disk, a solid state disk, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the computer-readable storage medium described above can be executed in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc., and in addition, in one example, the computer program and any associated data, data files, and data structures are distributed over a networked computer system so that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0074] According to embodiments of the disclosure, there can also be provided a computer program product including computer instructions which, when executed by at least one processor, implement the parallel computing method of distributed data according to embodiments of the disclosure.

[0075] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

[0076] It should be understood that the present disclosure is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the present disclosure. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for parallel computing of distributed data, comprising: Performing data splitting on the data to be processed according to the splitting strategy to obtain a plurality of data slices; Allocating the plurality of data slices to a plurality of computing units, wherein each computing unit performs calculations based on the allocated data slices to obtain slice calculation results; and Aggregating the slice calculation results; The splitting strategy includes at least one of a first splitting strategy and a second splitting strategy, wherein the first splitting strategy is to perform splitting based on the number of data partitions to be allocated to each computing unit, and the second splitting strategy is to perform splitting based on the number of data rows to be allocated to each computing unit or the total number of the plurality of computing units; The splitting strategy further includes: a third splitting strategy, wherein the third splitting strategy is to perform splitting according to resource scheduling information, wherein the resource scheduling information includes at least one of the expected total running time of the calculation and the computing resources expected to be used to perform the calculation.

2. The parallel computing method according to claim 1, wherein: The step of performing data splitting on the processed data according to the splitting strategy to obtain multiple data slices includes: performing data splitting according to the first splitting strategy to obtain multiple initial slices, and then adjusting the multiple initial slices according to the second splitting strategy to obtain the multiple data slices.

3. The parallel computing method according to claim 2, wherein: The step of adjusting the multiple initial slices according to the second splitting strategy is performed by repartitioning the multiple initial slices.

4. The parallel computing method according to claim 1, wherein: The step of performing data splitting according to the splitting strategy to obtain multiple data slices includes: performing data splitting according to the first splitting strategy and / or the second splitting strategy to obtain multiple initial slices, and then adjusting the multiple initial slices according to the third splitting strategy to obtain the multiple data slices.

5. The parallel computing method according to claim 1 or 4, characterized in that: The steps of executing the third splitting strategy include: Obtaining the resource scheduling information; Obtaining an operating resource indicator and a secondary splitting indicator, wherein the operating resource indicator is used to represent the usage of the computing resources, and the secondary splitting indicator is used to represent the time cost consumed when performing the adjustment; Splitting is performed based on the operating resource index and the secondary splitting index according to resource scheduling information.

6. The parallel computing method according to claim 5, characterized in that: The computing resources include a first computing unit that is performing the computing and a second computing unit that is not performing the computing, the first computing unit includes the plurality of computing units, The step of performing splitting based on the operating resource indicator and the secondary splitting indicator according to the resource scheduling information includes: Obtaining a computing unit to be allocated in the second computing unit based on the operating resource indicator; Estimate an estimated computing time based on the data to be processed or the multiple initial slices, the first computing unit, and the computing unit to be allocated; Estimate the change in total running time based at least on the estimated calculation time and the secondary splitting indicator; and The splitting is performed based on the total running time change according to resource scheduling information.

7. The parallel computing method according to claim 6, characterized in that: The operating resource indicator includes at least one of the collected CPU utilization, memory utilization, and bandwidth utilization of each computing unit.

8. The parallel computing method according to claim 6, wherein: The step of estimating the change in the total running time includes estimating the change in the total running time based on at least one of the data slice reading consumption and the preprocessing consumption, the expected calculation time and the secondary splitting index.

9. A parallel computing device for distributed data, comprising: A data splitting unit is configured to perform data splitting on the data to be processed according to a splitting strategy to obtain a plurality of data slices; a plurality of computing units configured to perform computing based on the allocated data slices to obtain slice computing results; as well as an aggregation unit, configured to aggregate the slice calculation results; The splitting strategy includes at least one of a first splitting strategy and a second splitting strategy, wherein the first splitting strategy is to perform splitting based on the number of data partitions to be allocated to each computing unit, and the second splitting strategy is to perform splitting based on the number of data rows to be allocated to each computing unit or the total number of the plurality of computing units; The splitting strategy further includes: a third splitting strategy, wherein the third splitting strategy is to perform splitting according to resource scheduling information, wherein the resource scheduling information includes at least one of the expected total running time of the calculation and the computing resources expected to be used to perform the calculation.

10. The parallel computing device according to claim 9, wherein: The data splitting unit is further configured to perform data splitting according to the first splitting strategy to obtain a plurality of initial slices, and then adjust the plurality of initial slices according to the second splitting strategy to obtain the plurality of data slices.

11. The parallel computing device according to claim 10, wherein: The data splitting unit is further configured to perform the step of adjusting the multiple initial slices according to the second splitting strategy by repartitioning the multiple initial slices.

12. The parallel computing device according to claim 9, wherein: The data splitting unit is configured to perform data splitting according to the first splitting strategy and / or the second splitting strategy to obtain a plurality of initial slices, and then adjust the plurality of initial slices according to the third splitting strategy to obtain the plurality of data slices.

13. The parallel computing device according to claim 9 or 12, characterized in that: The apparatus further includes an indicator acquisition unit configured to acquire an operating resource indicator and a secondary splitting indicator, wherein the operating resource indicator is used to represent the usage of the computing resource, and the secondary splitting indicator is used to represent the time cost consumed when performing the adjustment. When performing data splitting according to the third splitting strategy, the data splitting unit is configured to: Obtaining the resource scheduling information; Acquire the operating resource indicator and the secondary splitting indicator from the indicator acquisition unit; Splitting is performed based on the operating resource indicator and the secondary splitting indicator according to the resource scheduling information.

14. The parallel computing device according to claim 13, wherein: The computing resources include a first computing unit that is performing the computing and a second computing unit that is not performing the computing, the first computing unit includes the plurality of computing units, The data splitting unit is further configured to: Obtaining a computing unit to be allocated in the second computing unit based on the operating resource indicator; Estimate an estimated computing time based on the data to be processed or the multiple initial slices, the first computing unit, and the computing unit to be allocated; Estimate a change in total running time based at least on the estimated calculation time and the secondary splitting indicator; and The splitting is performed based on the total running time change according to resource scheduling information.

15. The parallel computing device according to claim 14, wherein: The operating resource indicator includes at least one of the collected CPU utilization, memory utilization, and bandwidth utilization of each computing unit.

16. The parallel computing device according to claim 14, wherein: The data splitting unit is further configured to infer the change in the total running time based on at least one of the data slice reading consumption, the preprocessing consumption, the expected calculation time and the secondary splitting index.

17. A system comprising at least one computing device and at least one storage device storing instructions, wherein: When the instructions are executed by the at least one computing device, the at least one computing device is prompted to perform the parallel computing method for distributed data according to any one of claims 1 to 8.

18. A computer-readable storage medium storing instructions, wherein: When the instructions are executed by at least one computing device, the at least one computing device is prompted to perform the parallel computing method for distributed data according to any one of claims 1 to 8.

Images