A data parallel aggregation method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a data parallel aggregation method, device, electronic equipment and storage medium. The method comprises: distributing data of a data source to a plurality of computing threads according to a first preset rule by a data thread; distributing computing results obtained by the plurality of computing threads in parallel calculation to a plurality of aggregation threads according to a second preset rule by a coordination thread; and obtaining an aggregation result obtained by the plurality of aggregation threads in parallel aggregation according to a third preset rule by a result thread. The technical solution provided by the present disclosure can ensure that the same parallel hash aggregation query instruction is executed on the data of the same data source at any time, and the summary aggregation result obtained each time is consistent.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of database query, and particularly relates to a data parallel aggregation method and device, electronic equipment and storage medium. BACKGROUND

[0002] In a database system, parallel hash aggregation query is a main way of data analysis. However, in practice, for the same data group, the same parallel hash aggregation query instruction is executed at different times, and the query results obtained may be inconsistent. For example, assuming that a table T has columns a and b, the table has 3 data (1, 2), (1, 3) and (2, 4), and a parallel hash aggregation query instruction select a, sum(b) from t group by a limit 1 is executed. The meaning of the instruction is to group the data of the table T according to the value of a, sum the value of b of each group, and output the first aggregation result. In practice, sometimes the output result is (1, 5), and sometimes the output result is (2, 4). Obviously, such results are unreasonable. Therefore, how to ensure that for the same data group, the same parallel hash aggregation query instruction is executed at different times, and the query results obtained are consistent is a problem to be solved at present. SUMMARY

[0003] In order to solve the above technical problems, the present disclosure provides a data parallel aggregation method, device, electronic equipment and storage medium.

[0004] In a first aspect, the present disclosure provides a data parallel aggregation method, characterized in that the method comprises:

[0005] distributing, by a data thread, data of a data source to a plurality of calculation threads according to a first preset rule;

[0006] distributing, by a coordination thread, calculation results obtained by the plurality of calculation threads through parallel calculation to a plurality of aggregation threads according to a second preset rule;

[0007] obtaining, by a result thread, aggregation results obtained by the plurality of aggregation threads through parallel aggregation according to a third preset rule.

[0008] In a second aspect, the present disclosure further provides a data parallel aggregation device, characterized in that the device comprises:

[0009] a first distribution unit, configured to distribute, by a data thread, data of a data source to a plurality of calculation threads according to a first preset rule;

[0010] a second distribution unit, configured to distribute, by a coordination thread, calculation results obtained by the plurality of calculation threads through parallel calculation to a plurality of aggregation threads according to a second preset rule;

[0011] The acquisition unit is configured to acquire, by the result thread, the aggregated result obtained by the plurality of aggregation threads in parallel aggregation according to a third preset rule.

[0012] In a third aspect, the present disclosure provides an electronic device, comprising: a processor and a memory.

[0013] The processor is configured to execute the steps of any of the above methods by invoking programs or instructions stored in the memory.

[0014] In a fourth aspect, the present disclosure provides a computer-readable storage medium storing programs or instructions, which cause a computer to execute the steps of any of the above methods.

[0015] Compared with the prior art, the technical solutions provided by the embodiments of the present disclosure have the following advantages:

[0016] In the technical solutions provided by the embodiments of the present disclosure, the data source data is distributed to the plurality of computing threads by the data thread according to the first preset rule, so that no matter when the same parallel hash aggregation query instruction is executed on the data of the same data source, the input data (i.e., the original data) and the output data (i.e., the calculation result or the intermediate result) of each computing thread are the same; the calculation results obtained by the plurality of computing threads in parallel calculation are distributed to the plurality of aggregation threads by the coordination thread according to the second preset rule, so that no matter when the same parallel hash aggregation query instruction is executed on the data of the same data source, the input data (i.e., the calculation result or the intermediate result) and the output data (i.e., the aggregation result or the final result) of each aggregation thread are the same; the aggregated result obtained by the plurality of aggregation threads in parallel aggregation is acquired by the result thread according to the third preset rule, so that no matter when the same parallel hash aggregation query instruction is executed on the data of the same data source, the order of each input data (i.e., the aggregation result or the final result) received by the result thread is consistent. Since in practice, the result thread usually outputs the input data according to the order of the input data it receives. If the order of each aggregation result received by the result thread is determined, the order of each aggregation result output by the result thread is also determined. Therefore, using the technical solutions provided by the present disclosure can ensure that no matter when the same parallel hash aggregation query instruction is executed on the data of the same data source, the summary aggregation result obtained each time is consistent. Here, the summary aggregation result consistent includes that each aggregation result and the output order of each aggregation result are consistent. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate one embodiment consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, those drawings can also provide other drawings based on the drawings without creative labor.

[0019] Figure 1 A flow chart of a data parallel aggregation method provided by the embodiment of the present disclosure;

[0020] Figure 2 A principle diagram of a parallel hash aggregation query execution stage provided by the embodiment of the present disclosure;

[0021] Figure 3 A flow chart of a data parallel aggregation method provided by the embodiment of the present disclosure;

[0022] Figure 4 A structural schematic diagram of a data parallel aggregation device provided by the embodiment of the present disclosure;

[0023] Figure 5 A hardware structural schematic diagram of an electronic device provided by the embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] In order to more clearly illustrate the above-mentioned purposes, features and advantages of the present disclosure, the solutions of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0025] In the following description, many specific details are set forth in order to fully understand the present disclosure, but the present disclosure can also be implemented in other ways different from those described herein; obviously, the embodiments in the description are only some of the embodiments of the present disclosure, not all the embodiments.

[0026] Figure 1 A flow chart of a data parallel aggregation method provided by the embodiment of the present disclosure. Figure 2 A principle diagram of a parallel hash aggregation query execution stage provided by the embodiment of the present disclosure. Referring to Figure 1 and Figure 2 The method comprises the following steps.

[0027] S110, distributing data of a data source to a plurality of computing threads by a data thread according to a first preset rule.

[0028] The data thread, namely DataWorker, is used to acquire data of a data source and distribute the acquired data to a plurality of computing threads.

[0029] The computing thread, i.e., the PartialWorker, is configured to pre-aggregate the raw data received by the data thread to form an intermediate result.

[0030] The first preset rule is a distribution rule of distributing the data (i.e., the raw data) of the data source to the computing threads. The first preset rule can specify an order in which the raw data is distributed to the computing threads. For example, referring to Figure 2 , it is assumed that there are three computing threads, and the first preset rule can specify that a raw data is sequentially sent to the computing threads in the order of the computing thread 1, the computing thread 2, the computing thread 1, the computing thread 2, and the computing thread 3. By setting the first preset rule, it can be ensured that, at any time, in response to the same parallel hash aggregation query instruction, the data of the same data source is distributed to the same computing thread. That is, at any time, the input data of the computing threads is the same when the same parallel hash aggregation query instruction is executed on the data of the same data source. As can be understood by those skilled in the art, for any computing thread, if the input data is the same, the output data will also be the same. That is, by setting the first preset rule, it can also be ensured that, at any time, the output data of the computing threads is the same when the same parallel hash aggregation query instruction is executed on the data of the same data source.

[0031] Optionally, the first preset rule is polling, i.e., a raw data is sequentially sent to the computing threads in the order of the computing thread number. For example, referring to Figure 2 , the first preset rule is that a raw data is sequentially sent to the computing threads in the order of the computing thread 1, the computing thread 2, and the computing thread 3. In this way, it can be ensured that all the computing threads participate in the data aggregation operation, and the data aggregation rate can be improved.

[0032] There are various implementation methods for this step, which are not limited in the present application. For example, the implementation method of this step includes:

[0033] First, the data thread acquires multiple data from the data source.

[0034] Second, the data thread distributes at least one data to each computing thread according to the first preset rule, so that the multiple computing threads perform parallel computation on the received data.

[0035] Third, after the parallel computation of the multiple computing threads is completed, the data thread again distributes at least one data to each computing thread according to the first preset rule until the multiple data is distributed.

[0036] S120, the coordination thread distributes the computing results obtained by the multiple computing threads in parallel to the multiple aggregation threads according to the second preset rule.

[0037] The coordinator worker is configured to distribute the intermediate results of the plurality of computing threads to the aggregation threads.

[0038] The final worker is configured to calculate the intermediate results received to form the final results.

[0039] The second preset rule is a rule for distributing the intermediate results calculated by the computing threads to the aggregation threads. The second preset rule is configured to ensure that, at any time, the same intermediate result is distributed to the same aggregation thread in response to the same parallel hash aggregation query instruction.

[0040] S130, acquiring the aggregation results obtained by the plurality of aggregation threads in parallel through the result thread according to a third preset rule.

[0041] The result thread is configured to receive the final results (i.e., the aggregation results) obtained by the aggregation threads and output the final results. In other words, the result thread is configured to aggregate all the aggregation results and output the aggregation results.

[0042] The third preset rule is a rule for sending the aggregation results (i.e., the final results) calculated by the aggregation threads to the result thread. The third preset rule is configured to ensure that, at any time, the result thread receives the final results (i.e., the aggregation results) in a consistent order in response to the same parallel hash aggregation query instruction.

[0043] Alternatively, there are multiple specific implementation methods for this step. For example, the specific implementation method for this step can be that the result thread sequentially acquires the corresponding aggregation results from the plurality of aggregation threads. In other words, the result thread is in an active acquisition state, and the result thread requests the aggregation results from the aggregation threads according to a preset order (e.g., the order of the aggregation thread numbers from small to large or the order of the aggregation thread numbers from large to small). For example, if the result thread requests the aggregation result from the mth aggregation thread, the mth aggregation thread has not completed the aggregation calculation, and the result thread waits. When the mth aggregation thread completes the aggregation calculation, the result thread acquires the aggregation result from the mth aggregation thread, and then requests the aggregation result from the m+1th aggregation thread. Here, m is a positive integer.

[0044] Alternatively, the specific implementation method of the present step can be that the result threads output the aggregation results of the plurality of aggregation threads in turn. That is, the aggregation threads are in a passive receiving state, and each result thread sends the aggregation result obtained by it to the result thread according to a preset order (for example, the order of the program numbers of the aggregation threads from small to large, or the order of the program numbers of the aggregation threads from large to small). For example, if the mth aggregation thread is currently scheduled to output the aggregation result to the result thread, if the mth aggregation thread has not completed the aggregation calculation at this time, the aggregation result is output to the result thread after the mth aggregation thread completes the aggregation calculation. After the mth aggregation thread outputs the aggregation result to the result thread, the m+1th aggregation thread is scheduled to output the aggregation result to the result thread. Wherein, m is a positive integer.

[0045] In the above technical solution, the data threads distribute the data of the data source to the plurality of computing threads according to the first preset rule, so that the input data (i.e., the original data) and the output data (i.e., the calculation result or the intermediate result) of each computing thread are the same when the same parallel hash aggregation query instruction is executed on the data of the same data source at any time; the coordination threads distribute the calculation results obtained by the plurality of computing threads to the plurality of aggregation threads according to the second preset rule, so that the input data (i.e., the calculation result or the intermediate result) and the output data (i.e., the aggregation result or the final result) of each aggregation thread are the same when the same parallel hash aggregation query instruction is executed on the data of the same data source at any time; and the result threads obtain the aggregation results obtained by the plurality of aggregation threads in parallel according to the third preset rule, so that the order of receiving each piece of input data (i.e., the aggregation result or the final result) by the result thread is consistent when the same parallel hash aggregation query instruction is executed on the data of the same data source at any time. Since in practice, the result thread usually forwards and outputs the input data according to the order of receiving the input data, if the result thread receives each piece of aggregation result in a certain order, the order of outputting each piece of aggregation result is also certain. Therefore, using the technical solution provided by the present disclosure can ensure that the summary aggregation result is consistent each time when the same parallel hash aggregation query instruction is executed on the data of the same data source. Here, the consistent summary aggregation result includes the consistent aggregation result and the consistent output order of the aggregation result.

[0046] It should be emphasized that the above aggregation method is not affected by the CPU scheduling mode. That is, no matter how the CPU schedules the threads, the original data input by each computing thread is consistent, and the calculation result output by each computing thread is consistent; the calculation result input by each aggregation thread is consistent, and the aggregation result output by each aggregation thread is consistent; and the summary aggregation result obtained by the result thread is consistent. Here, the consistent summary aggregation result includes the consistent aggregation result and the consistent output order of the aggregation result.

[0047] Figure 3 A flowchart illustrating a data parallel aggregation method provided in an embodiment of this disclosure. See also... Figure 2 and Figure 3 The parallel data aggregation method includes:

[0048] S210. Distribute the data from the data source to multiple computing threads according to the first preset rule through the data thread.

[0049] For example, suppose table T1 has columns a and b, and table T1 contains 3n rows of data, namely data1, data2, data3, data4, data5, data6, ..., data3n. Execute the parallel hash aggregation query command `select a,sum(b)from tgroup by a limit 1`. This command means to group the data in table T according to the value of 'a', sum the values ​​of 'b' in each group, and output the first row in the aggregation result.

[0050] See Figure 3 Data 1 is sent to computation thread 1, data 2 to computation thread 2, and data 3 to computation thread 3, so that the three computation threads can perform parallel computations. After all three threads have completed their computations, data 4 is sent to computation thread 1, data 5 to computation thread 2, and data 6 to computation thread 3, so that the three computation threads can perform parallel computations again. This process is repeated until all 3n data items have been distributed.

[0051] S220. Multiple computing threads compute the received data in parallel, and each computing thread stores its own computation result in its own hash table. The key of the hash table is the aggregation line program number, and the value of the hash table is the computation result.

[0052] Optionally, for a calculation result, the aggregation line program number corresponding to the calculation result is the remainder obtained by taking the modulo operation between the calculation result and the number of aggregation threads.

[0053] For example, see [link to previous article] Figure 2 Calculation thread 1 performs preliminary aggregation on the received data 1, data 4, ..., and data 3n-2, that is, grouping according to the value of 'a' and summing the value of 'b' for each group. Similarly, calculation thread 2 performs preliminary aggregation on the received data 2, data 5, ..., and data 3n-1, that is, grouping according to the value of 'a' and summing the value of 'b' for each group. Calculation thread 3 performs preliminary aggregation on the received data 3, data 6, ..., and data 3n, that is, grouping according to the value of 'a' and summing the value of 'b' for each group. The specific calculation methods of the calculation threads are existing technologies and will not be elaborated here.

[0054] Specifically, assume that the n data received by computation thread 1 can be divided into 6 groups based on the value of 'a': the group with a=1, the group with a=2, the group with a=3, the group with a=4, the group with a=5, and the group with a=6. Computation thread 1 calculates the sum of the data b belonging to the group with a=1, and uses (1, b11) as the result for the group with a=1. This yields the results for all computation threads. For example, Table 1 shows the results for three computation threads.

[0055] Table 1

[0056]

[0057]

[0058] When storing the calculation result (1, b11) into the hash table of calculation thread 1, the result is calculated using 1 (i.e., the value of a) and the number of aggregation threads. Figure 2 In this process, there are three aggregation threads, so the total number of aggregation threads is 3. The remainder of the modulo operation is 1, meaning the key in the hash table is 1 and the value is (1, b11). The key being 1 indicates that the calculation result (1, b11) will subsequently be input into aggregation thread 1. In this way, all calculation results can be stored in the hash table. For example, Table 2 shows the hash tables for each calculation thread.

[0059] Table 2

[0060]

[0061] S230. The coordinating thread sequentially distributes the calculation results to the corresponding aggregation thread from the hash tables of the multiple calculation threads, based on the aggregation thread program number.

[0062] There are several ways to implement this step. For example, one possible method is to distribute the results of all calculations in one hash table to the next hash table via a coordinating thread. For example, referring to Table 2, the calculation results of calculation thread 1 are stored in hash table 1, the calculation results of calculation thread 2 are stored in hash table 2, and the calculation results of calculation thread 3 are stored in hash table 3. When executing this step, the process starts with hash table 1, distributing all calculation results in hash table 1 to the corresponding aggregation thread, then distributing all calculation results in hash table 2 to the corresponding aggregation thread, and finally distributing all calculation results in hash table 3 to the corresponding aggregation thread.

[0063] Alternatively, the implementation method of the step includes: after completing the distribution of all the calculation results corresponding to the same aggregation thread program number in all the hash tables, the next aggregation thread program number corresponding to all the calculation results is distributed by the coordination thread. For example, referring to Table 2, the calculation result of the calculation thread 1 is stored in the hash table 1, the calculation result of the calculation thread 2 is stored in the hash table 2, and the calculation result of the calculation thread 3 is stored in the hash table 3. Each hash table includes 6 calculation results. When performing the step, first, all the calculation results corresponding to the aggregation thread program number 1 (i.e., Key is 1) in the hash table 1 are distributed to the aggregation thread 1, and then all the calculation results corresponding to the aggregation thread program number 1 (i.e., Key is 1) in the hash table 2 are distributed to the aggregation thread 1, and finally all the calculation results corresponding to the aggregation thread program number 1 (i.e., Key is 1) in the hash table 3 are distributed to the aggregation thread 1. Second, all the calculation results corresponding to the aggregation thread program number 2 (i.e., Key is 2) in the hash table 1 are distributed to the aggregation thread 2, and then all the calculation results corresponding to the aggregation thread program number 2 (i.e., Key is 2) in the hash table 2 are distributed to the aggregation thread 2, and finally all the calculation results corresponding to the aggregation thread program number 2 (i.e., Key is 2) in the hash table 3 are distributed to the aggregation thread 2. Third, all the calculation results corresponding to the aggregation thread program number 3 (i.e., Key is 3) in the hash table 1 are distributed to the aggregation thread 3, and then all the calculation results corresponding to the aggregation thread program number 3 (i.e., Key is 3) in the hash table 2 are distributed to the aggregation thread 3, and finally all the calculation results corresponding to the aggregation thread program number 3 (i.e., Key is 3) in the hash table 3 are distributed to the aggregation thread 3.

[0064] Table 3 is the data received by each aggregation thread after the data in Table 2 is distributed to each aggregation thread.

[0065] Table 3

[0066]

[0067]

[0068] S240, each of the plurality of aggregation threads performs parallel aggregation on the calculation result received by the aggregation thread based on the order of the received calculation result.

[0069] The specific aggregation method of the aggregation thread is a prior art, which is not described here.

[0070] S250, the result thread obtains the aggregation result obtained by the plurality of aggregation threads in parallel according to a third preset rule.

[0071] Optionally, the result thread sequentially obtains the corresponding aggregation result from the plurality of aggregation threads; or the result thread sequentially outputs the aggregation result of the plurality of aggregation threads.

[0072] The technical solution described above stores the respective calculation results of the plurality of calculation threads into respective hash tables, wherein the key of the hash table is the aggregation thread program number and the value of the hash table is the calculation result. The coordination thread sequentially distributes the calculation results to the corresponding aggregation thread based on the aggregation thread program number from the respective hash tables of the plurality of calculation threads. In essence, the same parallel hash aggregation query instruction is executed on the same data source at any time, and the input data (i.e., the calculation result or intermediate result) and the output data (i.e., the aggregation result or final result) of each aggregation thread are the same, thereby ensuring that the order of the final output of each aggregation result is consistent.

[0073] Optionally, when S210 is executed, the order in which the calculation thread receives the first data of each group from the data thread is recorded; S230 is replaced by: the coordination thread sequentially distributes the calculation results to the corresponding aggregation thread based on the aggregation thread program number and the order from the respective hash tables of the plurality of calculation threads. For example, assume that table T has columns a and b, and the table includes (1, 2), (1, 3), (2, 4), and (2, 8), and these four data will be sent to the calculation thread in turn. Grouping by the value of a, (1, 2) is the first data of the a = 1 group, and the time at which the data (1, 2) is received is recorded. (2, 4) is the first data of the a = 2 group, and the time at which the data (2, 4) is received is recorded. Thus, it can be determined that the first data of the a = 1 group is received before the first data of the a = 2 group. When the calculation result is output to the aggregation thread, the calculation result of the a = 1 group is output first, and then the calculation result of the a = 2 group is output. This arrangement ensures that the order in which each aggregation thread receives each calculation result is fixed. This arrangement ensures that when the same parallel hash aggregation query instruction is executed on the same data source, the order of the calculation result output by the same calculation thread is consistent, and further ensures that the order of the final output of each aggregation result is consistent.

[0074] Similarly, when performing S230, the record aggregation thread can optionally record the order in which the first data of each packet is received from the computing thread; S250 is replaced by: obtaining the aggregation results obtained by the plurality of aggregation threads in parallel aggregation by the result thread according to a third preset rule and the order. Assuming that a certain aggregation thread sequentially obtains intermediate results (1, 5), (1, 6), (2, 7) and (2, 8), and groups the values with a, the aggregation result obtained after aggregation is. Since the first data (1, 5) of the a = 1 group is received before the first data (2, 7) of the a = 2 group, when the result thread obtains the aggregation result from the aggregation thread, the aggregation result of the a = 1 group is obtained first, and then the aggregation result of the a = 2 group is obtained. This setting can ensure that when the same parallel hash aggregation query instruction is executed on the data of the same data source, the order of the aggregation results output by the same aggregation thread is consistent, and further ensures that the order of the final output aggregation results is consistent.

[0075] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the order of the described actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0076] Figure 4 A structural schematic diagram of a data parallel aggregation device proposed for the embodiments of the present disclosure is shown in FIG. 4. As shown in FIG. 4, the device includes: Figure 4

[0077] A first distribution unit 410 is configured to distribute the data of the data source to a plurality of computing threads by a data thread according to a first preset rule;

[0078] A second distribution unit 420 is configured to distribute the computing results obtained by the plurality of computing threads in parallel calculation to a plurality of aggregation threads by a coordination thread according to a second preset rule;

[0079] An obtaining unit 430 is configured to obtain the aggregation results obtained by the plurality of aggregation threads in parallel aggregation by a result thread according to a third preset rule.

[0080] Further, the first distribution unit 410 is configured to:

[0081] obtain a plurality of data from a data source by a data thread;

[0082] distribute at least one data to each computing thread by the data thread according to a first preset rule, so that the plurality of computing threads calculate the received data in parallel; ​

[0083] After the parallel computation of the plurality of computing threads is completed, the data threads are used to distribute at least one piece of data to each computing thread again according to a first preset rule until the plurality of pieces of data are distributed.

[0084] Further, the first preset rule is polling.

[0085] Further, the apparatus further comprises a computing unit configured to control the plurality of computing threads to perform parallel computation on the received data after the data threads distribute the data of the data source to the plurality of computing threads according to the first preset rule, and the plurality of computing threads store respective computation results in respective hash tables, wherein a key of the hash table is an aggregate thread program number, and a value of the hash table is the computation result.

[0086] Further, for one computation result, the aggregate thread program number corresponding to the computation result is a remainder obtained by performing a modulo operation on the computation result and the number of aggregate threads.

[0087] Further, the second distribution unit 420 is configured to:

[0088] The coordination threads sequentially distribute the computation results to corresponding aggregate threads based on the aggregate thread program number from the hash tables of the plurality of computing threads.

[0089] Further, the second distribution unit 420 is configured to:

[0090] After the distribution of all computation results in one hash table is completed, the coordination threads perform the distribution of computation results in the next hash table.

[0091] Further, the second distribution unit 420 is configured to:

[0092] After the distribution of all computation results corresponding to the same aggregate thread program number in all hash tables is completed, the coordination threads perform the distribution of all computation results corresponding to the next aggregate thread program number.

[0093] Further, the apparatus further comprises an aggregation unit configured to control the plurality of aggregate threads to perform parallel aggregation on the respective received computation results based on the order of the received computation results after the coordination threads distribute the computation results obtained by the parallel computation of the plurality of computing threads according to a second preset rule.

[0094] Further, the acquisition unit 430 is configured to:

[0095] The result threads sequentially acquire corresponding aggregate results from the plurality of aggregate threads; or

[0096] The aggregation results of the plurality of aggregation threads are output in sequence by a result thread.

[0097] The apparatuses disclosed in the above embodiments can implement the methods disclosed in the above method embodiments, and have the same or corresponding beneficial effects. To avoid repetition, no further elaboration is given here.

[0098] Figure 5 A hardware structure schematic diagram of an electronic device provided by the embodiments of the present disclosure is shown in FIG. 1, which can include a smart terminal such as a mobile phone or a PAD. The electronic device includes: Figure 5

[0099] one or more processors 301, Figure 5 In an example, the processor 301 is taken as an example.

[0100] a memory 302,

[0101] The electronic device can further include an input device 303 and an output device 304.

[0102] The processor 301, the memory 302, the input device 303 and the output device 304 in the electronic device can be connected through a bus or other means, Figure 5 In an example, the connection through the bus is taken as an example.

[0103] The memory 302 as a kind of non-transient computer readable storage medium, it can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the data parallel aggregation method in the embodiments of the present disclosure. The processor 301 executes the various functions of the server and data processing by running the software programs, instructions and modules stored in the memory 302, that is, the data parallel aggregation method of the above method embodiments is realized.

[0104] The memory 302 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required by at least one function; the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 302 can include a high-speed random access memory, and can also include a non-transient memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transient solid-state memory device. In some embodiments, the memory 302 can optionally include a memory remotely arranged with respect to the processor 301, which can be connected to the terminal device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0105] ​The input device 303 can be used to receive inputted digital or character information, and to generate key signal input related to user settings and function control of the electronic device. The output device 304 can include a display device such as a display screen.

[0106] The embodiments of the present disclosure further provide a computer readable storage medium storing programs or instructions, which cause a computer to execute a data parallel aggregation method, the method comprising:

[0107] distributing data of the data source to a plurality of computing threads according to a first preset rule by a data thread;

[0108] distributing computing results obtained by the plurality of computing threads in parallel computation to a plurality of aggregation threads according to a second preset rule by a coordination thread;

[0109] obtaining an aggregation result obtained by the plurality of aggregation threads in parallel aggregation according to a third preset rule by a result thread.

[0110] Optionally, the computer executable instructions, when executed by the computer processor, can further be used to execute the technical solutions of the data parallel aggregation method provided by any of the embodiments of the present disclosure.

[0111] From the above description about the embodiments, those skilled in the art can clearly understand that the present disclosure can be realized by means of software and necessary universal hardware, and of course can also be realized by hardware, but in many cases the former is a better implementation. Based on such understanding, the technical solutions of the present disclosure can be embodied in the form of a software product, and the computer software product can be stored in a computer readable storage medium, such as a floppy disk, a ROM, a RAM, a FLASH, a hard disk, or an optical disc, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present disclosure.

[0112] It has to be noted that, in the present document, relational terms are intended only to convey a possible relationship between elements or

[0113] The above description is merely that of the specific embodiments of the present disclosure and therefore is not intended to limit the present disclosure. Various modifications made to the embodiments of the present disclosure will be apparent to those skilled in the art to which the present disclosure pertains, and such modifications are not to be interpreted within the scope or spirit of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein but will be construed to include all modifications, equivalents, and substitutes falling within the scope of the present disclosure.

Claims

1. A data parallel aggregation method, characterized in that, The method comprises the following steps: distributing data of a data source to a plurality of computing threads by a data thread according to a first preset rule; distributing computing results obtained by the plurality of computing threads through parallel computing to a plurality of aggregation threads by a coordination thread according to a second preset rule; obtaining aggregation results obtained by the plurality of aggregation threads through parallel aggregation by a result thread according to a third preset rule; wherein the step of distributing data of a data source to a plurality of computing threads by a data thread according to a first preset rule comprises: obtaining a plurality of data from the data source by the data thread; distributing at least one piece of data to each computing thread by the data thread according to a first preset rule, so that the plurality of computing threads perform parallel computing on the received data; after the parallel computing of the plurality of computing threads is completed, distributing at least one piece of data to each computing thread again by the data thread according to the first preset rule, until the plurality of data is distributed completely.

2. The method of claim 1, wherein, The first preset rule is polling.

3. The method of claim 1, wherein, After the step of distributing data of a data source to a plurality of computing threads by a data thread according to a first preset rule, the method further comprises: the plurality of computing threads perform parallel computing on the received data, and the plurality of computing threads store respective computing results in respective hash tables, wherein the key of the hash table is an aggregation thread number, and the value of the hash table is a computing result.

4. The method of claim 3, wherein, For one computing result, the aggregation thread number corresponding to the computing result is the remainder obtained by taking the modulus of the computing result and the number of aggregation threads.

5. The method of claim 3, wherein, The step of distributing computing results obtained by the plurality of computing threads through parallel computing to a plurality of aggregation threads by a coordination thread according to a second preset rule comprises: distributing computing results from the respective hash tables of the plurality of computing threads to corresponding aggregation threads based on the aggregation thread number by the coordination thread in sequence.

6. The method of claim 5, wherein, The step of distributing computing results from the respective hash tables of the plurality of computing threads to corresponding aggregation threads based on the aggregation thread number by the coordination thread in sequence comprises: after the distribution of all computing results in one hash table is completed, the coordination thread performs the distribution of computing results in the next hash table.

7. The method of claim 5, wherein, The step of distributing computing results from the respective hash tables of the plurality of computing threads to corresponding aggregation threads based on the aggregation thread number by the coordination thread in sequence comprises: after the distribution of all computing results corresponding to one aggregation thread number in all hash tables is completed, the coordination thread performs the distribution of all computing results corresponding to the next aggregation thread number.

8. The method of claim 1, wherein, After the step of distributing computing results obtained by the plurality of computing threads through parallel computing to a plurality of aggregation threads by a coordination thread according to a second preset rule, the method further comprises: the plurality of aggregation threads perform parallel aggregation on the respective received computing results based on the order of the received computing results.

9. The method of claim 1, wherein, The step of obtaining aggregation results obtained by the plurality of aggregation threads through parallel aggregation by a result thread according to a third preset rule comprises: the result thread obtains corresponding aggregation results from the plurality of aggregation threads in sequence; or the result thread outputs the aggregation results of the plurality of aggregation threads in sequence.

10. A data parallel aggregation apparatus, characterized by The device comprises: The first distribution unit is configured to distribute data of a data source to a plurality of computing threads according to a first preset rule through a data thread; The second distribution unit is configured to distribute computing results obtained by the plurality of computing threads through a coordination thread according to a second preset rule to a plurality of aggregation threads; The acquisition unit is configured to acquire an aggregation result obtained by the plurality of aggregation threads through a result thread according to a third preset rule; The first distribution unit is configured to: acquire a plurality of data from the data source through the data thread; distribute at least one piece of data to each computing thread through the data thread according to the first preset rule, so that the plurality of computing threads perform parallel computation on the received data; after the parallel computation of the plurality of computing threads is completed, distribute at least one piece of data to each computing thread again through the data thread according to the first preset rule, until the plurality of data is completely distributed.

11. An electronic device, comprising: comprise: a processor and a memory; the processor is configured to execute the steps of the method according to any one of claims 1 to 9 by calling programs or instructions stored in the memory.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores programs or instructions, which enable the computer to execute the steps of the method according to any one of claims 1 to 9.

Images