Data processing method and device

Abstract

The present invention discloses a data processing method and device. The method comprises: obtaining a query request; judging whether the data in the query condition in the query request matches the pre-stored intermediate data; if the judgment result is yes, obtaining the corresponding data from the intermediate data, and determining the data as the query result; if the judgment result is no, re-obtaining the data, obtaining the intermediate data according to the number and the sum of the values ​​of the obtained data, and calculating the mean of the data according to the intermediate data to obtain the query result corresponding to the query request. The present invention solves the technical problem of high delay and low efficiency caused by the need to perform calculations on a large amount of data in a distributed algorithm.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a data processing method and device. Background Art

[0002] For big data computing, the Map Reduce algorithm is usually used. Each calculation requires reading data from the disk and performing calculations on a large amount of data, which results in high latency. For some different requests, the query conditions may be the same, but the covered data ranges may overlap.

[0003] That is, in the related art: caching the original data, reading the original data into the memory, can save the time of reading data from the disk the next time it is read, but the defect is that the read data still needs to be recalculated; in addition, there is also a big data calculation method: caching the summary results, the next time it is read, if the parameters are exactly the same, the same piece of data is hit and the calculation is also the same, then the cached results can be directly read. However, the defect is that if the parameters change, a part of the data needs to be calculated more, and the cache will be invalid.

[0004] Currently, no effective solution has been proposed to address the problems of high latency and low efficiency caused by the need to perform calculations on a large amount of data in the above-mentioned distributed algorithms. Summary of the invention

[0005] The embodiments of the present invention provide a data processing method and device to at least solve the technical problem of high latency and low efficiency caused by the need to perform calculations on a large amount of data in a distributed algorithm.

[0006] According to one aspect of an embodiment of the present invention, there is provided a data processing method, comprising: obtaining a query request; determining whether the data in the query condition in the query request matches the pre-stored intermediate data; if the judgment result is yes, obtaining corresponding data from the intermediate data, and determining the data as the query result; if the judgment result is no, re-obtaining the data, obtaining the intermediate data based on the number of acquired data and the sum of the values, and calculating the mean of the data based on the intermediate data to obtain the query result corresponding to the query request.

[0007] Optionally, the method also includes: before obtaining the query request, storing each group of data in shards; calculating the intermediate data of the data in each shard respectively; obtaining the intermediate data of the data in each shard, and calculating the mean of the intermediate data of the data in each shard to obtain the mean.

[0008] Further, optionally, storing each set of data in segments includes: acquiring each set of data from a first storage medium, and loading each set of data into a second storage medium.

[0009] Optionally, respectively calculating the intermediate data of the data in each shard includes: when the third storage medium includes the memory of each shard, obtaining the corresponding data from the memory of each shard to obtain the data in each shard; obtaining each group of data from the data in each shard; and obtaining the intermediate data of the data in each shard based on the number of data in each group and the sum of the values ​​of each data in each group of data.

[0010] Further, optionally, a mean is calculated for the intermediate data in each shard to obtain the mean, including: storing the intermediate data in each shard to a fourth storage medium, wherein the fourth storage medium includes the memory of the aggregation node; based on the intermediate data stored in the memory of the aggregation node, calculating the sum of the values ​​in each group of data and the number of data in each group of data to obtain the mean.

[0011] According to one aspect of an embodiment of the present invention, there is provided a data processing device, comprising: a request acquisition module, used to acquire a query request; a judgment module, used to judge whether the data in the query condition in the query request matches the pre-stored intermediate data; a first acquisition module, used to acquire corresponding data from the intermediate data when the judgment result is yes, and determine the data as the query result; a second acquisition module, used to acquire the data again when the judgment result is no, and acquire the intermediate data based on the number of acquired data and the sum of the values, and calculate the mean of the data based on the intermediate data to acquire the query result corresponding to the query request.

[0012] Optionally, the device also includes: a storage module, used to store each group of data in shards before obtaining a query request; a first calculation module, used to calculate the intermediate data of the data in each shard respectively; a second calculation module, used to obtain the intermediate data of the data in each shard, and perform mean calculation on the intermediate data of the data in each shard to obtain the mean.

[0013] Further, optionally, the first computing module includes: an extraction unit, used to obtain corresponding data from each shard memory to obtain data in each shard when the third storage medium includes each shard memory; a data acquisition unit, used to obtain each group of data from the data in each shard; and a computing unit, used to obtain intermediate data of the data in each shard based on the number of each group of data and the sum of the values ​​of each data in each group of data.

[0014] According to one aspect of an embodiment of the present invention, a storage medium is provided, wherein the storage medium includes a stored program, wherein when the program is running, a device where the storage medium is located is controlled to execute the above method.

[0015] According to one aspect of an embodiment of the present invention, a processor is provided, wherein the processor is used to run a program, wherein the above method is executed when the program is run.

[0016] In an embodiment of the present invention, a query request is obtained; it is determined whether the data in the query conditions in the query request matches the pre-stored intermediate data; when the judgment result is yes, the corresponding data is obtained from the intermediate data, and the data is determined as the query result; when the judgment result is no, the data is re-acquired, and the intermediate data is obtained based on the number of acquired data and the sum of the values, and the mean of the data is calculated based on the intermediate data to obtain the query result corresponding to the query request, thereby achieving the purpose of effectively increasing the utilization of the cache, thereby realizing the technical effect of improving the computing efficiency of the distributed algorithm, and further solving the technical problems of high latency and low efficiency caused by the need to perform calculations on a large amount of data in the distributed algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 It is a hardware structure block diagram of a computer terminal of a data processing method according to an embodiment of the present invention;

[0019] Figure 2 is a flow chart of a data processing method according to Embodiment 1 of the present invention;

[0020] Figure 2a is a schematic diagram of a data processing method according to Embodiment 1 of the present invention applied to a query scenario;

[0021] Figure 2b is a schematic diagram of data reading and mean value calculation in a data processing method according to Embodiment 1 of the present invention;

[0022] Figure 2c is a schematic diagram of the time proportion of each step in the data processing method according to the first embodiment of the present invention;

[0023] Figure 2d is a schematic diagram of mean value calculation in a data processing method according to Embodiment 1 of the present invention;

[0024] Figure 2e is a schematic diagram of the time proportion of each step in the data processing method according to the first embodiment of the present invention;

[0025] Figure 3 is a schematic diagram of a data processing device according to a second embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Technical terms involved in this application:

[0029] MapReduce: A distributed computing algorithm that performs preliminary calculations on multiple machines (Map) and sends the preliminary calculation results to one machine, or sends them to multiple machines according to the hash for summary calculations (Reduce).

[0030] Aggregate calculation: A summary calculation performed on data, with multiple data items as input and one data item as output. The corresponding calculation is transformation calculation, which performs calculation on each data item and outputs one result.

[0031] Original data: Map input data.

[0032] Intermediate results: The calculation results of the Map phase are called intermediate results, which are also the input of Reduce.

[0033] Summary result: The calculation result of the Reduce phase is called the summary result.

[0034] Cache: Cache the intermediate results to facilitate the use of data for the next calculation.

[0035] Example 1

[0036] According to an embodiment of the present invention, a method embodiment of a data processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a computer terminal of a data processing method according to an embodiment of the present invention. Figure 1 As shown, the computer terminal 10 may include one or more (only one is shown in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0038] The memory 104 can be used to store software programs and modules of application software, such as program instructions / modules corresponding to the data processing method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the data processing method of the above-mentioned application is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0039] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0040] Under the above operating environment, this application provides Figure 2 The data processing method shown. Figure 2 Flowchart of a data processing method according to Embodiment 1 of the present invention. The data processing method provided by the embodiment of the present application includes:

[0041] Step S202, obtaining a query request;

[0042] Step S204, determining whether the data in the query condition in the query request matches the pre-stored intermediate data;

[0043] Step S206, if the judgment result is yes, obtain the corresponding data from the intermediate data, and determine the data as the query result;

[0044] Step S208, when the judgment result is no, re-acquire the data, obtain the intermediate data according to the number and the sum of the values ​​of the acquired data, and calculate the mean of the data according to the intermediate data to obtain the query result corresponding to the query request.

[0045] Specifically, in combination with step S202 to step S208, the data processing method provided in the embodiment of the present application can be applied to a distributed storage system, especially to a scenario where massive data is to be processed.

[0046] When storing cache, calculation and storage are performed in shards. In subsequent queries, you first need to confirm which complete shards are hit by this query: for fully matched shards, read the intermediate result cache directly from the cache system; for shards that do not fully match, you need to re-read the original data for calculation.

[0047] exist Figure 2aIn the case of , the first shard only needs to calculate two numbers, so it needs to read data 2 and 3, and calculate sum=2+3=5 again; count is 2, and we get (sum=5, count=2). The second and third shards completely hit the cache, and directly read the results (sum=15, count=3) and (sum=24, count=3) from the cache system, and then send the results to the final node for aggregation. At this time, the average is 5.5=(5+15+24) / (2+3+3). In this calculation, only part of the data of the first shard needs to be re-read and calculated. This greatly saves IO and computing resources.

[0048] In an embodiment of the present invention, a query request is obtained; it is determined whether the data in the query conditions in the query request matches the pre-stored intermediate data; when the judgment result is yes, the corresponding data is obtained from the intermediate data, and the data is determined as the query result; when the judgment result is no, the data is re-acquired, and the intermediate data is obtained based on the number of acquired data and the sum of the values, and the mean of the data is calculated based on the intermediate data to obtain the query result corresponding to the query request, thereby achieving the purpose of effectively increasing the utilization of the cache, thereby realizing the technical effect of improving the computing efficiency of the distributed algorithm, and further solving the technical problems of high latency and low efficiency caused by the need to perform calculations on a large amount of data in the distributed algorithm.

[0049] Optionally, the data processing method provided in the embodiment of the present application further includes:

[0050] Step S196, before obtaining the query request, each set of data is stored in segments;

[0051] Among them, by dividing the acquired data into several slices (that is, "grouping the acquired data" in the embodiment of the present application), the data of each slice is obtained.

[0052] Specifically, taking obtaining 9 data as an example, the obtained data are "1, 2, 3, 4, 5, 6, 7, 8, 9" respectively. The above 9 data are grouped to obtain: the first group: "1, 2, 3"; the second group: "4, 5, 6"; the third group: "7, 8, 9".

[0053] Step S198, respectively calculating the intermediate data of the data in each shard;

[0054] Among them, based on each group of data obtained in step S196, the intermediate data is obtained according to the number of each group of data and the sum of the values ​​of each data in each group of data, that is, in the embodiment of the present application, the data structure of the intermediate data of each group may include: {sum, count}; for example, taking the first group of data "1,2,3" as an example, the intermediate data of the first group of data is {sum=6, count=3}, and the intermediate data indicates that the number of the first group of data is 3, and the sum of the values ​​is 6.

[0055] Similarly, the intermediate data of the second and third groups of data are obtained.

[0056] Step S200, obtaining the intermediate data of the data in each shard, and performing mean calculation on the intermediate data of the data in each shard to obtain the mean.

[0057] Among them, based on the intermediate data of each group of data obtained, the mean value is calculated according to the intermediate data, and the mean value of the corresponding 9 data in step S202 can be obtained.

[0058] For example, the intermediate data corresponding to the first, second, and third sets of data are as follows:

[0059] Group 1:

[0060] The first set of data: "1,2,3";

[0061] The middle data of the first set of data {sum=6, count=3};

[0062] Group 2:

[0063] The second set of data: "4,5,6";

[0064] The middle data of the second set of data: {sum=15, count=3};

[0065] Group 3:

[0066] The third set of data: "7,8,9"

[0067] The middle data of the third set of data: {sum=24, count=3};

[0068] By performing mean calculation based on the intermediate data of the above groups of data, the mean of the “9 data” in step S202 can be obtained: (6+15+24) / (3+3+3)=5.

[0069] In summary, the data processing method provided in the embodiment of the present application divides the original data into several shards for execution. After each shard is executed, the intermediate result is obtained and then sent to the final node for summary calculation. Take the calculation of the average value of 9 numbers as an example. The 9 numbers are divided into 3 shards, and each shard calculates its own intermediate result in parallel. The calculation process is divided into two steps: Figure 2b is a schematic diagram of data reading and mean value calculation in a data processing method according to Embodiment 1 of the present invention, such as Figure 2b As shown in the figure, the first step is to read the data from the disk and load it into the memory; the second step is to calculate the intermediate result of the mean. In the mean calculation, the intermediate result is in the form of the number of data and the sum of the values.

[0070] Further, optionally, storing each set of data in segments in step S196 includes: acquiring each set of data from a first storage medium, and loading each set of data into a second storage medium.

[0071] Among them, in the embodiment of the present application, the first storage medium takes a disk as an example, and the second storage medium takes a memory as an example. In the embodiment of the present application, obtaining data from the first storage medium and loading it into the second storage medium includes: reading data from the disk and loading it into the memory.

[0072] Optionally, calculating the intermediate data of the data in each shard separately in step S198 includes: when the third storage medium includes the memory of each shard, obtaining the corresponding data from the memory of each shard to obtain the data in each shard; obtaining each group of data from the data in each shard; and obtaining the intermediate data of the data in each shard based on the number of each group of data and the sum of the values ​​of each data in each group of data.

[0073] Specifically, the third storage medium may include: each shard memory, and in the process of grouping the acquired data, each group of data is stored in each shard memory respectively.

[0074] Further, optionally, a mean is calculated for the intermediate data in each shard to obtain the mean, including: storing the intermediate data in each shard to a fourth storage medium, wherein the fourth storage medium includes the memory of the aggregation node; based on the intermediate data stored in the memory of the aggregation node, calculating the sum of the values ​​in each group of data and the number of data in each group of data to obtain the mean.

[0075] Specifically, the fourth storage medium may include a memory of the aggregation node, and sending the intermediate data of each group of data stored in the third storage medium to the fourth storage medium includes: sending each group of intermediate data from each shard memory to the memory of the aggregation node.

[0076] According to the intermediate data stored in the memory of the aggregation node, the mean value of the data acquired in step S202 is obtained by calculating the sum of the values ​​in each group of data and the number of data in each group of data.

[0077] For example, taking 9 data as an example, the data obtained are "1, 2, 3, 4, 5, 6, 7, 8, 9" respectively. The above 9 data are grouped to obtain: the first group: "1, 2, 3"; the second group: "4, 5, 6"; the third group: "7, 8, 9"; the intermediate data of each group of data are as follows:

[0078] Group 1:

[0079] The first set of data: "1,2,3";

[0080] The middle data of the first set of data {sum=6, count=3};

[0081] Group 2:

[0082] The second set of data: "4,5,6";

[0083] The middle data of the second set of data: {sum=15, count=3};

[0084] Group 3:

[0085] The third set of data: "7,8,9"

[0086] The middle data of the third set of data: {sum=24, count=3};

[0087] By performing mean calculation based on the intermediate data of the above groups of data, the mean of the “9 data” in step S202 can be obtained: (6+15+24) / (3+3+3)=5.

[0088] In summary, the data processing method provided in the embodiment of the present application is different from the prior art in that the embodiment of the present application needs to perform the following four steps in the data calculation of distributed storage:

[0089] The first step is to read the original data from the disk and load it into the memory;

[0090] The second step is to calculate the intermediate data of the mean. In the mean calculation, the intermediate data is in the form of the number and sum of data. Among them, there are a lot of original data, and after calculation, the generated intermediate results occupy very little space.

[0091] In the third step, after the intermediate data of the three shards are calculated and divided, they are sent to the aggregation node for final mean calculation. The data is sent from the memory of each shard to the memory of the aggregation node through the network.

[0092] The fourth step is to calculate the final mean.

[0093] Among them, the first step is to read a large amount of data from the disk, and the disk reading operation takes a lot of time.

[0094] The second step involves the calculation of a large amount of data, which consumes a lot of CPU resources. For some complex calculation types, the delay will be relatively high.

[0095] Because the third step involves the cross-network transmission of a small amount of data, the latency is not very high.

[0096] The fourth step involves the calculation of a small amount of data, and the latency is not very high.

[0097] It can be seen that because the third and fourth steps involve a relatively small amount of data, the computational cost and delay are relatively small. Figure 2c As shown, Figure 2c is a schematic diagram of the time proportion of each step in the data processing method according to the first embodiment of the present invention, Figure 2c The time distribution of the entire calculation process, the first step and the second step together account for 90% of the total calculation time. In order to optimize this part of time and save resource consumption, the data processing method provided in the embodiment of the present application introduces an intermediate data cache. Figure 2d is a schematic diagram of mean value calculation in the data processing method according to the first embodiment of the present invention, such as Figure 2d As shown in the figure, the 9 numbers are divided into 3 shards. Each shard calculates its own intermediate result. The intermediate data corresponding to the mean is the number of data and the sum of the values. The intermediate data of the three shards are saved in the cache system and sent to the final node for summary calculation to obtain the mean of 5.

[0098] Depend on Figure 2d It can be seen that the intermediate data of each shard only contains two types of numbers, and each shard corresponds to three pieces of original data, which in actual use corresponds to tens of millions of data. The reading and calculation costs of the original data are very high. After the above preliminary calculation, the space becomes very small.

[0099] also, Figure 2a is a schematic diagram of a data processing method according to Embodiment 1 of the present invention applied to a query scenario, such as Figure 2a As shown in the figure, when storing cache, calculation and storage are performed in shards. In subsequent queries, you first need to confirm which complete shards are hit by this query: for shards that are fully matched, read the intermediate result cache directly from the cache system; for shards that are not fully matched, you need to re-read the original data for calculation.

[0100] exist Figure 2aIn the case of , the first shard only needs to calculate two numbers, so it needs to read data 2 and 3, and calculate sum=2+3=5 again; count is 2, and we get (sum=5, count=2). The second and third shards completely hit the cache, and directly read the results (sum=15, count=3) and (sum=24, count=3) from the cache system, and then send the results to the final node for aggregation. At this time, the average is 5.5=(5+15+24) / (2+3+3). In this calculation, only part of the data of the first shard needs to be re-read and calculated. This greatly saves IO and computing resources.

[0101] In calculations that utilize cache, Figure 2e is a schematic diagram of the time proportion of each step in the data processing method according to the first embodiment of the present invention, and the time distribution is as follows: Figure 2e As shown in the figure, since only one shard needs to be recalculated and only 2 / 3 of the data in this shard needs to be recalculated, the delay is 2 / 3 of the original delay. For the other two shards, the time is negligible because the cache is fully reused. Due to the use of cache, the computing resources of two shards are saved and can be used for more calculations, and the overall delay is reduced from 10 seconds to 7 seconds.

[0102] The data processing method provided in the embodiment of the present application is described using only 3 pieces of data as an example, but actual cases may have tens of millions of pieces of data, so the optimization effect will be more obvious. The biggest optimization effect is to fully reuse the cache of all shards, reducing the time from 10 seconds to 1 second.

[0103] It should be noted that the data processing method provided in the embodiment of the present application is only illustrated by taking the above example as an example, and is subject to the data processing method provided in the embodiment of the present application, and is not specifically limited.

[0104] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0105] Through the description of the above implementation methods, those skilled in the art can clearly understand that the data processing method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0106] Example 2

[0107] According to one aspect of an embodiment of the present invention, there is provided a data processing device. Figure 3 is a schematic diagram of a data processing device according to Embodiment 2 of the present invention. Figure 3 As shown, it includes: a request acquisition module 32, used to obtain a query request; a judgment module 34, used to judge whether the data in the query condition in the query request matches the pre-stored intermediate data; a first acquisition module 36, used to obtain corresponding data from the intermediate data when the judgment result is yes, and determine the data as the query result; a second acquisition module 38, used to re-acquire the data when the judgment result is no, obtain the intermediate data based on the number of acquired data and the sum of the values, and calculate the mean of the data based on the intermediate data to obtain the query result corresponding to the query request.

[0108] Optionally, the data processing device provided in the embodiment of the present application also includes: a storage module, used to store each group of data in shards before obtaining a query request; a first calculation module, used to separately calculate the intermediate data of the data in each shard; and a second calculation module, used to obtain the intermediate data of the data in each shard, and perform mean calculation on the intermediate data of the data in each shard to obtain the mean.

[0109] Further, optionally, the first computing module includes: an extraction unit, used to obtain corresponding data from each shard memory to obtain data in each shard when the third storage medium includes each shard memory; a data acquisition unit, used to obtain each group of data from the data in each shard; and a computing unit, used to obtain intermediate data of the data in each shard based on the number of each group of data and the sum of the values ​​of each data in each group of data.

[0110] Example 3

[0111] According to one aspect of an embodiment of the present invention, a storage medium is provided, wherein the storage medium includes a stored program, wherein when the program is running, a device where the storage medium is located is controlled to execute the method in the above-mentioned embodiment 1.

[0112] Example 4

[0113] According to one aspect of an embodiment of the present invention, a processor is provided, wherein the processor is used to run a program, wherein the method in the above-mentioned embodiment 1 is executed when the program is run.

[0114] Example 5

[0115] The embodiment of the present invention further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data processing method provided in the first embodiment.

[0116] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0117] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining a query request; determining whether the data in the query conditions in the query request matches the pre-stored intermediate data; if the judgment result is yes, obtaining corresponding data from the intermediate data, and determining the data as the query result; if the judgment result is no, re-obtaining the data, obtaining the intermediate data based on the number of acquired data and the sum of the values, and calculating the mean of the data based on the intermediate data to obtain the query result corresponding to the query request.

[0118] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: before obtaining a query request, storing each group of data in shards; calculating the intermediate data of the data in each shard respectively; obtaining the intermediate data of the data in each shard, and calculating the mean of the intermediate data of the data in each shard to obtain the mean.

[0119] Further, optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: Storing each set of data in segments includes: obtaining each set of data from a first storage medium, and loading each set of data into a second storage medium.

[0120] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: respectively calculating the intermediate data of the data in each shard includes: when the third storage medium includes the memory of each shard, obtaining the corresponding data from the memory of each shard to obtain the data in each shard; obtaining each group of data from the data in each shard; and obtaining the intermediate data of the data in each shard based on the number of each group of data and the sum of the values ​​of each data in each group of data.

[0121] Further, optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: performing mean calculation on the intermediate data of the data in each shard, and obtaining the mean includes: storing the intermediate data of the data in each shard to a fourth storage medium, wherein the fourth storage medium includes the memory of the aggregation node; based on the intermediate data stored in the memory of the aggregation node, obtaining the mean by calculating the sum of the numerical values ​​in each group of data and the number of data in each group of data.

[0122] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0123] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0124] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0125] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0126] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0127] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0128] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data processing method, wherein: include: Get the query request; Determining whether the data in the query condition in the query request matches the pre-stored intermediate data; If the judgment result is yes, obtaining corresponding data from the intermediate data, and determining the data as the query result; If the judgment result is negative, re-acquire the data, obtain the intermediate data according to the number of the acquired data and the sum of the values, and calculate the mean of the data according to the intermediate data to obtain the query result corresponding to the query request; The method further includes: when storing the cache, calculating and storing in units of shards; in querying, for completely matched shards, directly reading the intermediate result cache from the cache system; for shards that are not completely matched, re-reading the original data for calculation; The method further includes: dividing the original data into a number of shards for execution, obtaining intermediate results after each shard is executed, and then sending them to the final node for summary calculation; The method further includes: before obtaining a query request, storing each set of data in shards; calculating the intermediate data of the data in each shard respectively; obtaining the intermediate data of the data in each shard, and calculating the mean of the intermediate data of the data in each shard to obtain the mean.

2. The method according to claim 1, wherein: The sharding storage of each set of data includes: Each set of data is obtained from the first storage medium, and each set of data is loaded into the second storage medium.

3. The method according to claim 1, wherein: The intermediate data of the data in each shard is calculated separately, including: In the case where the third storage medium includes each shard memory, obtaining corresponding data from each shard memory to obtain the data in each shard; Obtain each group of data from the data in each shard; According to the number of each group of data and the sum of the values ​​of each data in each group of data, the intermediate data of the data in each slice is obtained.

4. The method according to claim 3, wherein: The calculating the mean of the intermediate data in each shard to obtain the mean includes: Storing intermediate data of the data in each shard in a fourth storage medium, wherein the fourth storage medium includes a memory of the aggregation node; The mean value is obtained by calculating the sum of the values ​​in each group of data and the number of data in each group of data according to the intermediate data stored in the memory of the aggregation node.

5. A data processing device, wherein: include: A request acquisition module is used to obtain query requests; A judgment module, used to judge whether the data in the query condition in the query request matches the pre-stored intermediate data; A first acquisition module, configured to acquire corresponding data from the intermediate data if the judgment result is yes, and determine the data as the query result; A second acquisition module is used to reacquire data when the judgment result is no, obtain intermediate data according to the number and value sum of the acquired data, and calculate the mean of the data according to the intermediate data to obtain the query result corresponding to the query request; The judgment module is also used to perform calculation and storage in units of shards when storing cache; in query, for completely matched shards, the intermediate result cache is directly read from the cache system; for shards that are not completely matched, the original data needs to be re-read for calculation; The judgment module is also used to divide the original data into several shards for execution. After each shard is executed, the intermediate result is obtained and then sent to the final node for summary calculation; Among them, the request acquisition module is also used to store each group of data in shards before obtaining the query request; calculate the intermediate data of the data in each shard respectively; obtain the intermediate data of the data in each shard, and calculate the mean of the intermediate data of the data in each shard to obtain the mean.

6. The device according to claim 5, wherein: The device also includes: The storage module is used to store each set of data in shards before obtaining a query request; The first calculation module is used to calculate the intermediate data of the data in each shard respectively; The second calculation module is used to obtain the intermediate data of the data in each shard, and perform mean calculation on the intermediate data of the data in each shard to obtain the mean.

7. The device according to claim 6, wherein: The first calculation module includes: An extraction unit, configured to obtain corresponding data from each shard memory when the third storage medium includes each shard memory, so as to obtain the data in each shard; A data acquisition unit, used for acquiring each group of data from the data in each shard; The calculation unit is used to obtain the intermediate data of the data in each slice according to the number of each group of data and the sum of the values ​​of each data in each group of data.

8. A storage medium, wherein: The storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 4.

9. A processor, wherein: The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 4 when running.

Images