A distributed key-value database column data and statistics system and method

Abstract

A distributed key-value database column data and statistics system and method includes more than two data storage servers and statistics servers; the data storage server includes a data shard basic statistical value management module; the data shard basic statistical value management module is used to manage basic statistical values, which correspond to data shards; the data shard statistical value change management module is used to manage statistical change values, which correspond to data shards, and are used to record the change statistics of data columns; the application decomposes the calculation operation of "the statistical sum of a column of data in an ultra-large-scale data list" into local calculations of each data storage data shard, summarizes the statistical sum of the entire data, and changes it into the sum of the basic statistical value and the statistical change value in each data storage data shard, greatly improving the operating efficiency of artificial intelligence or large-scale databases and greatly reducing power consumption.

Description

Technical Field

[0001] The present application belongs to the technical field of large-scale data storage, and in particular relates to a distributed key-value database column data and statistics system and method. Background Art

[0002] Artificial Intelligence: AI is a branch of computer science that seeks to develop systems and algorithms that can simulate and execute intelligent human behavior. AI technologies include machine learning, deep learning, natural language processing, and computer vision. By training on large amounts of data, AI systems can automatically perform data analysis, pattern recognition, and predictions.

[0003] Big data: Big data refers to data sets whose volume, velocity, and diversity exceed the capabilities of traditional data processing. Big data technologies encompass data collection, storage, processing, and analysis, using tools and platforms such as Hadoop, Spark, and NoSQL databases. The value of big data lies in uncovering hidden information and knowledge through analysis of vast amounts of data, thereby supporting decision-making.

[0004] Key-Value storage system, key-value distributed storage system has fast query speed, large data storage capacity, and supports high concurrency. It is very suitable for querying by primary key, but cannot perform complex conditional queries.

[0005] The statistical sum in this application refers to the sum of a large number of values ​​of the same data category of the same data type, such as the sum of the same column members of a large data table, or the sum of the same column data of the same data object in a distributed database.

[0006] If supplemented with a real-time search engine for complex conditional searches and full-text retrieval, it can replace relational databases like MySQL, which have lower concurrency performance, achieving high concurrency and performance while saving dozens of times the number of servers. Key-value distributed storage, such as MemcacheDB and Tokyo Tyrant, can easily complete high-speed queries with tens of thousands of concurrent connections.

[0007] Large-scale artificial intelligence systems or large-scale databases have highly complex big data storage systems at the bottom or middle layer. During the operation of artificial intelligence systems, large-scale data needs to be processed and calculated, such as obtaining the statistical sum of a column of data in a very large-scale data list in real time.

[0008] Obtaining the statistical sum of a column of data in a very large-scale data list in real time is a basic operation for artificial intelligence computing. This operation can be understood as thousands of computers performing data "addition" operations at the same time. This operation occurs transiently and randomly. This operation is the fundamental reason for the power consumption of large-scale artificial intelligence or databases, and has currently become a major bottleneck in the development of the field of artificial intelligence. Summary of the Invention

[0009] This application decomposes the calculation operation of "the statistical sum of a column of data in an ultra-large-scale data list" into local calculations of each data storage data slice, and summarizes the statistical sum of the entire data. During the local calculation process, the basic statistical value is used to save the stage statistical value, and the statistical change value is used to save the change value during the change process. The statistical sum operation that consumes huge electric energy is changed into the sum of the basic statistical value and the statistical change value in each data storage data slice. In this way, the calculation amount of the statistical sum is reduced to less than one ten-thousandth, and the power consumption is reduced to less than one millionth, which greatly improves the operating efficiency of artificial intelligence or large-scale databases and greatly reduces power consumption.

[0010] A distributed key-value database column data and statistics system comprises two or more data storage servers and a statistics server; the data storage server is used to store one or more data shards, the data shards are used to store data, the data are indexed by key values, and the key values ​​are stored in the data shards in order; the data storage server comprises a data shard basic statistical value management module; the data shard basic statistical value management module is used to manage basic statistical values, the basic statistical values ​​correspond to the data shards, and the basic statistical values ​​correspond to the statistical sum of a data column of the single-row multi-column distributed key-value data; the data storage server comprises a data shard statistical value change management module; the data shard statistical value change management module is used to manage statistical change values, the statistical change values ​​correspond to the data shards, and the statistical change values ​​correspond to the statistical sum and change values ​​of a data column of the single-row multi-column distributed key-value data; the statistical change values ​​are used to record the change statistics of the data column; the statistics server obtains the basic statistical values ​​and statistical change values ​​of the data columns corresponding to the data shards from the data storage server, calculates the sum of all basic statistical values ​​and statistical change values, and obtains the statistical sum of the data columns; the basic statistical values ​​are recorded according to versions, and the statistical change values ​​are recorded according to versions;

[0011] Each time a data shard is split, a new version base statistic value is generated, the base statistic value is recalculated, and a new version statistic change value is generated. The initial value of the statistic change value is zero. The recalculation of the base statistic value is a background calculation and does not lock the shard.

[0012] According to the above-mentioned distributed key-value database column data and statistical system, after each data shard is merged, a new version basic statistical value is generated, the basic statistical value is recalculated, and a new version statistical change value is generated. The initial value of the statistical change value is zero. The recalculation of the basic statistical value is a background calculation and does not lock the shard.

[0013] According to the above-mentioned distributed key-value database column data and statistical system, when a data shard is first counted, a new version basic statistical value is generated, the basic statistical value is calculated, and a new version statistical change value is generated. The initial value of the statistical change value is zero.

[0014] According to the above-mentioned distributed key-value database column data and statistical system, after the data shards are split or merged, after receiving the first statistical request sent by the statistics server, a new version basic statistical value is generated, the basic statistical value is calculated, and a new version statistical change value is generated. The initial value of the statistical change value is zero.

[0015] According to the above-mentioned distributed key-value database column data and statistical system, the above-mentioned statistical change value is recorded according to version; the version of the statistical change value corresponds to the version of the basic statistical value; and the starting value of the statistical change value of the new version is zero.

[0016] According to the above-mentioned distributed key-value database column data and statistical system, a key-value data is added to the above-mentioned data shard, and the above-mentioned statistical change value is equal to the existing value plus the column data corresponding to the newly added key-value.

[0017] According to the above-mentioned distributed key-value database column data and statistical system, when a key-value data is deleted from the above-mentioned data shard, the above-mentioned statistical change value is equal to the existing value minus the column data corresponding to the deleted key-value data.

[0018] A distributed key-value database column data and statistics method, wherein data is stored in data shards in index key value order; multiple data shards are stored in two or more servers; each data piece includes at least one data column, and one data column is selected as a statistical object; the sum of the data columns in each data shard is calculated and recorded as a basic statistical value; when data in each data shard changes, the change value of the sum of the data columns is calculated and recorded as a statistical change value; the value of the statistical object is equal to the sum of the basic statistical value and the statistical change value in each data shard; the basic statistical value is recorded by version, and the statistical change value is recorded by version;

[0019] Each time a data shard is split, a new version basic statistical value is generated and the basic statistical value is recalculated; a new version statistical change value is generated, and the initial value of the statistical change value is zero; the recalculation of the basic statistical value is a background calculation and does not lock the shard.

[0020] After each data shard is merged, a new version basic statistical value is generated, the basic statistical value is recalculated, and a new version statistical change value is generated. The initial value of the statistical change value is zero; the recalculation of the basic statistical value is a background calculation and does not lock the shard.

[0021] A version record is generated each time a shard is newly generated, split, or merged.

[0022] According to the above-mentioned distributed key-value database column data and statistical method, when the data shard is first counted, a new version basic statistical value is generated, the basic statistical value is calculated, and a new version statistical change value is generated. The initial value of the statistical change value is zero.

[0023] According to the above-mentioned distributed key-value database column data and statistical method, after the data shards are split or merged, after receiving the first statistical request, a new version basic statistical value is generated, the basic statistical value is recalculated, and a new version statistical change value is generated. The initial value of the statistical change value is zero.

[0024] According to the above-mentioned distributed key-value database column data and statistical method, when a data shard is first counted, a new version statistical change value is generated, and the initial value of the statistical change value is zero.

[0025] According to the above-mentioned distributed key-value database column data and statistical method, a key-value data is added to the above-mentioned data shard, and the above-mentioned statistical change value is equal to the existing value plus the column data corresponding to the newly added key-value.

[0026] According to the above-mentioned distributed key-value database column data and statistical method, when a key-value data is deleted from the above-mentioned data shard, the above-mentioned statistical change value is equal to the existing value minus the column data corresponding to the deleted key-value data.

[0027] The beneficial effects of the technical solution in the present application include: using basic statistical values ​​to save stage statistical values, using statistical change values ​​to count the change values ​​during the change process, and changing the statistical sum operations that consume huge amounts of electricity into the sum of basic statistical values ​​and statistical change values ​​in each data storage data shard. This method changes the query-triggered calculation mode to one calculation for multiple queries, directly eliminating the query calculation amount, and reducing the statistical sum calculation amount to less than one ten-thousandth.

[0028] The beneficial effects of the technical solution in the present application include: using basic statistical values ​​to save stage statistical values, using statistical change values ​​to count the change values ​​during the change process, the calculation of basic statistical values ​​after the data shards are split can be background calculation, and during the data insertion or deletion process, the statistical change values ​​can record temporary change values, and the calculation of statistical sums does not lock the shards, which greatly improves the operability of the data. In traditional methods, after the split, the calculation of statistical sums requires locking the shards, resulting in the inability to write operations to the data shards during the statistical period.

[0029] The beneficial effects of the technical solution in the present application include: using basic statistical values ​​to save stage statistical values, using statistical change values ​​to count the change values ​​during the change process, and the calculation of basic statistical values ​​after the data shards are split can be background calculation. In the background calculation process, the basic statistical values ​​are calculated only once, and there is no need to repeatedly calculate each time the statistical sum is queried, which greatly reduces the computing overhead and saves power consumption.

[0030] The beneficial effects of the technical solution in this application include: using basic statistical values ​​to save stage statistical values, calculating when sharding actions occur or merge, ensuring the relative stability of basic values, and using statistical change values ​​to record the changes brought about by each row and column data operation, which actually has the effect of changing the column data and the statistical values ​​also changing synchronously, and actually decomposing statistics and calculations into data change operations.

[0031] The beneficial effects of the technical solution in this application include: using versions to manage statistical change values ​​and statistical change values, and in a distributed computing environment, multi-version backup can be achieved to prevent data loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a block diagram of a distributed key-value database column data and statistical system;

[0033] Figure 2 This is a schematic diagram of data shard K2 before splitting;

[0034] Figure 3 This is a schematic diagram after the data shard K2 is split;

[0035] Figure 4 This is a schematic diagram of calculating statistics and by version;

[0036] Figure 5 This is a diagram showing the statistical sum calculated by version after the K2 split;

[0037] Figure 6 It is a data sharding diagram for recording urban electricity consumption;

[0038] Figure 7 This is a data recording diagram for the city electricity data slice K2;

[0039] Figure 8 This is a schematic diagram of the split of shard K2 recording the city electricity data;

[0040] Figure 9 This is a schematic diagram of the data fragment K21 after the city electricity consumption is generated. DETAILED DESCRIPTION

[0041] The following is a further detailed description of the present application in conjunction with the accompanying drawings. It should be noted that the following is a description of preferred embodiments of the present invention and does not constitute any limitation to the present invention. The description of the preferred embodiments of the present invention is only intended to illustrate the general principles of the present invention.

[0042] like Figure 1 A distributed key-value database column data and statistics system includes a statistics server and two or more data storage servers, such as server A, server B, and server C.

[0043] The data storage server is used to store one or more data shards, such as data shard K1, data shard K2, and data shard K3. The above data shards are used to store data, and the above data are indexed by key values. The above key values ​​are stored in the data shards in order.

[0044] The data storage server includes a data shard basic statistical value management module; the above-mentioned data shard basic statistical value management module is used to manage basic statistical values, the above-mentioned basic statistical values ​​correspond to the above-mentioned data shards, and the basic statistical values ​​correspond to the statistical sum of a data column of the above-mentioned single-row multi-column distributed key-value data.

[0045] The data storage server includes a data shard statistical value change management module; the above-mentioned data shard statistical value change management module is used to manage statistical change values, the above-mentioned statistical change values ​​correspond to the above-mentioned data shards, and the statistical change values ​​correspond to the statistical sum change values ​​of a data column of the above-mentioned single-row multi-column distributed key-value data; the statistical change values ​​are used to record the change statistics of the above-mentioned data columns; the statistical server obtains the basic statistical values ​​and statistical change values ​​of the data columns corresponding to the data shards from the data storage server, calculates the sum of all basic statistical values ​​and statistical change values, and obtains the statistical sum of the above-mentioned data columns.

[0046] A server can include multiple data shards. If the reliability of data shards is considered, each data shard includes one or more data backups, and the operations of the data backup shards can be consistent with those of the data shards. Alternatively, the statistical sum can be calculated only on the primary data shard. When the primary shard fails and the backup shard becomes the primary shard, the statistical sum can be calculated again.

[0047] like Figure 2 ,3,During the operation process, data slices are generally required to be read into the memory, so the size of each data slice needs to be controlled, e.g. Figure 2 , the data shard K2 on server B is split into data shard K2 and data shard K21, as shown in Figure 3 , data shard K21 is stored in server D, and part of the data in the original data shard K2 is moved to server D.

[0048] like Figure 4,5. After the data shards are split, the data statistics and the basic statistical values ​​of 3 shards plus the statistical change values ​​change to the basic statistical values ​​of 4 shards plus the statistical change values.

[0049] like Figure 4 After each data shard split, a new version of the basic statistical value is generated and the basic statistical value is recalculated; for example, for the K2 data shard, the basic statistical value, version 1 = B2_V1, and the statistical change value, version 1 = D2_V1;

[0050] like Figure 5 After the data shard K2 is split, the basic statistical value of the shard where the original K2 is located, version 2 = B2_V2, and the statistical change value, version 2 = D2_V2.

[0051] The newly generated data shard K21 has the basic statistical value after the split, version 1 = B21_V2, and the statistical change value, version 1 = D21_V2.

[0052] After each data shard is merged, a new version of the basic statistical value is generated and the basic statistical value is recalculated. Data merging occurs when the data in a data shard is deleted, resulting in the data shard becoming smaller. If the shard is too small, it will increase the management overhead. The smaller data shard can be merged into the adjacent data shard. The merged data shard needs to recalculate the basic statistical value and reset the statistical change value to zero.

[0053] Yes, after a data shard is split or merged, the version of the data shard will be incremented; after a key value is written to a shard (added or modified), the corresponding version can be the current version of the data shard.

[0054] It can be that the basic statistical value counts the statistical values ​​of all keys that are smaller than the current version of the data shard; the statistical change value is the statistical value that is increased at the same time after the data shard receives an insert / modify / delete request, that is, the change statistical value counts the statistical value of the key corresponding to the current shard version.

[0055] For data sharding, when statistics are first collected for a data shard, a new version of base statistics is generated and the base statistics are calculated. After a data shard is split or merged, the base statistics are recalculated upon receiving the first statistics request from the statistics server. Most column data does not require statistical sum calculations, so there is no need to proactively initiate statistical sum calculations. After the initial statistics are collected, base statistics are generated. Subsequent operations on column data require updating the statistical change value, which increases computational overhead. Therefore, the frequency of data statistics needs to be considered. For infrequent column sum calculations, it is not necessary to initiate the calculation of base statistics and statistical change values.

[0056] The above statistical change values ​​are recorded according to versions. After each data shard split, a new version statistical change value is generated, and the initial value of the statistical change value is zero.

[0057] Alternatively, each time a data shard is merged, a new version statistical change value is generated, and the initial value of the statistical change value is zero.

[0058] Alternatively, when statistics are collected for the first time for a data shard, a new version statistical change value is generated, and the initial value of the statistical change value is zero.

[0059] Alternatively, the versions of the basic statistical values ​​and statistical change values ​​are consistent with the latest version of the shard. The shards are split and merged. Through version number management, data rollback operations can be performed after data splitting or merging fails.

[0060] like Figure 4 , Figure 5 The above statistical change values ​​are recorded according to the version; the version of the statistical change value corresponds to the version of the basic statistical value; the starting value of the statistical change value of the new version is zero. Figure 4 In the total=B1_V1+D1_V1+B2_V1+D2_V1+B3_V1+D3_V1, the statistical change value and the basic statistical version are V1. Figure 5 In the example, total = B1_V1+D1_V1+B2_V2+D2_V2+B21_V2+D21_V2+B3_V1+D3_V1. Because data shard K2 was split, the data versions were upgraded to B2_V2 and D2_V2, and the versions of the newly added shards were B21_V2 and D21_V2. When performing statistics, total can calculate the statistical sum of the latest versions of each shard. With version distinction, new and old data can be calculated and saved during the sharding or merging process.

[0061] like Figures 6 to 9 ,gives the change process of a single-row multi-column distributed key-value data before and after the split of a data shard.,Taking the statistics of electricity consumption of users in a city as an example, many scenarios require the sum of annual electricity consumption of each user in a city.,In a city, electricity users change frequently and change over time.,The sum of annual electricity consumption also changes based on monthly statistics.

[0062] In the data column, each user corresponds to a single row of multi-column data, which is used to calculate the annual electricity consumption. When the monthly data is collected each month, the user's annual electricity consumption must be added to the newly added monthly electricity consumption.

[0063] like Figure 7In February, a new key value was added to the data shard. The statistical change value is equal to the existing value plus the column data corresponding to the newly added key value, which equals 1. In March, the statistical change value is equal to the existing value plus the column data corresponding to the newly added key value W003, and the additional March electricity consumption in rows W001 and W002. The total statistical change value is 4. This corresponds to the February electricity consumption of user W003, who was newly added in the second month of the figure, and the March electricity consumption of users W001, W002, and W003. The annual total electricity consumption of the basic statistical value is version 1, with a value of 4.

[0064] If a key value is deleted from a data shard, the above statistical change value is equal to the existing value minus the column data corresponding to the deleted key value.

[0065] A distributed key-value database column data and statistical method, such as Figures 7 to 9 , data is stored in the data shards in the order of index key values; Figure 4 , Figure 5 , data shards are stored in two or more servers;

[0066] Each data includes at least one data column, select a data column as the statistical object; Figures 7 to 9 , select annual electricity as the calculation object of statistical sum. When starting statistics, calculate the sum of the above data columns in each data slice and record it as the basic statistical value; based on the basic statistical value, when the data changes, calculate the change value of the sum of the above data columns and record it as the statistical change value; the value of the statistical object is equal to the sum of the basic statistical value and the statistical change value in each data slice.

[0067] like Figures 7 to 9 , basic statistical values ​​are recorded according to versions; when data is sharded or merged, the calculation update corresponds to a version record.

[0068] Each time a data shard is split, a new version basic statistics value is generated and the basic statistics value is recalculated.

[0069] It can be that after each data shard is merged, a new version basic statistical value is generated and the basic statistical value is recalculated.

[0070] When the data shard is first counted, a new version basic statistical value is generated and the basic statistical value is calculated.

[0071] After the data shards are split or merged, the basic statistical values ​​are recalculated after receiving the first statistical request.

[0072] It can be that after each data shard split, a new version statistical change value is generated, and the initial value of the statistical change value is zero; Figure 8In March, K2 data was sharded. After the split, the statistical change value generated a new version 2. The initial value of version 2 was 0. After the electricity was generated in April, the statistical value of version 2 became 2.

[0073] A new version statistical change value can be generated each time a data shard is merged, with the initial value of the statistical change value being zero. A new version statistical change value can be generated when a data shard is first counted, with the initial value of the statistical change value being zero.

[0074] Alternatively, the versions of the basic statistical values ​​and statistical change values ​​are consistent with the latest version of the shard. The shards are split and merged. Through version number management, data rollback operations can be performed after data splitting or merging fails.

[0075] A new key value data may be added to the data shard, and the above statistical change value is equal to the existing value plus the column data corresponding to the new key value.

[0076] It is possible that a key value data is deleted from the data shard, and the above statistical change value is equal to the existing value minus the column data corresponding to the deleted key value data.

[0077] Although the present invention is illustrated and described based on the preferred embodiment and several alternatives, the invention is not limited by the specific description in this specification. Other additional replacement or equivalent components can also be used to practice the present invention.

Claims

1. A distributed key-value database column data and statistics system, characterized by: Includes more than two data storage servers and statistical servers; The data storage server is used to store one or more data slices, wherein the data slices are used to store data, the data are indexed by key values, and the key values ​​are stored in the data slices in order; The data storage server includes a data sharding basic statistical value management module; The data shard basic statistical value management module is used to manage basic statistical values, the basic statistical values ​​correspond to the data shards, and the basic statistical values ​​correspond to the statistical sum of a data column of single-row multi-column distributed key-value data; The data storage server includes a data sharding statistics value change management module; The data shard statistical value change management module is used to manage statistical change values, the statistical change values ​​correspond to the data shards, and the statistical change values ​​correspond to the statistics and change values ​​of a data column of the single-row multi-column distributed key-value data; The statistical change value is used to record the change statistics of the data column; The statistics server obtains the basic statistical value and the statistical change value of the data column corresponding to the data shard from the data storage server, calculates the sum of all the basic statistical values ​​and the statistical change values, and obtains the statistical sum of the data column; The basic statistical values ​​are recorded according to the version, and the statistical change values ​​are recorded according to the version; Each time a data shard is split, a new version basic statistical value is generated, the basic statistical value is recalculated, and a new version statistical change value is generated. The initial value of the statistical change value is zero. The recalculation of the basic statistical value is a background calculation and does not lock the shard; After each data shard is merged, a new version basic statistical value is generated, the basic statistical value is recalculated, and a new version statistical change value is generated. The initial value of the statistical change value is zero. The recalculation of the basic statistical value is a background calculation and does not lock the shard.

2. The distributed key-value database column data and statistics system according to claim 1, characterized in that: Including: when the data shard is counted for the first time, a new version basic statistical value is generated, the basic statistical value is calculated, and a new version statistical change value is generated. The initial value of the statistical change value is zero; After the data shards are split or merged, after receiving the first statistical request sent by the statistics server, a new version basic statistical value is generated, the basic statistical value is calculated, and a new version statistical change value is generated. The initial value of the statistical change value is zero.

3. The distributed key-value database column data and statistics system according to claim 1, characterized in that: The version of the statistical change value corresponds to the version of the basic statistical value; the starting value of the statistical change value of the new version is zero.

4. The distributed key-value database column data and statistics system according to claim 1, characterized in that: include: The data shard adds a key value data, and the statistical change value is equal to the existing value plus the column data corresponding to the newly added key value; The data shard deletes a key value data, and the statistical change value is equal to the existing value minus the column data corresponding to the deleted key value data.

5. A distributed key-value database column data and statistical method, characterized in that: Data is stored in data shards in the order of index key values; Multiple data shards are stored in two or more servers; Each data includes at least one data column, and a data column is selected as the statistical object; Calculate the sum of the data columns in each data shard and record it as the basic statistical value; In each data shard, when the data changes, the change value of the sum of the data columns is calculated and recorded as the statistical change value; The value of the statistical object is equal to the sum of the basic statistical value and the statistical change value in each data slice; The basic statistical values ​​are recorded according to the version, and the statistical change values ​​are recorded according to the version; including: Each time a data shard is split, a new version basic statistical value is generated, the basic statistical value is recalculated, and a new version statistical change value is generated. The initial value of the statistical change value is zero; the recalculation of the basic statistical value is a background calculation and does not lock the shard; After each data shard is merged, a new version basic statistical value is generated, the basic statistical value is recalculated, and a new version statistical change value is generated. The initial value of the statistical change value is zero; the recalculation of the basic statistical value is a background calculation and does not lock the shard.

6. The distributed key-value database column data and statistical method according to claim 5, characterized in that: include: When the data shard is counted for the first time, a new version basic statistical value is generated, the basic statistical value is calculated, and a new version statistical change value is generated. The initial value of the statistical change value is zero. After the data shards are split or merged, after receiving the first statistical request, a new version basic statistical value is generated, the basic statistical value is recalculated, and a new version statistical change value is generated. The initial value of the statistical change value is zero.

7. The distributed key-value database column data and statistical method according to claim 5, characterized in that: include: The data shard adds a key value data, and the statistical change value is equal to the existing value plus the column data corresponding to the newly added key value; The data shard deletes a key value data, and the statistical change value is equal to the existing value minus the column data corresponding to the deleted key value data.

Images