Data storage and data query method and device

Abstract

The embodiments of this specification provide a data storage and data query method and device. The front-end node can route and distribute write requests or query requests so that requests from the same subject are distributed to the same server node. The server node stores aggregated fine-grained data from a recent period of time in a local storage unit, which improves throughput on the one hand and reduces query time on the other. In addition, the data in the storage unit is backed up in a data storage platform. Older cold data with a larger business span, that is, aggregated coarse-grained data, is stored in the data storage platform. Therefore, hot data during query is directly read from the local storage unit, and cold data covered during query can be retrieved from the data storage platform.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of data processing technology, and in particular, to a data storage and data query method and device. Background Art

[0002] With the development of society and technology, the analysis and processing of data are becoming more and more important. In the field of Internet finance, when users use payment tools to make payments, financial institutions or third-party payment institutions need to check whether the payment operation exceeds the prescribed limit. For example, historical transaction data is used to determine whether the payment operation exceeds the daily limit, monthly limit or annual limit. In actual applications, a large number of users will generate massive amounts of historical transaction data. The efficiency of processing massive amounts of data is even more important in the field of financial payment. Any delay will affect the user's payment experience.

[0003] Therefore, it is hoped that there will be an improved solution that can improve the efficiency of data storage and query. Summary of the invention

[0004] One or more embodiments of this specification describe a data storage and data query method and device to improve the efficiency of data storage and query. The specific technical solution is as follows.

[0005] In a first aspect, an embodiment provides a data storage method, which is executed by a server node, wherein the server node includes a storage unit, and the server node is communicatively connected to a data storage platform; the method includes:

[0006] receiving first data to be written;

[0007] Aggregating the first data with the data in the storage unit based on a first time unit, and writing the obtained first granularity aggregated data into the storage unit;

[0008] When the data in the storage unit meets a preset transfer condition, determining the data to be transferred from the storage unit;

[0009] Aggregate the first granularity aggregated data in the data to be transferred and the data obtained from the data storage platform based on a second time unit, and store the obtained second granularity aggregated data in the data storage platform; wherein the second time unit is larger than the first time unit.

[0010] In one embodiment, after determining the data to be transferred from the storage unit, the method further includes:

[0011] The data to be transferred is stored in the data storage platform.

[0012] In one implementation, the step of aggregating the first data with the data in the storage unit based on the first duration unit includes:

[0013] When the first data is the first data in a first time unit, directly writing the first data into the storage unit;

[0014] When the first data is not the first data, the first data is aggregated with corresponding data in the storage unit in a preset manner until complete data of the first time unit is obtained through aggregation.

[0015] In one embodiment, the storage unit is used to store the latest n first-granularity aggregated data of the first time units, and the maximum amount of stored data is the first amount of data; the preset transfer condition includes at least one of the following conditions:

[0016] When the number of first duration units included in the first granularity aggregate data in the storage unit reaches a first number; the first number is not greater than n;

[0017] When the amount of data stored in the storage unit reaches a data amount threshold; the data amount threshold is not greater than the first data amount.

[0018] In one embodiment, the step of storing the obtained second granularity aggregate data in the data storage platform includes:

[0019] When the obtained second granularity aggregated data has not yet aggregated to complete data of one second time unit, the obtained second granularity aggregated data is stored in the data storage platform.

[0020] In one embodiment, the method further comprises:

[0021] When the obtained second granularity aggregated data has been aggregated to complete data of a second time unit, the obtained second granularity aggregated data is written into the storage unit.

[0022] In one embodiment, the storage unit is further used to store the latest m second time units of second granularity aggregate data; the step of determining the data to be transferred from the storage unit includes:

[0023] The data to be transferred is determined from the second granularity aggregated data stored in the storage unit.

[0024] In one embodiment, the data obtained from the data storage platform carries the first version information; the step of storing the obtained second granularity aggregated data in the data storage platform includes:

[0025] The first version information is updated to the second version information, and the second version information and the obtained second granularity aggregated data are stored in association with each other in the data storage platform.

[0026] In one embodiment, after storing the obtained second granularity aggregate data in the data storage platform, the method further includes:

[0027] The data information stored in the data storage platform is updated in the metadata.

[0028] In one embodiment, the first data is data of a first user;

[0029] The step of aggregating the first data and the data in the storage unit based on the first duration unit includes:

[0030] Aggregate the first data and the data of the first user in the storage unit based on a first time unit to obtain first granularity aggregated data of the first user;

[0031] The step of aggregating the first granularity aggregated data in the data to be transferred and the data obtained from the data storage platform based on the second time unit includes:

[0032] The first granularity aggregated data of the first user in the data to be transferred and the data of the first user obtained from the data storage platform are aggregated based on a second time unit to obtain second granularity aggregated data of the first user.

[0033] In a second aspect, an embodiment provides a data query method, which is executed by a server node, wherein the server node includes a storage unit, and the server node is communicatively connected to a data storage platform; the method includes:

[0034] Determine the data to be queried;

[0035] Determining a storage location of aggregated data used to determine the data to be queried;

[0036] When the storage location indicates that part of the aggregated data is stored in the storage unit and part of the data is stored in the data storage platform, the corresponding first granularity aggregated data is obtained from the storage unit, and the corresponding second granularity aggregated data is obtained from the data storage platform; wherein the first granularity aggregated data is obtained by the server node aggregating the received first data with the data in the storage unit based on a first time unit; the second granularity aggregated data is obtained by the server node aggregating the first granularity aggregated data with the data in the data storage platform based on a second time unit;

[0037] The acquired first granularity aggregated data and the acquired second granularity aggregated data are aggregated, and the data to be queried is determined based on the aggregated data.

[0038] In one embodiment, the method further comprises:

[0039] When the storage location indicates that all the aggregated data are stored in the storage unit, obtaining corresponding first granularity aggregated data from the storage unit;

[0040] The data to be queried is determined based on the acquired first granularity aggregate data.

[0041] In one implementation, the storage unit is further used to store the latest m second time units of second granularity aggregate data; the method further includes:

[0042] When the storage location indicates that different parts of the aggregated data are stored in different locations of the storage unit, obtaining corresponding first granularity aggregated data and corresponding second granularity aggregated data from different locations of the storage unit;

[0043] The acquired first granularity aggregated data and the acquired second granularity aggregated data are aggregated, and the data to be queried is determined based on the aggregated data.

[0044] In one embodiment, the step of determining the storage location of the aggregated data for determining the data to be queried includes:

[0045] A storage location of aggregated data used to determine the data to be queried is determined from the metadata.

[0046] In a third aspect, an embodiment provides a data storage device, which is deployed in a server node, wherein the server node includes a storage unit, and the server node is communicatively connected to a data storage platform; the device includes:

[0047] A receiving module, configured to receive first data to be written;

[0048] A first aggregation module is configured to aggregate the first data with the data in the storage unit based on a first time unit, and write the obtained first granularity aggregated data into the storage unit;

[0049] A transfer module, configured to determine data to be transferred from the storage unit when the data in the storage unit meets a preset transfer condition;

[0050] The second aggregation module is configured to aggregate the first granularity aggregated data in the data to be transferred and the data obtained from the data storage platform based on a second time unit, and store the obtained second granularity aggregated data to the data storage platform; wherein the second time unit is larger than the first time unit.

[0051] In a fourth aspect, an embodiment provides a data query device, which is deployed in a server node, wherein the server node includes a storage unit, and the server node is communicatively connected to a data storage platform; the device includes:

[0052] A first determination module is configured to determine the data to be queried;

[0053] A second determination module is configured to determine a storage location of aggregated data used to determine the data to be queried;

[0054] A first acquisition module is configured to acquire corresponding first-granularity aggregated data from the storage unit and acquire corresponding second-granularity aggregated data from the data storage platform when the storage location indicates that part of the aggregated data is stored in the storage unit and part of the data is stored in the data storage platform; wherein the first-granularity aggregated data is obtained by the server node aggregating the received first data with the data in the storage unit based on a first time unit; and the second-granularity aggregated data is obtained by the server node aggregating the first-granularity aggregated data with the data in the data storage platform based on a second time unit;

[0055] The third determination module is configured to aggregate the acquired first granularity aggregated data and the acquired second granularity aggregated data, and determine the data to be queried based on the aggregated data.

[0056] In a fifth aspect, an embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute any one of the methods in the first to second aspects.

[0057] In a sixth aspect, an embodiment provides a computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in any one of the first aspect to the second aspect is implemented.

[0058] In the method and device provided in the embodiments of this specification, the server node first aggregates the data in its own storage unit in shorter units. When the preset transfer conditions are met, the data in the storage unit is aggregated with the data in the data storage platform in longer units and stored in the data storage platform. In this way, the latest small-granularity aggregated data can be stored in the storage unit, and the earlier large-granularity aggregated data can be stored in the data storage platform. When querying data, most query requests will hit the local storage unit, thereby reducing the network time consumed to retrieve data from the remote data storage platform. Therefore, this embodiment can improve the efficiency of data storage and query. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0060] Figure 1 A schematic diagram of an implementation scenario of an embodiment disclosed in this specification;

[0061] Figure 2 A schematic diagram of a data storage method provided in an embodiment;

[0062] Figure 3 A schematic diagram of a data query method is provided for an embodiment;

[0063] Figure 4 A schematic block diagram of a data storage device provided in an embodiment;

[0064] Figure 5 A schematic block diagram of a data query device provided in an embodiment. DETAILED DESCRIPTION

[0065] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0066] As mentioned above, in the fields of Internet finance and e-commerce where transaction data is generated, service providers have the need to store and query transaction data. A transaction data can usually contain attribute values ​​of multiple attributes. For example, a payment data includes the attribute values ​​of payment time, payment amount, payment method and purpose; a goods transaction data includes the attribute values ​​of delivery time, payment time, payment method, quantity of goods and transportation method. Each transaction can be stored in the form of data for subsequent query.

[0067] In the scenario of massive transaction data, distributed systems, clusters or multi-threading in a single device can be used to perform transaction data storage and query tasks. Figure 1 Schematic diagram of an implementation scenario of an embodiment disclosed in this specification. Among them, the front-end node (Frontend, FE) is responsible for receiving write requests or query requests for transaction data; the splitting unit (Sharding Manager) in the front-end node is responsible for routing and distributing write requests or query requests, and the hash strategy is used for routing and distribution by default; the back-end node (Backend, BE) is responsible for the aggregation processing, query and other operations of transaction data. The back-end node is locally provided with a storage unit (RocksDB), and the back-end node can communicate with the data storage platform (HBase) to transfer data to the data storage platform. The storage unit is used to store the latest data, that is, the data of the most recent period of time, and the data storage platform is used to back up the data in the storage unit and store data of an earlier period of time. The front-end node and the back-end node can be implemented by a server node. Multiple front-end nodes and multiple back-end nodes can be implemented by devices in a distributed system, or by a device cluster, or by multiple threads in a single device, which is not specifically limited in this application. The data storage platform can be implemented by a distributed storage device.

[0068] When querying data, the original transaction data is generally not used directly, but the aggregated data after aggregating multiple transaction data is needed. Therefore, according to business needs, multiple transaction data can be aggregated according to one or more attributes to obtain aggregated data. Depending on the different time units used in aggregation, aggregated data of different granularities can be obtained. For example, aggregation in days will obtain daily aggregated data; aggregation in months will obtain monthly aggregated data; aggregation in years will obtain annual aggregated data, etc.

[0069] The storage unit can be used to store the latest, smaller-granularity aggregated data, and the data storage platform can be used to store older, larger-granularity aggregated data. Generally speaking, query requests for the latest data are more popular, so high-hot query requests will hit the storage unit, and the backend node will take very little time to directly obtain data from the local storage unit; low-hot query requests will fall on the data storage platform, thereby minimizing time consumption and improving processing efficiency.

[0070] In the above content, transaction data is briefly described as an example. In the solution provided in the embodiment of this specification, the data processed is not only transaction data, but also payment data or interaction data, etc. These can be collectively referred to as data, which are the objects processed by the embodiment of this specification. The back-end node can be implemented as a server node. The specific embodiments of this application are described below through the accompanying drawings.

[0071] Figure 2 A flow chart of a data storage method provided in an embodiment. The method can be executed by a server node, and specifically can be executed by a control unit in the server node. The server node also includes a storage unit, and the storage unit can be a memory or a hard disk connected to the BE through a physical line. The server node is communicatively connected to the data storage platform. The server node can be implemented by any device, equipment, platform, device cluster, etc. with computing and processing capabilities. The data storage platform can be implemented by distributed storage. The method includes the following steps.

[0072] Step S210: the server node BE receives first data to be written.

[0073] The first data may be any piece of data in a large amount of data, and the large amount of data may be data that is continuously generated over time, so the first data may be any piece of data in a series of data generated over time. The first data includes attribute values ​​of multiple attributes, and in addition, may also include corresponding user, timestamp and other information.

[0074] The first data received by the server node BE may be sent by the front-end node FE. The front-end node FE and the server node BE may be implemented by different modules or different threads in a device, or may be implemented by different devices. The number of server nodes BE may be multiple, and the multiple server nodes BE are respectively connected to the multiple front-end nodes.

[0075] When the front-end node FE receives a write request for the first data, it can send the first information to the corresponding server node BE based on the routing information corresponding to the first data. The routing information contains the subject corresponding to the first data, and the subject may include, for example, a user or a card number. The server node BE receives the first information to be written sent by the front-end node FE. In this way, the data of the same subject can be distributed to the same BE through the hash routing strategy, and the data of the same subject can be processed by the same BE.

[0076] Step S220: the server node BE aggregates the first data and the data in the storage unit based on the first duration unit, and writes the obtained first granularity aggregated data into the storage unit.

[0077] The storage unit belongs to the local storage space of the server node BE. The server node BE has a very high read and write speed to the storage unit and a low time consumption. The storage unit is used to store a part of the data with the highest access popularity, such as the hot data with the most recent storage time, that is, the latest hot data.

[0078] When querying data, the first data is generally not used directly, but aggregated data after aggregating multiple first data. For example, from multiple historical transaction data of a user, the total transaction amount or total number of transactions of the user in a day can be aggregated.

[0079] The first duration unit may be a pre-set duration, for example, the first duration unit may be 1 hour, 2 hours or one day. Aggregating the data in the first duration unit may obtain first granularity aggregated data. The storage unit may be used to store the first granularity aggregated data.

[0080] When receiving the first data, the server node BE can directly write the data into the storage unit. When receiving subsequent data, the server node BE can aggregate the data with the corresponding data in the storage unit in a preset manner until the complete data of the first time unit is obtained.

[0081] Aggregating in a preset manner may include aggregating according to a specified attribute in the data, and / or accumulating and counting the amount of data, etc. For example, the specified attribute includes at least one of the following categories: payment amount, transaction volume, etc. The first granularity aggregated data may be stored in the form of a key-value pair (KV).

[0082] For example, when the first time unit is day and data is aggregated in day units, when the first data in a day is received, the data can be directly written into the storage unit. When the second data in the day is received, the first data is read from the storage unit, the second data is aggregated with the first data to obtain first granularity aggregated data, and the data is stored in the storage unit. When the third data in the day is received, the first granularity aggregated data is read from the storage unit, the third data is aggregated with the first granularity aggregated data. During the aggregation, the payment amount can be accumulated. In this way, the first granularity aggregated data includes: the payment amount accumulated continuously within one day. When the accumulation of one day is completed, the first granularity aggregated data corresponding to the day is obtained.

[0083] When writing and reading data to the storage unit, it can be achieved through the file system Rocksdb, and then the lock-based transaction management mechanism (Transaction) of Rocksdb is used to ensure data consistency and improve the efficiency of data writing and reading.

[0084] Step S230: When the data in the storage unit meets the preset transfer condition, the server node BE determines the data to be transferred from the storage unit.

[0085] The storage space of the storage unit is generally limited and cannot store too much aggregated data. The storage unit can be set to store the latest first-granularity aggregated data of the n first time units, and set the maximum value of the amount of stored data, such as the maximum value being the first data amount. For example, the storage unit can be set to store the latest 7 days of first-granularity aggregated data, that is, the 7 days of first-granularity aggregated data closest to the current time.

[0086] The above-mentioned preset transfer condition includes at least one of the following conditions:

[0087] When the number of first duration units included in the first granularity aggregate data in the storage unit reaches a first number;

[0088] When the amount of data stored in the storage unit reaches a data amount threshold.

[0089] The first quantity is not greater than n. For example, the storage unit is used to store the first-granularity aggregated data for the last 7 days, and the first quantity can be set to 7. When the storage unit has stored the first-granularity aggregated data for 7 days, the preset transfer condition is met. The data volume threshold is not greater than the first data volume. As long as one of the above conditions is met, data can be transferred from the storage unit to the data storage platform.

[0090] When determining the data to be transferred from the storage unit, the data to be transferred may be determined from the stored first-granularity aggregated data. For example, earlier first-granularity aggregated data may be determined as the data to be transferred. When the storage unit stores the first-granularity aggregated data of the last 7 days, whenever the storage unit aggregates complete data of a first time unit, the earliest first-granularity aggregated data of a first time unit in the storage unit may be used as the data to be transferred.

[0091] After the data to be transferred is determined from the storage unit, the data to be transferred can also be directly stored in the data storage platform. In this way, the small-grained aggregated data in the storage unit can be backed up to the data storage platform. After the data to be transferred is stored in the data storage platform, the data to be transferred can be eliminated from the storage unit so that the storage unit can accommodate hotter data.

[0092] In step S240, the server node BE aggregates the first granularity aggregated data in the data to be transferred and the data obtained from the data storage platform based on the second time unit, and stores the obtained second granularity aggregated data in the data storage platform.

[0093] Among them, the second duration unit is greater than the first duration unit, and the second duration unit can contain multiple first duration units. For example, when the first duration unit is "day", the second duration unit can be set to "month". That is, the server node BE can pre-calculate fine-grained data in units of days in real time, and then aggregate the data in units of days into data in units of months, for example, aggregate 30 days of granularity data into a complete monthly granularity data, and store it in the form of key-value pairs in the data storage platform (HBase), so that point query can be used to query monthly granularity data, thereby greatly reducing query time.

[0094] It takes a period of time to aggregate small-granularity aggregate data to large-granularity aggregate data. When the first-granularity aggregate data in the data to be transferred is the first data in a second time unit, the first-granularity aggregate data can be directly stored in the data storage platform.

[0095] When the first granularity aggregated data in the data to be transferred is not the first data mentioned above, the server node BE can obtain the corresponding data from the data storage platform, aggregate the first granularity aggregated data in the data to be transferred with the data obtained from the data storage platform, and obtain the second granularity aggregated data. When aggregating, the aggregation can be performed in a preset manner, for example, the aggregation can be performed according to the specified attributes in the data, and / or the number of data can be cumulatively counted, etc. The aggregation method corresponding to the first granularity aggregated data can be the same as or different from the aggregation method corresponding to the second granularity data. The second granularity aggregated data can be stored in the form of key-value pairs (KV).

[0096] When the obtained second-granularity aggregate data has not yet aggregated the complete data of a second time unit, the server node BE stores the obtained second-granularity aggregate data to the data storage platform so that when new data to be transferred in the storage unit is squeezed to the data storage platform, it continues to aggregate based on the second-granularity aggregate data.

[0097] When the obtained second-granularity aggregated data has been aggregated to complete data of a second time unit, the server node BE writes the obtained second-granularity aggregated data into the storage unit. For example, taking the second time unit "month" as an example, when the aggregated data of a certain month is aggregated, the aggregated data of that month can be written into the storage unit. The aggregated data of that month also belongs to the most recent aggregated data and is hot data. Writing it into the storage unit can make it easier to query.

[0098] Therefore, in this embodiment, the storage unit may also be used to store the latest m second time units of aggregated data of the second granularity. The latest may be understood as the data closest to the current moment.

[0099] Looking back at step S230, the preset transfer condition may also include: when the number of second duration units contained in the second granularity aggregate data in the storage unit reaches a second number. The second number is not greater than m. For example, the storage unit is also used to store the second granularity aggregate data of the last three months, and the second number may be set to 3. When the storage unit has stored three full days of second granularity aggregate data, it is considered that the preset transfer condition is met.

[0100] When determining the data to be transferred from the storage unit, the data to be transferred may be determined from the second granularity aggregated data stored in the storage unit, for example, the earlier second granularity aggregated data may be determined as the data to be transferred. When the storage unit stores the second granularity aggregated data of the last three months, whenever the storage unit aggregates complete data of one second time unit, the earliest second granularity aggregated data of one second time unit in the storage unit may be used as the data to be transferred.

[0101] The storage unit may be used to store the first granularity aggregate data first. In addition, when there is remaining space in the storage unit, the latest m second time units of aggregate data of the second granularity may continue to be stored.

[0102] When the amount of data in the storage unit reaches a data amount threshold, earlier second granularity aggregate data may be determined as data to be transferred, or all second granularity aggregate data may be determined as data to be transferred, and the data to be transferred may be stored in the data storage platform.

[0103] The server node BE may also aggregate the second granularity aggregate data with the corresponding data in the data storage platform based on the third time unit to obtain third granularity aggregate data. The third time unit is larger than the second time unit, and the third time unit may contain multiple first time units. For example, the third time unit may be "year". The third granularity aggregate data may be stored in the form of a key-value pair (KV).

[0104] When the obtained third granularity aggregated data has not yet aggregated to complete data of one third time unit, the server node BE may store the third granularity aggregated data in the data storage platform. When the obtained third granularity aggregated data has aggregated to complete data of one third time unit, the server node BE may write the third granularity aggregated data into the storage unit.

[0105] Looking back at step S240, when the complete data of a second time unit has not yet been aggregated, it is necessary to read the incomplete second-granularity aggregated data from the data storage platform to continue aggregation, which requires ensuring strong consistency of the data.

[0106] In one embodiment, corresponding version information is set for the data in the data storage platform. In step S240, the data obtained by the server node BE from the data storage platform carries the first version information. When the obtained second-granularity aggregated data is stored in the data storage platform, the first version information can be updated to the second version information, and the second version information and the obtained second-granularity aggregated data are stored in the data storage platform in association. In other words, each time the data in the storage platform is aggregated into the second-granularity aggregated data, its version information can be updated, so as to avoid reading the wrong version of dirty data.

[0107] For example, Table 1 is the day-granularity data aggregated in units of days in the storage unit (Rocksdb).

[0108] Table 1

[0109] key value a_b_c_20211211_count(d) 10 a_b_c_20211211_sum(d) 100

[0110] The second and third rows of data in Table 1 are the total number of transactions and the total transaction amount on December 11, 2021. Table 2 is the monthly granularity data aggregated in the data storage platform HBase on a monthly basis.

[0111] Table 2

[0112] key value Version a_b_c_202112_count(d) 900 1 a_b_c_202112_sum(d) 1900 1

[0113] The second and third rows of data in Table 2 are the total number of transactions, total transaction amount, and version information in December 2021. When reading incomplete monthly granularity data from the data storage platform, the highest version of the monthly granularity data in Table 2 can be read. After aggregating the daily granularity data in Table 1 to the monthly granularity data, the version is increased by 1 to obtain the monthly granularity data shown in Table 3.

[0114] Table 3

[0115] key value Version a_b_c_202112_count(d) 910 2 a_b_c_202112_sum(d) 2000 2

[0116] After the daily granularity data is aggregated into the monthly granularity data and the corresponding data in the storage unit is eliminated, the updated version information is used to read the monthly granularity data, which can avoid reading dirty data.

[0117] In order to facilitate the subsequent data query process, after the obtained second-granularity aggregate data is stored in the data storage platform, the data information stored in the data storage platform can also be updated in the metadata. Metadata is used to record the storage location of data of different granularities. When the aggregate data is transferred, for example, the first-granularity aggregate data is transferred to the data storage platform, or the second-granularity aggregate data is written to the storage unit, or the third-granularity aggregate data is written to the storage unit, the corresponding data information can be written into the metadata, so that the storage location and version information of the data can be determined according to the metadata during the query stage.

[0118] exist Figure 2 In the illustrated embodiment, the data may be aggregated regardless of the user. For example, when the data is aggregated in units of days, the total number and total amount of all transactions in a day may be obtained. In one embodiment, the data may also be aggregated for different users.

[0119] For example, the first data is data of the first user. When executing step S220, the first data and the data of the first user in the storage unit may be aggregated based on the first duration unit to obtain first granularity aggregated data of the first user.

[0120] When executing step S240, the first granularity aggregated data of the first user in the data to be transferred and the data of the first user obtained from the data storage platform may be aggregated based on the second time unit to obtain the second granularity aggregated data of the first user.

[0121] After aggregating massive amounts of data for different users, you can also provide services to query aggregated data to complete more services. Figure 2 Corresponding to the illustrated embodiment, the following embodiment provides a data query method.

[0122] Figure 3 The embodiment provides a flow chart of a data query method. The method is executed by a server node BE, which includes a storage unit, and the server node BE is connected to a data storage platform. For a description of the server node BE, the storage unit, and the data storage platform, see Figure 2 The embodiment shown is not described in detail here. The method comprises the following steps.

[0123] Step S310, the server node BE determines the data to be queried data1.

[0124] The server node BE may be connected to the front-end node FE. When the front-end node receives a query request for data in a certain period of time, it may send the query request to the corresponding server node BE based on the routing information of the query request.

[0125] When receiving a query request, the server node BE can analyze the logic in the query request to determine the data to be queried by the query request. For example, the logic in the query request indicates that the aggregated data of December is to be queried (this is the data to be queried data1). Assuming that the current date is December 11, according to the logical analysis, it can be determined that the aggregated data used to determine the data to be queried includes the aggregated data d1 to d11 from December 1 to 11.

[0126] In step S320, the server node BE determines the storage location of the aggregated data d1 to d11 used to determine the data to be queried. Specifically, the storage location of the aggregated data d1 to d11 can be determined from the metadata. At the same time, according to the information in the metadata, it can be determined that the aggregated data d1 to d11 contains first-granularity aggregated data and second-granularity aggregated data. For example, the aggregated data d1 to d11 specifically includes monthly-granularity data from December 1 to 4 (stored in the data storage platform), and daily-granularity data from December 5 to 11 (stored in the storage unit). Among them, the storage unit stores the latest 7-day daily-granularity data.

[0127] Step S330, when the above storage location indicates that part of the aggregated data d1~d11 is stored in the storage unit and part of the data is stored in the data storage platform, obtain the corresponding first granularity aggregated data from the storage unit and obtain the corresponding second granularity aggregated data from the data storage platform.

[0128] The first granularity aggregated data is obtained by the server node aggregating the received first data with the data in the storage unit based on the first time unit. The second granularity aggregated data is obtained by the server node aggregating the first granularity aggregated data with the data of the data storage platform based on the second time unit.

[0129] For example, the storage location indicates that the monthly granularity data corresponding to the aggregated data d1 to d4 is stored in the data storage platform, and the daily granularity data corresponding to d5 to d11 is stored in the local storage unit. Then, the daily granularity data from December 5 to 11 can be read from the local storage unit, and the monthly granularity data for December can be read from the data storage platform. The monthly granularity data has aggregated the daily granularity data from December 1 to 4.

[0130] Step S340, aggregate the acquired first granularity aggregated data and the acquired second granularity aggregated data, and determine the data to be queried based on the aggregated data. During the aggregation, the acquired first granularity aggregated data and the acquired second granularity aggregated data may be aggregated in a preset manner. The aggregated data may be directly used as the data to be queried, or the aggregated data may be used as the data to be queried after being subjected to preset processing.

[0131] For example, the monthly granularity data of December and the daily granularity data from December 5 to 11 may be aggregated to obtain the aggregated data from December 1 to 11, and the data to be queried may be determined based on the aggregated data.

[0132] Step S350: When the storage location indicates that all the aggregated data is stored in the storage unit, the corresponding first-granularity aggregated data is obtained from the storage unit. Step S360: Determine the data to be queried based on the obtained first-granularity aggregated data.

[0133] For example, assuming that the current date is December 3, the aggregated data from December 1 to 3 are all stored in the storage unit. In this case, the day-granularity data from December 1 to 3 can be directly obtained from the storage unit. These day-granularity data are further aggregated, and the data to be queried is determined based on the aggregated data.

[0134] In one implementation, the storage unit may also be used to store the latest m second time units of second granularity aggregate data. Figure 3 The method shown may also include the following steps.

[0135] Step S370: when the storage location indicates that different parts of the aggregated data are stored in different locations of the storage unit, obtain corresponding first granularity aggregated data and corresponding second granularity aggregated data from different locations of the storage unit.

[0136] Step S380: Aggregate the acquired first granularity aggregated data and the acquired second granularity aggregated data, and determine the data to be queried based on the aggregated data.

[0137] For example, the logic in the query request indicates that the aggregated data of the last three months is to be queried (this is the data to be queried data1). Assuming that the current date is December 7, it can be determined based on logical analysis that the aggregated data used to determine the data to be queried includes the aggregated data from October 1 to December 11. According to the information recorded in the metadata, it can be known that these aggregated data include: monthly granularity data for October and November, which is stored in the first storage area of ​​the storage unit; and daily granularity data from December 1 to 7, which is stored in the second storage area of ​​the storage unit. According to the different locations where different parts of the data are stored, the corresponding monthly granularity data and daily granularity data are obtained from the storage unit and aggregated.

[0138] In one implementation, when data is obtained from the data storage platform, version information is also carried, which can be queried from metadata.

[0139] In one embodiment, the query of data can be performed for different users. For example, in step S310, the data to be queried for the second user is determined. In step S330, the first granularity aggregated data of the corresponding second user is obtained from the storage unit, and the second granularity aggregated data of the corresponding second user is obtained from the data storage platform. In step S350, the first granularity aggregated data of the corresponding second user is obtained from the storage unit. In step S370, the first granularity aggregated data and the second granularity aggregated data of the corresponding second user are obtained from different locations of the storage unit.

[0140] In the above embodiment, a storage backend integrating local and remote storage is constructed by using local storage plus remote distributed storage. Since most queries hit the data in the local storage, the network time consumed in retrieving data from the remote storage is reduced, and the overall query time is further reduced. At the same time, the RocksDB embedded in the local storage provides complete transaction capabilities, which can ensure data consistency during the real-time pre-calculation stage of writing. Remote storage can adopt a multi-version concurrency control mechanism (MVCC), which can provide a fault-tolerant mechanism. After an unexpected downtime, the original version can be used to overwrite the dirty data of the last execution.

[0141] In this specification, the word "first" in terms such as first data, first time unit, first granularity aggregated data, first quantity, first version information and first user, as well as the corresponding "second" in the text, are merely for the convenience of distinction and description and do not have any limiting meaning.

[0142] The foregoing describes certain embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in an order different from that in the embodiments, and the desired results may still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0143] Figure 4 Schematic block diagram of a data storage device provided in an embodiment. The device 400 is deployed in a server node, the server node includes a storage unit, and the server node is communicatively connected to a data storage platform. Figure 2 The device 400 includes:

[0144] The receiving module 410 is configured to receive first data to be written;

[0145] A first aggregation module 420 is configured to aggregate the first data with the data in the storage unit based on a first time unit, and write the obtained first granularity aggregated data into the storage unit;

[0146] A transfer module 430, configured to determine data to be transferred from the storage unit when the data in the storage unit meets a preset transfer condition;

[0147] The second aggregation module 440 is configured to aggregate the first granularity aggregated data in the data to be transferred and the data obtained from the data storage platform based on a second time unit, and store the obtained second granularity aggregated data to the data storage platform; wherein the second time unit is larger than the first time unit.

[0148] In one embodiment, the device 400 further includes:

[0149] The storage module 450 is configured to store the data to be transferred to the data storage platform after determining the data to be transferred from the storage unit.

[0150] In one implementation, the first aggregation module 420 includes:

[0151] a writing submodule (not shown in the figure), configured to directly write the first data into the storage unit when the first data is the first data in a first time unit;

[0152] The aggregation submodule (not shown in the figure) is configured to aggregate the first data with the corresponding data in the storage unit in a preset manner when the first data is not the first data, until complete data of the first time unit is obtained through aggregation.

[0153] In one embodiment, the storage unit is used to store the latest n first-granularity aggregated data of the first time units, and the maximum amount of stored data is the first amount of data; the preset transfer condition includes at least one of the following conditions:

[0154] When the number of first duration units included in the first granularity aggregate data in the storage unit reaches a first number; the first number is not greater than n;

[0155] When the amount of data stored in the storage unit reaches a data amount threshold; the data amount threshold is not greater than the first data amount.

[0156] In one embodiment, when the second aggregation module 440 stores the obtained second granularity aggregated data in the data storage platform, it includes:

[0157] When the obtained second granularity aggregated data has not yet aggregated to complete data of one second time unit, the obtained second granularity aggregated data is stored in the data storage platform.

[0158] In one embodiment, the device 400 further includes:

[0159] The writing module 460 is configured to write the obtained second granularity aggregate data into the storage unit when the obtained second granularity aggregate data has been aggregated to reach complete data of a second time unit.

[0160] In one implementation, the storage unit is further used to store the latest m second time units of second granularity aggregate data; the transfer module 430 is specifically configured as follows:

[0161] The data to be transferred is determined from the second granularity aggregated data stored in the storage unit.

[0162] In one implementation, the data obtained from the data storage platform carries the first version information; and the second aggregation module 440, when storing the obtained second granularity aggregated data in the data storage platform, includes:

[0163] The first version information is updated to the second version information, and the second version information and the obtained second granularity aggregated data are stored in association with each other in the data storage platform.

[0164] In one embodiment, the device 400 further includes:

[0165] The update module (not shown in the figure) is configured to update the data information stored in the data storage platform in the metadata after storing the obtained second granularity aggregate data in the data storage platform.

[0166] In one embodiment, the first data is data of a first user;

[0167] The first aggregation module 420, when aggregating the first data with the data in the storage unit based on the first time unit, includes:

[0168] Aggregate the first data and the data of the first user in the storage unit based on a first time unit to obtain first granularity aggregated data of the first user;

[0169] The second aggregation module 440, when aggregating the first granularity aggregated data in the data to be transferred and the data obtained from the data storage platform based on the second time unit, includes:

[0170] The first granularity aggregated data of the first user in the data to be transferred and the data of the first user obtained from the data storage platform are aggregated based on a second time unit to obtain second granularity aggregated data of the first user.

[0171] Figure 5 Schematic block diagram of a data query device provided in an embodiment. The device 500 is deployed in a server node, the server node includes a storage unit, and the server node is connected to a data storage platform for communication. Figure 3 The method embodiment shown corresponds to the embodiment shown in FIG. 5 . The device 500 includes:

[0172] A first determination module 510 is configured to determine the data to be queried;

[0173] A second determination module 520 is configured to determine a storage location of the aggregated data used to determine the data to be queried;

[0174] The first acquisition module 530 is configured to acquire corresponding first-granularity aggregated data from the storage unit and acquire corresponding second-granularity aggregated data from the data storage platform when the storage location indicates that part of the aggregated data is stored in the storage unit and part of the data is stored in the data storage platform; wherein the first-granularity aggregated data is obtained by the server node aggregating the received first data with the data in the storage unit based on a first time unit; and the second-granularity aggregated data is obtained by the server node aggregating the first-granularity aggregated data with the data in the data storage platform based on a second time unit;

[0175] The third determination module 540 is configured to aggregate the acquired first granularity aggregated data and the acquired second granularity aggregated data, and determine the data to be queried based on the aggregated data.

[0176] In one embodiment, the apparatus 500 further includes:

[0177] A second acquisition module 550 is configured to acquire corresponding first granularity aggregate data from the storage unit when the storage location indicates that the aggregate data is all stored in the storage unit;

[0178] The fourth determination module 560 is configured to determine the data to be queried based on the acquired first granularity aggregated data.

[0179] In one implementation, the storage unit is further used to store the latest m second time units of second granularity aggregate data; the device 500 further includes:

[0180] A third acquisition module 570 is configured to acquire corresponding first granularity aggregate data and corresponding second granularity aggregate data from different locations of the storage unit when the storage location indicates that different parts of the aggregate data are stored in different locations of the storage unit;

[0181] The fifth determination module 580 is configured to aggregate the acquired first granularity aggregated data and the acquired second granularity aggregated data, and determine the data to be queried based on the aggregated data.

[0182] In one implementation, the second determining module 520 is specifically configured as follows:

[0183] A storage location of aggregated data used to determine the data to be queried is determined from the metadata.

[0184] The above-mentioned device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0185] The present specification also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute Figures 1 to 3 Any of the methods described above.

[0186] The embodiment of the present specification also provides a computing device, including a memory and a processor, wherein the memory stores an executable code, and when the processor executes the executable code, Figures 1 to 3 Any of the methods described above.

[0187] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium and computing device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0188] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention can be implemented by hardware, software, firmware, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0189] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present invention in detail. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A data storage method, executed by a server node, wherein the server node includes a storage unit, and the server node is communicatively connected to a data storage platform; the method comprises: receiving first data to be written; Aggregating the first data with the data in the storage unit based on a first time unit, and writing the obtained first granularity aggregated data into the storage unit; When the data in the storage unit meets a preset transfer condition, determining the data to be transferred from the storage unit; Aggregate the first granularity aggregated data in the data to be transferred and the data obtained from the data storage platform based on a second time unit, and store the obtained second granularity aggregated data in the data storage platform; wherein the second time unit is larger than the first time unit.

2. The method according to claim 1, after determining the data to be transferred from the storage unit, further comprises: The data to be transferred is stored in the data storage platform.

3. According to the method of claim 1, the step of aggregating the first data with the data in the storage unit based on the first time unit comprises: When the first data is the first data in a first time unit, directly writing the first data into the storage unit; When the first data is not the first data, the first data is aggregated with corresponding data in the storage unit in a preset manner until complete data of the first time unit is obtained through aggregation.

4. The method according to claim 1, wherein the storage unit is used to store the latest first-granularity aggregated data of n first time units, and the maximum amount of stored data is the first amount of data; and the preset transfer condition includes at least one of the following conditions: When the number of first duration units included in the first granularity aggregate data in the storage unit reaches a first number; the first number is not greater than n; When the amount of data stored in the storage unit reaches a data amount threshold; the data amount threshold is not greater than the first data amount.

5. According to the method of claim 1, the step of storing the obtained second granularity aggregate data in the data storage platform comprises: When the obtained second granularity aggregated data has not yet aggregated to complete data of one second time unit, the obtained second granularity aggregated data is stored in the data storage platform.

6. The method according to claim 5, further comprising: When the obtained second granularity aggregated data has been aggregated to complete data of a second time unit, the obtained second granularity aggregated data is written into the storage unit.

7. The method according to claim 6, wherein the storage unit is further used to store the latest m second-granularity aggregated data of the second time units; the step of determining the data to be transferred from the storage unit comprises: The data to be transferred is determined from the second granularity aggregated data stored in the storage unit.

8. According to the method of claim 1, the data obtained from the data storage platform carries the first version information; the step of storing the obtained second granularity aggregated data in the data storage platform comprises: The first version information is updated to the second version information, and the second version information and the obtained second granularity aggregated data are stored in association with each other in the data storage platform.

9. The method according to claim 1, after storing the obtained second granularity aggregate data in the data storage platform, further comprises: The data information stored in the data storage platform is updated in the metadata.

10. The method according to claim 1, wherein the first data is data of a first user; The step of aggregating the first data and the data in the storage unit based on the first duration unit includes: Aggregate the first data and the data of the first user in the storage unit based on a first time unit to obtain first granularity aggregated data of the first user; The step of aggregating the first granularity aggregated data in the data to be transferred and the data obtained from the data storage platform based on the second time unit includes: The first granularity aggregated data of the first user in the data to be transferred and the data of the first user obtained from the data storage platform are aggregated based on a second time unit to obtain second granularity aggregated data of the first user.

11. A data query method, executed by a server node, wherein the server node includes a storage unit, and the server node is communicatively connected to a data storage platform; the method comprises: Determine the data to be queried; Determining a storage location of aggregated data used to determine the data to be queried; When the storage location indicates that part of the aggregated data is stored in the storage unit and part of the data is stored in the data storage platform, the corresponding first granularity aggregated data is obtained from the storage unit, and the corresponding second granularity aggregated data is obtained from the data storage platform; wherein the first granularity aggregated data is obtained by the server node aggregating the received first data with the data in the storage unit based on a first time unit; the second granularity aggregated data is obtained by the server node aggregating the first granularity aggregated data with the data in the data storage platform based on a second time unit; The acquired first granularity aggregated data and the acquired second granularity aggregated data are aggregated, and the data to be queried is determined based on the aggregated data.

12. The method according to claim 11, further comprising: When the storage location indicates that all the aggregated data are stored in the storage unit, obtaining corresponding first granularity aggregated data from the storage unit; The data to be queried is determined based on the acquired first granularity aggregate data.

13. The method according to claim 11, wherein the storage unit is further used to store the latest m second-granularity aggregate data of the second time units; the method further comprises: When the storage location indicates that different parts of the aggregated data are stored in different locations of the storage unit, obtaining corresponding first granularity aggregated data and corresponding second granularity aggregated data from different locations of the storage unit; The acquired first granularity aggregated data and the acquired second granularity aggregated data are aggregated, and the data to be queried is determined based on the aggregated data.

14. The method according to claim 11, wherein the step of determining a storage location of the aggregated data for determining the data to be queried comprises: A storage location of aggregated data used to determine the data to be queried is determined from the metadata.

15. A data storage device, deployed in a server node, wherein the server node includes a storage unit, and the server node is communicatively connected to a data storage platform; the device comprises: A receiving module, configured to receive first data to be written; A first aggregation module is configured to aggregate the first data with the data in the storage unit based on a first time unit, and write the obtained first granularity aggregated data into the storage unit; A transfer module, configured to determine data to be transferred from the storage unit when the data in the storage unit meets a preset transfer condition; The second aggregation module is configured to aggregate the first granularity aggregated data in the data to be transferred and the data obtained from the data storage platform based on a second time unit, and store the obtained second granularity aggregated data to the data storage platform; wherein the second time unit is larger than the first time unit.

16. A data query device, deployed in a server node, wherein the server node includes a storage unit, and the server node is communicatively connected to a data storage platform; the device comprises: A first determination module is configured to determine the data to be queried; A second determination module is configured to determine a storage location of aggregated data used to determine the data to be queried; A first acquisition module is configured to acquire corresponding first-granularity aggregated data from the storage unit and acquire corresponding second-granularity aggregated data from the data storage platform when the storage location indicates that part of the aggregated data is stored in the storage unit and part of the data is stored in the data storage platform; wherein the first-granularity aggregated data is obtained by the server node aggregating the received first data with the data in the storage unit based on a first time unit; and the second-granularity aggregated data is obtained by the server node aggregating the first-granularity aggregated data with the data in the data storage platform based on a second time unit; The third determination module is configured to aggregate the acquired first granularity aggregated data and the acquired second granularity aggregated data, and determine the data to be queried based on the aggregated data.

17. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 14.

18. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 14 is implemented.

Images