Data storage method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure provide a data storage method and device, electronic equipment and storage medium. After obtaining target data, the first data feature of the target data is extracted, and the first data feature represents the data volume of the target data. According to the first data feature, the target number of fragments is obtained, and the target number of fragments represents the number of storage units for storing the target data. Based on the target number of fragments, the corresponding storage engine is obtained, and the target data is stored in the storage engine. By obtaining the target number of fragments according to the first data feature of the target data, and storing based on the target number of fragments, the effect of dynamically allocating storage units based on the data volume of the target data is realized, avoiding the problem of storing large volume data in one or a small number of storage units, thereby avoiding the problem of data hotspot, and improving the overall throughput and read-write performance of the storage system.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of database, and particularly relate to a data storage method and device, electronic equipment and storage medium. BACKGROUND

[0002] At present, for the application scenario of large-scale distributed storage, the existing technology usually splits the data to be stored, and stores the data based on the principle of load balancing, so as to improve the overall performance of the storage system.

[0003] However, the number of data shards in the existing technology is fixed, which is usually fixed based on the number of available storage units, which is easy to cause data hotspots, storage imbalance and other problems, and affects the overall throughput and read-write performance of the storage system. SUMMARY

[0004] Embodiments of the present disclosure provide a data storage method and device, electronic equipment and storage medium to overcome the problems of data hotspots, storage imbalance and the like.

[0005] In a first aspect, the embodiments of the present disclosure provide a data storage method, comprising:

[0006] reading target data to be stored in the current period from a message queue, extracting a first data feature of the target data, the first data feature representing the data amount of the target data; calling an adaptive consistent hash processing module, determining data shard information adapted to the target data according to the first data feature, and synchronizing the data shard information adapted to the target data to a configuration center, the data shard information being used to indicate at least one storage engine for storing the target data; the configuration center is used to record data shard information and refresh to a data write component at regular intervals; calling the data write component, writing the target data in the corresponding storage engine based on the data shard information, and the storage engine supporting query processing of the target data.

[0007] In a second aspect, the embodiments of the present disclosure provide a data storage device, comprising:

[0008] an extraction unit configured to read target data to be stored in the current period from a message queue, and extract a first data feature of the target data, the first data feature representing the data amount of the target data;

[0009] The processing unit is configured to invoke the adaptive consistency hash processing module, determine data shard information adapted to the target data according to the first data feature, and synchronize the data shard information adapted to the target data to a configuration center, wherein the data shard information is used to indicate at least one storage engine that stores the target data in shards; and the configuration center is configured to record the data shard information and refresh the data shard information to a data writing component at a timing.

[0010] The storage unit is configured to invoke the data writing component, write the target data in the corresponding storage engine based on the data shard information, and the storage engine supports query processing of the target data.

[0011] In a third aspect, an electronic device is provided, and the electronic device comprises:

[0012] a processor, and a memory connected to the processor in communication;

[0013] The memory stores computer-executable instructions.

[0014] The processor executes the computer-executable instructions stored in the memory to implement the data storage method as described in the first aspect and various possible designs of the first aspect.

[0015] In a fourth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the data storage method as described in the first aspect and various possible designs of the first aspect is implemented.

[0016] In a fifth aspect, a computer program product is provided, and the computer program product comprises a computer program. When a processor executes the computer program, the data storage method as described in the first aspect and various possible designs of the first aspect is implemented.

[0017] The data storage method, device, electronic equipment and storage medium provided by the embodiment obtain target data to be stored in the current period from a message queue, extract a first data feature of the target data, the first data feature representing a data volume of the target data, call an adaptive consistency hash processing module, determine data shard information adapted to the target data according to the first data feature, and synchronize the data shard information adapted to the target data to a configuration center, the data shard information being used to indicate at least one storage engine for storing the target data in shards; the configuration center is used to record the data shard information and refresh to a data writing component at a time; the data writing component is called to write the target data in the corresponding storage engine based on the data shard information, and the storage engine supports query processing of the target data. By extracting the first data feature of the target data, obtaining the data shard information based on the first data feature, and completing the shard storage process of the target data based on the data shard information, the effect of dynamically allocating a storage unit based on the data volume of the target data is realized, the problem of centralized storage of large-volume data in one or a small number of storage units is avoided, and the problems of data hotspots and the like are avoided, thereby improving the overall throughput and read-write performance of the storage system. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, a brief introduction will be given to the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present disclosure, and other drawings can also be obtained by those skilled in the art without creative labor.

[0019] Figure 1 An application scenario diagram of the data storage method provided by the embodiment of the present disclosure is shown in the figure.

[0020] Figure 2 A flowchart of the data storage method provided by the embodiment of the present disclosure is shown in the figure. Figure 1 ;

[0021] Figure 3 A data structure diagram of the target data provided by the embodiment of the present disclosure is shown in the figure.

[0022] Figure 4 A flowchart of the specific implementation of step S101 in the embodiment shown in the figure is shown in the figure. Figure 2

[0023] A flowchart of the specific implementation of step S102 in the embodiment shown in the figure is shown in the figure. Figure 5 Figure 2

[0024] Figure 6 ​​A process diagram for determining a target storage engine is provided for the embodiments of the present disclosure.

[0025] Figure 7 For Figure 5 A flow chart of the specific implementation of step S1022 in the embodiment shown.

[0026] Figure 8 A flow chart of the data storage method provided for the embodiments of the present disclosure Figure 1 ;

[0027] Figure 9 A process diagram of pre-aggregation is provided for the embodiments of the present disclosure.

[0028] Figure 10 For Figure 8 A flow chart of the specific implementation of step S204 in the embodiment shown.

[0029] Figure 11 A process diagram for determining a storage engine is provided for the embodiments of the present disclosure.

[0030] Figure 12 A structural block diagram of a data storage device is provided for the embodiments of the present disclosure.

[0031] Figure 13 A structural diagram of an electronic device is provided for the embodiments of the present disclosure.

[0032] Figure 14 A hardware structural diagram of an electronic device is provided for the embodiments of the present disclosure. DETAILED DESCRIPTION

[0033] To make the purposes, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present disclosure.

[0034] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0035] The application scenarios of the embodiments of the present disclosure are explained as follows:

[0036] Figure 1 This diagram illustrates an application scenario of the data storage method provided in this disclosure. The data storage method provided in this disclosure can be applied to distributed storage application scenarios. Specifically, it can be applied to distributed storage application scenarios for large-scale time-series data, where the time-series data includes, for example, device operation data, traffic monitoring data, and other data with time-series information. Figure 1 As shown in the embodiments of this disclosure, the method can be applied to a data storage server. After receiving and collecting time-series data to be stored, the data storage server adds the time-series data to a message queue (MQ) based on the timestamp of the time-series data. Then, the time-series data in the message queue is sequentially sharded for storage. That is, through the data storage method provided in this disclosure, the time-series data obtained from the message queue is stored in different storage engines, thereby achieving sharded storage of data and load balancing among the storage engines, improving the read and write performance of the storage system. Here, a storage engine can be the number of instances of backend storage devices, the number of physical servers or virtual machines, or the number of processes; it can be understood as an individual with independent computing and data processing capabilities. Further, based on its specific function, the data storage server can be further refined into a data collection unit and a data storage engine. The data collection unit is used to receive and collect target data and store it in the message queue, while the data storage engine is used to execute the data storage method provided in this embodiment, storing the target data in the message queue into the corresponding storage engine. The data collection unit and the data storage engine can be implemented based on the same electronic device, such as the aforementioned data storage server, or they can be implemented through two separate electronic devices, and the function of the data storage server is achieved through communication between the two.

[0037] In the prior art, in the application scenario of large-scale distributed storage, when a data storage server performs sharding or bucketing on target data, the target data is usually sharded based on a fixed number of shards. The number of shards is usually fixed based on the number of available storage engines, for example, if the number of available storage engines is 10, the number of shards is set to 10, and the target data is evenly distributed to the 10 available storage engines for storage to achieve load balancing between storage engines. However, in actual application, when obtaining the storage engine corresponding to the target data, the storage engine used to store the target data is usually obtained by hashing mapping based on a specific label of the target data, for example, based on a data type label represented by the target data, to obtain the corresponding storage engine. More specifically, the data type includes CPU running data, network traffic data, etc. When the data amount of the target data corresponding to the specific label is large (more), a large amount of data will be stored in the same or a few storage engines, resulting in storage hotspots and storage imbalance, affecting the read-write performance and stability of the storage system.

[0038] The embodiments of the present disclosure provide a data storage method to solve the above problems.

[0039] Reference Figure 2 , Figure 2 The flowchart of the data storage method provided by the embodiments of the present disclosure is shown in Figure 1 The method of the present embodiment can be applied in an electronic device, and the data storage method comprises:

[0040] Step S101: reading target data to be stored in the current period from a message queue, and extracting a first data feature of the target data, the first data feature representing the data amount of the target data.

[0041] For example, referring to the application scenario diagram shown in Figure 1 The execution subject of the present embodiment can be a data storage server as shown in Figure 1 , or an electronic device corresponding to a data storage unit. In the present embodiment, the data storage server (hereinafter referred to as server) is taken as the execution subject for introduction.

[0042] Specifically, the server reads target data to be stored in the current period from a message queue, and the target data is a collection of a series of sub-data (i.e. data records) belonging to the same data type. For example, a collection of 100 device running status information. In one possible implementation, the target data is time series data, and the target data includes a plurality of time series data records. Each time series data record represents the record information at a time point, for example, the device running status information at time point A.

[0043] Further, after obtaining the target data, the server processes the target data and extracts a first data feature of the target data, i.e., a data volume of the target data. For example, the data volume of the target data can refer to the data volume (represented by the number of bytes) of the target data as a whole, the number of data records (sub-data) in the target data, or the number of data dimensions of the target data. The data dimension refers to different meanings represented by the data stored in the target data. For example, the target data contains three data dimensions, i.e., data dimension A, data dimension B, and data dimension C. The data under data dimension A represents the data type of the target data, such as running state data and flow data. The data under data dimension B represents the IP address of the device. The data under data dimension C represents the data generation time.

[0044] For example, Figure 3 A data structure diagram of target data is provided for the embodiments of the present disclosure, as shown in Figure 3 The target data stores data information in the form of a data table, each row is a data record, and each column is a data dimension. Specifically, in the data table corresponding to the target data, the first column (data dimension A) corresponds to the data type of the target data, for example, as shown in the figure, the data type of the target data is "cpu_load", representing CPU load. The second column (data dimension B) corresponds to a specific single indicator of the target data, i.e., device identification. The combination of idc (data center identification) and host (device identification) can realize the identification of a specific device. The third column (data dimension C) corresponds to another specific single indicator of the target data, i.e., timestamp, representing the generation time of the data, for example, 20230419-000001, representing April 19, 2023, 0:00:01, and other similar examples, which will not be described in detail. The fourth column (data dimension D) corresponds to the data type value of the target data, i.e., the specific numerical value of the CPU load. In another aspect, each row in the to-be-processed data represents a data record, and each data record contains information of the above four dimensions (i.e., the data record includes four dimensions).

[0045] In a possible implementation, the target data is time series data, as shown in Figure 4 The specific implementation of step S101 includes:

[0046] Step S1011: detecting the generation rate of the time series data in the current time period based on the first interval duration.

[0047] Step S1012: obtaining the first data feature based on the generation rate.

[0048] Exemplarily, the data to be processed is time series data, the time series data is ordered data generated over time, and the server detects the number of time series data generated in the current period at a fixed detection period (first interval length) to obtain the generation rate of the time series data in the current period. Specifically, for example, after the server obtains the time series data, the time interval of each time series data record is calculated based on the timestamp information in the time series data. The longer the time interval, the lower the generation rate. Conversely, the shorter the time interval, the higher the generation rate. Thus, the generation rate of the time series data is obtained. The first interval length is the time interval at which the server reads the time series data. Then, a fixed mapping is performed according to the generation speed to obtain the corresponding first data feature. In this embodiment, the first data feature can be a discrete gear identification or a continuous floating point number. The specific use of the first data feature can be set according to the subsequent use, which will not be described here.

[0049] Step S102: calling an adaptive consistency hash processing module, determining data shard information adapted to the target data according to the first data feature, and synchronizing the data shard information adapted to the target data to a configuration center. The data shard information is used to indicate at least one storage engine for storing the target data. The configuration center is used to record the data shard information and refresh it to the data writing component at a regular time.

[0050] Exemplarily, the adaptive consistent hashing processing module is a preset software function module. After obtaining the first data feature, the first data feature is processed through the adaptive consistent hashing processing module, and based on the data volume of the target data represented by the first data feature, a corresponding target shard number is obtained, where the shard number represents the number of storage units for storing data, that is, the number of shards when the target data is stored in shards. Then, a mapping relationship between each data shard of the target data and the storage unit, that is, the data shard information, is established. By storing the target data shards in different storage units, load balancing between the storage units is achieved. Wherein, the greater the data volume represented by the first data feature, the greater the corresponding target shard number; on the contrary, the smaller the data volume represented by the first data feature, the smaller the corresponding target shard number. By obtaining the target shard number (the number of storage units) matched with the first data feature, when the data volume of the target data is small, the data is stored in the same storage unit, thereby improving the efficiency of continuous reading of data from the storage system, and when the data volume of the target data is large, the data is stored in multiple storage units, thereby reducing the load of a single storage unit, achieving load balancing, and improving the read-write efficiency of the storage system. Then, the data shard information obtained based on the first data feature is synchronized to the configuration center to achieve the storage of the data shard information adapted to the target data. The configuration center is used to record the data shard information and refresh to the data writing component at regular intervals. Subsequently, in response to an access request for the target data, the data shard information is obtained through the configuration center, and the access and query of the target data are realized based on the data shard information. Wherein, the configuration center can be a service or application program set in the server or outside the server, and the specific implementation manner is configured as needed, which is not limited here.

[0051] In a possible implementation manner, the first data feature has a many-to-one mapping relationship with the target shard number. For example, when the first data feature is located in a first feature interval (data interval), the target shard number is A, and when the first data feature is located in a second feature interval (data interval), the target shard number is B. At the same time, the target shard number has an upper limit value and a lower limit value. Exemplarily, the upper limit value of the target shard number is the number of available data storage units, and the lower limit value of the target shard number is 1.

[0052] In a possible implementation manner, the first data feature includes the dimension of the data record in the target data. According to the first data feature, the specific implementation manner of obtaining the target shard number is to obtain the target shard number according to the dimension of the data record. Specifically, the higher the dimension of the data record, the more the target shard number. Wherein, the meaning of the dimension of the data record can be referred to Figure 3The corresponding part of the data structure of the target data is shown. In this embodiment, since the higher the dimension of the data record is, the larger the data amount of a single data record is, and the larger the overall data amount of the corresponding target data is, the target number of shards is obtained through the dimension of the data record, and the target data shards are stored based on the target number of shards, so that excessive data can be avoided from being concentrated in the same storage unit, data hotspots can be avoided, and the read / write performance of the storage system can be improved.

[0053] In a possible implementation, the adaptive consistent hash processing module determines the mapping between the target data after sharding and the storage engine through a consistent hash algorithm, so as to generate data shard information adapted to the target data. Exemplarily, as shown in Figure 5 The specific implementation of step S102 includes the following steps.

[0054] Step S1021: determining the target number of shards according to the first data feature.

[0055] Step S1022: calling the adaptive consistent hash processing module to process the target data to obtain the target storage engine corresponding to the target number of shards.

[0056] Step S1023: generating data shard information based on the engine identifier of the target storage engine.

[0057] Exemplarily, the first data feature and the target number of shards have a preset mapping relationship, and the specific implementation has been introduced in the previous embodiment part, which will not be described here. After obtaining the target number of shards, the adaptive consistent hash processing module is called to process the target data through a consistent hash algorithm to obtain the target storage engine corresponding to the target number of shards. The consistent hash algorithm is a method for mapping binary values of any length to shorter fixed-length binary values. The shorter binary value obtained after hash operation is called hash value. The hash value is a unique and compact numerical representation of a piece of data. When data is stored in a distributed manner, a certain piece of information or index of the data can be used as a hash input value for hash operation, and the obtained hash value can be used to realize the mapping between the data and the storage unit. On this basis, in order to solve the problem of data mapping failure caused by the change (increase or decrease) of the storage unit, the prior art further proposes a consistent hash algorithm, which sets a ring-shaped hash space (i.e., a hash ring), and sets the hash identifier of the storage unit on the ring-shaped hash space, to realize the mapping between the data and the storage unit, thereby avoiding the problem of mapping failure between the data and the storage unit when the storage unit changes. The specific implementation and principle of the consistent hash algorithm are prior art, which will not be described in detail here.

[0058] On this basis, the consistent hash operation between the data and the storage unit needs to obtain the input value for performing the hash operation in the target data first, such as the data ID of the target data, the generation time, etc. Since the target data includes one or more dimensions of data, it is necessary to obtain which data in the target data is used as the input value for the hash operation, that is, to obtain the first target dimension of the target data. Then, the data under the first target dimension of the target data is obtained, and the consistent hash algorithm is called to perform the hash operation on the data under the first target dimension, and the modulo operation is performed based on the hash value obtained after the operation, so as to obtain the storage unit corresponding to the target data after the sharding, that is, the storage engine.

[0059] Further, the target number of shards has a preset mapping relationship with the first target data dimension of the target data, and according to the mapping relationship, the first target data dimension corresponding to the target number of shards can be obtained. Specifically, the mapping relationship includes: when the target number of shards is equal to 1, the first target data dimension is the first dimension, and the data under the first dimension represents the data category of the target data; when the target number of shards is greater than 1, the first target data dimension is the first dimension and at least one second dimension, and the data under the second dimension represents the single item index of the target data. In the embodiment step, when the target number of shards obtained according to the first data feature is equal to 1, it indicates that the data amount of the target data is very small, and the data category (first dimension) of the target data can be directly stored in the corresponding storage engine based on the data category (first dimension) of the target data, so as to ensure that the data of the same data category (such as CPU running temperature, device load) can be stored in the same storage unit, improve the data acquisition efficiency when reading and processing such data subsequently, and reduce the acquisition cost; when the target number of shards obtained according to the first data feature is greater than 1, it indicates that the data amount of the target data is large, at this time, the target data needs to be further stored based on the data category and at least one other single item index, and the data under the first dimension and the second dimension is used as the hash input value and is mapped to the corresponding storage engine, so as to realize the data distribution based on the single item index (second dimension), such as sharding based on the device identification, sharding based on the storage cluster identification, etc., so as to realize the load balancing between the storage units and improve the read-write efficiency of the storage system.

[0060] Figure 6 A process diagram for determining a target storage engine provided by the embodiments of the present disclosure is as follows: Figure 6As shown, the target data includes data dimension A, data dimension B, data dimension C, data dimension D, wherein the data under data dimension A represents the data type of the target data, such as an access request (denoted as Request in the figure); the data under data dimension B represents the device IP address, such as 1.1.1.1 and 1.1.1.2; the data under data dimension C represents the data generation time, such as 20230419 (representing April 19, 2023, which will not be repeated hereinafter) and 20230420. The data under data dimension D represents the access frequency, including 20, 38, 112, and 7. Among them, data dimension A is the first dimension; data dimension B, data dimension C, and data dimension D are the second dimension. Referring to the figure, when the first data feature corresponds to a target number of shards of 1 (denoted as target number of shards = 1 in the figure), the data under data dimension A (“Request”) is subjected to a hash operation, that is, hash(Request) is calculated, to obtain a corresponding hash value Hash_1, and then Hash_1 is subjected to a modulo operation based on a preset value, and the modulo result is mapped to the corresponding storage engine device_1. That is, the target data is stored in device_1 as a whole. When the first data feature corresponds to a target number of shards greater than 1 (denoted as target number of shards = 2 in the figure), in one possible implementation, the combination of the data under data dimension B (“1.1.1.1” or “1.1.1.2”) and the data under data dimension A (“Request”) is subjected to a hash operation, that is, hash(Request, 1.1.1.1) and hash(Request, 1.1.1.2) are calculated, to obtain corresponding hash values Hash_2 and Hash_3, respectively, and then Hash_2 and Hash_3 are subjected to a modulo operation, respectively, and are mapped to the corresponding storage engines device_2 and device_3. The shard storage of the target data is implemented. Of course, in other possible implementation, the combination of data dimension C and data dimension A can also be subjected to a hash operation based on data dimension C, to obtain a corresponding storage engine, and the specific implementation manner is similar and will not be repeated here.

[0061] Further, in one possible implementation, when the first target data dimension is the first dimension and at least one second dimension, that is, the target number of shards is greater than 1, as shown in Figure 7 The specific implementation of step S1022 includes the following steps.

[0062] Step S1022A: obtaining a first storage engine based on the data under the first dimension in the target data.

[0063] Step S1022B: obtaining a second storage engine based on the data in the second dimension in the target data, wherein the first storage engine and the second storage engine belong to the same storage cluster, and the second storage engine is a logically adjacent downstream node or a physically adjacent downstream node of the first storage engine.

[0064] Step S1022C: obtaining a target storage engine based on the first storage engine and the second storage engine.

[0065] For example, in combination with the method of obtaining the corresponding storage engine through the hash operation of the data in different dimensions in the target data introduced in the previous steps, first, based on the data in the first dimension, the first storage engine can be obtained, which is a storage device for storing data of the data type of the target data, for example, a storage device for storing CPU temperature data. Then, on this basis, based on the data in the second dimension in the target data, the second storage engine is obtained, which is a logically adjacent downstream node or a physically adjacent downstream node of the first storage engine, and the first storage engine and the second storage engine are used as the storage engine. In this embodiment, the first storage engine is determined through the data in the first dimension, and the second storage engine adjacent to the downstream of the first storage engine is determined based on the data in the second dimension, so that the target data belonging to the same data category can be stored in the same storage cluster, the data transmission time when reading and processing such data is reduced, and the read-write performance of the storage system is improved.

[0066] Step S103: calling a data writing component to write the target data in the corresponding storage engine based on the data sharding information, and the storage engine supports query processing of the target data.

[0067] For example, the data writing component is also a preset software function module. Further, after obtaining the data sharding information, the server stores the target data in the corresponding storage engine by calling the data writing component, wherein the mapping relationship between the sharded target data and the storage engine is determined based on the data sharding information. The data sharding information is sent to the data writing component by the configuration center. Therefore, by calling the data writing component, the sharding storage process of the target data can be completed based on the above data sharding information.

[0068] In this embodiment, by obtaining the target data to be stored in the current cycle from the message queue, the first data feature of the target data is extracted, and the first data feature represents the data volume of the target data; the adaptive consistent hash processing module is called, the data shard information adapted to the target data is determined according to the first data feature, and the data shard information adapted to the target data is synchronized to the configuration center, and the data shard information is used to indicate at least one storage engine for storing the target data in shards; the configuration center is used to record the data shard information, and is refreshed to the data writing component at regular intervals; the data writing component is called, and the target data is written in the corresponding storage engine based on the data shard information, and the storage engine supports query processing of the target data. By extracting the first data feature of the target data, obtaining the data shard information based on the first data feature, and completing the shard storage process of the target data based on the data shard information, the effect of dynamically allocating storage units based on the data volume of the target data is realized, the problem of storing large volume data in one or a small number of storage units is avoided, and the problem of data hotspot is avoided, thereby improving the overall throughput and read-write performance of the storage system.

[0069] Reference Figure 8 , Figure 8 The flowchart of the data storage method provided by the embodiments of the present disclosure is shown in Figure 1 The embodiments of the present disclosure further refine step S103 based on the embodiments shown in Figure 2 The method of the present embodiment includes:

[0070] Step S201: reading the target data to be stored in the current cycle from the message queue.

[0071] Step S202: querying from the configuration center whether the target data is registered with the pre-aggregation rule.

[0072] Exemplarily, before performing the pre-aggregation operation, the server can generate a pre-aggregation rule in the configuration center through a preset configuration instruction. Then, based on the pre-aggregation rule, the corresponding second target data dimension is determined, and data dimension reduction is performed with the second target data dimension as the target to obtain aggregated data. The pre-aggregation rule represents the mapping relationship between the first dimension of the target data and the second target data dimension of the target data, and describes the aggregation method of at least one type of target data during pre-aggregation. By generating the corresponding pre-aggregation rule through the configuration instruction, flexible aggregation of the target data can be realized, the personalized storage and reading needs of users can be met, and the read-write performance of the storage system can be improved.

[0073] Step S203: When it is queried that the target data is registered with the pre-aggregation rule, the target data is pre-aggregated according to the pre-aggregation rule to obtain pre-aggregated data, and the data dimension of the aggregated data is less than the data dimension of the target data.

[0074] Exemplarily, after obtaining the target data, the target data can be pre-aggregated first, so as to realize dimension reduction of the target data, reduce the data amount, and provide efficiency of subsequent storage and reading of data. Specifically, the pre-aggregation rule is used to represent the mapping relationship between the first dimension of the target data and the second target data dimension of the target data, and the pre-aggregation of the second target data dimension of the target data means that the data of one or more data dimensions in the target data is combined, so as to realize dimension reduction of the data, Figure 9 A pre-aggregation process diagram provided by the embodiment of the present disclosure is shown in Figure 9 Exemplarily, the target data M is time series data, which includes service_name, host, cluster, time_stamp and value, and 5 data dimensions (all of which are second dimensions) representing service name (service_name), device address (host), cluster identifier (cluster), data timestamp (time_stamp) and data value (value). According to the pre-configured pre-aggregation rule, service_name, cluster and time_stamp are taken as the second target data dimension, and are aggregated by the aggregation mode of cumulative summation (Sum), while the data under the host dimension is deleted. More specifically, for a data record M(1) in the target data M, the corresponding data content is: service_name=abc, host=10.10.10.10, cluster=c1, time_stamp=20230419, value=1. After pre-aggregation of the second target data dimension of the target data, the data under the host dimension is discarded, and the cumulative summation operation is performed on the data points with the same service_name dimension, dc dimension and time_stamp dimension, to obtain aggregated data M_r. The data corresponding to the data record M(1) is aggregated data record Mr(1), and the corresponding data content is: service_name=abc, cluster=c1, time_stamp=20230419, value=8. The meaning represented thereby is that the sum of the data values of all devices corresponding to the cluster identifier c1 on 20230419 for the service with the service name abc is 8.

[0075] In the above embodiment, the second target data dimension refers to the data dimension retained in the target data. In another possible implementation, the second target data dimension can also refer to the data dimension deleted from the target data. The subsequent execution manner is adjusted accordingly, and details are not described herein.

[0076] For example, the target data is time series data, and the specific implementation of step S203 includes:

[0077] Step S2031: obtaining a data generation timestamp corresponding to the time series data.

[0078] Step S2032: performing pre-aggregation processing on the target data based on the data generation timestamp and a pre-aggregation rule to obtain pre-aggregated data, wherein the pre-aggregation rule is a rule for aggregating the time series data based on a time dimension.

[0079] For example, when the target data is time series data, the generation of the target data has a time domain feature. For example, each record data in the target data has a specific feature in the corresponding generation time, which can be represented by a data generation timestamp. Based on this, the pre-aggregation rule based on the data generation timestamp and the rule for aggregating the time series data based on the time dimension can be used to perform pre-aggregation processing on the data, so as to realize data aggregation based on the time dimension. Specifically, for example, the record data generated between A time point and B time point in the time domain data is aggregated, so as to realize statistics for a feature time period. This pre-aggregation rule specific to time series data can effectively reduce the data volume while realizing aggregation statistics of the time dimension (multiple time periods) and improving the data quality and content richness of the generated pre-aggregated data.

[0080] Step S204: calling a data writing component to write the pre-aggregated data in the corresponding storage engine based on the data shard information, and the storage engine supports query processing for the pre-aggregated data.

[0081] For example, as shown in Figure 10 the specific implementation steps of step S204 include:

[0082] Step S2041: extracting a first data feature of the aggregated data.

[0083] Step S2042: determining a target shard number according to the first data feature.

[0084] For example, the specific implementation of extracting the first data feature based on the aggregated data and obtaining the target shard number according to the first data feature in steps S203-S204 is the same as the implementation of the corresponding steps in the embodiment shown in Figure 2 For example, the specific implementation of extracting the first data feature based on the aggregated data and obtaining the target shard number according to the first data feature in steps S203-S204 is the same as the implementation of the corresponding steps in the embodiment shown in Figure 2The related description of the corresponding part in the illustrated embodiment will not be repeated here.

[0085] Step S2043: obtaining the second target data dimension of the pre-aggregated data according to the target number of shards.

[0086] Exemplarily, after obtaining the target number of shards, the second target data dimension is determined to be the first dimension or the first dimension and at least one second dimension based on the specific target number of shards, and the specific implementation manner is described in detail in the following. Figure 2 The detailed description of the illustrated embodiment has been described above, and will not be repeated here. In the subsequent steps, the case where the target number of shards is greater than 1, i.e., the second target data dimension is the first dimension and at least one second dimension, is further introduced.

[0087] Step S2044: obtaining the data of the data record in the pre-aggregated data under the second target data dimension.

[0088] Step S2045: calling a consistent hashing algorithm to calculate the data of the data record under the second target data dimension to obtain the target storage engine corresponding to the data record.

[0089] Further, the pre-aggregated data includes at least one data record. When the second target data dimension is the first dimension and at least one second dimension, the data of the data record in the pre-aggregated data under the first dimension and at least one second dimension is obtained. Then, the consistent hashing algorithm is called to calculate the data of the data record under the first dimension and at least one second dimension to obtain the target storage engine corresponding to the data record.

[0090] Exemplarily, the specific implementation manner of step S2045 includes:

[0091] Step S2045A: obtaining the first data of the data record under the first dimension, and calculating the first data based on the target hashing algorithm to obtain the first hash value.

[0092] Step S2045B: obtaining the second data of the data record under the at least one second dimension, and calculating the second data under the at least one second dimension based on the target hashing algorithm to obtain the second hash value.

[0093] Step S2045C: respectively performing modulo operation on the preset hash ring length by using the first hash value and the second hash value to obtain the first hash ring position corresponding to the first hash value and the second hash ring position corresponding to the second hash value.

[0094] Step S2045D: obtaining the target storage engine corresponding to the data record according to the sum of the first hash ring position and the second hash ring position.

[0095] Figure 11This is a schematic diagram illustrating a process for determining a storage engine according to an embodiment of the present disclosure. The following is in conjunction with... Figure 11 The above steps will be described in detail, for example, as follows: Figure 11 As shown, for data record M in the pre-aggregated data, firstly, the first data M1 (shown as M1 in the figure) of data record M under the first dimension is obtained, where the first dimension represents the data type of data record M, such as CPU temperature data. Then, a hash operation is performed on the first data M1 based on the target hash algorithm (shown as hash(M1) in the figure) to obtain the first hash value (shown as Hash_1 in the figure). Afterwards, following similar steps, the second data of data record M1 under at least one second dimension is obtained, namely the second data M2_1 and the second data M2_2 (shown as M2_1 and M2_2 in the figure, respectively). The same target hash algorithm is then used to perform a hash operation on the combination of the second data M2_1 and the second data M2_2 (shown as hash(M2_1, M2_2) in the figure) to obtain the second hash value (shown as Hash_2 in the figure). Next, modulo operations (Hash_1%L and Hash_2%L) are performed on the preset hash ring length (L in the diagram) using the first hash value and the second hash value respectively, to obtain the remainders of the first hash value and the second hash value relative to the hash ring length, i.e., the first hash ring position corresponding to the first hash value (pos_1 in the diagram) and the second hash ring position corresponding to the second hash value (pos_2 in the diagram). Then, the sum of the first hash ring position and the second hash ring position is calculated, and a modulo operation is performed on the hash ring length ((pos_1+pos_2)%L in the diagram) to obtain the storage engine T corresponding to the data record.

[0096] Step S2046: Generate data sharding information based on the engine identifier of the target storage engine.

[0097] Step S2047: Based on the data sharding information, store each data record to the corresponding target storage engine.

[0098] Furthermore, after determining the storage engine corresponding to each data record in the pre-aggregated data, each data record is stored in its corresponding storage engine, thereby achieving the purpose of sharding storage based on the data characteristics of each data record in the pre-aggregated data. In this embodiment, by performing consistent hashing operations on the data in the first and second dimensions respectively, and then superimposing the calculated hash values, sharding based on single data indicators is achieved in the form of a two-level hash ring (the first-level hash ring corresponding to the first dimension and the second-level hash ring corresponding to the second dimension). This allows the pre-aggregated data to be evenly distributed across multiple corresponding storage units, improving the uniformity of data sharding mapping on the hash ring (corresponding storage unit).

[0099] It should be noted that the above process of processing based on the pre-aggregation data, realizing writing the pre-aggregation data in the corresponding storage engine based on the data shard information, is also applicable to processing based on target data (data that has not been pre-aggregated), realizing writing the target data in the corresponding storage engine based on the data shard information, that is, the further refinement of the process of writing the target data in the target storage engine based on the embodiment shown in Figure 2 will not be described here.

[0100] Optionally, in a possible implementation, after step S204, further comprising:

[0101] Step S2051: receiving a first query instruction for the pre-aggregation data.

[0102] Step S2052: obtaining, based on the first query instruction, a pre-aggregation rule corresponding to the pre-aggregation data from the configuration center.

[0103] Step S2053: obtaining the pre-aggregation data from the search engine according to the pre-aggregation rule corresponding to the pre-aggregation data.

[0104] Exemplarily, after writing the pre-aggregation data (and / or target data) into the storage engine, the query of the pre-aggregation data (and / or target data) can be further realized based on the first query instruction, and this process can be realized based on a data query component. Specifically, after receiving the first query instruction for the pre-aggregation data, the pre-aggregation rule corresponding to the pre-aggregation data is obtained through the configuration center, and then the target data corresponding to the first query instruction is obtained based on the data shard information, and the target data is aggregated based on the pre-aggregation rule, so as to obtain the pre-aggregation data.

[0105] Optionally, in another possible implementation, after step S204, further comprising:

[0106] Step S2054: receiving a second query instruction for the pre-aggregation data.

[0107] Step S2055: calling an adaptive consistency hash processing module to query the configuration center to obtain data shard information corresponding to the pre-aggregation data.

[0108] Step S2056: calling a data query component to query data according to the data shard information corresponding to the pre-aggregation data, to obtain the pre-aggregation data.

[0109] Exemplarily, in another possible implementation, the pre-aggregated data corresponds to data shard information, after the pre-aggregated data is written into the plurality of storage engines, the querying of the pre-aggregated data can be further implemented based on the second query instruction, and the process can be implemented based on the data query component. Specifically, after receiving the second query instruction for the pre-aggregated data, the data shard information corresponding to the pre-aggregated data is obtained through the configuration center, and then the pre-aggregated data is obtained. It should be noted that the steps of this embodiment are also applicable to the querying of the target data, and will not be described here.

[0110] The data storage method corresponding to the above embodiment, Figure 12 A structural block diagram of a data storage device provided by an embodiment of the present disclosure is shown. For ease of illustration, only parts related to the embodiment of the present disclosure are shown.

[0111] With reference to Figure 12 The data storage device 3 comprises:

[0112] The extraction unit 31 is configured to read the target data to be stored in the current period from the message queue, extract a first data feature of the target data, and the first data feature represents the data volume of the target data.

[0113] The processing unit 32 is configured to call an adaptive consistency hash processing module, determine data shard information adapted to the target data according to the first data feature, and synchronize the data shard information adapted to the target data to a configuration center. The data shard information is used to indicate at least one storage engine for storing the target data in shards. The configuration center is used to record the data shard information and refresh the data shard information to a data writing component at a regular time.

[0114] The storage unit 33 is configured to call the data writing component, write the target data in the corresponding storage engine based on the data shard information, and the storage engine supports the query processing of the target data.

[0115] In an embodiment of the present disclosure, the storage unit 33 is further configured to query whether the target data is registered with a pre-aggregation rule from the configuration center, and when it is found that the target data is registered with the pre-aggregation rule, perform pre-aggregation processing on the target data according to the pre-aggregation rule to obtain pre-aggregated data, the data dimension of the aggregated data is less than the data dimension of the target data, call the data writing component, and write the pre-aggregated data in the corresponding storage engine based on the data shard information, and the storage engine supports the query processing of the pre-aggregated data.

[0116] In an embodiment of the present disclosure, the target data is time series data, and the storage unit 33 is specifically configured to: obtain a data generation timestamp corresponding to the time series data when performing pre-aggregation processing on the target data according to a pre-aggregation rule to obtain pre-aggregation data, and perform pre-aggregation processing on the target data based on the data generation timestamp and the pre-aggregation rule to obtain the pre-aggregation data, wherein the pre-aggregation rule is a rule of aggregating the time series data based on a time dimension.

[0117] In an embodiment of the present disclosure, the storage unit 33 is further configured to: receive a first query instruction for the pre-aggregation data, obtain the pre-aggregation rule corresponding to the pre-aggregation data from the configuration center based on the first query instruction, and obtain the pre-aggregation data from the search engine according to the pre-aggregation rule corresponding to the pre-aggregation data.

[0118] In an embodiment of the present disclosure, the storage unit 33 is further configured to: receive a second query instruction for the target data, call the adaptive consistent hash processing module to query the configuration center to obtain data sharding information of the target data, and call the data query component to perform data query according to the data sharding information of the target data to obtain the target data.

[0119] In an embodiment of the present disclosure, the processing unit 32 is specifically configured to: determine a target number of shards according to the first data feature, call the adaptive consistent hash processing module to process the target data to obtain a target storage engine corresponding to the target number of shards, and generate the data sharding information based on an engine identifier of the target storage engine.

[0120] In an embodiment of the present disclosure, when the processing unit 32 calls the adaptive consistent hash processing module to process the target data to obtain the target storage engine corresponding to the target number of shards, the processing unit 32 is specifically configured to: obtain a first target data dimension of the target data according to the target number of shards, and process data in the target data in the first target data dimension based on a consistent hash algorithm to obtain the target storage engine.

[0121] In an embodiment of the present disclosure, when the processing unit 32 obtains the first target data dimension of the target data according to the target number of shards, the processing unit 32 is specifically configured to: when the target number of shards is equal to 1, the first target data dimension is a first dimension, and data in the first dimension represents a data category of the target data; and when the target number of shards is greater than 1, the first target data dimension is the first dimension and at least one second dimension, and data in the second dimension represents a single-item index of the target data.

[0122] In an embodiment of the present disclosure, when the first target data dimension is a first dimension and at least one second dimension, the processing unit 32, when processing the data of the first target data dimension in the target data based on the consistent hashing algorithm to obtain the target storage engine, is specifically configured to: obtain a first storage engine based on the data of the first dimension in the target data; obtain a second storage engine based on the data of the second dimension in the target data, wherein the first storage engine and the second storage engine belong to the same storage cluster, and the second storage engine is a logically adjacent downstream node or a physically adjacent downstream node of the first storage engine; and obtain the target storage engine according to the first storage engine and the second storage engine.

[0123] In an embodiment of the present disclosure, the target data includes at least one data record, and the processing unit 32, when processing the data of the first target data dimension in the target data based on the consistent hashing algorithm to obtain the target storage engine, is specifically configured to: obtain the data of the data record in the first target data dimension; and call the consistent hashing algorithm to calculate the data of the data record in the first target data dimension to obtain the target storage engine corresponding to the data record.

[0124] In an embodiment of the present disclosure, the processing unit 32, when calling the consistent hashing algorithm to calculate the data of the data record in the first target data dimension to obtain the target storage engine corresponding to the data record, is specifically configured to: obtain first data of the data record in the first dimension, and calculate the first data based on the target hashing algorithm to obtain a first hash value; obtain second data of the data record in the at least one second dimension, and calculate the second data of the at least one second dimension based on the target hashing algorithm to obtain a second hash value; perform a modulo operation on a preset hash ring length by using the first hash value and the second hash value respectively to obtain a first hash ring position corresponding to the first hash value and a second hash ring position corresponding to the second hash value; and obtain the target storage engine corresponding to the data record according to the sum of the first hash ring position and the second hash ring position.

[0125] In an embodiment of the present disclosure, the first data feature includes a dimension of the data record in the target data.

[0126] In an embodiment of the present disclosure, the target data is time series data, and the extraction unit 31, when extracting the first data feature of the target data, is specifically configured to: detect a generation rate of the time series data in a current time period based on a first interval duration; and obtain the first data feature based on the generation rate.

[0127] The extraction module 31, the processing module 32, and the storage module 33 are connected in sequence. The data storage device 3 provided in the embodiment can execute the technical solutions of the above method embodiments, and has similar implementation principles and technical effects. Details are not described herein again.

[0128] Figure 13 A structural diagram of an electronic device is provided for the embodiments of the present disclosure, as shown in Figure 13 The electronic device 4 includes:

[0129] a processor 41, and a memory 42 connected with the processor 41 in communication;

[0130] The memory 42 stores computer-executable instructions.

[0131] The processor 41 executes the computer-executable instructions stored in the memory 42 to implement the data storage method in the embodiments as shown in Figures 2-11

[0132] Optionally, the processor 41 and the memory 42 are connected through a bus 43.

[0133] The related descriptions can be understood by referring to the related descriptions and effects of the steps in the corresponding embodiments, which will not be repeated here. Figures 2-11

[0134] The embodiments of the present disclosure provide a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to implement the data storage method in any one of the embodiments of the present disclosure when executed by a processor. Figures 2-11

[0135] The embodiments of the present disclosure provide a computer program product, which includes a computer program, and the computer program is used to implement the data storage method in the embodiments as shown in Figures 2-11

[0136] Reference is made to Figure 14 , which shows a structural diagram of an electronic device 900 suitable for implementing the embodiments of the present disclosure. The electronic device 900 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDA), tablet computers (PAD), portable media players (PMP), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 14 The electronic device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0137] As shown in Figure 14 ​​​​As shown, the electronic device 900 can include a processing device (e.g., a central processor, a graphics processor, etc.) 901 that can perform various suitable actions and processes in accordance with programs stored in a Read Only Memory (ROM) 902 or loaded from a storage device 908 into a Random Access Memory (RAM) 903. Various programs and data required by the electronic device 900 for operation are also stored in the RAM 903. The processing device 901, the ROM 902, and the RAM 903 are connected to each other by a bus 904. An Input / Output (I / O) interface 905 is also connected to the bus 904.

[0138] Generally, the following devices can be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 907 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 909. The communication devices 909 can allow the electronic device 900 to communicate wirelessly or wired with other devices to exchange data. Although Figure 14 The electronic device 900 is shown with various devices, but it should be understood that not all of the shown devices are required to be implemented or present. More or fewer devices can alternatively be implemented or present.

[0139] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 909, or installed from the storage devices 908, or installed from the ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0140] It should be noted that the computer-readable medium in the above disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, an RF (radio frequency) or the like, or any suitable combination of the above.

[0141] The computer-readable medium described above can be contained in the electronic device described above; or can exist separately and not be assembled into the electronic device.

[0142] The computer-readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0143] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0144] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0145] The units described in the embodiments of the present disclosure can be implemented by software, or by hardware. In some cases, the name of the unit does not constitute a limitation on the unit itself. For example, the first obtaining unit can also be described as a unit for obtaining at least two Internet protocol addresses.

[0146] The functions described in this specification can be performed at least in part by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0147] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0148] In a first aspect, according to one or more embodiments of the present disclosure, a data storage method is provided, comprising:

[0149] reading target data to be stored in the current period from a message queue, extracting a first data feature of the target data, the first data feature representing a data volume of the target data; calling an adaptive consistent hash processing module, determining data shard information adapted to the target data according to the first data feature, and synchronizing the data shard information adapted to the target data to a configuration center, the data shard information being used to indicate at least one storage engine for storing the target data in shards; the configuration center is used to record data shard information and refresh to a data writing component at regular intervals; calling the data writing component, writing the target data in the corresponding storage engine based on the data shard information, and the storage engine supporting query processing of the target data.

[0150] According to one or more embodiments of the present disclosure, the method further comprises: querying from the configuration center whether the target data registers a pre-aggregation rule; when it is queried that the target data registers a pre-aggregation rule, performing pre-aggregation processing on the target data according to the pre-aggregation rule to obtain pre-aggregation data, the data dimension of the aggregation data being less than the data dimension of the target data; calling the data writing component, writing the pre-aggregation data in the corresponding storage engine based on the data shard information, and the storage engine supporting query processing of the pre-aggregation data.

[0151] According to one or more embodiments of the present disclosure, the target data is time series data, and the pre-aggregation processing of the target data according to the pre-aggregation rule to obtain pre-aggregated data comprises: obtaining a data generation timestamp corresponding to the time series data; and performing pre-aggregation processing on the target data based on the data generation timestamp and the pre-aggregation rule to obtain pre-aggregated data, wherein the pre-aggregation rule is a rule for aggregating time series data based on a time dimension.

[0152] According to one or more embodiments of the present disclosure, the method further comprises: receiving a first query instruction for the pre-aggregated data; obtaining, based on the first query instruction, a pre-aggregation rule corresponding to the pre-aggregated data from the configuration center; and obtaining the pre-aggregated data from the search engine according to the pre-aggregation rule corresponding to the pre-aggregated data.

[0153] According to one or more embodiments of the present disclosure, the method further comprises: receiving a second query instruction for the target data; calling an adaptive consistent hash processing module to query the configuration center to obtain data shard information of the target data; and calling a data query component to perform data query according to the data shard information of the target data to obtain the target data.

[0154] According to one or more embodiments of the present disclosure, the calling of the adaptive consistent hash processing module to determine data shard information adapted to the target data according to the first data feature comprises: determining a target shard number according to the first data feature; calling the adaptive consistent hash processing module to process the target data to obtain a target storage engine corresponding to the target shard number; and generating the data shard information based on an engine identifier of the target storage engine.

[0155] According to one or more embodiments of the present disclosure, the calling of the adaptive consistent hash processing module to process the target data to obtain a target storage engine corresponding to the target shard number comprises: obtaining a first target data dimension of the target data according to the target shard number; and processing data in the first target data dimension in the target data based on a consistent hash algorithm to obtain the target storage engine.

[0156] According to one or more embodiments of the present disclosure, the obtaining of the first target data dimension of the target data according to the target shard number comprises: when the target shard number is equal to 1, the first target data dimension is a first dimension, and data in the first dimension represents a data category of the target data; and when the target shard number is greater than 1, the first target data dimension is the first dimension and at least one second dimension, and data in the second dimension represents a single index of the target data.

[0157] According to one or more embodiments of the present disclosure, when the first target data dimension is the first dimension and at least one second dimension, the processing the data in the first target data dimension in the target data based on the consistent hashing algorithm to obtain the target storage engine comprises: obtaining a first storage engine based on the data in the first dimension in the target data; obtaining a second storage engine based on the data in the second dimension in the target data, wherein the first storage engine and the second storage engine belong to the same storage cluster, and the second storage engine is a logically adjacent downstream node or a physically adjacent downstream node of the first storage engine; and obtaining the target storage engine according to the first storage engine and the second storage engine.

[0158] According to one or more embodiments of the present disclosure, the target data comprises at least one data record, and the processing the data in the first target data dimension in the target data based on the consistent hashing algorithm to obtain the target storage engine comprises: obtaining the data in the first target data dimension of the data record in the target data; and invoking a consistent hashing algorithm to calculate the data in the first target data dimension of the data record to obtain the target storage engine corresponding to the data record.

[0159] According to one or more embodiments of the present disclosure, the invoking the consistent hashing algorithm to calculate the data in the first target data dimension of the data record to obtain the target storage engine corresponding to the data record comprises: obtaining first data in the first dimension of the data record, and calculating the first data based on a target hashing algorithm to obtain a first hash value; obtaining second data in at least one second dimension of the data record, and calculating the second data in the at least one second dimension based on the target hashing algorithm to obtain a second hash value; respectively performing modulo operation on a preset hash ring length by using the first hash value and the second hash value to obtain a first hash ring position corresponding to the first hash value and a second hash ring position corresponding to the second hash value; and obtaining the target storage engine corresponding to the data record according to the sum of the first hash ring position and the second hash ring position.

[0160] According to one or more embodiments of the present disclosure, the first data feature comprises a dimension of the data record in the target data.

[0161] According to one or more embodiments of the present disclosure, the target data is time series data, and the extracting the first data feature of the target data comprises: detecting a generation rate of the time series data in a current time period based on a first interval duration; and obtaining the first data feature based on the generation rate.

[0162] In a second aspect, according to one or more embodiments of the present disclosure, a data storage device is provided, comprising:

[0163] an extraction unit configured to read target data to be stored in the current period from a message queue, extract a first data feature of the target data, the first data feature representing a data volume of the target data;

[0164] a processing unit configured to call an adaptive consistency hashing processing module, determine data shard information adapted to the target data according to the first data feature, and synchronize the data shard information adapted to the target data to a configuration center, the data shard information being used to indicate at least one storage engine for storing the target data by shards; the configuration center being used to record the data shard information and refresh the data shard information to a data writing component at a regular time;

[0165] a storage unit configured to call the data writing component, write the target data in the corresponding storage engine based on the data shard information, and the storage engine supporting query processing of the target data.

[0166] According to one or more embodiments of the present disclosure, the storage unit is further configured to: query whether the target data is registered with a pre-aggregation rule from the configuration center; when it is found that the target data is registered with the pre-aggregation rule, perform pre-aggregation processing on the target data according to the pre-aggregation rule to obtain pre-aggregated data, the data dimension of the aggregated data being less than the data dimension of the target data; call the data writing component, and write the pre-aggregated data in the corresponding storage engine based on the data shard information, and the storage engine supporting query processing of the pre-aggregated data.

[0167] According to one or more embodiments of the present disclosure, the target data is time-series data, and when the storage unit performs pre-aggregation processing on the target data according to the pre-aggregation rule to obtain pre-aggregated data, the storage unit is specifically configured to: obtain a data generation timestamp corresponding to the time-series data; perform pre-aggregation processing on the target data based on the data generation timestamp and the pre-aggregation rule to obtain the pre-aggregated data, wherein the pre-aggregation rule is a rule for aggregating the time-series data based on a time dimension.

[0168] According to one or more embodiments of the present disclosure, the storage unit is further configured to: receive a first query instruction for the pre-aggregated data; obtain the pre-aggregation rule corresponding to the pre-aggregated data from the configuration center based on the first query instruction; and obtain the pre-aggregated data from a search engine according to the pre-aggregation rule corresponding to the pre-aggregated data.

[0169] According to one or more embodiments of the present disclosure, the storage unit is further configured to: receive a second query instruction for the target data; call the adaptive consistency hashing processing module, query the configuration center, and obtain the data shard information of the target data; call a data query component to perform data query according to the data shard information of the target data, and obtain the target data.

[0170] According to one or more embodiments of the present disclosure, the processing unit is specifically configured to: determine a target number of shards according to a first data feature; invoke an adaptive consistent hashing processing module to process target data to obtain a target storage engine corresponding to the target number of shards; and generate data shard information based on an engine identifier of the target storage engine.

[0171] According to one or more embodiments of the present disclosure, when the processing unit invokes the adaptive consistent hashing processing module to process the target data to obtain the target storage engine corresponding to the target number of shards, the processing unit is specifically configured to: obtain a first target data dimension of the target data according to the target number of shards; and process data in the target data at the first target data dimension based on a consistent hashing algorithm to obtain the target storage engine.

[0172] According to one or more embodiments of the present disclosure, when the processing unit obtains the first target data dimension of the target data according to the target number of shards, the processing unit is specifically configured to: when the target number of shards is equal to 1, the first target data dimension is a first dimension, and data at the first dimension represents a data category of the target data; and when the target number of shards is greater than 1, the first target data dimension is the first dimension and at least one second dimension, and data at the second dimension represents a single-item index of the target data.

[0173] According to one or more embodiments of the present disclosure, when the first target data dimension is the first dimension and the at least one second dimension, and the processing unit processes the data at the first target data dimension in the target data based on the consistent hashing algorithm to obtain the target storage engine, the processing unit is specifically configured to: obtain a first storage engine based on data at the first dimension in the target data; obtain a second storage engine based on data at the second dimension in the target data, wherein the first storage engine and the second storage engine belong to the same storage cluster, and the second storage engine is a logically adjacent downstream node or a physically adjacent downstream node of the first storage engine; and obtain the target storage engine according to the first storage engine and the second storage engine.

[0174] According to one or more embodiments of the present disclosure, the target data includes at least one data record, and when the processing unit processes the data at the first target data dimension in the target data based on the consistent hashing algorithm to obtain the target storage engine, the processing unit is specifically configured to: obtain data of the data record at the first target data dimension in the target data; and invoke the consistent hashing algorithm to calculate the data of the data record at the first target data dimension to obtain a target storage engine corresponding to the data record.

[0175] According to one or more embodiments of the present disclosure, when the processing unit invokes the consistent hash algorithm to calculate data of the data record under the first target data dimension, to obtain the target storage engine corresponding to the data record, the processing unit is specifically configured to: obtain first data of the data record under the first dimension, and calculate the first data based on the target hash algorithm to obtain a first hash value; obtain second data of the data record under at least one second dimension, and calculate the second data under the at least one second dimension based on the target hash algorithm to obtain a second hash value; perform modulo operation on a preset hash ring length by using the first hash value and the second hash value respectively to obtain a first hash ring position corresponding to the first hash value and a second hash ring position corresponding to the second hash value; and obtain the target storage engine corresponding to the data record according to a sum of the first hash ring position and the second hash ring position.

[0176] According to one or more embodiments of the present disclosure, the first data feature includes a dimension of the data record in the target data.

[0177] According to one or more embodiments of the present disclosure, the target data is time series data, and when the extraction unit extracts the first data feature of the target data, the extraction unit is specifically configured to: detect a generation rate of the time series data in a current time period based on a first interval duration; and obtain the first data feature based on the generation rate.

[0178] In a third aspect, according to one or more embodiments of the present disclosure, an electronic device is provided, including: a processor, and a memory connected to the processor in communication;

[0179] The memory stores computer execution instructions;

[0180] The processor executes the computer execution instructions stored in the memory, to implement the data storage method as described in the above first aspect and various possible designs of the first aspect.

[0181] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer readable storage medium is provided, and the computer readable storage medium stores computer execution instructions, when a processor executes the computer execution instructions, the data storage method as described in the above first aspect and various possible designs of the first aspect is implemented.

[0182] In a fifth aspect, the present disclosure provides a computer program product, including a computer program, when the computer program is executed by a processor, the data storage method as described in the above first aspect and various possible designs of the first aspect is implemented.

[0183] The above description merely illustrates the preferred embodiments of the disclosure and a principle for applying the technologies. It is understood by those skilled in the art that the disclosed scope of the disclosure is not limited to the technical solutions formed by the specific combinations of the technical features described above, and should also cover other technical solutions formed by the combinations of the technical features described above or their equivalent features without departing from the disclosed concept. For example, the technical solutions formed by the mutual replacement of the above-described features and the technical features with similar functions disclosed in the disclosure (but not limited to) can be formed.

[0184] Further, although operations are depicted in a particular, sequential order, this should not be understood as requiring or implying that the operations are performed in the order illustrated or sequentially. In certain circumstances, multitasking and parallel processing can be advantageous. Likewise, although specific implementation details are included for the purpose of providing a thorough disclosure, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0185] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A data storage method, characterized by, The method comprises: reading target data to be stored in the current cycle from the message queue, extracting a first data feature of the target data, the first data feature representing a data volume of the target data, and the first data feature comprising a dimension of data records in the target data; calling an adaptive consistency hash processing module to determine data shard information adapted to the target data according to the first data feature, and synchronizing the data shard information adapted to the target data to a configuration center, the data shard information being used to indicate at least one storage engine for storing the target data by sharding; the configuration center is used to record data shard information and refresh to a data writing component at regular intervals; calling the data writing component to write the target data in the corresponding storage engine based on the data shard information, and the storage engine supporting query processing of the target data.

2. The method of claim 1, wherein, The method further comprises: querying from the configuration center whether the target data is registered with a pre-aggregation rule; when it is found that the target data is registered with a pre-aggregation rule, performing pre-aggregation processing on the target data according to the pre-aggregation rule to obtain pre-aggregated data, the data dimension of the aggregated data being less than the data dimension of the target data; calling the data writing component to write the pre-aggregated data in the corresponding storage engine based on the data shard information, and the storage engine supporting query processing of the pre-aggregated data.

3. The method of claim 2, wherein, The target data is time series data, and the pre-aggregation processing of the target data according to the pre-aggregation rule to obtain pre-aggregated data comprises: obtaining a data generation timestamp corresponding to the time series data; performing pre-aggregation processing on the target data based on the data generation timestamp and the pre-aggregation rule to obtain pre-aggregated data, wherein the pre-aggregation rule is a rule for aggregating time series data based on a time dimension.

4. The method of claim 3, wherein, The method further comprises: receiving a first query instruction for the pre-aggregated data; obtaining the pre-aggregation rule corresponding to the pre-aggregated data from the configuration center based on the first query instruction; obtaining the pre-aggregated data from the storage engine according to the pre-aggregation rule corresponding to the pre-aggregated data.

5. The method of claim 1, wherein, The method further comprises: receiving a second query instruction for the target data; calling an adaptive consistency hash processing module to query the configuration center to obtain data shard information of the target data; calling a data query component to perform data query according to the data shard information of the target data to obtain the target data.

6. The method of claim 1, wherein, The calling of the adaptive consistency hash processing module to determine data shard information adapted to the target data according to the first data feature comprises: determining a target shard number according to the first data feature; calling the adaptive consistency hash processing module to process the target data to obtain a target storage engine corresponding to the target shard number; generating the data shard information based on an engine identifier of the target storage engine.

7. The method of claim 6, wherein, The calling of the adaptive consistency hash processing module to process the target data to obtain a target storage engine corresponding to the target shard number comprises: According to the target number of fragments, a first target data dimension of the target data is obtained; Based on a consistent hash algorithm, the data in the first target data dimension of the target data is processed to obtain the target storage engine.

8. The method of claim 7, wherein, According to the target number of fragments, a first target data dimension of the target data is obtained; When the target number of fragments is equal to 1, the first target data dimension is a first dimension, and the data in the first dimension represents the data category of the target data; When the target number of fragments is greater than 1, the first target data dimension is the first dimension and at least one second dimension, and the data in the second dimension represents a single indicator of the target data.

9. The method of claim 8, wherein, When the first target data dimension is the first dimension and at least one second dimension, the target storage engine is obtained by processing the data in the first target data dimension of the target data based on a consistent hash algorithm, comprising: Based on the data in the first dimension of the target data, a first storage engine is obtained; Based on the data in the second dimension of the target data, a second storage engine is obtained, wherein the first storage engine and the second storage engine belong to the same storage cluster, and the second storage engine is the adjacent downstream logical node or the adjacent downstream physical node of the first storage engine; According to the first storage engine and the second storage engine, the target storage engine is obtained.

10. The method of claim 7, wherein, The target data includes at least one data record, and the target storage engine is obtained by processing the data in the first target data dimension of the target data based on a consistent hash algorithm, comprising: Obtain the data of the data record in the first target data dimension; Call a consistent hash algorithm to calculate the data of the data record in the first target data dimension to obtain the target storage engine corresponding to the data record.

11. The method of claim 10, wherein, The call consistent hash algorithm calculates the data of the data record in the first target data dimension to obtain the target storage engine corresponding to the data record, comprising: Obtain the first data of the data record in the first dimension, and calculate the first data based on the target hash algorithm to obtain the first hash value; Obtain the second data of the data record in at least one second dimension, and calculate the second data of the at least one second dimension based on the target hash algorithm to obtain the second hash value; Respectively use the first hash value and the second hash value to perform modulo operation on the preset hash ring length to obtain the first hash ring position corresponding to the first hash value and the second hash ring position corresponding to the second hash value; According to the sum of the first hash ring position and the second hash ring position, the target storage engine corresponding to the data record is obtained.

12. The method of claim 1, wherein, The target data is time series data, and the first data feature of the target data is extracted, comprising: Based on a first interval duration, the generation rate of the time series data in the current period is detected; Based on the generation rate, the first data feature is obtained.

13. A data storage device, characterized by Comprising: An extraction unit is configured to read target data to be stored in the current period from a message queue, and extract a first data feature of the target data, the first data feature representing a data volume of the target data, and the first data feature including a dimension of data records in the target data. A processing unit is configured to invoke an adaptive consistent hashing processing module, determine data shard information adapted to the target data according to the first data feature, and synchronize the data shard information adapted to the target data to a configuration center, the data shard information being used to indicate at least one storage engine for storing the target data by sharding; the configuration center is configured to record the data shard information and refresh the data shard information to a data writing component at a timing. A storage unit is configured to invoke the data writing component, and write the target data in a corresponding storage engine based on the data shard information, the storage engine supporting query processing of the target data.

14. An electronic device, comprising: The method comprises: a processor, and a memory connected to the processor in communication; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method of any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, the data storage method of any one of claims 1 to 12 is implemented.

16. A computer program product, characterised in that, The computer program is executed by the processor to implement the data storage method of any one of claims 1 to 12.

Images