Time series data processing method, system, storage medium and electronic device

Abstract

The application discloses a time series data processing method and system, a storage medium and an electronic device. The method comprises the following steps: in response to a client request, obtaining time series data to be processed from a time series database, wherein the time series data is generated by a data source according to a time period and stored in the time series database; storing a sharding result corresponding to the time series data in a message queue, wherein the sharding result is used to enable the message queue to store a corresponding log file; and writing the sharding result read from the message queue into a memory. The application solves the technical problem of low data storage efficiency.

Description

Technical Field

[0001] This invention relates to the fields of distributed computing and data processing, and more specifically, to a time-series data processing method, system, storage medium, and electronic device. Background Technology

[0002] Currently, when storing time-series data, log storage and data storage are tightly coupled on the same data node. When it is necessary to horizontally scale the data node or increase the number of data replicas, the number of log replicas will also increase, resulting in low data storage efficiency.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides a time-series data processing method, system, storage medium, and electronic device to at least solve the technical problem of low data storage efficiency.

[0005] According to one aspect of the present invention, a time-series data processing method is provided, comprising: responding to a client request to obtain time-series data to be processed from a time-series database, wherein the time-series data is generated by a data source according to a time period and stored in the time-series database; storing the sharding results corresponding to the time-series data in a message queue, wherein the sharding results are used to enable the message queue to store corresponding log files; and writing the sharding results read from the message queue into memory.

[0006] Optionally, the method includes: performing sharding on the time-series data to obtain sharding results.

[0007] Optionally, the time-series data is segmented to obtain segmentation results, including: determining the first segmentation information matched by the timestamp of the time-series data, wherein the first segmentation information is used to characterize the mapping relationship between the first time period and the segments in the first segmentation information, and the first time period includes the timestamp of the time-series data; and segmenting the time-series data based on the first segmentation information to obtain segmentation results.

[0008] Optionally, the time series data is sharded based on the first sharding information to obtain sharding results, including: determining the target shard corresponding to the time series data based on the number of shards in the first sharding information and the key corresponding to the time series data, and obtaining sharding results.

[0009] Optionally, determining the first shard information matched by the timestamp of the time-series data, the method includes: determining the first shard information based on the number of storage nodes; wherein, writing the time-series data read from the message queue into memory includes: writing the time-series data read from the message queue into memory based on the storage nodes.

[0010] Optionally, the first shard information is determined based on the number of storage nodes. The method includes: creating at least one first shard in the message queue based on the first shard information; and creating a second shard on the storage nodes with the same number as the first shard in the message queue, wherein the second shard corresponds one-to-one with the first shard.

[0011] Optionally, the method further includes: adding a third shard to the message queue corresponding to the number of newly added storage nodes; determining the third shard information based on the number of third shards, the timestamps corresponding to the newly added storage nodes, and the number of first shards in the message queue, wherein the third shard information is used to characterize the mapping relationship between the third time period and the shard set obtained from the first shard and the third shard, and the third time period is associated with the timestamps corresponding to the newly added storage nodes.

[0012] Optionally, writing the sharding results read from the message queue to memory includes: reading the sharding results from the message queue on multiple storage nodes and writing the read sharding results to memory; or reading multiple sharding results under the same topic from the message queue on multiple storage nodes and writing the read multiple sharding results to memory.

[0013] Optionally, obtaining the time-series data to be processed includes: if the data proxy node in the first cluster is in a normal working state, then obtaining the time-series data to be processed based on the data proxy node in the first cluster; if the data proxy node in the first cluster is in an abnormal working state, then obtaining the time-series data to be processed based on the data proxy node in the second cluster.

[0014] Optionally, the method further includes: determining a target component based on a data subscription message, wherein the data subscription message is used to characterize the task to be subscribed to; and writing time-series data read from the message queue into the target component.

[0015] According to another aspect of the present invention, a time-series data processing apparatus is provided, comprising: a response unit, configured to respond to a client request and obtain time-series data to be processed from a time-series database, wherein the time-series data is generated by a data source according to a time period and stored in the time-series database; a storage unit, configured to store sharding results corresponding to the time-series data in a message queue, wherein the sharding results are used to enable the message queue to store corresponding log files; and a writing unit, configured to write the sharding results read from the message queue to memory.

[0016] According to another aspect of the present invention, a time-series data processing system is provided, comprising: a data proxy component, configured to respond to a client request and retrieve time-series data to be processed from a time-series database, wherein the time-series data is generated by a data source according to a time period and stored in the time-series database; a message queue component, configured to store sharding results corresponding to the time-series data in a message queue, wherein the sharding results are used to enable the message queue to store corresponding log files; and a data storage component, configured to write the sharding results read from the message queue into memory.

[0017] This invention also provides a computer-readable storage medium. The computer-readable storage medium includes a stored program, wherein, when the program is run by a processor, it controls the device where the computer-readable storage medium is located to perform the timing data processing method of this invention.

[0018] This invention also provides a processor. The processor is used to run a program, wherein the program executes the timing data processing method of this invention during runtime.

[0019] This invention also provides an electronic device, including: a processor; and a memory connected to the processor, configured to provide the processor with instructions to perform the following processing steps: responding to a client request, retrieving time-series data to be processed from a time-series database, wherein the time-series data is generated by a data source according to a time period and stored in the time-series database; storing the sharding results corresponding to the time-series data in a message queue, wherein the sharding results are used to enable the message queue to store corresponding log files; and writing the sharding results read from the message queue into memory.

[0020] In this embodiment of the invention, by using message queues to separate log and data storage, loose coupling between log storage and data storage is achieved. This effectively reduces the complexity of data replication (high availability) and horizontal scaling of storage nodes, thereby improving the efficiency of data storage and solving the technical problem of low data storage efficiency. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0022] Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a time-series data processing method according to an embodiment of the present invention.

[0023] Figure 2 This is a flowchart of a time-series data processing method according to an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of an LSM-Tree engine in the prior art;

[0025] Figure 4 This is a schematic diagram of a distributed cluster architecture in existing technology;

[0026] Figure 5 This is a schematic diagram of the logical structure of a TDengine distributed architecture in the prior art;

[0027] Figure 6 This is a schematic diagram of the write process of a Master Vnode in the existing technology;

[0028] Figure 7 This is a schematic diagram of the write process of a slave vnode in existing technology;

[0029] Figure 8 This is a schematic diagram of an overall architecture for loosely coupled time-series data storage based on a message queue, according to an embodiment of the present invention.

[0030] Figure 9 This is a schematic diagram of a time-slicing process according to an embodiment of the present invention;

[0031] Figure 10 This is a schematic diagram of a storage mapping according to an embodiment of the present invention;

[0032] Figure 11 This is a schematic diagram of a database creation process according to an embodiment of the present invention;

[0033] Figure 12 This is a schematic diagram illustrating the changes in a cluster before and after expanding a storage node according to an embodiment of the present invention;

[0034] Figure 13 This is a schematic diagram of a fragmented data replication according to an embodiment of the present invention;

[0035] Figure 14 This is a schematic diagram of database data replication according to an embodiment of the present invention;

[0036] Figure 15 This is a schematic diagram of cluster data replication according to an embodiment of the present invention;

[0037] Figure 16 This is a schematic diagram of a data subscription according to an embodiment of the present invention;

[0038] Figure 17 This is a schematic diagram of a time-series data processing apparatus according to an embodiment of the present invention;

[0039] Figure 18 This is a schematic diagram of a time-series data processing system according to an embodiment of the present invention;

[0040] Figure 19 This is a structural block diagram of a computer terminal according to an embodiment of the present invention. Detailed Implementation

[0041] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0042] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0043] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0044] Time-series databases provide a range of unique data access operations, including aggregation, downsampling, and interpolation, while also offering a data lifecycle (data timeliness) management mechanism.

[0045] Aggregation is a computational operation that performs grouping, summing, counting, and other calculations on data points on the same or different timelines.

[0046] Downsampling is used when the query time interval is long and the original data sampling frequency is high. Downsampling can reduce the precision of the data display in the query. For example, data collected by second can be sampled by hour during the query to reduce the number of data points in the result set.

[0047] Interpolation is used to automatically fill in missing data points in the middle of a timeline using interpolation algorithms.

[0048] Data Time To Live (TTL) refers to the validity period of data retention. Data that has exceeded its validity period will be automatically deleted. The default data validity period is permanent retention.

[0049] A metric is similar to a table in a relational database, representing a collection of similar time-series data. For example, a table can be created for air quality sensors to store the monitoring data of all sensors.

[0050] Tags are used to describe the characteristics of a data source and usually do not change over time. For example, a sensor device contains tag information such as device ID and region. The database will automatically create an index for tags and support multi-dimensional retrieval queries based on tags. A tag consists of a tag key and a tag value, both of which are strings.

[0051] A timestamp represents the point in time when data was generated. It can be specified when writing data or automatically generated by the system.

[0052] A field is used to describe the measurement metrics of a data source. These metrics typically change over time. For example, sensor devices may contain fields such as temperature and humidity. Field columns do not require a pre-created fixed schema and can be dynamically added or removed during operation. A field consists of a field key and a field value. The field key is of type String, and the field value supports types such as Boolean, Int, Double, String, and Byte.

[0053] A data point is a specific measurement value (field value) generated by a data source at a certain time. When querying or writing data, the number of data points is used as a statistical indicator.

[0054] A time series is a data source whose metrics change over time. A time series is defined by the combination of Metric, Tags, and Fields. Calculations for time series data, including downsampling, aggregation (sum, count, max, min, etc.), and interpolation, are all performed based on the time series dimension. When storing data, the database will cluster data from the same time series as much as possible to improve the efficiency of time series data access and better support time series data compression.

[0055] Example 1

[0056] According to an embodiment of the present invention, a method embodiment for processing time-series data is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0057] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a time-series data processing method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0058] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be wholly or partially embodied in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or wholly or partially integrated into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuit serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0059] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the timing data processing method in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned timing data processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0060] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0061] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0062] Under the aforementioned operating environment, this application provides the following: Figure 2 This illustrates a data processing method.

[0063] Figure 2 This is a flowchart of a time-series data processing method according to an embodiment of the present invention. Figure 2 As shown, the time-series data processing method may include the following steps:

[0064] Step S202: In response to the client request, retrieve the time series data to be processed from the time series database. The time series data is generated by the data source according to the time period and stored in the time series database.

[0065] In the technical solution provided by step S202 of the present invention, the time-series data can be newly written data by the user, and a series of unique data access operations can be provided, including aggregation, downsampling, interpolation, etc. At the same time, a data lifecycle (data timeliness) management mechanism is also provided. The characteristic of writing time-series data is that the writing volume is large.

[0066] In this embodiment, the data source can be a typical time-series data source such as the Internet of Things, application monitoring, and industrial Internet. Time-series data can be generated by the data source according to a time period and stored in a time-series database. For example, the data source usually generates time-series data continuously at a certain period. A piece of time-series data is described by elements such as metric, tags, timestamp, and field.

[0067] In this embodiment, a client request can be responded to, and time series data to be processed can be obtained from the time series database. For example, the client initiates a request, generates a signal representing the request, and responds to the signal to obtain the time series data to be processed from the time series database.

[0068] In this embodiment, client requests can be received through a data proxy (TSProxy).

[0069] Step S204: Store the sharding results corresponding to the time-series data in the message queue, wherein the sharding results are used to enable the message queue to store the corresponding log files.

[0070] In the technical solution provided by step S704 of the present invention, the sharding results corresponding to the time-series data can be stored in the message queue. For example, the data broker performs sharding according to the sharding strategy and writes the sharded results into the message queue.

[0071] In this embodiment, data sharding can be performed according to a data sharding strategy before storing the sharding results corresponding to time-series data in the message queue.

[0072] For example, message queues, as log systems, store data written by users and ensure data persistence. It should be noted that the message queues here refer to distributed messaging systems that support the "publish-subscribe" model, such as Kafka, RocketMQ, and Pulsar. Distributed messaging systems that support the "publish-subscribe" model generally support the creation of topics, and multiple partitions can be created under a topic.

[0073] It should be noted that the above guarantee of data persistence refers to the internal guarantee of data persistence by the message queue component. This is a basic function of message queues. For example, Kafka, RocketMQ, and Pulsar all write received messages to disk.

[0074] Step S206: Write the sharding results read from the message queue to memory.

[0075] In the technical solution provided by step S206 of the present invention, the sharding result read from the message queue can be written to memory. For example, messages are first pulled from the message queue and written to memory. After the data in memory reaches a certain threshold, the data in memory is flushed to the disk and submitted to the message queue.

[0076] It should be noted that the above messages refer to time-series data points, which include metrics, tags, timestamps, fields, etc. The maximum message drift mentioned above can be calculated by the message queue returning an offset to the client after the client submits a message. The client can then use this offset to perform precise operations on the message, such as reading or deleting, according to the message queue component. In addition, the offset can be monotonically increasing, so the maximum offset can be calculated. After the client consumes a message, it needs to submit the offset of the consumed message to the message queue. The message queue will mark messages with offsets less than this offset as consumed, and will not consume the same message again on subsequent consumption attempts.

[0077] Through steps S202 to S206 of this application, in response to client requests, time-series data to be processed is obtained from the time-series database. This time-series data is generated by the data source according to a time period and stored in the time-series database. The sharding results corresponding to the time-series data are stored in a message queue, where the sharding results are used to store the corresponding log files in the message queue. The sharding results read from the message queue are written to memory. In other words, this application uses a message queue to separate log and data storage, achieving loose coupling between log storage and data storage. This effectively reduces the complexity of data replication (high availability) and horizontal scaling implementation of storage nodes, solving the technical problem of low data storage efficiency and achieving the technical effect of improving data storage efficiency.

[0078] The method described in this embodiment will be further described below.

[0079] As an optional implementation, the method further includes: performing sharding processing on the time-series data to obtain sharding results.

[0080] In this embodiment, time-series data can be sharded to obtain sharding results. For example, data sharding can be performed according to a data sharding strategy. For instance, for the cloud-native database service data proxy library (Lindorm TSDB), the overall sharding is hash sharding. Sharding strategies can include: metric hash sharding, tag hash sharding, metric and tag hash sharding, etc.

[0081] In this embodiment, the time-series data can be sharded by first performing time sharding and then hash sharding. Time sharding can be achieved by performing the following steps: determining the first shard information matched by the timestamp of the time-series data, wherein the first shard information is used to characterize the mapping relationship between the first time period and the shards in the first shard information, and the first time period includes the timestamp of the time-series data; and performing sharding on the time-series data based on the first shard information to obtain sharding results. Hash sharding can be achieved by performing the following steps: determining the target shard corresponding to the time-series data based on the number of shards in the first shard information and the key corresponding to the time-series data to obtain sharding results.

[0082] As an optional implementation, the time-series data is segmented to obtain segmentation results, including: determining the first segmentation information matched by the timestamp of the time-series data, wherein the first segmentation information is used to characterize the mapping relationship between the first time period and the segments in the first segmentation information, and the first time period includes the timestamp of the time-series data; and performing segmentation processing on the time-series data based on the first segmentation information to obtain segmentation results.

[0083] In this embodiment, the timestamp can be the timestamp of data newly written by the user, and the first shard information can be a routing table corresponding to different time periods.

[0084] In this embodiment, compared to other types of data records, each piece of time-series data carries a timestamp, which can increase over time.

[0085] In this embodiment, time sharding can be a routing table that matches the timestamp of the time series data to different time periods, and then the time series data is sharded based on the routing table to obtain the sharding result.

[0086] For example, suppose the cluster initially has 3 shards. During system initialization, a routing table is created: (0-max) = {P1, P2, P3}. When a node is added, a shard P4 is added, and a new routing table is created: (t1-max) = {P1, P2, P3, P4}. The original routing table is modified to (0-t1) = {P1, P2, P3}, where t1 is the timestamp when the system expands. After expansion, the system has two routing tables: (0-t1) and (t1, max). Assuming the timestamp of newly written data is timestamp, if 0 ≤ timestamp < t1, the (0-t1) routing table is used to write data; otherwise, the (t1, max) routing table is used. When the system needs to expand by another node, and the timestamp of the expansion is t2, the system's routing table splits into 3, as follows:

[0087] (0-t1)=P1, P2, P3, (t1, t2)=P1, P2, P3, P4, (t2, max)=P1, P2, P3, P4, P5

[0088] It should be noted that, since the timestamps of records written to time-series data usually increase over time, the advantage of using time sharding is that when the system is expanded, there is no need to migrate the data; it can simply remain on the original nodes.

[0089] As an optional implementation, the time-series data is segmented based on the first segmentation information to obtain the segmentation result, including: determining the target segment corresponding to the time-series data based on the number of segments in the first segmentation information and the key corresponding to the time-series data, and obtaining the segmentation result.

[0090] In this embodiment, the key corresponding to the time-series data can be the key of the data point.

[0091] In this embodiment, the target shard corresponding to the time-series data can be determined based on the number of shards in the first sharding information and the key corresponding to the time-series data, thus obtaining the sharding result. For example, in hash sharding, when writing data, the corresponding routing table is first found based on the timestamp of the data, and then the hash value (hashCode) of the key of the data point is calculated to finally determine the shard where the data is located.

[0092] As an optional implementation, the time-series data is segmented based on the first segmentation information to obtain the segmentation result. The method further includes: generating second segmentation information based on the newly added segment, the timestamp corresponding to the newly added segment, and the first segmentation information. The second segmentation information is used to characterize the mapping relationship between the second time period and the segmentation set obtained by the segmentation in the first segmentation information and the newly added segment. The second time period is associated with the timestamp corresponding to the newly added segment.

[0093] In this embodiment, the newly added shard can be a shard added when expanding a node, and the timestamp corresponding to the newly added shard can be the timestamp during the expansion.

[0094] In this embodiment, second fragment information can be generated based on the fragments added when expanding the node, the timestamp during expansion, and the routing table corresponding to different time periods.

[0095] For example, suppose the cluster initially has 3 shards. During system initialization, a routing table is created: (0-max) = {P1, P2, P3}. When a node is added, a shard P4 is added, and a new routing table is created: (t1-max) = {P1, P2, P3, P4}. At the same time, the original routing table is modified to (0-t1) = {P1, P2, P3}, where t1 is the timestamp when the system is expanded.

[0096] As an optional implementation, the method for determining the first shard information matched by the timestamp of the time-series data includes: determining the first shard information based on the number of storage nodes; wherein, writing the time-series data read from the message queue into memory includes: writing the time-series data read from the message queue into memory based on the storage nodes.

[0097] In this embodiment, the first shard information can be determined based on the number of storage nodes. For example, a routing table can be created based on user parameters and the number of storage nodes. For the cloud-native database service data proxy library (Lindorm TSDB), user parameters may include database name, data retention period, sharding strategy, pre-deprecision rules, etc.

[0098] In this embodiment, time-series data read from the message queue can be written to memory based on the storage nodes. For example, an array (routeTable) of size equal to the number of nodes can be created, and the contents of the array are the Internet Protocol (IP) addresses of the nodes, thereby writing the read time-series data to memory.

[0099] As an optional implementation, the method involves writing time-series data read from the message queue into memory based on the storage node. The method includes: creating at least a first shard in the message queue based on the first shard information; and creating a second shard on the storage node with the same number of shards as the first shards in the message queue, wherein the second shard corresponds one-to-one with the first shard.

[0100] In this embodiment, at least a first shard can be created in the message queue based on the first shard information, for example, by creating a topic and shard in the message queue according to the routing table.

[0101] In this embodiment, a second shard with the same number of shards as the first shard in the message queue can be created on the storage node. For example, a producer of the message queue can be created on the data broker node, and shards with the same number of shards as the message queue can be created on the data storage node. A consumer can then be created to consume the shard data of the corresponding message queue.

[0102] It should be noted that in this application, the data proxy node (TSProxy) is the message producer of the message queue, i.e., writing messages, and the data storage node (TSCore) is the message consumer of the message queue, i.e. reading messages.

[0103] In this embodiment, the second shard corresponds one-to-one with the first shard. For example, the shards of the storage node are mapped one-to-one with the shards under the topics in the message queue.

[0104] As an optional implementation, the method further includes: adding a third shard to the message queue corresponding to the number of newly added storage nodes; determining the third shard information based on the number of third shards, the timestamps corresponding to the newly added storage nodes, and the number of first shards in the message queue, wherein the third shard information is used to characterize the mapping relationship between the third time period and the shard set obtained from the first shard and the third shard, and the third time period is associated with the timestamps corresponding to the newly added storage nodes.

[0105] In this embodiment, a third shard corresponding to the number of newly added storage nodes can be added to the message queue. For example, the storage nodes can be expanded according to the expansion conditions, and the number of topic shards corresponding to the storage nodes can be added under the corresponding topic in the message queue.

[0106] It should be noted that the above expansion conditions refer to the number of nodes to be expanded. For example, if the original number of storage nodes is 3, the expansion condition is to add 2 storage nodes, and the total number of storage nodes will be 5.

[0107] In this embodiment, the third shard can be a topic shard corresponding to the newly added and storage nodes.

[0108] In this embodiment, the information of the third shard can be determined based on the number of third shards, the timestamp corresponding to the newly added storage node, and the number of first shards in the message queue. For example, a new routing table can be created based on the change in the number of shards in the message queue and the timestamp during expansion, and the original routing table can be modified.

[0109] For example, when expanding a storage node, two shards need to be added under the topic of the message queue when a storage node is added. After the expansion, the mapping relationship between the shards in the original message queue and the shards in the storage node does not change. Therefore, there is no need to migrate or rebalance the data during the expansion, and it will not affect the original data writing and querying process.

[0110] As an optional implementation, step S206 involves writing the sharding results read from the message queue to memory. This method includes: reading the sharding results from the message queue in multiple storage nodes and writing the read sharding results to memory; or reading multiple sharding results under the same topic from the message queue in multiple storage nodes and writing the read multiple sharding results to memory.

[0111] In this embodiment, in a distributed cluster, each storage node is generally only responsible for storing a portion of the data in the entire cluster. When some nodes fail, some data in the system will become inaccessible and unreadable. Therefore, in order to ensure high availability of data, data is usually replicated to other nodes via the network to create multiple copies of the same data.

[0112] In this embodiment, sharding results can be read from message queues on multiple storage nodes and written to memory. For example, sharded data replication: for a certain shard of data, multiple copies of the same data shard are formed on storage nodes by consuming shards in the same message queue on multiple storage nodes. High availability of data is ensured by consuming shards in the same message queue on multiple storage nodes.

[0113] In this embodiment, multiple shard results under the same topic can be read from the message queue in multiple storage nodes, and the read multiple shard results can be written to memory. For example, database data replication: the data of the entire database (Data Basic, abbreviated as DB) is replicated to generate multiple replicas in the cluster. That is, all shards under the topic in the message queue are consumed by multiple consumers.

[0114] As an optional implementation, step S202, obtaining time-series data to be processed, includes: if the data proxy node in the first cluster is in a normal working state, then obtaining the time-series data to be processed based on the data proxy node in the first cluster; if the data proxy node in the first cluster is in an abnormal working state, then obtaining the time-series data to be processed based on the data proxy node in the second cluster.

[0115] In this embodiment, the first cluster can be the main cluster (cluster1), and the abnormal state can be that the data proxy point of the main cluster (cluster1) is in a state of being down or unavailable. The second cluster can be a cluster different from the main cluster (cluster2).

[0116] For example, cluster data replication refers to copying data to multiple different clusters to form a primary and secondary cluster, thereby further improving data availability. Figure 15 ,in, Figure 15 This is a schematic diagram of cluster data replication according to an embodiment of the present invention. Initially, cluster1 is the master cluster, and the data proxy (TSProxy) in cluster1 is responsible for writing data to the message queue. The data storage nodes (TSCore) in both cluster1 and cluster2 consume data from the message queue. When cluster1 becomes unavailable due to failure or other reasons, the data proxy (TSProxy) of cluster2 is responsible for writing data to the message queue.

[0117] As an optional implementation, the method further includes: determining a target component based on a data subscription message, wherein the data subscription message is used to characterize the task to be subscribed to; and writing time-series data read from the message queue into the target component.

[0118] In this embodiment, the target component can be determined based on the data subscription message, and the time-series data read from the message queue can be written to the target component. For example, the message queue can be used as a log system. Since the message queue naturally supports the "publish-subscribe" model, for tasks that require data subscription, such as stream computing and anomaly detection, data subscription can be achieved directly by adding consumers to the message queue.

[0119] In this embodiment, data subscription is essentially adding a consumer to the message queue to read messages from the message queue. For example, if only TSCore was consuming data before, one or more consumers can be added to read the data from the message queue into other components, such as distributed streaming data stream engines (Apache Flink).

[0120] In this embodiment, the data subscription task can be a stream computing task, anomaly detection task, or similar tasks.

[0121] In this embodiment of the invention, time-series data is sharded to obtain sharding results; sharding results are read from message queues in multiple storage nodes and written to memory; multiple sharding results under the same topic are read from message queues in multiple storage nodes and written to memory; if the data proxy node in the first cluster is in normal working condition, the time-series data to be processed is obtained based on the data proxy node in the first cluster; if the data proxy node in the first cluster is in abnormal working condition, the time-series data to be processed is obtained based on the data proxy node in the second cluster; a target component is determined based on a data subscription message, wherein the data subscription message is used to characterize the task to be subscribed to; the time-series data read from the message queue is written to the target component, thus solving the technical problem of low data storage efficiency and achieving the technical effect of improving data storage efficiency.

[0122] Example 2

[0123] The preferred embodiments of the method described above in this example will be further described below.

[0124] In existing technologies, in typical time-series scenarios such as the Internet of Things, application monitoring, and the Industrial Internet, the data source (indicating who generates the data) typically generates time-series data continuously at certain periods. A piece of time-series data is described by elements such as metrics, tags, timestamps, and fields. The characteristics of writing time-series data are continuous high-concurrency and high-throughput writing, with more writes than reads. It is very suitable to use the log-structured merged-tree (LSM-Tree) structure to write data. The LSM-Tree is a hierarchical, ordered, disk-oriented data structure that fully utilizes the characteristic that sequential writes in batches on the disk are much more efficient than random writes.

[0125] Figure 3 This is a schematic diagram of an existing LSM-Tree engine. (Example) Figure 3As shown, the LSM-Tree engine comprises four parts: Write-Ahead Log (WAL), Memtable, File, and Compaction. The WAL is a crucial component of the LSM-Tree engine's data writing process, ensuring data persistence and atomicity of updates. Furthermore, the write throughput of the WAL largely determines the write throughput of the LSM-Tree.

[0126] To ensure high availability and scalability of the system, we typically build a distributed cluster where each machine node stores data shards and replicas.

[0127] Figure 4 This is a schematic diagram of a distributed cluster architecture in existing technology. For example... Figure 4 As shown, there are 4 nodes in total, with 4 data shards: P1, P2, P3, and P4. Each shard has 3 replicas, so each node actually stores data from 3 shards. Each node uses an LSM-Tree engine and has its own independent local Write-Ahead Log (WAL). Data replication is achieved through distributed consistency protocols such as the Raft management log algorithm, which copies the WAL.

[0128] However, based on this distributed architecture implementation, the following problems exist:

[0129] First, data subscription is difficult, specifically because the log WAL is distributed across multiple nodes without a unified log storage, which makes it difficult to implement or replicate data based on the log WAL and increases the cost of operation and maintenance.

[0130] Secondly, the data is tightly coupled, specifically because the log WAL is coupled with the data on the same node. When the number of data replicas is increased, the number of log replicas will also increase, leading to higher storage costs.

[0131] Finally, data redistribution manifests itself in the following way: when the system is expanded or shrunk, the change in data storage nodes will cause a change in the data sharding method, which usually leads to the migration of data between nodes.

[0132] Figure 5 This is a schematic diagram of the logical structure of a TDengine distributed architecture in existing technology. For example... Figure 5As shown, a complete temporal spatial big data engine (TDengine) system runs on one or more physical nodes. Logically, it includes data nodes (dnodes), TDengine application drivers (taosc), and applications (APPs). The system contains one or more data nodes, which form a cluster. Applications interact with the TDengine cluster through the taosc application programming interface (API).

[0133] Figure 6 This is a schematic diagram of the write process of a Master Vnode in existing technology. For example... Figure 6 As shown, the process includes the following steps:

[0134] In step S61, the virtual node manager (Master vnode) receives the application's data insertion request. If the verification is successful, it proceeds to the next step.

[0135] In step S62, if the system configuration parameter (walLevel) is greater than 0, the virtual node (vnode) will write the original data packet of the request to the database log file (WAL). If walLevel is set to 2 and network file synchronization (fsync) is set to 0, TDengine will also immediately write the WAL data to disk to ensure that even if there is a system crash, the data can be recovered from the database log file, avoiding data loss.

[0136] In step S63, if there are multiple replicas, the vnode will forward the data packet to the slave vnodes within the same virtual node group. The forwarded packet carries the version number of the data.

[0137] Step S64: Write to memory and add the record to the skip list;

[0138] In step S65, the Master vnode returns an acknowledgment message to the application, indicating that the write was successful;

[0139] It should be noted that if any step two, three, or four fails, an error will be returned directly to the application.

[0140] Figure 7 This is a schematic diagram of the write process of a slave vnode in existing technology. For example... Figure 7 As shown, the process includes the following steps:

[0141] Step S71: The Slave vnode receives the data insertion request forwarded by the Master vnode;

[0142] In step S72, if the system configuration parameter (walLevel) is greater than 0, the vnode will write the original data packet of the request to the database log file WAL. If walLevel is set to 2 and fsync is set to 0, TDengine will also immediately write the WAL data to disk to ensure that even if there is a crash, the data can be recovered from the database log file and avoid data loss.

[0143] Step S73: Write to memory and update the skip list in memory.

[0144] In TDEngine's architecture, logs and data are coupled together. This distributed architecture presents the following problems:

[0145] (1) Difficulty in data subscription: The log WAL is distributed across multiple nodes without a unified log storage, which makes it difficult to implement data subscription or replication based on the log WAL and increases the operation and maintenance costs.

[0146] (2) Tight data coupling: The log WAL is coupled with the data on the same node. When the node needs to be horizontally scaled up or the number of data replicas needs to be increased, the number of log replicas will also increase, resulting in higher storage costs.

[0147] (3) Data redistribution: When the system is expanded or reduced, the data storage nodes will change, which will cause the data sharding method to change. Therefore, data will usually be migrated before the nodes.

[0148] To address the problems in the prior art, this application proposes a loosely coupled storage scheme for time-series data based on message queues.

[0149] This application proposes a loosely coupled time-series data storage scheme based on message queues, mainly comprising four parts: overall architecture, data sharding, data replication, and data subscription. Based on the principle of "logs as data," this application uses message queues as the log system, which can effectively support functions such as data subscription, while effectively solving the problems of high availability and elastic scaling in time-series data storage.

[0150] The overall architecture of this embodiment is described below:

[0151] Figure 8 This is a schematic diagram of an overall architecture for loosely coupled time-series data storage based on a message queue, according to an embodiment of the present invention. Figure 8 As shown, the architecture includes a data proxy (TSProxy) component, a message queue (Message Queue) component, and a data storage (TSCore) component.

[0152] The data proxy (TSProxy) component is responsible for receiving client requests, splitting the data according to the data splitting strategy, and writing the splitting results into the message queue.

[0153] The message queue component, acting as a logging system, stores user-written data, ensuring data persistence. It's important to note that the message queue here refers to a distributed messaging system that supports a publish-subscribe model, such as Kafka, RocketMQ, and Pulsar. Distributed messaging systems supporting the publish-subscribe model typically support creating topics, and multiple partitions can be created under a topic. For example... Figure 8 The topic Topic1 contains four sub-sections: P1, P2, P3, and P4.

[0154] The data storage (TSCore) component is responsible for storing data. It first pulls messages from the message queue and writes them into memory. Once the data in memory reaches a certain threshold, it flushes the data in memory to the disk and submits the maximum message offset of the flushed data to the message queue.

[0155] The data proxy (TSProxy), message queue, and data storage (TSCore) are connected via a network and are independent of each other, so each can be horizontally scaled.

[0156] The data sharding of this embodiment will be described below.

[0157] Database systems typically support the creation and use of multiple databases, and the solution presented in this paper is no exception. Therefore, the data sharding mentioned in this paper is performed on a single database. Multiple databases can use the same or different sharding strategies, meaning different databases can generate different routing tables. In this application, data sharding can first use time sharding and then hash sharding.

[0158] The time slicing of this embodiment will be described below.

[0159] Compared to other types of data records, each piece of time-series data has a timestamp, and generally, this timestamp increases over time.

[0160] Figure 9 This is a schematic diagram of a time-slicing process according to an embodiment of the present invention. Figure 9As shown, assuming the cluster initially has three shards, a routing table is created during system initialization: (0-max) = {P1, P2, P3}. When a node is added, a shard P4 is added, and a new routing table is created: (t1-max) = {P1, P2, P3, P4}. At the same time, the original routing table is modified to (0-t1) = {P1, P2, P3}, where t1 is the timestamp when the system expands.

[0161] After the system is expanded, there will be two routing tables: (0-t1) and (t1,max). Assuming that the timestamp of the data newly written by the user is timestamp, when 0≤timestamp<t1, the (0-t1) routing table is used to write the data; otherwise, the (t1,max) routing table is used to write the data.

[0162] When the system needs to expand by one node, and the timestamp of the expansion is t2, the system's routing table will split into three, as follows:

[0163] (0-t1)=P1, P2, P3, (t1, t2)=P1, P2, P3, P4, (t2, max)=P1, P2, P3, P4, P5

[0164] Because the timestamps of records written to time-series data usually increase over time, the advantage of using time sharding is that when the system expands, there is no need to migrate the data; it can simply remain on the original nodes.

[0165] The hash sharding of this embodiment will be described below.

[0166] When writing data, the corresponding routing table is first looked up based on the data's timestamp. Then, the hash value (hashCode) of the data point's key is calculated. Finally, the shard where the data belongs can be determined using the following formula:

[0167] partitionIdx=hasj(key)%numPartitions

[0168] Where `hash` is a hash function, such as the `murmur` hash function, `key` is the key of the data point, and `numPatititions` is the number of shards in the routing table matched based on the timestamp of the data point written. For example, the following routing table has 3 shards:

[0169] (0-t1) = {P1, P2, P3}

[0170] The storage mapping of this embodiment will be described below.

[0171] Storage mapping refers to the mapping of shards in the routing table to topics and partitions in the message queue, and the mapping of storage nodes (TSCore) to topics and partitions in the consumer message queue.

[0172] Figure 10 This is a schematic diagram of a storage mapping according to an embodiment of the present invention. Figure 10 As shown, databases (DB) are mapped one-to-one to message queue topics, and database partitions are mapped one-to-one to message queue partitions; storage node (TSCore) buckets are mapped one-to-one to message queue partitions, as shown in... Figure 10 In the example, there are two databases, each with four shards. These four shards are mapped to four shards in a messaging system, and each of the four shards in the messaging system is bound to a shard on a storage node.

[0173] Figure 11 This is a schematic diagram illustrating a process for creating a database according to an embodiment of the present invention. Figure 11 As shown, the process may include the following steps:

[0174] Step S1101: Create a routing table based on user parameters and the number of storage nodes;

[0175] Step S1102: Create topics and shards in the message queue according to the routing table;

[0176] Step S1103: Create a message queue producer on the data broker node;

[0177] Step S1104: Create a shard on the data storage node with the same number of shards as the message queue, and create a consumer to consume the shard data of the corresponding message queue.

[0178] In this application, node expansion is achieved through the following steps:

[0179] Step 1: Expand the corresponding storage nodes according to the expansion conditions;

[0180] Step 2: Add the number of topic shards corresponding to the storage node under the corresponding topic in the message queue;

[0181] Step 3: Based on the changes in the number of shards in the message queue and the timestamp during expansion, create a new routing table and modify the original routing table.

[0182] Figure 12 This is a schematic diagram illustrating the changes in a cluster before and after expanding a storage node according to an embodiment of the present invention. Figure 12As shown, each storage node has two shards. Therefore, when expanding a storage node, two new shards need to be added under the message queue topic. It's also worth noting that after expansion, the mapping relationship between the original shards (P1-P4) in the message queue and the shards in the storage node remains unchanged. Therefore, expansion does not require data migration or rebalancing, and it will not affect the original data writing and querying processes.

[0183] The data copying of this embodiment will be described below.

[0184] In a distributed cluster, each storage node is generally only responsible for storing a portion of the data in the entire cluster. When some nodes fail, some data in the system will become inaccessible and unreadable. Therefore, in order to ensure high availability of data, data is usually replicated to other nodes over the network, creating multiple copies of the same data.

[0185] Using a message queue as the logging system, the message queue itself generally has a multi-replica mechanism to ensure high data availability. Therefore, the data replication here is the generation of replicas of the storage node (TSCore).

[0186] In this application, the shards of the storage node are mapped one-to-one with the shards under the topics in the message queue. When consuming messages, a single shard of the specified consumption queue can be consumed directly, and the database is mapped to the topic of the message queue. At the same time, the message queue naturally supports multiple consumers to consume the same data independently at the same time. Therefore, data replication at three granularities can be achieved: shard data replication, database data replication, and cluster data replication.

[0187] The following describes the fragmented data replication of this embodiment.

[0188] Sharded data replication refers to creating multiple copies of a single data shard by consuming shards from the same message queue on multiple storage nodes.

[0189] Figure 13 This is a schematic diagram of fragmented data replication according to an embodiment of the present invention. Figure 13 As shown, the data in shard P2 in the message queue is consumed simultaneously by three storage nodes, thus generating three copies of the same data.

[0190] The following describes the database data copying process in this embodiment.

[0191] Figure 14 This is a schematic diagram illustrating database data replication according to an embodiment of the present invention. Figure 14As shown, database data replication refers to copying the entire database data and generating multiple replicas in the cluster. In this application, all shards under the topic in the message queue are consumed by multiple consumers.

[0192] The following describes the cluster data replication in this embodiment.

[0193] Cluster data replication refers to copying data to multiple different clusters to form a primary and backup cluster, thereby further improving data availability.

[0194] Figure 15 This is a schematic diagram of cluster data replication according to an embodiment of the present invention. Figure 15 As shown, initially, cluster1 is the master cluster, and the data proxy (TSProxy) in cluster1 is responsible for writing data to the message queue. The data storage nodes (TSCore) in both cluster1 and cluster2 consume data from the message queue. When cluster1 fails or becomes unavailable, the data proxy (TSProxy) in cluster2 is responsible for writing data to the message queue.

[0195] The data subscription method for this embodiment will be described below.

[0196] Figure 16 This is a schematic diagram of a data subscription according to an embodiment of the present invention. Figure 16 As shown, a message queue is used as the log system. Since message queues naturally support the "publish-subscribe" model, for tasks that require data subscription, such as stream computing and anomaly detection, data subscription can be achieved directly by adding consumers to the message queue.

[0197] In the above embodiments of the present invention, a distributed message queue is used as log storage, providing a unified data subscription interface. The "publish-subscribe" model based on the message queue can effectively support data subscription requirements. The message queue separates log and data storage, and the two-layer sharding method of time sharding and hash sharding is used to achieve effective support for data subscription requirements. The message queue and data storage nodes can be scaled horizontally independently without affecting each other. The loose coupling between log and data storage solves the problem of not needing data migration or redistribution operations after cluster expansion, thereby solving the technical problem of low data storage efficiency and achieving the technical effect of improving data storage efficiency.

[0198] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0199] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0200] Example 3

[0201] According to embodiments of the present invention, a method for implementing the above is also provided. Figure 2 The time series data processing device of the time series data processing method shown.

[0202] Figure 17 This is a schematic diagram of a time-series data processing apparatus according to an embodiment of the present invention. Figure 17 As shown, the timing data processing device 170 may include a response unit 1701, a storage unit 1702, and a writing unit 1703.

[0203] The response unit 1701 is used to respond to client requests and obtain time series data to be processed from the time series database. The time series data is generated by the data source according to the time period and stored in the time series database.

[0204] Storage unit 1702 is used to store the sharding results corresponding to time-series data in the message queue, wherein the sharding results are used to enable the message queue to store the corresponding log files;

[0205] Write unit 1703 is used to write the sharding results read from the message queue to memory.

[0206] Optionally, the system further includes a sharding unit, wherein the sharding unit may include a determination module and a sharding module. The sharding unit is used to shard the time-series data to obtain sharding results; the determination module is used to determine the first sharding information matched to the timestamp of the time-series data, wherein the first sharding information is used to characterize the mapping relationship between a first time period and the shards in the first sharding information, and the first time period includes the timestamp of the time-series data; the sharding module is used to shard the time-series data based on the first sharding information to obtain sharding results.

[0207] The sharding module performs sharding processing on time-series data based on the first sharding information to obtain sharding results through the following steps: Based on the number of shards in the first sharding information and the key corresponding to the time-series data, the target shard corresponding to the time-series data is determined, and the sharding results are obtained.

[0208] The device is also used to perform the following steps: generating second fragment information based on the newly added fragment, the timestamp corresponding to the newly added fragment, and the first fragment information, wherein the second fragment information is used to characterize the mapping relationship between the second time period and the fragment set obtained from the fragments in the first fragment information and the newly added fragment, and the second time period is associated with the timestamp corresponding to the newly added fragment.

[0209] The device is also used to perform the following steps: determining first shard information based on the number of storage nodes; wherein writing time-series data read from the message queue into memory includes: writing time-series data read from the message queue into memory based on the storage nodes.

[0210] The device is also used to perform the following steps: creating at least a first shard in the message queue based on the first shard information; creating a second shard on the storage node with the same number of shards as the first shards in the message queue, wherein the second shard corresponds one-to-one with the first shard.

[0211] The device is also used to perform the following steps: adding a third shard to the message queue corresponding to the number of newly added storage nodes; determining the third shard information based on the number of third shards, the timestamp corresponding to the newly added storage nodes, and the number of first shards in the message queue, wherein the third shard information is used to characterize the mapping relationship between the third time period and the shard set obtained from the first shard and the third shard, and the third time period is associated with the timestamp corresponding to the newly added storage nodes.

[0212] Optionally, the write unit 1703 includes: a first write module and a second write module. The first write module is used to read sharding results from message queues in multiple storage nodes and write the read sharding results to memory; the second write module is used to read multiple sharding results under the same topic from message queues in multiple storage nodes and write the read multiple sharding results to memory.

[0213] Optionally, the response unit 1701 includes an acquisition module. The acquisition module is configured to acquire time-series data to be processed by performing the following steps: if the data proxy node in the first cluster is in a normal working state, then acquire the time-series data to be processed based on the data proxy node in the first cluster; if the data proxy node in the first cluster is in an abnormal working state, then acquire the time-series data to be processed based on the data proxy node in the second cluster.

[0214] Optionally, the device further includes a determining unit and a third writing module. The determining unit is used to determine a target component based on a data subscription message, wherein the data subscription message characterizes a task to be subscribed to. The third writing unit is used to write time-series data read from the message queue into the target component.

[0215] In this embodiment of the invention, a response unit 1701 is used to respond to a client request and obtain time-series data to be processed from a time-series database. The time-series data is generated by a data source according to a time period and stored in the time-series database. A storage unit 1702 is used to store the sharding results corresponding to the time-series data in a message queue. The sharding results are used to enable the message queue to store the corresponding log files. A writing unit 1703 is used to write the sharding results read from the message queue to memory, thereby improving the efficiency of data storage and solving the technical problem of low data storage efficiency.

[0216] It should be noted that the response unit 1701, storage unit 1702, and writing unit 1703 mentioned above correspond to steps S202 to S206 in Embodiment 1. The three units and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0217] Example 4

[0218] According to embodiments of the present invention, a method for implementing the above is also provided. Figure 2 The time series data processing system shown is a time series data processing method.

[0219] Figure 18 This is a schematic diagram of a time-series data processing system according to an embodiment of the present invention. Figure 18 As shown, the time-series data processing system 180 may include: a data broker component 1801, a message queue component 1802, and a data storage component 1803.

[0220] Data proxy component 1801 is used to respond to client requests and obtain time series data to be processed from the time series database. The time series data is generated by the data source according to the time period and stored in the time series database.

[0221] The message queue component 1802 is used to store the sharding results corresponding to time-series data in the message queue, wherein the sharding results are used to enable the message queue to store the corresponding log files;

[0222] Data storage component 1803 is used to write the sharded results read from the message queue to memory.

[0223] In this embodiment of the invention, the data proxy component 1801 responds to client requests and retrieves time-series data to be processed from the time-series database. The time-series data is generated by the data source according to a time period and stored in the time-series database. The message queue component 1802 stores the sharding results corresponding to the time-series data in the message queue. The sharding results are used to enable the message queue to store the corresponding log files. The data storage component 1803 writes the sharding results read from the message queue into memory, thereby improving the efficiency of data storage and solving the technical problem of low data storage efficiency.

[0224] It should be noted that the data proxy component 1801, message queue component 1802, and data storage component 1803 mentioned above correspond to steps S202 to S206 in Embodiment 1. The three components and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the device, can run on the computer terminal 10 provided in Embodiment 1.

[0225] Example 5

[0226] Embodiments of the present invention may provide an electronic device, which may include: a processor; and a memory connected to the processor, for providing the processor with instructions to perform the following processing steps: responding to a client request, retrieving time-series data to be processed from a time-series database, wherein the time-series data is generated by a data source according to a time period and stored in the time-series database; storing the sharding results corresponding to the time-series data in a message queue, wherein the sharding results are used to enable the message queue to store the corresponding log files; and writing the sharding results read from the message queue into memory.

[0227] Optionally, the electronic device in this embodiment may include a computer terminal, which may be any one of the computer terminal devices in a group of computer terminals. Optionally, in this embodiment, the computer terminal may also be replaced by a mobile terminal or other terminal device.

[0228] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0229] In this embodiment, the computer terminal described above can execute the program code for the following steps in the time-series data processing method of this invention: responding to a client request, obtaining time-series data to be processed from a time-series database, wherein the time-series data is generated by a data source according to a time period and stored in the time-series database; storing the sharding results corresponding to the time-series data in a message queue, wherein the sharding results are used to enable the message queue to store the corresponding log files; and writing the sharding results read from the message queue into memory.

[0230] Optionally, Figure 19 This is a structural block diagram of a computer terminal according to an embodiment of the present invention. Figure 19 As shown, the computer terminal A may include one or more (only one is shown in the figure) processors 1902, memory 1904, and transmission devices 1906.

[0231] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the timing data processing method and apparatus in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned timing data processing method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0232] The processor can invoke information and applications stored in memory via a transmission device to perform the following steps: responding to a client request, retrieving time-series data to be processed from a time-series database, wherein the time-series data is generated by a data source according to a time period and stored in the time-series database; storing the sharding results corresponding to the time-series data in a message queue, wherein the sharding results are used to enable the message queue to store the corresponding log files; and writing the sharding results read from the message queue into memory.

[0233] Optionally, the processor may also execute program code that performs the following steps: slices the timing data to obtain the sliced ​​results.

[0234] Optionally, the processor may also execute program code for the following steps: determining the first segment information matched by the timestamp of the time series data, wherein the first segment information is used to characterize the mapping relationship between the first time period and the segments in the first segment information, and the first time period includes the timestamp of the time series data; performing segmentation processing on the time series data based on the first segment information to obtain the segmentation result.

[0235] Optionally, the processor may also execute program code that performs the following steps: based on the number of fragments in the first fragmentation information and the key corresponding to the timing data, determine the target fragment corresponding to the timing data, and obtain the fragmentation result.

[0236] Optionally, the processor may also execute program code that performs the following steps: generating second fragment information based on the newly added fragment, the timestamp corresponding to the newly added fragment, and the first fragment information, wherein the second fragment information is used to characterize the mapping relationship between the second time period and the fragment set obtained from the fragments in the first fragment information and the newly added fragment, and the second time period is associated with the timestamp corresponding to the newly added fragment.

[0237] Optionally, the processor may also execute program code that performs the following steps: determining the first shard information based on the number of storage nodes; wherein writing the timing data read from the message queue to memory includes: writing the timing data read from the message queue to memory based on the storage nodes.

[0238] Optionally, the processor may also execute program code that performs the following steps: creating at least a first shard in the message queue based on the first shard information; creating a second shard on the storage node with the same number of shards as the first shards in the message queue, wherein the second shard corresponds one-to-one with the first shard.

[0239] Optionally, the processor may also execute program code that performs the following steps: adding a third shard to the message queue corresponding to the number of new storage nodes; determining the third shard information based on the number of third shards, the timestamp corresponding to the new storage nodes, and the number of first shards in the message queue, wherein the third shard information is used to characterize the mapping relationship between the third time period and the shard set obtained from the first shard and the third shard, and the third time period is associated with the timestamp corresponding to the new storage nodes.

[0240] Optionally, the processor may also execute program code that reads the sharding results from the message queue in multiple storage nodes and writes the read sharding results into memory.

[0241] Optionally, the processor may also execute program code that reads multiple shard results under the same topic from the message queue in multiple storage nodes and writes the read multiple shard results into memory.

[0242] Optionally, the processor may also execute program code that performs the following steps: if the data proxy node in the first cluster is in normal working condition, then obtain the time-series data to be processed based on the data proxy node in the first cluster; if the data proxy node in the first cluster is in abnormal working condition, then obtain the time-series data to be processed based on the data proxy node in the second cluster.

[0243] Optionally, the processor may also execute program code that performs the following steps: determining a target component based on a data subscription message, wherein the data subscription message is used to characterize the task to be subscribed to; and writing the time-series data read from the message queue into the target component.

[0244] This invention provides a scheme for loosely coupled time-series data storage based on message queues. By using message queues to separate log and data storage, loose coupling between log and data storage is achieved. This effectively reduces the complexity of data replication (high availability) and horizontal scaling implementation on storage nodes, improves data storage efficiency, solves the technical problem of low data storage efficiency, and achieves the technical effect of improving data storage efficiency.

[0245] Those skilled in the art will understand that Figure 19 The structure shown is for illustrative purposes only. The computer terminal can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a mobile internet device (MID), a PAD, and other terminal devices. Figure 19 This does not limit the structure of the aforementioned electronic device. For example, computer terminal 19 may also include components that are more advanced than those described above. Figure 19 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 19 The different configurations shown.

[0246] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0247] Embodiments of the present invention also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the timing data processing method provided in Embodiment 1.

[0248] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0249] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: responding to a client request, retrieving time-series data to be processed from a time-series database, wherein the time-series data is generated by a data source according to a time period and stored in the time-series database; storing the sharding results corresponding to the time-series data in a message queue, wherein the sharding results are used to enable the message queue to store the corresponding log files; and writing the sharding results read from the message queue into memory.

[0250] Optionally, the computer-readable storage medium is also configured to store program code for performing the following steps: fragmenting the time-series data to obtain fragmentation results.

[0251] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: determining first fragment information matched by the timestamp of the time series data, wherein the first fragment information is used to characterize the mapping relationship between the first time period and the fragments in the first fragment information, the first time period including the timestamp of the time series data; and performing fragmentation processing on the time series data based on the first fragment information to obtain fragmentation results.

[0252] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: determining the target fragment corresponding to the time-series data based on the number of fragments in the first fragmentation information and the key corresponding to the time-series data, and obtaining the fragmentation result.

[0253] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: generating second fragment information based on the newly added fragment, the timestamp corresponding to the newly added fragment, and the first fragment information, wherein the second fragment information is used to characterize the mapping relationship between the second time period and the fragment set obtained from the fragments in the first fragment information and the newly added fragment, and the second time period is associated with the timestamp corresponding to the newly added fragment.

[0254] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: determining first shard information based on the number of storage nodes; wherein writing timing data read from the message queue into memory includes: writing timing data read from the message queue into memory based on the storage nodes.

[0255] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: creating at least a first shard in a message queue based on the first shard information; creating a second shard on a storage node with the same number as the first shard in the message queue, wherein the second shard corresponds one-to-one with the first shard.

[0256] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: adding a third shard to the message queue corresponding to the number of newly added storage nodes; determining third shard information based on the number of third shards, the timestamps corresponding to the newly added storage nodes, and the number of first shards in the message queue, wherein the third shard information is used to characterize the mapping relationship between a third time period and the shard set obtained from the first shard and the third shard, and the third time period is associated with the timestamps corresponding to the newly added storage nodes.

[0257] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: reading sharding results from message queues in multiple storage nodes and writing the read sharding results into memory.

[0258] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: reading multiple shard results under the same topic from message queues in multiple storage nodes respectively, and writing the read multiple shard results to memory.

[0259] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: if the data broker node in the first cluster is in a normal working state, then obtain the time-series data to be processed based on the data broker node in the first cluster; if the data broker node in the first cluster is in an abnormal working state, then obtain the time-series data to be processed based on the data broker node in the second cluster.

[0260] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: determining a target component based on a data subscription message, wherein the data subscription message is used to characterize a task to be subscribed to; and writing time-series data read from the message queue into the target component.

[0261] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0262] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0263] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0264] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0265] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0266] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0267] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A time-series data processing method, characterized in that, include: When the response data broker receives a client request, it retrieves the time-series data to be processed from the time-series database, wherein the time-series data is generated by the data source according to a time period and stored in the time-series database; The sharding results corresponding to the time-series data are stored in a message queue, wherein the sharding results are used to enable the message queue to store the corresponding log files; The sharding results read from the message queue are written to memory; The method further includes: performing time sharding and hash sharding on the time-series data sequentially to obtain the sharding result, wherein the hash sharding process includes at least one of the following: metric hash sharding, label hash sharding, and the metric hash sharding and label hash sharding. The method further includes: in response to the time-series data in the memory reaching a threshold, flushing the time-series data in the memory to the log file, and submitting the flushed time-series data to the message queue; and performing read or delete operations on the corresponding messages in the message queue based on the flushed time-series data. The method further includes: adding a third shard to the message queue according to the expansion conditions, the expansion conditions being used to represent the number of storage nodes to be expanded; determining third shard information based on the number of third shards in the message queue, the timestamp corresponding to the newly added storage nodes, and the number of first shards in the message queue, wherein the third shard information is used to characterize the mapping relationship between a third time period and the shard set obtained from the first shard and the third shard, the third time period being associated with the timestamp corresponding to the newly added storage nodes, the first shard being created in the message queue based on the first shard information, the first shard information being used to characterize the mapping relationship between a first time period and the shards in the first shard information, the first time period including the timestamp of the time-series data.

2. The method according to claim 1, characterized in that, The time-series data is sequentially processed by time sharding and hash sharding to obtain the sharding results, including: Determine the first segment information matched by the timestamp of the time series data; The time-series data is hash-sharded based on the first sharding information to obtain the sharding result.

3. The method according to claim 2, characterized in that, Based on the first sharding information, the time-series data is subjected to hash sharding processing to obtain the sharding result, including: Based on the number of shards in the first sharding information and the key corresponding to the time series data, the target shard corresponding to the time series data is determined, and the sharding result is obtained.

4. The method according to claim 2, characterized in that, The method further includes: A second shard information is generated based on the newly added shard, the timestamp corresponding to the newly added shard, and the first shard information. The second shard information is used to characterize the mapping relationship between the second time period and the shard set obtained by the shards in the first shard information and the newly added shard. The second time period is associated with the timestamp corresponding to the newly added shard.

5. The method according to claim 2, characterized in that, The method further includes: The first shard information is determined based on the number of storage nodes; Specifically, writing the time-series data read from the message queue into memory includes: writing the time-series data read from the message queue into memory based on the storage node.

6. The method according to claim 5, characterized in that, The method further includes: On the storage node, a second shard is created in the same number as the first shard in the message queue, wherein the second shard corresponds one-to-one with the first shard.

7. The method according to claim 1, characterized in that, The sharding results read from the message queue are written to memory, including: The sharding results are read from the message queue on multiple storage nodes and written to the memory; or Multiple sharding results under the same topic are read from the message queue in multiple storage nodes respectively, and the read multiple sharding results are written to the memory.

8. The method according to claim 1, characterized in that, Obtain the time series data to be processed, including: If the data proxy node in the first cluster is in normal working condition, the time-series data to be processed is obtained based on the data proxy node in the first cluster; If the data proxy node in the first cluster is in an abnormal working state, the time-series data to be processed is obtained based on the data proxy node in the second cluster.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: The target component is determined based on the data subscription message, wherein the data subscription message is used to characterize the task to be subscribed to. The time-series data read from the message queue is written to the target component.

10. A time-series data processing system, characterized in that, include: A data proxy component is used to respond to a client request received by the data proxy and retrieve time-series data to be processed from a time-series database, wherein the time-series data is generated by the data source according to a time period and stored in the time-series database; A message queue component is used to store the sharding results corresponding to the time-series data in a message queue, wherein the sharding results are used to enable the message queue to store the corresponding log files; A data storage component is used to write the sharding results read from the message queue into memory; The system is also used to: sequentially perform time sharding and hash sharding on the time-series data to obtain the sharding result, wherein the hash sharding process includes at least one of the following: metric hash sharding, label hash sharding, the metric hash sharding process and the label hash sharding process; In response to the time-series data in the memory reaching a threshold, the time-series data in the memory is flushed to the log file, and the flushed time-series data is submitted to the message queue; based on the flushed time-series data, the corresponding message in the message queue is read or deleted. According to the expansion conditions, a third shard corresponding to the number of newly added storage nodes is added to the message queue, wherein the expansion conditions are used to indicate the number of storage nodes to be expanded; the third shard information is determined based on the number of the third shards in the message queue, the timestamp corresponding to the newly added storage nodes, and the number of the first shards in the message queue, wherein the third shard information is used to characterize the mapping relationship between the third time period and the shard set obtained from the first shard and the third shard, the third time period is associated with the timestamp corresponding to the newly added storage nodes, the first shard is created in the message queue based on the first shard information, the first shard information is used to characterize the mapping relationship between the first time period and the shards in the first shard information, and the first time period includes the timestamp of the time-series data.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein when the program is run by a processor, it controls the device in which the computer-readable storage medium resides to perform the method according to any one of claims 1 to 9.

12. An electronic device, characterized in that, include: processor; A memory, connected to the processor, is used to provide the processor with instructions to process the following steps: responding to a data broker receiving a client request, retrieving time-series data to be processed from a time-series database, wherein the time-series data is generated by a data source according to a time period and stored in the time-series database; storing the sharding results corresponding to the time-series data in a message queue, wherein the sharding results are used to enable the message queue to store corresponding log files; and writing the sharding results read from the message queue into memory. The memory is also used to provide the processor with instructions to perform the following processing steps: sequentially perform time sharding and hash sharding on the time-series data to obtain the sharding result, wherein the hash sharding process includes at least one of the following: metric hash sharding, tag hash sharding, the metric hash sharding process and the tag hash sharding process; In response to the time-series data in the memory reaching a threshold, the time-series data in the memory is flushed to the log file, and the flushed time-series data is submitted to the message queue; based on the flushed time-series data, the corresponding message in the message queue is read or deleted. According to the expansion conditions, a third shard corresponding to the number of newly added storage nodes is added to the message queue, wherein the expansion conditions are used to indicate the number of storage nodes to be expanded; the third shard information is determined based on the number of the third shards in the message queue, the timestamp corresponding to the newly added storage nodes, and the number of the first shards in the message queue, wherein the third shard information is used to characterize the mapping relationship between the third time period and the shard set obtained from the first shard and the third shard, the third time period is associated with the timestamp corresponding to the newly added storage nodes, the first shard is created in the message queue based on the first shard information, the first shard information is used to characterize the mapping relationship between the first time period and the shards in the first shard information, and the first time period includes the timestamp of the time-series data.

Images