Index construction methods and apparatus

Abstract

This specification provides an index construction method and apparatus. The index construction method includes: acquiring a set of messages to be processed and writing the set of messages to be processed into a message collection queue; when a construction thread reads the set of messages to be processed from the message collection queue, constructing a message index for the messages to be processed contained in the set of messages to be processed and writing it into a target memory space; converting a first data structure of the target memory space into a second data structure and writing it into a data processing queue; reading the second data structure from the data processing queue and storing it into a target disk.

Description

Technical Field

[0001] This specification relates to the field of database technology, and in particular to index construction methods and apparatus. Background Technology

[0002] With the development of internet technology and the increasing scale of cloud services, coupled with the growing complexity of cloud infrastructure environments, performance management scenarios for large-scale computer systems are facing various challenges. For example, with the application of containerization, elastic scaling, and large-scale hybrid deployment technologies, the operating environment of distributed services is becoming increasingly dynamic, leading to shorter lifecycles for individual processes and a larger number of entities requiring management. Simultaneously, the timeliness of managed data is becoming increasingly important. For instance, elastic scaling strategies rely on recent aggregated metrics for scaling decisions; if the indexing delay of metrics exceeds a set time, the aggregation results will be inaccurate, causing the elastic strategy to make incorrect judgments. Furthermore, in performance management scenarios, a set number of timelines are typically queried, representing a very small proportion compared to the timelines in the database. Scanning massive amounts of data consumes significant computing resources. Therefore, the indexing overhead and latency in existing technologies impact the performance management quality of cloud services, and there is a lack of effective solutions to this problem; an effective solution is urgently needed. Summary of the Invention

[0003] In view of this, embodiments of this specification provide an index construction method. One or more embodiments of this specification also relate to an index construction apparatus, a time-series database, a message determination method, a message determination apparatus, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0004] According to a first aspect of the embodiments of this specification, an index construction method is provided, comprising:

[0005] Obtain the set of messages to be processed and write the set of messages to be processed into the message collection queue;

[0006] When the build thread reads the set of messages to be processed from the message collection queue, it builds a message index for the messages to be processed contained in the set of messages to be processed and writes it into the target memory space;

[0007] The first data structure of the target memory space is converted into a second data structure and written into the data processing queue;

[0008] The second data structure is read from the data processing queue and stored to the target disk.

[0009] According to a second aspect of the embodiments of this specification, an index building apparatus is provided, comprising:

[0010] The acquisition module is configured to acquire a set of messages to be processed and write the set of messages to be processed into a message collection queue;

[0011] The build module is configured to build a message index for the messages to be processed contained in the message collection queue and write it to the target memory space when the build thread reads the message collection set from the message collection queue.

[0012] The write module is configured to convert a first data structure in the target memory space into a second data structure and write it to the data processing queue.

[0013] The read module is configured to read the second data structure from the data processing queue and store it to the target disk.

[0014] According to a third aspect of the embodiments of this specification, another index construction method is provided, including:

[0015] A set of messages to be processed is determined in the time series database, and the set of messages to be processed is written into the message collection queue;

[0016] When the build thread reads the set of messages to be processed from the message collection queue, it builds a message index for the messages to be processed contained in the set of messages to be processed and writes it into the target memory space;

[0017] The first data structure of the target memory space is converted into a second data structure and written into the data processing queue;

[0018] Read the second data structure from the data processing queue and store it to the target disk;

[0019] Upon receiving a message detection request, the set of messages to be processed is determined based on the first data structure, the second data structure, and the target disk, and is used as a response to the message detection request.

[0020] According to a fourth aspect of the embodiments of this specification, another index building apparatus is provided, comprising:

[0021] The set determination module is configured to determine the set of messages to be processed in the time series database and write the set of messages to be processed into the message collection queue;

[0022] The write space module is configured to, when the build thread reads the set of messages to be processed from the message collection queue, build a message index for the messages to be processed contained in the set of messages to be processed, and write it to the target memory space;

[0023] The write queue module is configured to convert a first data structure in the target memory space into a second data structure and write it into the data processing queue.

[0024] The thread module is configured to read the second data structure from the data processing queue and store it to the target disk;

[0025] The read set module is configured to, upon receiving a message detection request, determine the set of messages to be processed based on the first data structure, the second data structure, and the target disk, as a response to the message detection request.

[0026] According to a fifth aspect of the embodiments of this specification, yet another index construction method is provided, comprising:

[0027] The set of messages to be processed is determined in the time-series database of the anomaly detection service, and the set of messages to be processed is written into the message collection queue;

[0028] When the build thread reads the set of messages to be processed from the message collection queue, it builds a message index for the messages to be processed contained in the set of messages to be processed and writes it into the target memory space;

[0029] The first data structure of the target memory space is converted into a second data structure and written into the data processing queue;

[0030] The second data structure is read from the data processing queue and stored in the target disk associated with the anomaly detection service.

[0031] According to a sixth aspect of the embodiments of this specification, yet another index building apparatus is provided, comprising:

[0032] The set determination module is configured to determine the set of messages to be processed in the time-series database of the anomaly detection service, and write the set of messages to be processed into the message collection queue;

[0033] The write space module is configured to, when the build thread reads the set of messages to be processed from the message collection queue, build a message index for the messages to be processed contained in the set of messages to be processed, and write it to the target memory space;

[0034] The conversion structure module is configured to convert the first data structure of the target memory space into a second data structure and write it into the data processing queue;

[0035] The storage disk module is configured to read the second data structure from the data processing queue and store it to the target disk associated with the anomaly detection service.

[0036] According to a seventh aspect of the embodiments of this specification, a message determination method is provided, comprising:

[0037] Receive message detection requests submitted to the time-series database;

[0038] Read the second data structure from the target disk according to the message detection request;

[0039] The second data structure is converted into the first data structure, and a message index is generated based on the first data structure;

[0040] Based on the message index, a recall process is performed on the first data structure, the second data structure, and the target disk, and a set of messages to be processed is obtained based on the recall process results.

[0041] According to an eighth aspect of the embodiments of this specification, a message determination apparatus is provided, comprising:

[0042] The receiving module is configured to receive message detection requests submitted to the time series database;

[0043] The reading module is configured to read a second data structure from the target disk according to the message detection request;

[0044] The generation module is configured to convert the second data structure into a first data structure and generate a message index based on the first data structure;

[0045] The recall module is configured to perform recall processing on the first data structure, the second data structure, and the target disk based on the message index, and obtain a set of messages to be processed based on the recall processing results.

[0046] According to a ninth aspect of the embodiments of this specification, a time-series database is provided, comprising:

[0047] Index building end and target disk;

[0048] The target disk is used to store messages and indexes, and the index building end is used to execute index building instructions. When the index building end executes the index building instructions, it implements the steps of the above-described index building method.

[0049] According to a tenth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0050] Memory and processor;

[0051] The memory is used to store computer-executable instructions, and the processor is used to implement the steps of any of the above-mentioned index construction methods or message determination methods when executing the computer-executable instructions.

[0052] According to an eleventh aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-described index construction method or message determination method.

[0053] According to a twelfth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described index construction method or message determination method.

[0054] The index building method provided in this manual improves index building performance by transforming the single-message processing flow into a batch processing flow. Specifically, messages to be processed are organized into message batches and written to a message collection queue. Then, a build thread reads a message batch from the message collection queue and builds a message index for each message within it. After construction, the index is written to the target memory space, stored based on a first data structure. To reduce memory usage, the first data structure in the target memory space can be converted into a smaller second data structure and written to a data processing queue. Finally, the second data structure is read from the data processing queue and stored on the target disk, completing the persistence of the index and messages. This improves index building performance and reduces space usage. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of an index construction method provided in one embodiment of this specification;

[0056] Figure 2 This is a flowchart illustrating an index construction method provided in one embodiment of this specification;

[0057] Figure 3 This is a schematic diagram of the prefix tree insertion process in an index construction method provided in one embodiment of this specification;

[0058] Figure 4 This is a schematic diagram of an algorithm in an index construction method provided in one embodiment of this specification;

[0059] Figure 5 This is a flowchart illustrating the processing steps of an index construction method provided in one embodiment of this specification.

[0060] Figure 6 This is a schematic diagram of the structure of an index building apparatus provided in one embodiment of this specification;

[0061] Figure 7 This is a flowchart of another index construction method provided in one embodiment of this specification;

[0062] Figure 8 This is a schematic diagram of another index building apparatus provided in one embodiment of this specification;

[0063] Figure 9 This is a flowchart of yet another index construction method provided in one embodiment of this specification;

[0064] Figure 10 This is a schematic diagram of the structure of another index building apparatus provided in one embodiment of this specification;

[0065] Figure 11 This is a flowchart of a message determination method provided in one embodiment of this specification;

[0066] Figure 12 This is a schematic diagram of the structure of a message determination device provided in one embodiment of this specification;

[0067] Figure 13 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0068] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0069] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0070] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0071] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0072] A time-series database is a database used to ingest, process, and store timestamped data. It primarily handles data with time tags (data that changes chronologically, i.e., time-series data), which is also known as time-series data.

[0073] An index, in a database, is a separate, physical storage structure that sorts the values ​​of one or more columns in a database table. It's a collection of values ​​from one or more columns in a table and a list of logical pointers to the data pages in the table that physically identify those values. An index is analogous to a book's index, allowing you to quickly find the desired content by referring to page numbers. An index provides pointers to data values ​​stored in specified columns of a table, and these pointers are then sorted according to a specified sort order. The database uses the index to find a specific value and then follows the pointers to find the row containing that value. This makes the SQL statements corresponding to the table execute faster, allowing for quick access to specific information in the database table.

[0074] Indexing latency refers to the time difference between when metric data enters the system and when it becomes observable by users.

[0075] Append-only: This is a property of computer data storage that allows new data to be appended to the storage, while existing data remains immutable. Many data structures and databases implement immutable objects, effectively making their data structures append-only. Implementing append-only data structures has many advantages, such as ensuring data consistency, improving performance, and allowing rollback.

[0076] This specification provides an index building method, and also relates to an index building apparatus, a time-series database, a message determination method, a message determination apparatus, a computing device, a computer-readable storage medium, and a computer program, which will be described in detail in the following embodiments.

[0077] In practical applications, time-series databases play a crucial role in providing observability for large-scale computer systems. Their applications include system anomaly detection and early warning, data dashboard analysis and display, anomaly localization, and root cause analysis. Timeline indexing technology is a core technical module of time-series databases. An efficient timeline indexing module is critical for the write performance, index timeliness, and query performance of time-series databases. However, indexing schemes designed for large-scale performance management scenarios suffer from insufficient construction performance and timeliness. Therefore, an effective solution is urgently needed to address these issues.

[0078] The index building method provided in this manual improves index building performance by transforming the single-message processing flow into a batch processing flow. Specifically, messages to be processed are organized into message batches and written to a message collection queue. Then, a build thread reads a message batch from the message collection queue and builds a message index for each message within it. After construction, the index is written to the target memory space, stored based on a first data structure. To reduce memory usage, the first data structure in the target memory space can be converted into a smaller second data structure and written to a data processing queue. Finally, the second data structure is read from the data processing queue and stored on the target disk, completing the persistence of the index and messages. This improves index building performance and reduces space usage.

[0079] In other words, see Figure 1 The diagram illustrates that during index building at the index building node, the read thread reads a batch of messages to be processed from the message queue, encapsulates them into a set of messages to be processed, and then places the set of messages to be processed into the message collection queue. The build thread, on the other hand, reads one set of messages to be processed from the message collection queue at a time for batch building. When a message to be processed is read, a message index is built for each message in the set and written to the target memory space. To save memory, a first asynchronous thread can convert the first data structure in the target memory space into a more compact and query-friendly second data structure and write it to the data processing queue. The asynchronous thread then reads the second data structure from the data processing queue one by one, processes it, and finally flushes the second data structure to the target disk.

[0080] It should be noted that any user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of such data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0081] Figure 2 A flowchart of an index construction method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0082] Step S202: Obtain the set of messages to be processed and write the set of messages to be processed into the message collection queue.

[0083] The index building method provided in this embodiment is applied to time-series databases. By leveraging the characteristics of time-series databases in building associated timelines, it improves the efficiency of timeline index building. Specifically, the set of messages to be processed refers to the collection of messages waiting in the message queue but not yet indexed. Each message to be processed represents a timeline, and the data belonging to that timeline is recorded in the message to be processed. Correspondingly, the message collection queue refers to a document collection queue that temporarily stores messages to be processed, and the message collection queue processes messages according to a first-in, first-out (FIFO) rule.

[0084] Therefore, in order to facilitate the subsequent construction of the index for the messages to be processed, after obtaining the set of messages to be processed, the set of messages to be processed can be written into the message collection queue for processing. When the set of messages meets the processing conditions of the message collection queue, the construction thread will construct the index.

[0085] Furthermore, before writing the set of messages to be processed into the message collection queue, considering that the messages to be processed are waiting to be processed in the message queue and exist in a single form, in order to improve the efficiency of index building, the messages can be read and encapsulated in batches. In this embodiment, the specific implementation method is as follows:

[0086] The reading thread reads a set number of messages to be processed from the message queue and encapsulates the set number of messages to be processed to obtain the message set to be processed; the reading thread writes the message set to be processed into the message collection queue, wherein the message collection queue processes the message set according to the order processing rules.

[0087] Specifically, the ReadThread refers to the thread used to read messages from the message queue and encapsulate them for processing; correspondingly, the message queue is used to store messages that need to be indexed and is used to receive messages from upstream services; and correspondingly, the sequential processing rule refers to the rules for writing and writing messages to the message collection queue, which follows a first-in, first-out (FIFO) rule.

[0088] Based on this, when building an index at an index building node, a set number of pending messages can be read from the pending message queue outside the node by a read thread. At this time, the messages read in this batch exist in a single form, so the read thread can encapsulate the pending messages in this batch to obtain a set of pending messages. Then, the read thread writes the set of pending messages into a message collection queue, so that when building the index later, it can be processed according to the first-in-first-out rule of the message collection queue, so as to ensure that the index building is more orderly.

[0089] In other words, the ReadThread in the index building node reads a batch of messages to be processed from the message queue outside the node each time, and then encapsulates the batch of messages to be processed into a vector pointer (PointVector). After that, the ReadThread puts the PointVector into the document collection queue so that the building thread in the subsequent index building node can read the set of messages to be processed from the queue to build the message index.

[0090] In practical applications, when reading a set number of messages to be processed from the message queue, the set number can be set according to actual computing needs. This embodiment does not impose any limitations on this.

[0091] In summary, when building the index for pending messages in the index building node, the pending messages will be read and encapsulated in batches, which effectively improves the efficiency of index building. At the same time, since the index building is applied to a time-series database, the messages in the message collection queue can be processed according to the sequential processing rules, thereby ensuring that the index building is processed in time.

[0092] Step S204: When the construction thread reads the set of messages to be processed from the message collection queue, a message index is constructed for the messages to be processed contained in the set of messages to be processed, and written to the target memory space.

[0093] Specifically, after the set of messages to be processed is written to the message collection queue, new sets of messages to be processed will be continuously written to the message collection queue over time. The sets of messages to be processed are indexed according to the first-in-first-out rule, and indexes are built for the first set of messages to be processed that enter the queue. Therefore, when the build thread reads the set of messages to be processed from the message collection queue, it means that an index needs to be built for the messages to be processed in the set. So, the build thread can read the set of messages to be processed from the message collection queue, and then build message indexes for the messages to be processed in the set. This combines the characteristics of time-series databases, builds indexes for the messages written to the database according to the timeline, and writes them to the target memory space, so that the indexes and data can be read from the target storage space for persistence later.

[0094] Specifically, the BuildThread refers to the thread used to build message indexes for messages to be processed in batches. Correspondingly, the message index is a pointer to the index built for each message in the set of messages to be processed, and this index corresponds to all the data in the timeline to which the message belongs. The target memory space refers to the memory space storing the indexes and data that have not yet been persisted. It should be noted that the internal data structure of the target memory space can use an append-only lock-free data structure to improve the throughput of the index building nodes.

[0095] Furthermore, during the construction of the message index, in order to ensure that the content subsequently written to the disk includes both the index and the data, the message point data corresponding to the message can be determined after the construction is completed. In this embodiment, the specific implementation method is as follows:

[0096] For the messages to be processed in the set of messages to be processed, a message index is constructed, and the message point data corresponding to the messages to be processed in the set of messages to be processed is determined, wherein the message point data includes a time parameter and the parameter value of the time parameter; the message index and message point data corresponding to the messages to be processed in the set of messages to be processed are written into the target memory space.

[0097] Specifically, message point data refers to point data on the timeline to which the message to be processed belongs. That is, message point data includes a timestamp, a string-type metric, multiple tags, and multiple fields; each tag is a string-type key-value pair; each field contains a string-type field name and a numerical value of any type, such as integer (int), double, string, etc., which is not limited in this embodiment.

[0098] Based on this, when the construction thread reads the set of messages to be processed from the message collection queue, message indexes can be constructed for each message to be processed in the set. In order to ensure that the content written to the target memory space includes both indexes and data, the message point data corresponding to each message to be processed in the set needs to be determined. Finally, the message index and message point data corresponding to each message to be processed are written to the target memory space in sequence.

[0099] In other words, after the BuildThread reads the PointVector from the message collection queue, it constructs a Series-Oriented index for each message in the PointVector. Simultaneously, it writes the timestamp, a string metric, multiple tags, and multiple fields from the message point data (sample) into the target memory space, i.e., the segment, for subsequent processing. It should be noted that each tag is a string key-value pair; each field contains a string field name and a value of any type.

[0100] In the target memory space, the message indexes and message point data being written exist in a full-memory data storage structure. That is, the target memory space for writing the indexes and data is a building segment (BuildingSegment). The internal data structure of the BuildingSegment can be implemented using an append-only lock-free data structure, thereby improving the construction throughput of the index building nodes. In other words, the BuildingSegment is a full-memory data storage structure that stores message indexes and message point data and receives real-time data writes.

[0101] In summary, by using a full-memory data storage structure to store the message index and message point data of the message to be processed, it can be ensured that the content written to the target memory space is in its original state and will not be changed relative to the original state due to writing / processing operations, thereby ensuring that the index and data subsequently written to the disk are more accurate.

[0102] Furthermore, in order to reduce memory usage, message point data from the same timeline can be encoded in the target memory space after being written to it, resulting in smaller content. In this embodiment, the specific implementation method is as follows:

[0103] An original time series point set is constructed based on the message point data corresponding to the message to be processed; the original time series point set is subjected to time series encoding processing to obtain a target time series point set; the message index corresponding to the message to be processed and the target time series point set are written into the target memory space.

[0104] Specifically, the original time-series point set refers to the set of message point data corresponding to the message to be processed, which can be uniformly encoded. Correspondingly, the target time-series point set refers to the time-series point set obtained after encoding. Based on this, after constructing the message index for the message to be processed, in order to reduce memory usage, we can first construct the original time-series point set for the message point data corresponding to the message to be processed, that is, combine the point data belonging to the same timeline to obtain the original time-series point set; secondly, we can perform time-series encoding on the original time-series point set to obtain a smaller target time-series point set; finally, we write the message index corresponding to the message to be processed and the target time-series point set into the target memory space.

[0105] In other words, within the target memory space, the original time series points belonging to the same timeline can be time-series encoded to save on construction memory usage. Furthermore, once the target memory space reaches a certain threshold, writing new message indices and data corresponding to pending messages can be stopped, allowing for subsequent asynchronous processing.

[0106] In summary, by performing time-series encoding on the original time series set belonging to the same timeline, the data occupancy rate in the target memory space can be further reduced, thereby allowing more data to be written into the target memory space, which can improve processing efficiency during subsequent persistence processing.

[0107] In addition, to improve index building efficiency and reduce computational resource consumption, indexes can be built in the following ways:

[0108] An inverted index and a prefix tree index are constructed for the messages to be processed contained in the set of messages to be processed; the inverted index and the prefix tree index are used as the message index of the messages to be processed.

[0109] In other words, when building an index for messages to be processed in a set of messages to be processed, an inverted index and a prefix tree index can be built simultaneously for each message to be processed, which can then be used as the message index for subsequent persistence.

[0110] The process of building the index is described above using the following structure of the message to be processed as an example. It should be noted that the message structure in this embodiment only illustrates the content required when building the index. The message structure in actual applications is not limited in this embodiment. Correspondingly, the construction of the index corresponding to the message can also refer to the same or corresponding description in this embodiment, and is not limited in this embodiment. In the message to be processed, some fields of its original content are metric:"cpu"; tags:{"ip":"10.10.10.1","appName":"tb"}.

[0111] Furthermore, to identify timelines, a timeline primary key (seriesKey) can be used to distinguish the timelines to which pending messages belong. In other words, the timeline primary key can be used to determine whether the currently received pending message and the previous pending message belong to the same timeline. Taking the above pending message as an example, its corresponding timeline primary key is seriesKey = cpu@ip = 10.10.10.1, appName = tb. When it is determined that the pending message corresponding to this timeline primary key does not belong to the same timeline as the previous pending message, an index needs to be built for this pending message.

[0112] During the inverted index construction phase, the mapping relationship between tokens and timeline identifiers (seriesId) is actually recorded using an inverted linked list. Tokens are created separately for metric and tag. For example, if the token is "cpu", the inverted linked list records the set of IDs for all seriesKeys with the metric "cpu" {id0, id1, id2, ...}. Similarly, if the token is "ip = 10.10.10.1", the inverted linked list records the set of timeline IDs for all tags containing "ip = 10.10.10.1" {id1, id2, id5, id9}. Likewise, if the token is "appName = tb", the inverted linked list records the set of timeline IDs for all tags containing "appName = tb" {id1, id11, id12}. In other words, the message to be processed contains a metric field and a tag field. The token is obtained through these two fields, and then a mapping relationship is established between the token and the timeline set. The timeline set is represented by the timeline ID, thereby completing the construction of the inverted index of the message to be processed.

[0113] During the construction of the prefix tree index, two nodes are actually inserted into the prefix tree: cpu@ip=10.10.10.1 and cpu@appName=tb. In other words, the timeline primary key (seriesKey) is converted to the form metric+tagk+tagv. Since a seriesKey corresponds to many tags, the timeline primary key (seriesKey) can be converted to the form metric@tagk=tagv based on multiple sets of tags. Then, inserting cpu@ip=10.10.10.1 and cpu@appName=tb completes the construction of the prefix tree index for the message to be processed. This type of prefix tree index is constructed because multiple messages to be processed may have the same metric but different tags, or the same metric and tagk but different tagv. Therefore, constructing the prefix tree index in this way saves memory and allows for quick retrieval of the required message during the query phase.

[0114] See Figure 3 The diagram illustrates this. Assuming the two data sets are "abc" and "abd", the prefix tree stores the common prefix "ab" in an intermediate node, and only once, reducing space usage. Furthermore, if a query requires finding a character sequence with "ab" as a prefix, it can traverse from the root node until it finds "abc" and "abd" at a leaf node, thus achieving a prefix query.

[0115] Furthermore, when performing concurrent read and write operations on a prefix tree structure, the most intuitive approach is to lock the prefix tree and execute reads and writes serially. However, locking incurs additional overhead. Therefore, a lock-free write optimization scheme based on epoch-version can be proposed. Specifically, each node in the prefix tree has a version number. When a node needs to be split or replaced, the node and its current version number are placed in a garbage collection queue. Simultaneously, each query thread maintains its own version number at the time of the query. During garbage collection, when actually releasing node space, only the smallest version number in the garbage collection queue held by the query thread is released. Based on this, locking is eliminated, and the data accessed by the query thread is not released.

[0116] In summary, by constructing prefix tree indexes and inverted indexes for messages, message processing speed can be effectively accelerated, facilitating subsequent persistence, and avoiding excessive memory usage.

[0117] Furthermore, when constructing the message index corresponding to the messages to be processed, in order to optimize memory access and avoid introducing a large number of cache invalidations, the index can be constructed in the following way:

[0118] The construction thread processes the messages to be processed contained in the message set to be processed, and obtains at least one timeline data sequence; according to the arrangement order of the at least one timeline data sequence, the memory address prefetch instructions of the timeline data sequence are sent to the processor in sequence; when the memory address prefetch instruction of the last timeline data sequence in the at least one timeline data sequence is sent to the processor, in response to the data prefetch result of the processor, an index is constructed sequentially for the timeline data sequence as the message index of the message to be processed in the message set to be processed.

[0119] Specifically, processing a message involves organizing it into a data sequence, that is, organizing the message points within the message into a data sequence corresponding to a timeline. The timeline data sequence is the sequence obtained by concatenating all the message points associated with the message. The arrangement order refers to the order in which the timeline data sequences are sorted according to the timeline. The memory address prefetch instruction is an instruction sent to the processor to prefetch each timeline data sequence into the cache. The data prefetch result refers to the result of the processor prefetching each timeline data sequence.

[0120] Therefore, during the construction of inverted indexes and prefix tree indexes, there may be a large number of random access operations to storage space, which can lead to significant cache invalidation and TLB (Translation Lookaside Buffer) failures. Therefore, coroutine anticipation and large page memory allocators can be used to optimize memory access. Specifically, firstly, a thread is constructed to process the messages to be processed in the message set to be processed, obtaining the timeline data sequence corresponding to each timeline. Then, according to the arrangement order of each timeline data sequence, the memory address prefetch instruction of each timeline data sequence is sent to the processor in sequence. At this time, the memory address prefetch instruction of each timeline data sequence will be sent to the CPU in sequence. When the memory address prefetch instruction of the last timeline data sequence in at least one timeline data sequence is sent to the processor, it means that the memory address prefetch instruction corresponding to the last timeline data sequence has been submitted to the CPU. In order to improve memory access efficiency, the first timeline data sequence in at least one timeline data sequence can be prefetched into the CPU cache. Then, in response to the processor's data prefetch result, an index can be constructed for each timeline data sequence in sequence as the message index of the messages to be processed in the message set to be processed.

[0121] In other words, coroutine prefetching can be used to improve the throughput efficiency of the CPU pipeline. The construction thread organizes the message point data corresponding to the messages to be processed into data batches containing k timelines (s1 to s2). k The build thread can use a coroutine to send the memory address prefetch instruction of s1 to the CPU first, and then suspend the index building of s1. At the same time, to improve efficiency, s2 to s... k until s k When index building for s1 is suspended, index building for s1 is resumed. During this process, the data for s1 has already been prefetched into the CPU cache, and so on, from s2 to s... k The processing is similar, thus masking memory access latency with CPU computation to improve CPU pipeline efficiency.

[0122] For example, see Figure 4The diagram in (a) shows an example where, assuming k=4, the memory address prefetch instruction for s1 is first sent to the CPU, and then the index construction of s1 is suspended. The same applies to s2, s3, and s4. When the index construction of s1 is resumed, s1 has already been prefetched into the CPU. Similarly, the processing from s2 to s4 is similar. This process masks the memory access latency with CPU computation, thereby improving CPU pipeline efficiency.

[0123] Building upon this, the memory access overhead during the build process is not only caused by cache misses but also by TLB misses. Transparent huge pages (TWPS) are a memory access optimization mechanism provided by the Linux operating system. However, relying entirely on the OS-provided TWPS mechanism can lead to memory bloat and performance fluctuations. Therefore, to avoid these problems, a chunk-based memory pool can be used to manage frequently accessed memory such as seriesMaps and data buffers. The memory pool reduces the cost of using TWPS by creating memory huge pages using the madvise system call and reusing memory chunks between segments, thus achieving the desired effect.

[0124] In summary, by combining coroutine prefetching and big page memory allocators to improve memory access efficiency, it is possible to ensure improved access efficiency during memory access.

[0125] Furthermore, considering that data corresponding to the same timeline may exist in target memory spaces of different periods during index construction, the indexes built in this way will contain duplicates, affecting index construction efficiency and wasting computing resources. Therefore, the complement method can be used to build the index. In this embodiment, the specific implementation is as follows:

[0126] If there are local unprocessed messages in the set of unprocessed messages, obtain the historical message set of the period adjacent to the set of unprocessed messages; compare the set of unprocessed messages with the historical message set, and determine the first message to be constructed and the second message to be constructed in the unprocessed messages according to the comparison result; construct the message index of the first message to be constructed and load the message index corresponding to the local unprocessed message; write the first message to be constructed and its corresponding message index, the local unprocessed message and its corresponding message index, and the second message to be constructed into the target memory space.

[0127] Specifically, "partial pending messages" refers to messages corresponding to a subset of message points on the same timeline, while another subset of pending messages exists in the pending message set of the previous period. Correspondingly, the historical message set refers to the set of pending messages from the previous index building period, which includes another subset of the partial pending messages from the current index building period. Similarly, "first pending messages" refers to messages in the pending message set that require index building, and "second pending messages" refers to messages in the pending message set that have already completed index building in the previous period.

[0128] Based on this, considering the temporal locality of time series data, that is, the reporting of a timeline is continuous during its life cycle, for example, from 9:00 to 10:00, the point data from 9:00 to 9:30 may be written into one set of pending messages, while the point data from 9:30 to 10:00 may be written into the next set of pending messages. Since pending messages belonging to the same timeline should have an index, if indexes are built separately for the pending messages contained in the two sets of pending messages, the same timeline will be indexed twice. Therefore, to avoid this problem, the complement method can be used to build the index.

[0129] In other words: First, if it is determined that there are local unprocessed messages in the set of unprocessed messages, it means that the previous index building cycle has already built the index for the timeline to which the local unprocessed message belongs. Therefore, at this time, the historical message set of the cycle adjacent to the set of unprocessed messages can be obtained first. The set of unprocessed messages is compared with the historical message set. Based on the comparison result, the first message to be built and the second message to be built are determined in the set of unprocessed messages. The first message to be built is the message that needs to be indexed. Therefore, the corresponding message index can be built for the first message to be built. The second message to be built has already completed the construction of the message index in the previous cycle, so it does not need to be built. This is to determine the unprocessed messages that have not been indexed. Then, the message index of the first message to be built is built, and the message index corresponding to the local unprocessed message is loaded. Finally, the first message to be built and its corresponding message index, the local unprocessed message and its corresponding message index, and the second message to be built are written into the target memory space.

[0130] The aforementioned index building process introduces a dependency problem between index building cycles. That is, both the index building and query processes for the current cycle depend on the index from the previous cycle, and the index from the previous cycle may in turn depend on the index from the cycle before that. To avoid infinite recursive dependencies on previous cycles, the index can be completed during the dump phase. In this embodiment, the specific implementation is as follows:

[0131] If the message index of the message to be built exists in the previous build cycle, obtain the first index identifier assigned to the message to be built in the previous build cycle and record it in the index identifier table; if the message index of the message to be built does not exist in the previous build cycle, call the identifier allocation function to assign a second index identifier to the message to be built and record it in the index identifier table; based on the index identifier recorded in the index identifier table and the timeline primary key corresponding to the message to be built, establish a prefix tree index and an inverted index for the message to be built.

[0132] Furthermore, the prefix tree index and inverted index corresponding to the message to be constructed can be determined; the prefix tree index corresponding to the message to be constructed and the inverted index corresponding to the message to be constructed can be completed; and the full set index corresponding to the message to be constructed can be determined based on the completion results.

[0133] Based on this, when building an index, considering that the index already exists in the previous build cycle, building the index again in the current cycle would cause index redundancy and waste computing resources. Therefore, in order to ensure the successful construction of the index, if it is determined that the message index of the message to be built exists in the previous build cycle, it means that it is not necessary to build the index again. At this time, the first index identifier of the message to be built in the previous build cycle can be obtained and recorded in the index identifier table; it is no longer necessary to build the index.

[0134] If the message index of the message to be built does not exist in the previous build cycle, it means that the index has not been built. The identifier allocation function can be called to assign a second index identifier to the message to be built and record it in the index identifier table. Then, based on the index identifier recorded in the index identifier table and the timeline primary key corresponding to the message to be built, a prefix tree index and an inverted index are built for the message to be built.

[0135] In other words, there are many duplicate timelines between adjacent segments. If a timeline *s* has already been indexed in a previous segment, there's no need to index it again in subsequent segments. Therefore, a complement approach can be used for indexing. With complement indexing, only the complement elements of the previous segment are indexed in each segment. However, during the dump phase, we need to complete the timeline index set because each segment only builds indices for lines that didn't exist in the previous segment. Then, further processing can proceed.

[0136] In practice, if timeline I enters segment[i] of the current stage, it is first necessary to determine whether timeline I has already been indexed by segment[i] (see [link]). Figure 4 In the algorithm shown in (b), if line 1-line 2 is used, it means that timeline I has already been indexed, and therefore an index cannot be created for it. In this case, the point data of timeline I can be written directly. During this process, if timeline I is indexed by segment[i-1] (see...), Figure 4 In the algorithm shown in (b), lines 3-6 indicate that timeline I does not need to build an index at the current stage. However, it should be noted that since timeline I has already been indexed, in order to avoid affecting its use, it is necessary to obtain the seriesId assigned to timeline I in segment[i-1] and record it in the timeline identifier linked list seriesIdList[i] for convenient use later.

[0137] If timeline I has not been indexed by segment[i-1] and segment[i] (see...) Figure 4 As shown in lines 7-12 of the algorithm in (b), timeline I is a new timeline that did not exist in the previous or current stage. In this case, a seriesId can be assigned to it using the identifier allocation function and recorded in seriesIdList[i]. Then, a prefix tree index and an inverted index are built for timeline I using the timeline primary key (seriesKey) and seriesId. Finally, the point data of timeline I is written into segment[i].

[0138] It should be noted that the identifier allocation function is used to allocate seriesId, and it needs to satisfy the property of allocating monotonically increasing integer numbers of Id, such as using the Mex function or the Max function. In practical applications, the choice can be made according to actual needs, and this embodiment will not impose too many restrictions here.

[0139] Furthermore, after utilizing the complement index, only the complement element index of the previous segment is constructed in the segment. However, during the dump stage, the timeline index set needs to be completed because only the indices of lines that did not exist in the previous segment are constructed in each segment. Therefore, during the dump stage, we need to complete the timeline index set. Based on this, according to the calculation logic in the above algorithm, only the timelines of the complement set are indexed in segment[i]. Therefore, during the dump stage, segment[i] will be completed into a complete, self-explanatory segment. By completing it into a self-explanatory segment, read amplification caused by excessively long dependency chains between segments can be avoided, and data management is more convenient.

[0140] For details, see Figure 4 The algorithm shown in (c) first completes the inverted index using lines 1-7. Line 1 merges the inverted index sInv[i-1] of segment[i-1] and the token set of the inverted index of segment[i] to obtain the candidate token set D. Then, lines 2-7 iterate through each token T in D, with the inverted list of token T identified as sPost. i (T), the calculation rule is to place T in the inverted chain sPost of the previous segment. i-1 Find the intersection of (T) and the current seriesId set, and then combine it with the linked list post based on the completed set. i(T) The union can be used to complete the index. Lines 8-17 can be used to complete the prefix tree index. Line 8 finds the union and intersection of the seriesId sets of segment[i-1] and segment[i], which gives the set E of IDs that appear in both adjacent segments. Then, line 15 inserts the timeline primary key (seriesKey) corresponding to set E back into the prefix tree sTrie[i] to complete the index construction.

[0141] In summary, employing a timeline-based asynchronous index building framework can improve the overall throughput of index building nodes. Furthermore, designing a compact and lock-free data structure can reduce memory consumption and enhance index building performance. The proposed index building algorithm, based on coroutine prefetching and large page memory optimization, can further improve index building performance. Finally, the proposed complement index algorithm can reduce throughput jitter and latency jitter during the cold start phase of index building.

[0142] Step S206: Convert the first data structure of the target memory space into a second data structure and write it into the data processing queue.

[0143] Specifically, after writing the message index and message point data into the target memory space and storing them in the form of a first data structure, in order to make the data more compact and easier to use during the query phase, the first data structure of the target memory space can be converted into a second data structure. The second data structure is more compact than the first data structure, and the second data structure is then written into the data processing queue for subsequent persistence processing.

[0144] Specifically, the second data structure refers to the data structure obtained after transforming the first data structure corresponding to the target memory space. It is smaller and more compact than the first data structure. The transformation process involves reordering and deduplicating the data in the first data structure. Correspondingly, the data processing queue is a queue that stores the second data structure, waiting for the disk to be ready for persistent processing.

[0145] Furthermore, in the process of converting the first data structure into the second data structure, in order to ensure that the processed second data structure can be written into the data processing queue for persistent processing, the following method can be used:

[0146] The first data structure of the target memory space is determined, wherein the first data structure is a lock-free data structure; the first asynchronous thread is invoked to convert the first data structure to obtain a second data structure; when the target memory space is detected to have reached a storage threshold, the second data structure is written into the data processing queue through the first asynchronous thread.

[0147] Specifically, the first asynchronous thread refers to the asynchronous dump thread, i.e., AsyncDumpThread, used to transform and process the first data structure. Correspondingly, the storage threshold refers to the threshold used to detect whether the target storage space has reached its storage limit. The storage threshold can be set according to actual needs, and this embodiment does not impose any limitations on it.

[0148] Based on this, once the first data structure corresponding to the target storage space is determined, the first asynchronous thread can be invoked to transform the first data structure into a more compact and query-friendly second data structure. Subsequently, if the target storage space reaches its storage threshold, indicating that no new data can be written to it, the first asynchronous thread can be invoked to write the second data structure to the data processing queue for later persistence processing.

[0149] In other words, once the memory usage of a BuildingSegment reaches the storage threshold, no more new data can be written. The BuildingSegment can then be pushed into the Dump queue, awaiting processing by the asynchronous dump thread (AsyncDumpThread). During this process, because the BuildingSegment's memory usage has reached the storage threshold, a new BuildingSegment can be created to receive new data writes. The BuildingSegment pushed into the Dump queue is transformed by the asynchronous dump thread (AsyncDumpThread) into a more compact and query-friendly data structure called ImMutable Segment. This ImMutable Segment can then be pushed into the data processing (Flush) queue by the asynchronous dump thread for persistent processing.

[0150] When the asynchronous dump thread performs the transformation, it actually reorders and deduplicates the BuildingSegment data in the dump queue. This involves reorganizing the data from in-memory format to a persistent format. After the transformation, the data is more compact and has lower storage overhead due to the deduplication and sorting operations.

[0151] In summary, by calling the first asynchronous thread to transform the first data structure, a more compact data structure that is more conducive to querying can be obtained. Furthermore, the second data structure is in a persistent form, which can be stored on disk for easier use during the query phase.

[0152] Step S208: Read the second data structure from the data processing queue and store it to the target disk.

[0153] Specifically, after writing the second data structure into the data processing queue, to further persist the message index and data, the second data structure can be read from the data processing queue and stored to the target disk. This process can be implemented using an asynchronous thread, that is, calling an asynchronous thread to read the second data structure from the data processing queue and store it to the target disk. Specifically, the asynchronous thread refers to the Flush thread, i.e., AsyncFlushThread, which reads messages from the queue and performs persistence processing.

[0154] In other words, after writing the ImMutable Segment to the Flush queue, the ImMutable Segments in the Flush queue can be flushed to the target disk one by one by the asynchronous Flush thread (AsyncFlushThread) to achieve persistence of point data and indexes, which is convenient for use in the query phase.

[0155] The index building method provided in this manual improves index building performance by transforming the single-message processing flow into a batch processing flow. Specifically, messages to be processed are organized into message batches and written to a message collection queue. Then, a build thread reads a message batch from the message collection queue and builds a message index for each message within it. After construction, the index is written to the target memory space, stored based on a first data structure. To reduce memory usage, the first data structure in the target memory space can be converted into a smaller second data structure and written to a data processing queue. Finally, the second data structure is read from the data processing queue and stored on the target disk, completing the persistence of the index and messages. This improves index building performance and reduces space usage.

[0156] The following is in conjunction with the appendix Figure 5 Taking the application of the index construction method provided in this specification in a practical application scenario as an example, the index construction method will be further explained. Figure 5 The flowchart of an index construction method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0157] Step S502: The reading thread reads a set number of messages to be processed from the message queue and encapsulates the set number of messages to be processed to obtain a set of messages to be processed.

[0158] Step S504: The set of messages to be processed is written into the message collection queue by the reading thread, wherein the message collection queue processes the message set according to the sequential processing rules.

[0159] Step S506: When the construction thread reads the set of messages to be processed from the message collection queue, a message index is constructed for the messages to be processed in the set of messages to be processed.

[0160] Step S508: Determine the message point data corresponding to the message to be processed in the message set to be processed, wherein the message point data includes time parameters and parameter values ​​of the time parameters.

[0161] Step S510: Construct an original time series point set based on the message point data corresponding to the message to be processed.

[0162] Step S512: Perform time-series encoding on the original time-series point set to obtain the target time-series point set.

[0163] Step S514: Write the message index and target time point set corresponding to the message to be processed into the target memory space.

[0164] Step S516: Determine the first data structure of the target memory space, wherein the first data structure is a lock-free data structure.

[0165] Step S518: Call the first asynchronous thread to convert the first data structure to obtain the second data structure.

[0166] In step S520, when the target memory space is detected to have reached the storage threshold, the second data structure is written into the data processing queue through the first asynchronous thread.

[0167] Step S522: Call the second asynchronous thread to read the second data structure from the data processing queue and store it to the target disk.

[0168] The index building method provided in this manual improves index building performance by transforming the single-message processing flow into a batch processing flow. Specifically, messages to be processed are organized into message batches and written to a message collection queue. Then, a build thread reads a message batch from the message collection queue and builds a message index for each message within it. After construction, the index is written to the target memory space, stored based on a first data structure. To reduce memory usage, the first data structure in the target memory space can be converted into a smaller second data structure and written to a data processing queue. Finally, the second data structure is read from the data processing queue and stored on the target disk, completing the persistence of the index and messages. This improves index building performance and reduces space usage.

[0169] Corresponding to the above method embodiments, this specification also provides embodiments of an index building apparatus. Figure 6 A schematic diagram of an index building apparatus according to one embodiment of this specification is shown. Figure 6 As shown, the device includes:

[0170] The acquisition module 602 is configured to acquire a set of messages to be processed and write the set of messages to be processed into a message collection queue;

[0171] The construction module 604 is configured to, when the construction thread reads the set of messages to be processed from the message collection queue, construct a message index for the messages to be processed contained in the set of messages to be processed, and write it to the target memory space.

[0172] The writing module 606 is configured to convert the first data structure of the target memory space into a second data structure and write it into the data processing queue.

[0173] The read module 608 is configured to read the second data structure from the data processing queue and store it to the target disk.

[0174] In an optional embodiment, the acquisition module 602 is further configured to:

[0175] The reading thread reads a set number of messages to be processed from the message queue and encapsulates the set number of messages to be processed to obtain the message set to be processed; the reading thread writes the message set to be processed into the message collection queue, wherein the message collection queue processes the message set according to the order processing rules.

[0176] In an optional embodiment, the building module 604 is further configured to:

[0177] For the messages to be processed in the set of messages to be processed, a message index is constructed, and the message point data corresponding to the messages to be processed in the set of messages to be processed is determined, wherein the message point data includes a time parameter and the parameter value of the time parameter; the message index and message point data corresponding to the messages to be processed in the set of messages to be processed are written into the target memory space.

[0178] In an optional embodiment, the building module 604 is further configured to:

[0179] An original time series point set is constructed based on the message point data corresponding to the message to be processed; the original time series point set is subjected to time series encoding processing to obtain a target time series point set; the message index corresponding to the message to be processed and the target time series point set are written into the target memory space.

[0180] In an optional embodiment, the writing module 606 is further configured to:

[0181] The first data structure of the target memory space is determined, wherein the first data structure is a lock-free data structure; the first asynchronous thread is invoked to convert the first data structure to obtain a second data structure; when the target memory space is detected to have reached a storage threshold, the second data structure is written into the data processing queue through the first asynchronous thread.

[0182] In an optional embodiment, the building module 604 is further configured to:

[0183] An inverted index and a prefix tree index are constructed for the messages to be processed contained in the set of messages to be processed; the inverted index and the prefix tree index are used as the message index of the messages to be processed.

[0184] In an optional embodiment, the building module 604 is further configured to:

[0185] The construction thread processes the messages to be processed contained in the message set to be processed, and obtains at least one timeline data sequence. According to the arrangement order of the at least one timeline data sequence, the memory address prefetch instruction of each timeline data sequence is sent to the processor in sequence. When the memory address prefetch instruction of the last timeline data sequence in the at least one timeline data sequence is sent to the processor, in response to the data prefetch result of the processor, an index is constructed for each timeline data sequence in sequence, which serves as the message index of the message to be processed in the message set to be processed.

[0186] In an optional embodiment, the building module 604 is further configured to:

[0187] If there are local unprocessed messages in the set of unprocessed messages, obtain the historical message set of the period adjacent to the set of unprocessed messages; compare the set of unprocessed messages with the historical message set, and determine the first message to be constructed and the second message to be constructed in the unprocessed messages according to the comparison result; construct the message index of the first message to be constructed and load the message index corresponding to the local unprocessed message; write the first message to be constructed and its corresponding message index, the local unprocessed message and its corresponding message index, and the second message to be constructed into the target memory space.

[0188] In an optional embodiment, the building module 604 is further configured to:

[0189] If the message index of the message to be built exists in the previous build cycle, obtain the first index identifier assigned to the message to be built in the previous build cycle and record it in the index identifier table; if the message index of the message to be built does not exist in the previous build cycle, call the identifier allocation function to assign a second index identifier to the message to be built and record it in the index identifier table; based on the index identifier recorded in the index identifier table and the timeline primary key corresponding to the message to be built, establish a prefix tree index and an inverted index for the message to be built.

[0190] In an optional embodiment, the building module 604 is further configured to:

[0191] Determine the prefix tree index and inverted index corresponding to the message to be constructed; perform completion processing on the prefix tree index and the inverted index corresponding to the message to be constructed; determine the global index corresponding to the message to be constructed based on the completion processing results.

[0192] The index building apparatus provided in this specification improves index building performance by transforming the single-message processing flow into a batch processing flow. Specifically, messages to be processed are organized into message batches and written to a message collection queue. Then, a building thread reads a message batch from the message collection queue and builds a message index for each message to be processed within it. After construction, the index is written to the target memory space and stored based on a first data structure. To reduce memory usage, the first data structure in the target memory space can be converted into a smaller second data structure and written to a data processing queue. Finally, the second data structure is read from the data processing queue and stored on the target disk, completing the persistence of the index and messages. This improves index building performance and reduces space usage during index construction.

[0193] The above is a schematic scheme of an index building apparatus according to this embodiment. It should be noted that the technical solution of this index building apparatus and the technical solution of the index building method described above belong to the same concept. For details not described in detail in the technical solution of the index building apparatus, please refer to the description of the technical solution of the index building method described above.

[0194] Corresponding to the above method embodiments, this specification also provides another index construction method embodiment. Figure 7 A flowchart illustrating another index construction method provided in one embodiment of this specification is shown. Figure 7 As shown, the method includes:

[0195] Step S702: Determine the set of messages to be processed in the time series database and write the set of messages to be processed into the message collection queue;

[0196] Step S704: When the construction thread reads the set of messages to be processed from the message collection queue, a message index is constructed for the messages to be processed contained in the set of messages to be processed, and written to the target memory space.

[0197] Step S706: Convert the first data structure of the target memory space into a second data structure and write it into the data processing queue;

[0198] Step S708: Read the second data structure from the data processing queue and store it to the target disk.

[0199] Step S710: Upon receiving a message detection request, determine the set of messages to be processed based on the first data structure, the second data structure, and the target disk, as a response to the message detection request.

[0200] The same or corresponding content in this embodiment can be found in the corresponding descriptions in the above embodiments, and will not be repeated here. When querying the set of messages to be processed, since the set of messages to be processed undergoes multiple processing steps before being written to the target disk, in order to obtain the content of the message detection request, it is necessary to retrieve the messages from the first data structure, the second data structure, and the target disk respectively, so as to obtain the set of messages to be processed as the response to the message detection request.

[0201] Optionally, determining the set of messages to be processed in the time-series database and writing the set of messages to be processed into the message collection queue includes:

[0202] The reading thread reads a set number of messages to be processed from the message queue and encapsulates the set number of messages to be processed to obtain the message set to be processed; the reading thread writes the message set to be processed into the message collection queue, wherein the message collection queue processes the message set according to the order processing rules.

[0203] Optionally, the step of constructing a message index for the messages to be processed contained in the set of messages to be processed and writing it into the target memory space includes:

[0204] For the messages to be processed in the set of messages to be processed, a message index is constructed, and the message point data corresponding to the messages to be processed in the set of messages to be processed is determined, wherein the message point data includes a time parameter and the parameter value of the time parameter; the message index and message point data corresponding to the messages to be processed in the set of messages to be processed are written into the target memory space.

[0205] Optionally, writing the message index and message point data corresponding to the messages to be processed in the set of messages to be processed into the target memory space includes:

[0206] An original time series point set is constructed based on the message point data corresponding to the message to be processed; the original time series point set is subjected to time series encoding processing to obtain a target time series point set; the message index corresponding to the message to be processed and the target time series point set are written into the target memory space.

[0207] Optionally, converting the first data structure of the target memory space into a second data structure and writing it into the data processing queue includes:

[0208] The first data structure of the target memory space is determined, wherein the first data structure is a lock-free data structure; the first asynchronous thread is invoked to convert the first data structure to obtain a second data structure; when the target memory space is detected to have reached a storage threshold, the second data structure is written into the data processing queue through the first asynchronous thread.

[0209] Optionally, constructing a message index for the messages to be processed contained in the set of messages to be processed includes:

[0210] An inverted index and a prefix tree index are constructed for the messages to be processed contained in the set of messages to be processed; the inverted index and the prefix tree index are used as the message index of the messages to be processed.

[0211] Optionally, constructing a message index for the messages to be processed contained in the set of messages to be processed includes:

[0212] The construction thread processes the messages to be processed contained in the message set to be processed, and obtains at least one timeline data sequence. According to the arrangement order of the at least one timeline data sequence, the memory address prefetch instruction of each timeline data sequence is sent to the processor in sequence. When the memory address prefetch instruction of the last timeline data sequence in the at least one timeline data sequence is sent to the processor, in response to the data prefetch result of the processor, an index is constructed for each timeline data sequence in sequence, which serves as the message index of the message to be processed in the message set to be processed.

[0213] Optionally, the step of constructing a message index for the messages to be processed contained in the set of messages to be processed and writing it into the target memory space includes:

[0214] If there are local unprocessed messages in the set of unprocessed messages, obtain the historical message set of the period adjacent to the set of unprocessed messages; compare the set of unprocessed messages with the historical message set, and determine the first message to be constructed and the second message to be constructed in the unprocessed messages according to the comparison result; construct the message index of the first message to be constructed and load the message index corresponding to the local unprocessed message; write the first message to be constructed and its corresponding message index, the local unprocessed message and its corresponding message index, and the second message to be constructed into the target memory space.

[0215] Optionally, constructing the message index of the message to be constructed includes:

[0216] If the message index of the message to be built exists in the previous build cycle, obtain the first index identifier assigned to the message to be built in the previous build cycle and record it in the index identifier table; if the message index of the message to be built does not exist in the previous build cycle, call the identifier allocation function to assign a second index identifier to the message to be built and record it in the index identifier table; based on the index identifier recorded in the index identifier table and the timeline primary key corresponding to the message to be built, establish a prefix tree index and an inverted index for the message to be built.

[0217] Optionally, it also includes:

[0218] Determine the prefix tree index and inverted index corresponding to the message to be constructed; perform completion processing on the prefix tree index and the inverted index corresponding to the message to be constructed; determine the global index corresponding to the message to be constructed based on the completion processing results.

[0219] In summary, to improve index building performance, the single-message processing flow is transformed into a batch processing flow. Specifically, messages to be processed are organized into message batches and written to a message collection queue. Then, the build thread reads a message batch from the message collection queue and builds a message index for each message within it. After construction, the index is written to the target memory space, stored based on a first data structure. To reduce memory usage, the first data structure in the target memory space can be converted into a smaller second data structure and written to the data processing queue. Finally, the second data structure is read from the data processing queue and stored on the target disk, completing the persistence of the index and messages. This improves index building performance and reduces space usage during index construction.

[0220] Corresponding to the above method embodiments, this specification also provides another embodiment of an index building apparatus. Figure 8 A schematic diagram of another index building apparatus provided in one embodiment of this specification is shown. Figure 8 As shown, the device includes:

[0221] The set determination module 802 is configured to determine the set of messages to be processed in the time series database and write the set of messages to be processed into the message collection queue;

[0222] The write space module 804 is configured to, when the build thread reads the set of messages to be processed from the message collection queue, build a message index for the messages to be processed contained in the set of messages to be processed, and write it to the target memory space.

[0223] The write queue module 806 is configured to convert the first data structure of the target memory space into a second data structure and write it into the data processing queue.

[0224] The thread module 808 is configured to read the second data structure from the data processing queue and store it to the target disk;

[0225] The read set module 810 is configured to, upon receiving a message detection request, determine the set of messages to be processed based on the first data structure, the second data structure, and the target disk, as a response to the message detection request.

[0226] In an optional embodiment, the determining set module 802 is further configured to:

[0227] The reading thread reads a set number of messages to be processed from the message queue and encapsulates the set number of messages to be processed to obtain the message set to be processed; the reading thread writes the message set to be processed into the message collection queue, wherein the message collection queue processes the message set according to the order processing rules.

[0228] In an optional embodiment, the write space module 804 is further configured to:

[0229] For the messages to be processed in the set of messages to be processed, a message index is constructed, and the message point data corresponding to the messages to be processed in the set of messages to be processed is determined, wherein the message point data includes a time parameter and the parameter value of the time parameter; the message index and message point data corresponding to the messages to be processed in the set of messages to be processed are written into the target memory space.

[0230] In an optional embodiment, the write space module 804 is further configured to:

[0231] An original time series point set is constructed based on the message point data corresponding to the message to be processed; the original time series point set is subjected to time series encoding processing to obtain a target time series point set; the message index corresponding to the message to be processed and the target time series point set are written into the target memory space.

[0232] In an optional embodiment, the write queue module 806 is further configured to:

[0233] The first data structure of the target memory space is determined, wherein the first data structure is a lock-free data structure; the first asynchronous thread is invoked to convert the first data structure to obtain a second data structure; when the target memory space is detected to have reached a storage threshold, the second data structure is written into the data processing queue through the first asynchronous thread.

[0234] In an optional embodiment, the write space module 804 is further configured to:

[0235] An inverted index and a prefix tree index are constructed for the messages to be processed contained in the set of messages to be processed; the inverted index and the prefix tree index are used as the message index of the messages to be processed.

[0236] In an optional embodiment, the write space module 804 is further configured to:

[0237] The construction thread processes the messages to be processed contained in the message set to be processed, and obtains at least one timeline data sequence. According to the arrangement order of the at least one timeline data sequence, the memory address prefetch instruction of each timeline data sequence is sent to the processor in sequence. When the memory address prefetch instruction of the last timeline data sequence in the at least one timeline data sequence is sent to the processor, in response to the data prefetch result of the processor, an index is constructed for each timeline data sequence in sequence, which serves as the message index of the message to be processed in the message set to be processed.

[0238] In an optional embodiment, the write space module 804 is further configured to:

[0239] If there are local unprocessed messages in the set of unprocessed messages, obtain the historical message set of the period adjacent to the set of unprocessed messages; compare the set of unprocessed messages with the historical message set, and determine the first message to be constructed and the second message to be constructed in the unprocessed messages according to the comparison result; construct the message index of the first message to be constructed and load the message index corresponding to the local unprocessed message; write the first message to be constructed and its corresponding message index, the local unprocessed message and its corresponding message index, and the second message to be constructed into the target memory space.

[0240] In an optional embodiment, the write space module 804 is further configured to:

[0241] If the message index of the message to be built exists in the previous build cycle, obtain the first index identifier assigned to the message to be built in the previous build cycle and record it in the index identifier table; if the message index of the message to be built does not exist in the previous build cycle, call the identifier allocation function to assign a second index identifier to the message to be built and record it in the index identifier table; based on the index identifier recorded in the index identifier table and the timeline primary key corresponding to the message to be built, establish a prefix tree index and an inverted index for the message to be built.

[0242] In an optional embodiment, the write space module 804 is further configured to:

[0243] Determine the prefix tree index and inverted index corresponding to the message to be constructed; perform completion processing on the prefix tree index and the inverted index corresponding to the message to be constructed; determine the global index corresponding to the message to be constructed based on the completion processing results.

[0244] In summary, to improve index building performance, the single-message processing flow is transformed into a batch processing flow. Specifically, messages to be processed are organized into message batches and written to a message collection queue. Then, the build thread reads a message batch from the message collection queue and builds a message index for each message within it. After construction, the index is written to the target memory space, stored based on a first data structure. To reduce memory usage, the first data structure in the target memory space can be converted into a smaller second data structure and written to the data processing queue. Finally, the second data structure is read from the data processing queue and stored on the target disk, completing the persistence of the index and messages. This improves index building performance and reduces space usage during index construction.

[0245] The above is an illustrative scheme of another index building apparatus according to this embodiment. It should be noted that the technical solution of this index building apparatus and the technical solution of the index building method described above belong to the same concept. For details not described in detail in the technical solution of the index building apparatus, please refer to the description of the technical solution of the index building method described above.

[0246] Corresponding to the above method embodiments, this specification also provides another embodiment of an index construction method. Figure 9 A flowchart illustrating yet another index construction method provided in one embodiment of this specification is shown. Figure 9 As shown, the method includes:

[0247] Step S902: Determine the set of messages to be processed in the time-series database of the anomaly detection service, and write the set of messages to be processed into the message collection queue;

[0248] Step S904: When the construction thread reads the set of messages to be processed from the message collection queue, a message index is constructed for the messages to be processed contained in the set of messages to be processed, and written to the target memory space.

[0249] Step S906: Convert the first data structure of the target memory space into a second data structure and write it into the data processing queue;

[0250] Step S908: Read the second data structure from the data processing queue and store it to the target disk associated with the anomaly detection service.

[0251] Optionally, determining the set of messages to be processed in the time-series database of the anomaly detection service and writing the set of messages to be processed into the message collection queue includes:

[0252] The reading thread reads a set number of messages to be processed from the message queue and encapsulates the set number of messages to be processed to obtain the message set to be processed; the reading thread writes the message set to be processed into the message collection queue, wherein the message collection queue processes the message set according to the order processing rules.

[0253] Optionally, the step of constructing a message index for the messages to be processed contained in the set of messages to be processed and writing it into the target memory space includes:

[0254] For the messages to be processed in the set of messages to be processed, a message index is constructed, and the message point data corresponding to the messages to be processed in the set of messages to be processed is determined, wherein the message point data includes a time parameter and the parameter value of the time parameter; the message index and message point data corresponding to the messages to be processed in the set of messages to be processed are written into the target memory space.

[0255] Optionally, writing the message index and message point data corresponding to the messages to be processed in the set of messages to be processed into the target memory space includes:

[0256] An original time series point set is constructed based on the message point data corresponding to the message to be processed; the original time series point set is subjected to time series encoding processing to obtain a target time series point set; the message index corresponding to the message to be processed and the target time series point set are written into the target memory space.

[0257] Optionally, converting the first data structure of the target memory space into a second data structure and writing it into the data processing queue includes:

[0258] The first data structure of the target memory space is determined, wherein the first data structure is a lock-free data structure; the first asynchronous thread is invoked to convert the first data structure to obtain a second data structure; when the target memory space is detected to have reached a storage threshold, the second data structure is written into the data processing queue through the first asynchronous thread.

[0259] Optionally, constructing a message index for the messages to be processed contained in the set of messages to be processed includes:

[0260] An inverted index and a prefix tree index are constructed for the messages to be processed contained in the set of messages to be processed; the inverted index and the prefix tree index are used as the message index of the messages to be processed.

[0261] Optionally, constructing a message index for the messages to be processed contained in the set of messages to be processed includes:

[0262] The construction thread processes the messages to be processed contained in the message set to be processed, and obtains at least one timeline data sequence. According to the arrangement order of the at least one timeline data sequence, the memory address prefetch instruction of each timeline data sequence is sent to the processor in sequence. When the memory address prefetch instruction of the last timeline data sequence in the at least one timeline data sequence is sent to the processor, in response to the data prefetch result of the processor, an index is constructed for each timeline data sequence in sequence, which serves as the message index of the message to be processed in the message set to be processed.

[0263] Optionally, the step of constructing a message index for the messages to be processed contained in the set of messages to be processed and writing it into the target memory space includes:

[0264] If there are local unprocessed messages in the set of unprocessed messages, obtain the historical message set of the period adjacent to the set of unprocessed messages; compare the set of unprocessed messages with the historical message set, and determine the first message to be constructed and the second message to be constructed in the unprocessed messages according to the comparison result; construct the message index of the first message to be constructed and load the message index corresponding to the local unprocessed message; write the first message to be constructed and its corresponding message index, the local unprocessed message and its corresponding message index, and the second message to be constructed into the target memory space.

[0265] Optionally, constructing the message index of the message to be constructed includes:

[0266] If the message index of the message to be built exists in the previous build cycle, obtain the first index identifier assigned to the message to be built in the previous build cycle and record it in the index identifier table; if the message index of the message to be built does not exist in the previous build cycle, call the identifier allocation function to assign a second index identifier to the message to be built and record it in the index identifier table; based on the index identifier recorded in the index identifier table and the timeline primary key corresponding to the message to be built, establish a prefix tree index and an inverted index for the message to be built.

[0267] Optionally, it also includes:

[0268] Determine the prefix tree index and inverted index corresponding to the message to be constructed; perform completion processing on the prefix tree index and the inverted index corresponding to the message to be constructed; determine the global index corresponding to the message to be constructed based on the completion processing results.

[0269] In summary, to improve index building performance, the single-message processing flow is transformed into a batch processing flow. Specifically, messages to be processed are organized into message batches and written to a message collection queue. Then, the build thread reads a message batch from the message collection queue and builds a message index for each message within it. After construction, the index is written to the target memory space, stored based on a first data structure. To reduce memory usage, the first data structure in the target memory space can be converted into a smaller second data structure and written to the data processing queue. Finally, the second data structure is read from the data processing queue and stored on the target disk, completing the persistence of the index and messages. This improves index building performance and reduces space usage during index construction.

[0270] Corresponding to the above method embodiments, this specification also provides another embodiment of an index building apparatus. Figure 10 A schematic diagram of another index building apparatus provided in one embodiment of this specification is shown. Figure 10 As shown, the device includes:

[0271] The set determination module 1002 is configured to determine the set of messages to be processed in the time-series database of the anomaly detection service, and write the set of messages to be processed into the message collection queue;

[0272] The write space module 1004 is configured to, when the build thread reads the set of messages to be processed from the message collection queue, build a message index for the messages to be processed contained in the set of messages to be processed, and write it to the target memory space.

[0273] The conversion structure module 1006 is configured to convert the first data structure of the target memory space into a second data structure and write it into the data processing queue;

[0274] The storage disk module 1008 is configured to read the second data structure from the data processing queue and store it to the target disk associated with the anomaly detection service.

[0275] In an optional embodiment, the determining set module 1002 is further configured to:

[0276] The reading thread reads a set number of messages to be processed from the message queue and encapsulates the set number of messages to be processed to obtain the message set to be processed; the reading thread writes the message set to be processed into the message collection queue, wherein the message collection queue processes the message set according to the order processing rules.

[0277] In an optional embodiment, the write space module 1004 is further configured to:

[0278] For the messages to be processed in the set of messages to be processed, a message index is constructed, and the message point data corresponding to the messages to be processed in the set of messages to be processed is determined, wherein the message point data includes a time parameter and the parameter value of the time parameter; the message index and message point data corresponding to the messages to be processed in the set of messages to be processed are written into the target memory space.

[0279] In an optional embodiment, the write space module 1004 is further configured to:

[0280] An original time series point set is constructed based on the message point data corresponding to the message to be processed; the original time series point set is subjected to time series encoding processing to obtain a target time series point set; the message index corresponding to the message to be processed and the target time series point set are written into the target memory space.

[0281] In an optional embodiment, the write queue module 1006 is further configured to:

[0282] The first data structure of the target memory space is determined, wherein the first data structure is a lock-free data structure; the first asynchronous thread is invoked to convert the first data structure to obtain a second data structure; when the target memory space is detected to have reached a storage threshold, the second data structure is written into the data processing queue through the first asynchronous thread.

[0283] In an optional embodiment, the write space module 1004 is further configured to:

[0284] An inverted index and a prefix tree index are constructed for the messages to be processed contained in the set of messages to be processed; the inverted index and the prefix tree index are used as the message index of the messages to be processed.

[0285] In an optional embodiment, the write space module 1004 is further configured to:

[0286] The construction thread processes the messages to be processed contained in the message set to be processed, and obtains at least one timeline data sequence. According to the arrangement order of the at least one timeline data sequence, the memory address prefetch instruction of each timeline data sequence is sent to the processor in sequence. When the memory address prefetch instruction of the last timeline data sequence in the at least one timeline data sequence is sent to the processor, in response to the data prefetch result of the processor, an index is constructed for each timeline data sequence in sequence, which serves as the message index of the message to be processed in the message set to be processed.

[0287] In an optional embodiment, the write space module 1004 is further configured to:

[0288] If there are local unprocessed messages in the set of unprocessed messages, obtain the historical message set of the period adjacent to the set of unprocessed messages; compare the set of unprocessed messages with the historical message set, and determine the first message to be constructed and the second message to be constructed in the unprocessed messages according to the comparison result; construct the message index of the first message to be constructed and load the message index corresponding to the local unprocessed message; write the first message to be constructed and its corresponding message index, the local unprocessed message and its corresponding message index, and the second message to be constructed into the target memory space.

[0289] In an optional embodiment, the write space module 1004 is further configured to:

[0290] If the message index of the message to be built exists in the previous build cycle, obtain the first index identifier assigned to the message to be built in the previous build cycle and record it in the index identifier table; if the message index of the message to be built does not exist in the previous build cycle, call the identifier allocation function to assign a second index identifier to the message to be built and record it in the index identifier table; based on the index identifier recorded in the index identifier table and the timeline primary key corresponding to the message to be built, establish a prefix tree index and an inverted index for the message to be built.

[0291] In an optional embodiment, the write space module 1004 is further configured to:

[0292] Determine the prefix tree index and inverted index corresponding to the message to be constructed; perform completion processing on the prefix tree index and the inverted index corresponding to the message to be constructed; determine the global index corresponding to the message to be constructed based on the completion processing results.

[0293] In summary, to improve index building performance, the single-message processing flow is transformed into a batch processing flow. Specifically, messages to be processed are organized into message batches and written to a message collection queue. Then, the build thread reads a message batch from the message collection queue and builds a message index for each message within it. After construction, the index is written to the target memory space, stored based on a first data structure. To reduce memory usage, the first data structure in the target memory space can be converted into a smaller second data structure and written to the data processing queue. Finally, the second data structure is read from the data processing queue and stored on the target disk, completing the persistence of the index and messages. This improves index building performance and reduces space usage during index construction.

[0294] The above is an illustrative scheme of another index building apparatus according to this embodiment. It should be noted that the technical solution of this index building apparatus and the technical solution of the above-described index building method belong to the same concept. For details not described in detail in the technical solution of the index building apparatus, please refer to the description of the technical solution of the above-described index building method.

[0295] Corresponding to the above method embodiments, this specification also provides a message determination method embodiment. Figure 11 A flowchart illustrating a message determination method according to one embodiment of this specification is shown. Figure 11 As shown, the method includes:

[0296] Step S1102: Receive message detection requests submitted to the time series database;

[0297] Step S1104: Read the second data structure from the target disk according to the message detection request;

[0298] Step S1106: Convert the second data structure into a first data structure, and generate a message index based on the first data structure;

[0299] Step S1108: Based on the message index, perform recall processing on the first data structure, the second data structure, and the target disk, and obtain a set of messages to be processed according to the recall processing result.

[0300] The same or corresponding content in this embodiment can be found in the corresponding descriptions in the above embodiments, and will not be repeated here. When querying the set of messages to be processed, since the set of messages to be processed undergoes multiple processing steps before being written to the target disk, in order to obtain the content of the message detection request, it is necessary to retrieve the messages from the first data structure, the second data structure, and the target disk respectively, so as to obtain the set of messages to be processed as the response to the message detection request.

[0301] Corresponding to the above method embodiments, this specification also provides an embodiment of a message determination device. Figure 12 A schematic diagram of a message determination device according to one embodiment of this specification is shown. Figure 12 As shown, the device includes:

[0302] The receiving module 1202 is configured to receive message detection requests submitted to the time series database;

[0303] The reading module 1204 is configured to read a second data structure from the target disk according to the message detection request;

[0304] The generation module 1206 is configured to convert the second data structure into a first data structure and generate a message index based on the first data structure;

[0305] The recall module 1208 is configured to perform recall processing on the first data structure, the second data structure and the target disk based on the message index, and obtain a set of messages to be processed based on the recall processing result.

[0306] The above is an illustrative scheme of a message determination device according to this embodiment. It should be noted that the technical solution of this message determination device and the technical solution of the message determination method described above belong to the same concept. For details not described in detail in the technical solution of the message determination device, please refer to the description of the technical solution of the message determination method described above.

[0307] Figure 13 A structural block diagram of a computing device 1300 according to one embodiment of this specification is shown. The components of the computing device 1300 include, but are not limited to, a memory 1310 and a processor 1320. The processor 1320 is connected to the memory 1310 via a bus 1330, and a database 1350 is used to store data.

[0308] The computing device 1300 also includes an access device 1340, which enables the computing device 1300 to communicate via one or more networks 1360. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 1340 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0309] In one embodiment of this application, the aforementioned components of the computing device 1300 and Figure 13 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 13 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0310] The computing device 1300 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1300 can also be a mobile or stationary server.

[0311] The processor 1320 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described index construction method.

[0312] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the index construction method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the index construction method described above.

[0313] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described index construction method or message determination method.

[0314] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the index construction method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the index construction method described above.

[0315] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described index construction method or message determination method.

[0316] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the index construction method described above belong to the same concept. Details not described in detail in the technical solution of the computer program can be found in the description of the technical solution of the index construction method described above.

[0317] An embodiment of this specification also provides a time-series database, including an index building end and a target disk; the target disk is used to store messages and indexes, and the index building end is used to execute index building instructions, wherein when the index building end executes the index building instructions, it implements the steps of the above-described index building method or message determination method.

[0318] The above is an illustrative scheme of a time-series database according to this embodiment. It should be noted that the technical solution of this time-series database and the technical solution of the index construction method described above belong to the same concept. For details not described in detail in the technical solution of the time-series database, please refer to the description of the technical solution of the index construction method described above.

[0319] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0320] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0321] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0322] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0323] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An index construction method, comprising: Obtain the set of messages to be processed and write the set of messages to be processed into the message collection queue; When the build thread reads the set of messages to be processed from the message collection queue, it builds a message index for the messages to be processed contained in the set of messages to be processed and writes it into the target memory space; Converting a first data structure of the target memory space into a second data structure and writing it into a data processing queue includes: determining the first data structure of the target memory space, wherein the first data structure is a lock-free data structure; calling a first asynchronous thread to convert the first data structure to obtain a second data structure, wherein the second data structure refers to the data structure obtained by converting the first data structure corresponding to the target memory space, the conversion process including data reordering and deduplication; and writing the second data structure into the data processing queue through the first asynchronous thread when the target memory space is detected to have reached a storage threshold. The second data structure is read from the data processing queue and stored to the target disk.

2. The method according to claim 1, wherein obtaining the set of messages to be processed and writing the set of messages to be processed into a message collection queue comprises: The set of messages to be processed is obtained by reading a set number of messages to be processed from the message queue by the reading thread and encapsulating the set number of messages to be processed. The message collection queue is written to the message set to be processed by the reading thread, wherein the message collection queue processes the message set according to the sequential processing rules.

3. The method according to claim 1, wherein constructing a message index for the messages to be processed contained in the set of messages to be processed and writing it into the target memory space comprises: Message indexes are constructed for the messages to be processed in the set of messages to be processed, and message point data corresponding to the messages to be processed in the set of messages to be processed are determined. Write the message index and message point data corresponding to the messages to be processed in the set of messages to be processed into the target memory space.

4. The method according to claim 3, wherein writing the message index and message point data corresponding to the messages to be processed in the set of messages to be processed into the target memory space comprises: Construct an original time-series point set based on the message point data corresponding to the message to be processed; The original time series point set is subjected to time series encoding processing to obtain the target time series point set; Write the message index and target time point set corresponding to the message to be processed into the target memory space.

5. The method according to claim 1, wherein constructing a message index for the messages to be processed contained in the set of messages to be processed includes: Construct an inverted index and a prefix tree index for the messages to be processed contained in the set of messages to be processed; The inverted index and the prefix tree index are used as the message index of the message to be processed.

6. The method according to claim 1, wherein constructing a message index for the messages to be processed contained in the set of messages to be processed includes: The construction thread processes the messages to be processed contained in the message set to be processed to obtain at least one timeline data sequence. The memory address prefetch instructions for at least one timeline data sequence are sent to the processor sequentially according to the arrangement order of the timeline data sequence. When the memory address prefetch instruction of the last timeline data sequence in the at least one timeline data sequence is sent to the processor, in response to the data prefetch result of the processor, an index is sequentially constructed for the timeline data sequence as the message index of the message to be processed in the message to be processed set.

7. The method according to claim 1, wherein constructing a message index for the messages to be processed contained in the set of messages to be processed and writing it into the target memory space comprises: If there are local unprocessed messages in the set of unprocessed messages, obtain the set of historical messages from the period adjacent to the set of unprocessed messages. The set of messages to be processed is compared with the set of historical messages, and the first message to be constructed and the second message to be constructed are determined from the messages to be processed based on the comparison results. Construct the message index of the first message to be constructed, and load the message index corresponding to the local message to be processed. Write the first message to be constructed and its corresponding message index, the local message to be processed and its corresponding message index, and the second message to be constructed into the target memory space.

8. The method according to claim 7, constructing a message index for the message to be constructed, comprising: If the message index of the message to be built exists in the previous build cycle, obtain the first index identifier of the message to be built that was assigned in the previous build cycle and record it in the index identifier table; If the message index of the message to be built does not exist in the previous build cycle, the identifier allocation function is called to allocate a second index identifier for the message to be built and record it in the index identifier table; Based on the index identifiers recorded in the index identifier table and the timeline primary key corresponding to the message to be constructed, a prefix tree index and an inverted index are built for the message to be constructed.

9. The method according to claim 8, further comprising: Determine the prefix tree index and inverted index corresponding to the message to be constructed; The prefix tree index corresponding to the message to be constructed is completed, and the inverted index corresponding to the message to be constructed is also completed. The complete set index corresponding to the message to be constructed is determined based on the completion processing result.

10. An index construction method, comprising: A set of messages to be processed is determined in the time series database, and the set of messages to be processed is written into the message collection queue; When the build thread reads the set of messages to be processed from the message collection queue, it builds a message index for the messages to be processed contained in the set of messages to be processed and writes it into the target memory space; Converting a first data structure of the target memory space into a second data structure and writing it into a data processing queue includes: determining the first data structure of the target memory space, wherein the first data structure is a lock-free data structure; calling a first asynchronous thread to convert the first data structure to obtain a second data structure, wherein the second data structure refers to the data structure obtained by converting the first data structure corresponding to the target memory space, the conversion process including data reordering and deduplication; and writing the second data structure into the data processing queue through the first asynchronous thread when the target memory space is detected to have reached a storage threshold. Read the second data structure from the data processing queue and store it to the target disk; Upon receiving a message detection request, the set of messages to be processed is determined based on the first data structure, the second data structure, and the target disk, and is used as a response to the message detection request.

11. An index construction method, comprising: The set of messages to be processed is determined in the time-series database of the anomaly detection service, and the set of messages to be processed is written into the message collection queue; When the build thread reads the set of messages to be processed from the message collection queue, it builds a message index for the messages to be processed contained in the set of messages to be processed and writes it into the target memory space; Converting a first data structure of the target memory space into a second data structure and writing it into a data processing queue includes: determining the first data structure of the target memory space, wherein the first data structure is a lock-free data structure; calling a first asynchronous thread to convert the first data structure to obtain a second data structure, wherein the second data structure refers to the data structure obtained by converting the first data structure corresponding to the target memory space, the conversion process including data reordering and deduplication; and writing the second data structure into the data processing queue through the first asynchronous thread when the target memory space is detected to have reached a storage threshold. The second data structure is read from the data processing queue and stored in the target disk associated with the anomaly detection service.

12. A message determination method, comprising: Receive message detection requests submitted to the time-series database; Read the second data structure from the target disk according to the message detection request; The second data structure is converted into the first data structure, and a message index is generated based on the first data structure. The second data structure is obtained by calling the first asynchronous thread to convert the first data structure. The first data structure is a lock-free data structure. The second data structure refers to the data structure obtained by converting the first data structure corresponding to the target memory space. The conversion process includes data reordering and deduplication. Based on the message index, a recall process is performed on the first data structure, the second data structure, and the target disk, and a set of messages to be processed is obtained based on the recall process results.

13. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 12.

Images