Data indexing method, system and related device

Abstract

The invention provides a data indexing method and system and a related device, and relates to the field of cloud computing. The method is performed by a data indexing system, the system comprising a first indexer and a first merger, the first merger and the first indexer being located on different compute instances. The method comprises the following steps: a first indexer generates a plurality of first segment files, packs the plurality of first segment files into a first data packet, and sends the first data packet to a first combiner; the first combiner combines the plurality of first segment files in the first data packet to generate a second segment file, and sends the second segment file to the first indexer in the form of the data packet; and the first indexer stores the second segment files and deletes the plurality of first segment files packaged into the first data packet from the disk. According to the embodiment of the invention, the merge operation is independent, and the merge operation is executed by the independent merger, so that the problem that an indexer delays in processing a write request and a read request due to processing of the merge operation can be avoided, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of cloud computing, and in particular to a data indexing method, system and related apparatus. Background Technology

[0002] Log structured merge tree (LSM) is a data storage structure designed to optimize write performance. It stores the data to be written in a MemTable in memory first, and then periodically merges this data into an SSTable on disk.

[0003] The merge operation in the LSM model combines multiple small SSTable files into a larger SSTable file. In practice, this typically involves first reading the multiple small SSTable files from the disk, merging them into a larger SSTable file, and storing it on the hard drive. Subsequently, the larger SSTable file is read from the disk again, merged into an even larger SSTable file, and stored on the hard drive.

[0004] In actual operation, when performing a merge operation, the system needs to consume a large amount of central processing unit (CPU) resources, memory resources, and disk input / output (I / O) resources to merge data. This will cause the system to write data at a significantly slower speed during the merge operation, with a large number of write requests being blocked or delayed. When users read data, the response time for read requests increases. Due to the increased utilization of CPU, memory, and disk I / O resources, the overall performance of the system deteriorates, affecting the user experience. Summary of the Invention

[0005] This application provides a data indexing method, system, and related apparatus that separates the merge operation and has it performed by an independent merger. This avoids the problem of latency in the indexer's handling of write and read requests due to the processing of merge operations, thereby improving the user experience.

[0006] Firstly, this application provides a data indexing method, which is executed by a data indexing system. The data indexing system includes a first indexer and a first merger, the first merger and the first indexer residing on different computing instances, including bare metal, a host machine, a virtual machine, and a container. The method includes: the first indexer generating multiple first segment files, each first segment file generated by the first indexer based on one or more received files, and packaging the multiple first segment files into a first data packet, which is then sent to the first merger. The first merger merges the multiple first segment files in the first data packet to generate a second segment file, and sends the second segment file to the first indexer in the form of a data packet. The first indexer saves the second segment file and deletes the multiple first segment files packaged into the first data packet from the disk.

[0007] In traditional methods, the first indexer stores multiple received files to disk. Subsequently, it reads these files from disk again and performs a merge operation, merging smaller files into medium-sized files, which are then stored back to disk. This process is repeated, with the medium-sized files being read again and merged into a larger file, which is also stored back to disk. However, the merge operation consumes significant CPU, memory, and disk I / O resources, leading to latency in processing write and read requests during the merge process, resulting in a poor user experience. This proposed solution separates the merge operation into a dedicated merger, avoiding the latency in processing write and read requests caused by the merge operation and improving the user experience.

[0008] Furthermore, traditional methods require the indexer to write and read files from disk multiple times, resulting in severe read and write amplification. In this application, the indexer performs only two write operations and one read operation: two write operations (writing multiple first-segment files to disk and writing the merged second-segment file to disk) and one read operation (reading multiple first-segment files from disk). Compared to traditional methods, this application significantly alleviates the problems of severe write and read amplification.

[0009] Based on the first aspect, in a possible implementation, after the first indexer stores the second file segment to disk, it updates the metadata, updating the metadata of the multiple first file segments packaged in the first data packet to the metadata of the second file segments. The metadata includes: a mapping relationship between the identifier of each second file segment, the storage path of each second file segment, and the time range involved in the files included in each second file segment. This is so that when reading data, the required files or data can be read based on the metadata.

[0010] Based on the first aspect, in possible implementations, when the first indexer merges multiple files into a first segment file, it can merge the multiple files according to a first time interval threshold, for example, merging files received within one minute into the first segment file every minute. Alternatively, it can merge the multiple files according to a first quantity threshold, for example, performing a merge every three or five files received. Alternatively, it can merge according to whichever condition—the first time interval threshold or the first quantity threshold—is reached first.

[0011] Based on the first aspect, in possible implementations, when the first indexer packages multiple first segment files into a data packet, it can do so according to a second time interval threshold. For example, it can package multiple first segment files generated within 3 minutes into a data packet every 3 minutes. Alternatively, it can be based on a second quantity threshold; for example, it can perform packaging once every 3 first segment files. Alternatively, it can be that whichever condition—the second time interval threshold or the second quantity threshold—is reached first, that method will be used for packaging.

[0012] Secondly, this application also provides a data indexing method, which is executed by a data indexing system. In this method, the data indexing system includes at least one additional storage node, which includes a first storage node. Unlike the method provided in the first aspect, in this method, the first merger sends the second file segment to the first storage node in the form of a data packet, instead of the first indexer. The first storage node stores the second file segment.

[0013] In this application, the merge operation is separated and performed by a dedicated merger. This avoids the latency issues that arise when the indexer processes write and read requests due to the merge operation, thus improving the user experience. Furthermore, this method involves one write operation and one read operation by the indexer: one write operation writes multiple first-segment files to disk, and one read operation reads multiple first-segment files from disk and sends them to the first merger. Compared to traditional methods, this application's solution alleviates the problems of severe write amplification and read amplification.

[0014] In addition, in this embodiment, the data adopts a multi-level storage format. The first indexer stores the most recently received files, which have not yet met the conditions for being packaged into the first merger and are temporarily stored in the first indexer. The merged files have already been stored in the storage nodes. This reduces the storage resource usage in the first indexer, allowing the resources in the indexer to better serve the user.

[0015] Based on the second aspect, in a possible implementation, after the first merger sends the second file segment to the first storage node in the form of a data packet, the first indexer deletes the multiple first file segments packaged in the first data packet from the disk where the first indexer is located.

[0016] Based on the second aspect, in a possible implementation, after receiving the second file segment, the first storage node saves or updates the metadata of the second file segment. This facilitates the subsequent retrieval of the required files or data from the disk based on the metadata.

[0017] Based on the second aspect, in a possible implementation, after the disk capacity occupied in the first storage node meets the set conditions, the first storage node stores some or all of the data in the object storage service OBS bucket, deletes the data stored in the OBS bucket from the disk where the first storage node is located, and updates the metadata.

[0018] Based on the second aspect, in a possible implementation, the data indexing system also includes a controller. The controller sends read requests to a first indexer and a first storage node. Upon receiving the read request, the first indexer retrieves first target data based on the user ID and the time range of the data to be read in the read request. The first storage node retrieves second target data based on the user ID and the time range of the data to be read in the read request. The first indexer sends the first target data to the controller, and the first storage node sends the second target data to the controller. The controller aggregates the received first and second target data to obtain an aggregated result, and sends the aggregated result to the user.

[0019] Based on the second aspect, in possible implementations, when the first indexer merges multiple files into a first segment file, it can merge the multiple files according to a first time interval threshold, for example, merging files received within one minute into the first segment file every minute. Alternatively, it can merge the multiple files according to a first quantity threshold, for example, performing a merge every three or five files received. Or, the merge can proceed according to whichever condition—the first time interval threshold or the first quantity threshold—is reached first.

[0020] Based on the second aspect, in possible implementations, when the first indexer packages multiple first segment files into a single data packet, it can do so according to a second time interval threshold. For example, it could package multiple first segment files generated within 3 minutes into a single data packet every 3 minutes. Alternatively, it could be based on a second quantity threshold; for example, packaging could be performed every time the number of first segment files reaches 3. Alternatively, the packaging method could be determined by whichever condition, the second time interval threshold or the second quantity threshold, is reached first.

[0021] Thirdly, this application provides a data indexing system, comprising at least one indexer and at least one merger, wherein the at least one indexer is communicatively connected to the at least one merger, and any indexer and any merger are located on different computing instances, the computing instances including any one of bare metal, host machine, virtual machine, and container, the at least one indexer including a first indexer, and the at least one merger including a first merger, the system comprising:

[0022] The first indexer is used to: generate multiple first segment files according to preset rules, each first segment file including a first original file and a first index file. The first original file is obtained by merging one or more files received by the first indexer, and the first index file is obtained by arranging the index information of one or more files in the order of their source file generation time, wherein the index information of each file is extracted from each file; package the multiple first segment files that meet the preset conditions into a first data packet; and send the first data packet to the first merger.

[0023] The first merger is used to: merge multiple first segment files in the first data packet to obtain a second data packet, the second data packet including a second segment file, the second segment file including a second original file and a second index file, the second original file being obtained by merging multiple first original files included in the first data packet, and the second index file being obtained by sequentially arranging the index information in the multiple first index files included in the first data packet according to the generation time order of the source files of each index information; and send the second data packet to the first indexer.

[0024] The first indexer is also used to: store the second segment file in the second data packet to the disk, and delete the multiple first segment files packaged in the first data packet from the disk.

[0025] Based on the third aspect, in a possible implementation, the first indexer is also used to: update the metadata of multiple first segment files packaged into the first data packet to the metadata of the second segment files.

[0026] Based on the third aspect, in possible implementations, the first indexer is used to generate multiple first segment files according to preset rules in any of the following cases:

[0027] 1) Whenever the set first time interval threshold is reached, the first indexer generates a first segment file based on one or more files received within the first time interval threshold range;

[0028] 2) Whenever the number of files received by the first indexer reaches a first quantity threshold, the first indexer generates a first segment file based on the files within the first quantity threshold range;

[0029] 3) The first indexer is set with condition a) the first time interval threshold is reached and condition b) the number of files received by the first indexer reaches the first quantity threshold. Whenever condition a) occurs first, the first indexer generates a first segment file based on one or more files received within the first time interval threshold. Whenever condition b) occurs first, the first indexer generates a first segment file based on files within the first quantity threshold.

[0030] Based on the third aspect, in possible implementations, the first indexer is used to package multiple first segment files that meet preset conditions into a first data packet in any of the following cases:

[0031] 1) Whenever the set second time interval threshold is reached, the first indexer packages multiple first segment files generated within the second time interval threshold range into a first data packet;

[0032] 2) Whenever the number of first segment files generated by the first indexer reaches the second number threshold, the first indexer packages multiple first segment files within the range of the second number threshold into a first data packet;

[0033] 3) The first indexer is set with condition c) the second time interval threshold is reached and condition d) the number of first segment files generated by the first indexer reaches the second quantity threshold. Whenever condition c) occurs first, the first indexer packages the multiple first segment files generated within the second time interval threshold into a first data packet. Whenever condition d) occurs first, the first indexer packages the multiple first segment files within the second quantity threshold into a first data packet.

[0034] Fourthly, this application provides a data indexing system, which includes at least one indexer, at least one merger, and at least one storage node. The at least one merger is communicatively connected to both the at least one indexer and the at least one storage node. Any indexer, any merger, and any storage node reside on different computing instances, including any of bare metal, host machine, virtual machine, and container. The at least one indexer includes a first indexer, the at least one merger includes a first merger, and the at least one storage node includes a first storage node.

[0035] The first indexer is used to: generate multiple first segment files according to preset rules, each first segment file including a first original file and a first index file, the first original file being obtained by merging one or more files received by the first indexer, and the first index file being obtained by arranging the index information of one or more files in the order of their source file generation time, wherein the index information of each file is extracted from each file; package the multiple first segment files that meet the first preset conditions into a first data packet; and send the first data packet to the first merger.

[0036] The first merger is used to: merge multiple first segment files in the first data packet to obtain a second data packet, the second data packet including a second segment file, the second segment file including a second original file and a second index file, the second original file being obtained by merging multiple first original files included in the first data packet, and the second index file being obtained by sequentially arranging the index information in the multiple first index files included in the first data packet according to the generation time order of the source files of each index information; and send the second data packet to the first storage node.

[0037] The first storage node is used to store the second file segment in the second data packet.

[0038] Based on the fourth aspect, in a possible implementation, the first indexer is used to: delete multiple first segment files packaged into the first data packet from the disk where the first indexer resides.

[0039] Based on the fourth aspect, in a possible implementation, the first storage node is used to: store the metadata of the second segment of the file.

[0040] Based on the fourth aspect, in a possible implementation, the first storage node is used to: store some or all of the data in the object storage service OBS bucket when the disk capacity occupied by the first storage node meets the second preset condition, and delete the data stored in the OBS bucket from the disk where the first storage node is located; and update metadata.

[0041] Based on the fourth aspect, in possible implementations, the system also includes a controller, which is communicatively connected to at least one indexer and at least one storage node, respectively.

[0042] The system also includes:

[0043] The first indexer and the first storage node are respectively used for: receiving a read request, the read request carrying a user identity ID and the time range of the data to be read;

[0044] The first indexer is used to: obtain the first target data according to the user ID and the time range of the data to be read through a preset mapping relationship, and send the first target data to the controller; the mapping relationship includes the mapping relationship between the data located on the disk of the first indexer and the user ID and the data generation time;

[0045] The first storage node is used to: obtain the second target data by querying metadata based on the user ID and the time range of the data to be read, and send the second target data to the controller;

[0046] The controller is used to: aggregate the first target data and the second target data to obtain the aggregated result; and send the aggregated result to the user.

[0047] Based on the fourth aspect, in possible implementations, the first indexer is used to generate multiple first segment files according to preset rules in any of the following cases:

[0048] 1) Whenever the set first time interval threshold is reached, the first indexer generates a first segment file based on one or more files received within the first time interval threshold range;

[0049] 2) Whenever the number of files received by the first indexer reaches a first quantity threshold, the first indexer generates a first segment file based on the files within the first quantity threshold range;

[0050] 3) The first indexer is set with condition a) the first time interval threshold is reached and condition b) the number of files received by the first indexer reaches the first quantity threshold. Whenever condition a) occurs first, the first indexer generates a first segment file based on one or more files received within the first time interval threshold. Whenever condition b) occurs first, the first indexer generates a first segment file based on files within the first quantity threshold.

[0051] Based on the fourth aspect, in possible implementations, the first indexer is used to package multiple first segment files that satisfy the first preset condition into a first data packet in any of the following cases:

[0052] 1) Whenever the set second time interval threshold is reached, the first indexer packages multiple first segment files generated within the second time interval threshold range into a first data packet;

[0053] 2) Whenever the number of first segment files generated by the first indexer reaches the second number threshold, the first indexer packages multiple first segment files within the range of the second number threshold into a first data packet;

[0054] 3) The first indexer is set with condition c) the second time interval threshold is reached and condition d) the number of first segment files generated by the first indexer reaches the second quantity threshold. Whenever condition c) occurs first, the first indexer packages the multiple first segment files generated within the second time interval threshold into a first data packet. Whenever condition d) occurs first, the first indexer packages the multiple first segment files within the second quantity threshold into a first data packet.

[0055] Fifthly, this application provides a computing device cluster including at least one computing device, the at least one computing device including a memory and a processor, the processor of the at least one computing device being configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the methods described in the first aspect and any possible implementation thereof, or to cause the computing device cluster to perform the methods described in the second aspect and any possible implementation thereof.

[0056] In a sixth aspect, this application provides a computer storage medium including program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to implement the method described in the first aspect and any possible implementation thereof, or cause the cluster of computing devices to implement the method described in the second aspect and any possible implementation thereof.

[0057] In a seventh aspect, this application provides a computer program product, including program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to implement the method described in the first aspect and any possible implementation thereof, or cause the cluster of computing devices to implement the method described in the second aspect and any possible implementation thereof. Attached Figure Description

[0058] Figure 1 A schematic diagram of a system architecture is provided for this application;

[0059] Figure 2A flowchart illustrating a data indexing method provided in this application;

[0060] Figure 3A A schematic diagram of the structure of a first data packet provided in this application;

[0061] Figure 3B A schematic diagram of the structure of a first-segment document provided in this application;

[0062] Figure 4 An example scenario diagram provided for this application;

[0063] Figure 5 Another system architecture diagram provided in this application;

[0064] Figure 6 A flowchart illustrating yet another data indexing method provided in this application;

[0065] Figure 7 A schematic diagram of a system structure is provided for this application;

[0066] Figure 8 Another system structure diagram provided in this application;

[0067] Figure 9 A schematic diagram of a computing device structure is provided for this application;

[0068] Figure 10 This application provides a schematic diagram of a computing device cluster structure;

[0069] Figure 11 This is a schematic diagram of another computing device cluster structure provided in this application. Detailed Implementation

[0070] This application provides a system, such as Figure 1 As shown, Figure 1 This application provides a schematic diagram of a system architecture. The system includes at least one indexer, at least one merger, and a controller.

[0071] The controller manages and schedules at least one indexer and at least one merger in the system. For example, the controller can increase or decrease the number of indexers and mergers in the system. For instance, if initially there were two indexers and two mergers handling user 1's traffic, but user 1's traffic increased, making the two indexers and two mergers insufficient, the controller can add more indexers and mergers to allocate to user 1, or schedule idle indexers and mergers for user 1's use.

[0072] The controller is also used to record the status data of each indexer and each merger in the system. For example, the controller records how many indexers and mergers are in the system, the status data of each indexer, the status data of each merger, the resource utilization and memory utilization of each indexer, and the resource utilization and memory utilization of each merger. The status data includes idle status and busy status.

[0073] The controller is also used to schedule traffic within the system. For example, when the controller receives traffic, it determines which indexer to send that traffic to. Or, when one indexer is busy and another is idle, the controller schedules traffic from the busy indexer to the idle one. Similarly, when one merger is busy and another is idle, the controller schedules traffic from the busy merger to the idle one.

[0074] The indexer receives write requests from the controller, each request carrying at least one file, and generates a first segment file based on one or more files. The indexer also packages multiple first segment files into a group and sends it as a data packet to the merger. The merger merges the multiple first segment files from the received data packet to obtain a second segment file, which is then sent as a data packet to the indexer. The indexer also stores the received second segment file to disk and updates its metadata.

[0075] The indexer is also used to receive read requests sent by the controller, read data according to the read requests, and return the read data to the user through the controller.

[0076] In this system architecture, any indexer and any merger reside on different compute instances, which can be bare metal, a host machine, a virtual machine, or a container. The controller can reside on a separate compute instance, or on the same compute instance as one of the indexers, or on the same compute instance as one of the mergers.

[0077] This system architecture is an example provided in this application; the system involved in this application can also have other architectures. Figure 1 The system architecture shown does not constitute a limitation on the solution proposed in this application.

[0078] based on Figure 1 As shown in the system architecture, this application provides a data indexing method. Figure 2 As shown, Figure 2 This application provides a flowchart illustrating a data indexing method, which can be implemented by... Figure 1The system execution includes at least one indexer and at least one merger. For ease of description, the method embodiment of this application is described using a first indexer and a first merger as examples. The first indexer is any one of the at least one indexer, and the first merger is any one of the at least one merger. The method includes, but is not limited to, the following description.

[0079] S110. The first indexer generates multiple first segment files according to preset rules. Each first segment file includes a first original file and a first index file.

[0080] The first indexer receives one or more write requests, each carrying at least one file. The first indexer generates a first segment file from the one or more files according to preset rules. The first segment file includes a first original file and a first index file. The one or more files can be carried in a single write request or multiple write requests. The first original file is obtained by merging the one or more files, and the first index file is obtained by sequentially arranging the index information of each of the one or more files. The index information of each file is extracted from each file, and the order of arrangement is based on the chronological order in which the one or more files were generated.

[0081] The first indexer generates the first file segment according to a preset rule. In one example, the preset rule is: whenever a set first time interval threshold is reached, the first indexer generates a first file segment based on one or more files received within that first time interval threshold. For example, if the first time interval threshold is set to 1 minute, the first indexer generates a first file segment based on one or more files received within that minute every 1 minute, and so on, generating multiple first file segments.

[0082] In another example, the preset rule is: whenever the number of files received by the first indexer reaches a first quantity threshold, the first indexer generates a first segment file based on the files within the first quantity threshold range. For example, if the first quantity threshold is set to 2, then whenever the first indexer receives 2 files, the first indexer generates a first segment file from the 2 received files. When the number of files received by the first indexer reaches 2 again, the first indexer generates a first segment file from these 2 files again... and so on, the first indexer generates multiple first segment files.

[0083] In another example, the first indexer sets two conditions: condition a) a first time interval threshold is reached; condition b) the number of files received by the first indexer reaches a first quantity threshold. The first indexer generates a first file segment if either condition is met. Specifically, if condition a) is met first, the first indexer generates a first file segment based on files received within the first time interval threshold range; if condition b) is met first, the first indexer generates a first file segment based on files within the first quantity threshold range. The preset rule can also take other forms, which are not limited in this application.

[0084] S120, The first indexer packages multiple first segment files that meet the preset conditions into a first data packet.

[0085] The first indexer packages multiple first-segment files that meet preset conditions into a single data packet. In one example, the preset condition is: whenever a set second time interval threshold is reached, the first indexer packages multiple first-segment files generated within that second time interval threshold into a single data packet. For example, if the second time interval threshold is set to 3 minutes, every 3 minutes, the first indexer packages multiple first-segment files generated within that 3 minutes into a single data packet. When the 3-minute interval is reached again, the first indexer again packages multiple first-segment files generated within that 3 minutes into a single data packet.

[0086] In another example, the preset condition is: whenever the number of first segment files generated by the first indexer reaches a second quantity threshold, the first indexer packages multiple first segment files within the second quantity threshold range into a single data packet. For example, if the second quantity threshold is set to 3, then whenever the number of first segment files generated by the first indexer reaches 3, the first indexer packages the 3 generated first segment files into a single data packet. When the number of first segment files generated by the first indexer again reaches 3, the first indexer again packages these 3 first segment files into a single data packet.

[0087] In another example, the first indexer sets two conditions: condition c) the second time interval threshold is reached, and condition d) the number of first segment files generated by the first indexer reaches a second quantity threshold. When either condition is met first, the first indexer performs packaging. Specifically, if condition c) occurs first, the first indexer packages multiple first segment files generated within the second time interval threshold range into a first data packet; if condition d) occurs first, the first indexer packages multiple first segment files within the second quantity threshold range into a first data packet. The preset conditions can also take other forms, which are not limited in this application.

[0088] See Figure 3A and Figure 3BAs shown, Figure 3A This application provides a schematic diagram of the structure of a first data packet. Figure 3B This is a schematic diagram of the structure of a first-segment document provided for this application. For example... Figure 3A As shown, the first data packet includes multiple first segment files, and the structure of each first segment file is as follows: Figure 3B As shown. The first file segment includes a first original file and a first index file. The first original file is generated by merging multiple files, and the first index file is obtained by extracting index information from each file in the multiple files and arranging the index information of each file in the order of their creation time. Figure 3B In the diagram, the first original file is obtained by merging the doc1 and doc2 files, and the first index file is obtained by extracting the index information from the doc1 and doc2 files respectively and arranging the index information of the two files in the order of their generation time. Figure 3B The first original file is obtained by merging two files. The first original file can also be obtained by merging more files, which is not limited in this application.

[0089] S130, the first indexer sends the first data packet to the first merger, and correspondingly, the first merger receives the first data packet sent by the first indexer.

[0090] S140. The first merger merges multiple first segment files in the first data packet to obtain a second data packet, which includes the second segment files.

[0091] The first merger receives the first data packet, merges multiple first segment files within the first data packet to obtain a second segment file, and packages the second segment file into a second data packet for transmission to the first indexer. The second segment file includes a second original file and a second index file. The second original file is obtained by merging multiple first original files included in the first data packet, and the second index file is obtained by arranging the index information from the multiple first index files included in the first data packet according to the chronological order in which the source files of each index information were generated. It can be understood that the first data packet includes multiple first segment files, each first segment file includes a first index file, and each first index file includes index information for one or more files, where the one or more files can be carried in a single write request or multiple write requests. The second index file is obtained by arranging the index information included in the multiple first index files in the first data packet according to the chronological order in which the source files of each index information were generated.

[0092] It can be understood that the first indexer and the first merger transmit data in the form of data packets. Therefore, after obtaining the second file segment, the first merger merges the second file segment to generate a second data packet. The second data packet is then sent to the first indexer.

[0093] S150, the first merger sends the second data packet to the first indexer, and correspondingly, the first indexer receives the second data packet sent by the first merger.

[0094] S160, the first indexer stores the second segment file in the second data packet to the disk, and deletes the multiple first segment files packaged in the first data packet from the disk.

[0095] The first indexer stores the second segment file in the second data packet to the disk and deletes multiple first segment files packaged in the first data packet from the disk to avoid duplicate file storage and save storage resources.

[0096] Optionally, the first indexer also stores metadata, which includes a mapping relationship between the identifier of each second segment file, the storage path of each second segment file, and the time range involved in the files included in each second segment file. After the first indexer stores the second segment files to the disk, the method further includes: the first indexer updating the metadata, updating the metadata of the multiple first segment files packaged in the first data packet to the metadata of the second segment files.

[0097] Optionally, the system also includes a controller that receives write requests from users and forwards them to at least one indexer. Each indexer and each merger processes the data according to the steps described above, ultimately storing the generated second segment file on disk. The second segment file includes a second original file and a second index file. The controller also receives read requests from users and forwards them to at least one indexer. The read requests carry the time range of the data to be read. Each indexer reads the data to be read from the disk by querying metadata and then sends the data to be read to the user through the controller.

[0098] See Figure 4 , Figure 4This is a scenario example diagram provided for this application. ① indicates that the first indexer generates multiple first segment files, with Seg11, Seg12, and Seg13 each representing a first segment file. The first indexer groups Seg11, Seg12, and Seg13 together and packages them into a data packet. ② indicates that the first indexer data packet is sent to the first merger. After receiving the data packet, the first merger merges Seg11, Seg12, and Seg13 into a segment file Seg(11-13), as shown in ③. The first merger sends the segment file Seg(11-13) to the first indexer in the form of a data packet, as shown in ④. When the second time interval threshold set in the first indexer is reached, or when the number of first segment files generated by the first indexer reaches a certain number threshold, the first indexer packages the first segment files that meet the preset conditions again, for example, packaging Seg21, Seg22, and Seg23 into a data packet, and then sending the data packet to the first merger. The first merger merges Seg21, Seg22 and Seg23 in the data packet into a segment file Seg(21-23) and sends the segment file Seg(21-23) to the first indexer.

[0099] In this embodiment of the method, the indexer needs to perform two write operations and one read operation. The two write operations are as follows: First, the indexer generates the first file segment from the file carried in the received write request and stores it on disk; second, the indexer receives the second data packet sent by the merger and writes the second file segment from the second data packet to disk. The one read operation occurs when the indexer reads multiple first file segments from the disk, packages them, and sends them to the merger.

[0100] As can be seen, in this application's solution, the merge operation is separated and performed by an independent merger. This avoids the latency issue in the indexer's handling of write and read requests caused by processing the merge operation, thus improving the user experience. In this solution, the indexer only needs to perform two write operations and one read operation: the two write operations are writing multiple first-segment files to disk and writing the merged second-segment file to disk; the one read operation is reading multiple first-segment files from disk. Compared to traditional methods, this application's solution alleviates the problems of severe write amplification and severe read amplification.

[0101] This application also provides a system, such as Figure 5 As shown, Figure 5This is another system architecture diagram provided for this application. The system includes a controller, a write proxy server, a read proxy server, at least one indexer, at least one merger, at least one storage node, and an object storage service (OBS) bucket.

[0102] In this system, the controller's functions are... Figure 1 The controllers in the system function similarly. They can be used to manage and schedule at least one indexer and at least one merger in the system, for example, by increasing or decreasing the number of indexers or mergers. They are also used to record the status data of each indexer and each merger. Furthermore, they can be used to schedule traffic within the system.

[0103] In addition to the functions mentioned above, the controller is also used to determine whether a user's traffic is a write request or a read request, and to route write requests and read requests accordingly. Specifically, write requests are sent to a write proxy server, and read requests are sent to a read proxy server.

[0104] Optionally, the controller can use a load balancing strategy to determine which write requests are assigned to which indexers based on the status data of each indexer and each merger. The write request proxy server executes the controller's allocation strategy, routing the corresponding write requests to the appropriate indexers.

[0105] The indexer receives write requests from the controller, each request carrying at least one file, and generates a first segment file based on one or more files. The indexer also packages multiple first segment files into a group and sends it as a data packet to the merger. The merger combines the multiple first segment files in the received data packet to obtain a second segment file, which is then sent as a data packet to the storage node. The storage node stores the received second segment file on disk and updates its metadata.

[0106] The storage node is also used to dump some or all of the data on the disk to the OBS bucket when a second preset condition is met. The second preset condition may be, for example, that the storage node's occupied capacity reaches a threshold or that the storage node's capacity is fully occupied.

[0107] The read proxy server sends read requests to the storage nodes and indexer. The storage nodes and indexer process the read requests and return the read data to the read proxy server. The read proxy server then aggregates the data returned by the storage nodes and indexer and sends the aggregated result to the user.

[0108] Optionally, write request proxy servers and read request proxy servers may not exist. In the absence of write request proxy servers and read request proxy servers, their functionality is performed by the controller.

[0109] In this system, any indexer, any merger, and any storage node reside on different compute instances, which can be any of the following: bare metal, host machine, virtual machine, and container.

[0110] Figure 5 The system architecture shown is merely an example. In practical applications, the system may include more or fewer modules or devices. Figure 5 This does not constitute a limitation on this application.

[0111] based on Figure 5 In addition to the system architecture shown, this application also provides a data indexing method, which can be implemented by... Figure 5 The system execution is shown. Figure 6 This is a flowchart illustrating yet another data indexing method provided in this application, which includes, but is not limited to, the following description.

[0112] This method embodiment is described using a first indexer, a first merger, and a first storage node as examples. The first indexer is any one of at least one indexer, the first merger is any one of at least one merger, and the first storage node is any one of at least one storage node. Other indexers, mergers, and storage nodes in the system are similar to the first indexer, first merger, and first storage node, and they have the same functions.

[0113] S210. The first indexer generates multiple first segment files according to preset rules. Each first segment file includes a first original file and a first index file.

[0114] S220, the first indexer packages multiple first segment files that meet the first preset conditions into a first data packet.

[0115] S230, the first indexer sends the first data packet to the first merger, and correspondingly, the first merger receives the first data packet sent by the first indexer.

[0116] S240, The first merger merges multiple first segment files in the first data packet to obtain a second data packet, which includes the second segment files.

[0117] S250, the first merger sends the second data packet to the first storage node, and correspondingly, the first storage node receives the second data packet sent by the first merger.

[0118] Steps S210 to S240 and Figure 2 Steps S110 to S140 in the method embodiments are similar or the same, and will not be repeated here for the sake of brevity.

[0119] and Figure 2 The difference between this method embodiment and the previous one is that in this embodiment, the first merger sends the second data packet to the first storage node, not the first indexer. The first storage node is independent of the first indexer and is dedicated to storing data.

[0120] S260, The first storage node stores the second segment of the file in the second data packet.

[0121] After receiving the second data packet, the first storage node stores the second file segment from the second data packet onto its disk. Optionally, after the first merger sends the second data packet to the first storage node, it sends a response message to the first indexer. Upon receiving the response message, the first indexer deletes the multiple first file segments packaged in the first data packet from the disk where the first indexer resides.

[0122] After receiving the second data packet, the first storage node saves the metadata of the second file segment in the second data packet. The metadata includes the mapping relationship between the identifier of each second file segment, the storage path of each second file segment, the time range involved in the files included in each second file segment, and the user identity ID to which each file segment belongs.

[0123] Optionally, if the disk capacity of the first storage node meets a second preset condition, the first storage node will store some or all of the data in the OBS bucket. The second preset condition could be, for example, the disk capacity of the first storage node reaching a threshold or the disk capacity of the first storage node being full. The first storage node is also used to delete data stored in the OBS bucket from the disk where the first storage node resides and update the metadata.

[0124] Optionally, when determining which write requests are assigned to which indexers, the controller also records which write requests are assigned to which indexers. Specifically, the write request carries a user identity ID, one or more files, and the creation time of each file within those files. The controller records the correspondence between each indexer and the user ID of the file it processes.

[0125] This method embodiment further includes: the controller receiving a read request, the read request carrying a user ID and the time range of the data to be read. Based on the user ID in the read request, the controller determines the indexer that has processed requests for that user as the first indexer, and sends the received read request to the first indexer. The controller is also used to send the read request to each storage node. After receiving the read request, each storage node determines whether it has processed a request for that user ID based on its metadata. If it has not processed it, it does not perform any operation; if it has processed it, it executes the read request.

[0126] Assuming the first indexer and the first storage node have processed requests for the user ID, upon receiving a read request, the first indexer retrieves the first target data based on the user ID and the time range of the data to be read, using a preset mapping relationship. This preset mapping relationship, stored in the first indexer, refers to the metadata of data not yet stored on at least one storage node. The mapping relationship includes the mapping between data located on the first indexer's disk and the user ID and data generation time. The first indexer then sends the first target data to the read proxy server.

[0127] After receiving a read request, the first storage node retrieves the second target data by querying metadata based on the user ID and time range of the data to be read in the read request, and then sends the second target data to the read broker server. The second target data may be located on the disk where the first storage node resides, or it may be located in an OBS bucket. The first storage node retrieves the second target data from the disk or from the OBS bucket by querying metadata.

[0128] The read proxy server receives the first target data sent by the first indexer and the second target data sent by the first storage node, summarizes the first target data and the second target data, obtains the summary result, and sends the summary result to the user.

[0129] Optionally, in the absence of a read proxy server in the system architecture, the controller directly distributes read requests to the first indexer and multiple storage nodes. If the first storage node determines from metadata that it has processed the user's request, it retrieves the second target data by querying the metadata. The first storage node sends the second target data to the controller, and the first indexer sends the first target data to the controller. The controller aggregates the first and second target data to obtain the aggregated result, which is then sent to the user.

[0130] In this embodiment of the method, the indexer only needs to perform one write operation and one read operation. The write operation involves generating the first segment file from the file carried in the received write request and writing it to the disk where the indexer resides. This is the only write operation. Subsequent operations do not require reading the files from the disk again, merging them, or writing the merged result back to the disk, thus avoiding the drawbacks of write amplification. The read operation involves reading multiple first segment files from the disk, packaging them, and sending them to the merger.

[0131] As can be seen, in this application's solution, the merge operation is separated and performed by an independent merger. This avoids the latency issue in the indexer's handling of write and read requests caused by the merge operation, thus improving the user experience. Furthermore, in this method, the indexer performs one write operation and one read operation: one write operation writes multiple first-segment files to disk, and one read operation reads multiple first-segment files from disk and sends them to the first merger. Compared to traditional methods, this application's solution alleviates the problems of severe write amplification and read amplification. Additionally, in this embodiment, data is stored in a multi-level storage format. The first indexer stores the most recently received files, which have not yet met the conditions for being packaged into the first merger and are temporarily stored there. The merged files have already been stored in the storage nodes. This reduces the storage resource consumption in the first indexer, allowing the indexer's resources to better serve the user.

[0132] This application provides a data indexing system 500. For example... Figure 7 As shown, Figure 7 A schematic diagram of the structure of a data indexing system 500 provided in this application.

[0133] System 500 includes at least one indexer and at least one merger, the at least one indexer being communicatively connected to the at least one merger, and any indexer and any merger residing on different compute instances, including any of bare metal, host machine, virtual machine, and container; the at least one indexer includes a first indexer 510, and the at least one merger includes a first merger 520. System 500 includes:

[0134] The first indexer 510 is used to: generate multiple first segment files according to preset rules, each first segment file including a first original file and a first index file, the first original file being obtained by merging one or more files received by the first indexer, and the first index file being obtained by arranging the index information of one or more files in the order of their source file generation time, wherein the index information of each file is extracted from each file; package the multiple first segment files that meet the preset conditions into a first data packet; and send the first data packet to the first merger 520.

[0135] The first merger 520 is used to: merge multiple first segment files in the first data packet to obtain a second data packet, the second data packet including a second segment file, the second segment file including a second original file and a second index file, the second original file being obtained by merging multiple first original files included in the first data packet, and the second index file being obtained by sequentially arranging the index information in the multiple first index files included in the first data packet according to the generation time order of the source files of each index information; and send the second data packet to the first indexer 510;

[0136] The first indexer 510 is also used to: store the second segment file in the second data packet to the disk, and delete the multiple first segment files packaged in the first data packet from the disk.

[0137] In a possible implementation, the first indexer 510 is also used to update the metadata of a plurality of first segment files packaged into a first data packet to the metadata of the second segment files.

[0138] In a possible implementation, the first indexer 510 is used to generate multiple first segment files according to preset rules in any of the following cases:

[0139] 1) Whenever the set first time interval threshold is reached, the first indexer 510 generates a first segment file based on one or more files received within the first time interval threshold range;

[0140] 2) Whenever the number of files received by the first indexer 510 reaches a first quantity threshold, the first indexer 510 generates a first segment file based on the files within the first quantity threshold range;

[0141] 3) The first indexer 510 is set with condition a) the first time interval threshold is reached and condition b) the number of files received by the first indexer 510 reaches the first quantity threshold. Whenever condition a) occurs first, the first indexer 510 generates a first segment file based on one or more files received within the first time interval threshold. Whenever condition b) occurs first, the first indexer 510 generates a first segment file based on files within the first quantity threshold.

[0142] In a possible implementation, the first indexer 510 is used to package multiple first segment files that meet preset conditions into a first data packet in any of the following situations:

[0143] 1) Whenever the set second time interval threshold is reached, the first indexer 510 packages multiple first segment files generated within the second time interval threshold range into a first data packet;

[0144] 2) Whenever the number of first segment files generated by the first indexer 510 reaches the second quantity threshold, the first indexer 510 packages multiple first segment files within the range of the second quantity threshold into a first data packet;

[0145] 3) The first indexer 510 is set with condition c) the second time interval threshold is reached and condition d) the number of first segment files generated by the first indexer 510 reaches the second quantity threshold. Whenever condition c) occurs first, the first indexer 510 packages the multiple first segment files generated within the second time interval threshold into a first data packet. Whenever condition d) occurs first, the first indexer 510 packages the multiple first segment files within the second quantity threshold into a first data packet.

[0146] This application also provides a data indexing system 600. For example... Figure 8 As shown, Figure 8 A schematic diagram of the structure of another data indexing system 600 provided in this application.

[0147] Data indexing system 600 includes at least one indexer, at least one merger, and at least one storage node. The at least one merger is communicatively connected to both the at least one indexer and the at least one storage node. Each indexer, merger, and storage node resides on different computing instances, which may be bare metal, a host machine, a virtual machine, or a container. The at least one indexer includes a first indexer 610, the at least one merger includes a first merger 620, and the at least one storage node includes a first storage node 630.

[0148] The first indexer 610 is used to: generate multiple first segment files according to preset rules, each first segment file including a first original file and a first index file, the first original file being obtained by merging one or more files received by the first indexer 610, and the first index file being obtained by arranging the index information of one or more files in the order of their source file generation time, wherein the index information of each file is extracted from each file; package the multiple first segment files that meet the first preset conditions into a first data packet; and send the first data packet to the first merger 620.

[0149] The first merger 620 is used to: merge multiple first segment files in the first data packet to obtain a second data packet, the second data packet including a second segment file, the second segment file including a second original file and a second index file, the second original file being obtained by merging multiple first original files included in the first data packet, and the second index file being obtained by arranging the index information in the multiple first index files included in the first data packet according to the generation time order of the source files of each index information; and send the second data packet to the first storage node 630.

[0150] The first storage node 630 is used to store the second file segment in the second data packet.

[0151] In a possible implementation, the first indexer 610 is used to delete multiple first segment files packaged into the first data packet from the disk where the first indexer 610 is located.

[0152] In a possible implementation, the first storage node 630 is used to store the metadata of the second file segment.

[0153] In a possible implementation, the first storage node 630 is used to: store some or all of the data in the object storage service OBS bucket when the disk capacity occupied by the first storage node 630 meets the second preset condition, and delete the data stored in the OBS bucket from the disk where the first storage node 630 is located; and update metadata.

[0154] In a possible implementation, system 600 also includes a controller, which is communicatively connected to at least one indexer and at least one storage node, respectively.

[0155] System 600 also includes:

[0156] The first indexer 610 and the first storage node 630 are respectively used to: receive a read request, wherein the read request carries a user identity ID and a time range of the data to be read;

[0157] The first indexer 610 is used to: obtain first target data according to the user ID and the time range of the data to be read through a preset mapping relationship, and send the first target data to the controller; the mapping relationship includes the mapping relationship between the data located on the disk of the first indexer 610 and the user ID and the data generation time;

[0158] The first storage node 630 is used to: obtain the second target data by querying metadata based on the user ID and the time range of the data to be read, and send the second target data to the controller;

[0159] The controller is used to: aggregate the first target data and the second target data to obtain the aggregated result; and send the aggregated result to the user.

[0160] In a possible implementation, the first indexer 610 is used to generate multiple first segment files according to preset rules in any of the following cases:

[0161] 1) Whenever the set first time interval threshold is reached, the first indexer 610 generates a first segment file based on one or more files received within the first time interval threshold range;

[0162] 2) Whenever the number of files received by the first indexer 610 reaches a first quantity threshold, the first indexer 610 generates a first segment file based on the files within the first quantity threshold range;

[0163] 3) The first indexer 610 is configured with condition a) the first time interval threshold is reached and condition b) the number of files received by the first indexer 610 reaches the first quantity threshold. Whenever condition a) occurs first, the first indexer 610 generates a first segment file based on one or more files received within the first time interval threshold. Whenever condition b) occurs first, the first indexer 610 generates a first segment file based on files within the first quantity threshold.

[0164] In a possible implementation, the first indexer 610 is used to package multiple first segment files that satisfy a first preset condition into a first data packet in any of the following situations:

[0165] 1) Whenever the set second time interval threshold is reached, the first indexer 610 packages multiple first segment files generated within the second time interval threshold range into a first data packet;

[0166] 2) Whenever the number of first segment files generated by the first indexer 610 reaches the second quantity threshold, the first indexer 610 packages multiple first segment files within the range of the second quantity threshold into a first data packet;

[0167] 3) The first indexer 610 is set with condition c) the second time interval threshold is reached and condition d) the number of first segment files generated by the first indexer 610 reaches the second quantity threshold. Whenever condition c) occurs first, the first indexer 610 packages the multiple first segment files generated within the second time interval threshold into a first data packet; whenever condition d) occurs first, the first indexer 610 packages the multiple first segment files within the second quantity threshold into a first data packet.

[0168] In the data indexing system 500, the first indexer 510 and the first merger 520 can be implemented in software or in hardware. Similarly, in the data indexing system 600, the first indexer 610, the first merger 620, and the first storage node 630 can all be implemented in software or in hardware.

[0169] For example, the implementation of the first merger 520 will be described below. Similarly, the implementation of the first indexer 510 can refer to the implementation of the first merger 520. The implementations of the first indexer 610, the first merger 620, and the first storage node 630 in the data indexing system 600 can also refer to the implementation of the first merger 520.

[0170] A module is an example of a software functional unit. The first merger 520 may include code running on compute instances, which may include, for example, bare metal, host machines, virtual machines, containers, etc. Further, a compute instance may be one or more. For example, the first merger 520 may include code running on multiple bare metal / host machines / virtual machines / containers. It should be noted that the multiple compute instances used to run the code can be distributed in the same region or in different regions. Further, the multiple compute instances used to run the code can be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple availability zones (AZs).

[0171] Similarly, multiple compute instances used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a single region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0172] The module serves as an example of a hardware functional unit. The first merger 520 may include at least one computing device, such as a bare machine, a host machine, a virtual machine, a container, etc. Alternatively, the first merger 520 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0173] The multiple computing devices included in the first merger 520 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the first merger 520 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the first merger 520 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0174] This application provides a computing device 700, see [link to documentation] Figure 9 , Figure 9 This application provides a schematic diagram of a computing device 700, which can be configured as a data indexing system 500 or a data indexing system 600. The computing device 700 includes a bus 702, a processor 704, a memory 706, and a communication interface 708. The processor 704, the memory 706, and the communication interface 708 communicate via the bus 702. It should be understood that this application does not limit the number of processors and memories in the computing device 700.

[0175] The 702 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 9The bus 702 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 702 may include a path for transmitting information between various components of the computing device 700 (e.g., memory 706, processor 704, communication interface 708).

[0176] Processor 704 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0177] The memory 706 may include volatile memory, such as random access memory (RAM). The processor 704 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0178] When the computing device 700 is configured as a data indexing system 500, the memory 706 stores executable code, and the processor 704 executes the executable code to implement the functions of the aforementioned first indexer 510 and first merger 520, thereby implementing a data indexing method. That is, the memory 706 stores instructions for executing a data indexing method.

[0179] When the computing device 700 is configured as a data indexing system 600, the memory 706 stores executable code, and the processor 704 executes the executable code to implement the functions of the aforementioned first indexer 610, first merger 620, and first storage node 630, thereby implementing a data indexing method. That is, the memory 706 stores instructions for executing a data indexing method.

[0180] The communication interface 708 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 700 and other devices or communication networks.

[0181] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, virtual machine, or container, such as a central server, edge server, or sidecar container.

[0182] like Figure 10 As shown, Figure 10 This application provides a schematic diagram of the structure of a computing device cluster, which includes at least one computing device 700. The memory 706 of one or more computing devices 700 in the computing device cluster may store the same instructions for executing a data indexing method.

[0183] In some possible implementations, the memory 706 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing a data indexing method. In other words, a combination of one or more computing devices 700 can be used to jointly execute instructions for a data indexing method.

[0184] When at least one computing device in the computing device cluster is configured as computing device 700, the memories 706 in different computing devices 700 in the computing device cluster can store different instructions, each used to execute a part of the functions of the computing device 700. That is, the instructions stored in the memories 706 of different computing devices 700 can implement the functions of one or more modules in the first indexer 510 and the first merger 520. Alternatively, the instructions stored in the memories 706 of different computing devices 700 can implement the functions of one or more modules in the first indexer 610, the first merger 620, and the first storage node 630.

[0185] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network, which can be a wide area network (WAN) or a local area network (LAN), etc. Figure 11 This illustrates the structure of yet another type of computing device cluster, such as... Figure 11 As shown, two computing devices 700A and 700B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 706 in computing device 700A stores instructions for the function of the first indexer 510, and the memory 706 in computing device 700B stores instructions for executing the function of the first merger 520. Computing device 700A is used to generate multiple first segment files from multiple received files, package the multiple first segment files into a first data packet, and send the first data packet to computing device 700B. Computing device 700B is used to merge the multiple first segment files in the first data packet into a second segment file, and send the second segment file to computing device 700A in the form of a second data packet. Computing device 700A is also used to store the second segment file in the second data packet.

[0186] It should be understood that Figure 11The functions of computing device 700A shown can also be performed by multiple computing devices 700, or a cluster of computing devices can include multiple computing devices with the same functions as computing device 700A. Similarly, the functions of computing device 700B can also be performed by multiple computing devices 700, or a cluster of computing devices can include multiple computing devices with the same functions as computing device 700B.

[0187] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 10 and Figure 11 The described computing device cluster is connected in a manner that differs in that the memory 706 of one or more computing devices 700 in the cluster may store different instructions for executing a data indexing method. In some possible implementations, the memory 706 of one or more computing devices 700 in the cluster may also each store a subset of instructions for executing a data indexing method. In other words, a combination of one or more computing devices 700 can jointly execute the instructions of a data indexing method.

[0188] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a data indexing method.

[0189] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device or cluster of computing devices to execute a data indexing method.

[0190] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A data indexing method, characterized in that, The method is applied to a data indexing system, the data indexing system including at least one indexer and at least one merger, the at least one indexer being communicatively connected to the at least one merger, any indexer and any merger residing on different computing instances, the computing instances including any one of bare metal, host machine, virtual machine, and container, the at least one indexer including a first indexer, the at least one merger including a first merger, the method comprising: According to preset rules, the first indexer generates multiple first segment files. Each first segment file includes a first original file and a first index file. The first original file is obtained by merging one or more files received by the first indexer. The first index file is obtained by arranging the index information of the one or more files in the order of their source file generation time. The index information of each file is extracted from each file. The first indexer packages multiple first segment files that meet preset conditions into a first data packet; The first indexer sends the first data packet to the first merger; The first merger merges multiple first segment files in the first data packet to obtain a second data packet. The second data packet includes a second segment file, which includes a second original file and a second index file. The second original file is obtained by merging multiple first original files included in the first data packet. The second index file is obtained by arranging the index information in the multiple first index files included in the first data packet according to the generation time order of the source files of each index information. The first merger sends the second data packet to the first indexer; The first indexer stores the second segment file in the second data packet to the disk and deletes the multiple first segment files packaged in the first data packet from the disk.

2. The method according to claim 1, characterized in that, The method further includes: The first indexer updates the metadata of the multiple first segment files packaged into the first data packet to the metadata of the second segment files.

3. The method according to claim 1 or 2, characterized in that, The first indexer generates multiple first segment files according to preset rules, including any of the following cases: 1) Whenever a set first time interval threshold is reached, the first indexer generates a first segment file based on one or more files received within the first time interval threshold range; 2) Whenever the number of files received by the first indexer reaches a first quantity threshold, the first indexer generates a first segment file based on the files within the first quantity threshold range; 3) The first indexer is set with condition a) the first time interval threshold is reached and condition b) the number of files received by the first indexer reaches the first number threshold. Whenever condition a) occurs first, the first indexer generates a first segment file based on one or more files received within the first time interval threshold range. Whenever condition b) occurs first between condition a) and condition b), the first indexer generates a first segment file based on files within the first quantity threshold range.

4. The method according to any one of claims 1 to 3, characterized in that, The first indexer packages multiple first segment files that meet preset conditions into a first data packet, including any of the following cases: 1) Whenever the set second time interval threshold is reached, the first indexer packages multiple first segment files generated within the second time interval threshold range into a first data packet; 2) Whenever the number of first segment files generated by the first indexer reaches the second quantity threshold, the first indexer packages multiple first segment files within the range of the second quantity threshold into the first data packet; 3) The first indexer is set with condition c) the second time interval threshold is reached and condition d) the number of first segment files generated by the first indexer reaches the second quantity threshold. Whenever condition c) occurs first, the first indexer packages multiple first segment files generated within the second time interval threshold range into the first data packet. Whenever condition d) occurs first, the first indexer packages multiple first segment files within the second quantity threshold range into the first data packet.

5. A data indexing method, characterized in that, An application is made to a data indexing system, the data indexing system comprising at least one indexer, at least one merger, and at least one storage node, wherein the at least one merger is communicatively connected to the at least one indexer and the at least one storage node, and any indexer, any merger, and any storage node are located on different computing instances, the computing instances including any one of bare metal, host machine, virtual machine, and container, the at least one indexer including a first indexer, the at least one merger including a first merger, and the at least one storage node including a first storage node, the method comprising: According to preset rules, the first indexer generates multiple first segment files. Each first segment file includes a first original file and a first index file. The first original file is obtained by merging one or more files received by the first indexer. The first index file is obtained by arranging the index information of the one or more files in the order of their source file generation time. The index information of each file is extracted from each file. The first indexer packages multiple first segment files that meet the first preset conditions into a first data packet; The first indexer sends the first data packet to the first merger; The first merger merges multiple first segment files in the first data packet to obtain a second data packet. The second data packet includes a second segment file, which includes a second original file and a second index file. The second original file is obtained by merging multiple first original files included in the first data packet. The second index file is obtained by arranging the index information in the multiple first index files included in the first data packet according to the generation time order of the source files of each index information. The first merger sends the second data packet to the first storage node; The first storage node stores the second segment file in the second data packet.

6. The method according to claim 5, characterized in that, After the first storage node stores the second segment file from the second data packet, the method further includes: The first indexer deletes multiple first segment files packaged in the first data packet from the disk where the first indexer is located.

7. The method according to claim 5 or 6, characterized in that, After the first storage node stores the second segment file from the second data packet, the method further includes: The first storage node stores the metadata of the second file segment.

8. The method according to claim 7, characterized in that, After the first storage node stores the second segment of the file in the second data packet to the disk, the method further includes: If the disk capacity occupied in the first storage node meets the second preset condition, the first storage node will store some or all of the data in the object storage service OBS bucket and delete the data stored in the OBS bucket from the disk where the first storage node is located. The first storage node updates the metadata.

9. The method according to claim 7 or 8, characterized in that, The data indexing system further includes a controller, which is communicatively connected to the at least one indexer and the at least one storage node, respectively. The method further includes: The first indexer and the first storage node respectively receive a read request, the read request carrying a user identity ID and the time range of the data to be read; The first indexer obtains the first target data according to the user ID and the time range of the data to be read through a preset mapping relationship, and sends the first target data to the controller; the mapping relationship includes the mapping relationship between the data located on the disk of the first indexer and the user ID and the data generation time; The first storage node obtains the second target data by querying the metadata based on the user ID and the time range of the data to be read, and sends the second target data to the controller; The controller aggregates the first target data and the second target data to obtain an aggregated result. The controller sends the summarized results to the user.

10. The method according to any one of claims 5 to 9, characterized in that, The first indexer generates multiple first segment files according to preset rules, including any of the following cases: 1) Whenever a set first time interval threshold is reached, the first indexer generates a first segment file based on one or more files received within the first time interval threshold range; 2) Whenever the number of files received by the first indexer reaches a first quantity threshold, the first indexer generates a first segment file based on the files within the first quantity threshold range; 3) The first indexer is set with condition a) the first time interval threshold is reached and condition b) the number of files received by the first indexer reaches the first number threshold. Whenever condition a) occurs first, the first indexer generates a first segment file based on one or more files received within the first time interval threshold range. Whenever condition b) occurs first between condition a) and condition b), the first indexer generates a first segment file based on files within the first quantity threshold range.

11. The method according to any one of claims 5 to 10, characterized in that, The first indexer packages multiple first segment files that meet the first preset conditions into a first data packet, including any of the following cases: 1) Whenever the set second time interval threshold is reached, the first indexer packages multiple first segment files generated within the second time interval threshold range into a first data packet; 2) Whenever the number of first segment files generated by the first indexer reaches the second quantity threshold, the first indexer packages multiple first segment files within the range of the second quantity threshold into the first data packet; 3) The first indexer is set with condition c) the second time interval threshold is reached and condition d) the number of first segment files generated by the first indexer reaches the second quantity threshold. Whenever condition c) occurs first, the first indexer packages multiple first segment files generated within the second time interval threshold range into the first data packet. Whenever condition d) occurs first, the first indexer packages multiple first segment files within the second quantity threshold range into the first data packet.

12. A data indexing system, characterized in that, The system includes at least one indexer and at least one merger, the at least one indexer being communicatively connected to the at least one merger, and any indexer and any merger residing on different compute instances, the compute instances including any one of bare metal, host machine, virtual machine, and container, the at least one indexer including a first indexer, and the at least one merger including a first merger, the system comprising: The first indexer is configured to: generate multiple first segment files according to preset rules, each first segment file including a first original file and a first index file, wherein the first original file is obtained by merging one or more files received by the first indexer, and the first index file is obtained by arranging the index information of the one or more files in the order of their source file generation time, wherein the index information of each file is extracted from each file; package the multiple first segment files that meet preset conditions into a first data packet; and send the first data packet to the first merger. The first merger is configured to: merge multiple first segment files in the first data packet to obtain a second data packet, the second data packet including a second segment file, the second segment file including a second original file and a second index file, the second original file being obtained by merging multiple first original files included in the first data packet, and the second index file being obtained by sequentially arranging the index information in the multiple first index files included in the first data packet according to the generation time order of the source files of each index information; and send the second data packet to the first indexer. The first indexer is further configured to: store the second segment file in the second data packet to a disk, and delete the plurality of first segment files packaged in the first data packet from the disk.

13. The system according to claim 12, characterized in that, The first indexer is further configured to: update the metadata of a plurality of first segment files packaged in the first data packet to the metadata of the second segment files.

14. The system according to claim 12 or 13, characterized in that, The first indexer is used to generate multiple first segment files according to the preset rules in any of the following situations: 1) Whenever a set first time interval threshold is reached, the first indexer generates a first segment file based on one or more files received within the first time interval threshold range; 2) Whenever the number of files received by the first indexer reaches a first quantity threshold, the first indexer generates a first segment file based on the files within the first quantity threshold range; 3) The first indexer is configured with condition a) the first time interval threshold is reached and condition b) the number of files received by the first indexer reaches the first quantity threshold. Whenever condition a) occurs first, the first indexer generates a first segment file based on one or more files received within the first time interval threshold. Whenever condition b) occurs first, the first indexer generates a first segment file based on files within the first quantity threshold.

15. The system according to any one of claims 12 to 14, characterized in that, The first indexer is configured to package multiple first segment files that meet preset conditions into a first data packet in any of the following situations: 1) Whenever the set second time interval threshold is reached, the first indexer packages multiple first segment files generated within the second time interval threshold range into a first data packet; 2) Whenever the number of first segment files generated by the first indexer reaches the second quantity threshold, the first indexer packages multiple first segment files within the range of the second quantity threshold into the first data packet; 3) The first indexer is set with condition c) the second time interval threshold is reached and condition d) the number of first segment files generated by the first indexer reaches the second quantity threshold. Whenever condition c) occurs first, the first indexer packages multiple first segment files generated within the second time interval threshold range into the first data packet. Whenever condition d) occurs first, the first indexer packages multiple first segment files within the second quantity threshold range into the first data packet.

16. A data indexing system, characterized in that, The data indexing system includes at least one indexer, at least one merger, and at least one storage node. The at least one merger is communicatively connected to both the at least one indexer and the at least one storage node. Each indexer, merger, and storage node resides on different computing instances, including bare metal, a host machine, a virtual machine, and a container. The at least one indexer includes a first indexer, the at least one merger includes a first merger, and the at least one storage node includes a first storage node. The first indexer is configured to: generate multiple first segment files according to preset rules, each first segment file including a first original file and a first index file, wherein the first original file is obtained by merging one or more files received by the first indexer, and the first index file is obtained by arranging the index information of the one or more files in the order of their source file generation time, wherein the index information of each file is extracted from each file; package the multiple first segment files that meet the first preset conditions into a first data packet; and send the first data packet to the first merger. The first merger is configured to: merge multiple first segment files in the first data packet to obtain a second data packet, the second data packet including a second segment file, the second segment file including a second original file and a second index file, the second original file being obtained by merging multiple first original files included in the first data packet, and the second index file being obtained by sequentially arranging the index information in the multiple first index files included in the first data packet according to the generation time order of the source files of each index information; and send the second data packet to the first storage node. The first storage node is used to store the second segment file in the second data packet.

17. The system according to claim 16, characterized in that, The first indexer is used to delete multiple first segment files packaged in the first data packet from the disk where the first indexer is located.

18. The system according to claim 16 or 17, characterized in that, The first storage node is used to store the metadata of the second segment file.

19. The system according to claim 18, characterized in that, The first storage node is used to: store some or all of the data in the Object Storage Service (OBS) bucket when the disk capacity occupied by the first storage node meets the second preset condition, and delete the data stored in the OBS bucket from the disk where the first storage node is located; and update the metadata.

20. The system according to claim 18 or 19, characterized in that, The system also includes a controller, which is communicatively connected to the at least one indexer and the at least one storage node, respectively. The system also includes: The first indexer and the first storage node are respectively used to: receive a read request, wherein the read request carries a user identity ID and a time range of the data to be read; The first indexer is configured to: obtain first target data according to the user ID and the time range of the data to be read through a preset mapping relationship, and send the first target data to the controller; the mapping relationship includes the mapping relationship between data located on the disk of the first indexer and user ID and data generation time; The first storage node is configured to: obtain second target data by querying the metadata based on the user ID and the time range of the data to be read, and send the second target data to the controller; The controller is used to: aggregate the first target data and the second target data to obtain an aggregated result; and send the aggregated result to the user.

21. The system according to any one of claims 16 to 20, characterized in that, The first indexer is used to generate multiple first segment files according to the preset rules in any of the following situations: 1) Whenever a set first time interval threshold is reached, the first indexer generates a first segment file based on one or more files received within the first time interval threshold range; 2) Whenever the number of files received by the first indexer reaches a first quantity threshold, the first indexer generates a first segment file based on the files within the first quantity threshold range; 3) The first indexer is configured with condition a) the first time interval threshold is reached and condition b) the number of files received by the first indexer reaches the first quantity threshold. Whenever condition a) occurs first, the first indexer generates a first segment file based on one or more files received within the first time interval threshold. Whenever condition b) occurs first, the first indexer generates a first segment file based on files within the first quantity threshold.

22. The system according to any one of claims 16 to 21, characterized in that, The first indexer is configured to package multiple first segment files that satisfy a first preset condition into a first data packet in any of the following situations: 1) Whenever the set second time interval threshold is reached, the first indexer packages multiple first segment files generated within the second time interval threshold range into a first data packet; 2) Whenever the number of first segment files generated by the first indexer reaches the second quantity threshold, the first indexer packages multiple first segment files within the range of the second quantity threshold into the first data packet; 3) The first indexer is set with condition c) the second time interval threshold is reached and condition d) the number of first segment files generated by the first indexer reaches the second quantity threshold. Whenever condition c) occurs first, the first indexer packages multiple first segment files generated within the second time interval threshold range into the first data packet. Whenever condition d) occurs first, the first indexer packages multiple first segment files within the second quantity threshold range into the first data packet.

23. A computing device cluster, characterized in that, The system includes at least one computing device, the at least one computing device comprising a memory and a processor, the processor of the at least one computing device being configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as claimed in any one of claims 1 to 4, or to cause the cluster of computing devices to perform the method as claimed in any one of claims 5 to 11.

24. A computer storage medium, characterized in that, Includes program instructions that, when executed by a computing device cluster, cause the computing device cluster to implement the method as described in any one of claims 1 to 4, or cause the computing device cluster to implement the method as described in any one of claims 5 to 11.

25. A computer program product, characterized in that, Includes program instructions that, when executed by a computing device cluster, cause the computing device cluster to implement the method as described in any one of claims 1 to 4, or cause the computing device cluster to implement the method as described in any one of claims 5 to 11.

Images