Index construction and data access processing method, device, equipment and medium

Abstract

The present application provides an index construction and data access processing method, device, equipment and medium. The index construction method includes: distinguishing between first-category files and second-category files, the second-category files including data after updating at least one data in the first-category files; constructing indexes for the first-category files and the second-category files in parallel, and generating corresponding index files respectively; determining the order of the index files according to the first-category files or the second-category files corresponding to the index files, and the order is used to determine the target index file from at least two index files including the target data, and the target data in the target index file includes the response data. The present application constructs indexes for the first-category files and the second-category files in parallel, avoiding indexing the first-category files first and then the second-category files, or indexing the second-category files first and then the first-category files, thereby reducing the time delay for indexing the first-category files or the second-category files.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to an index construction and data access processing method, device, system, equipment and medium. Background Art

[0002] The database stores data. When the database receives a request to access data, it can access the data by primary key. Since the primary keys in the database are usually sorted in order, the access speed is fast. However, when accessing data by attribute, the attribute columns are not sorted in order, resulting in a slow access speed. In order to solve the problem of slow access speed of attribute columns, you can build indexes for the attribute columns.

[0003] In the prior art, there are two main schemes for building indexes: asynchronous indexing and synchronous indexing. For asynchronous indexing, the existing stock data is first indexed. During this construction process, new data, i.e., incremental data, is allowed to be written. However, the incremental data needs to be indexed after the index construction of the stock data is completed, i.e., the stock data and incremental data are indexed serially in time. For synchronous indexing, the existing stock data is indexed first. In order to ensure the correctness of the index, new data is avoided from being written within a certain period of time or even during the entire process of indexing the stock data.

[0004] After studying the above two solutions, the applicant found that asynchronous indexing and synchronous indexing lead to a large time delay in building an index for incremental data. Summary of the invention

[0005] The embodiments of the present application provide an index construction method and a data access processing method to reduce the time delay in constructing an index file.

[0006] Correspondingly, the embodiments of the present application also provide an index construction and data access processing device, equipment and medium to ensure the implementation and application of the above method.

[0007] In order to solve the above problems, an embodiment of the present application discloses an index construction method, which includes: distinguishing between a first category of files and a second category of files, the second category of files including data after updating at least one item of data in the first category of files; constructing indexes for the first category of files and the second category of files in parallel, and generating corresponding index files respectively; determining the order of the index files according to the first category of files or the second category of files corresponding to the index files, the order being used to determine a target index file from at least two index files including target data, the target data in the target index file including response data.

[0008] An embodiment of the present application also discloses a data access processing method, which includes: receiving a data access request; determining response data corresponding to the data access request from at least two index files according to the order of index files; the at least two index files include an index file corresponding to a first category of files and an index file corresponding to a second category of files, the second category of files including data after updating at least one item of data in the first category of files, the index files are constructed in parallel, and the order of the index files is determined by the first category of files or the second category of files corresponding to the index files.

[0009] An embodiment of the present application also discloses another index construction method, which includes: distinguishing between existing files and incremental files; constructing indexes for the existing files and incremental files in parallel, and generating corresponding index files respectively; determining the order of the index files based on the existing files or incremental files corresponding to the index files, and the order is used to determine a target index file from at least two index files including target data, and the target data in the target index file includes response data.

[0010] An embodiment of the present application also discloses another index construction method, which includes: performing snapshot processing on existing files to obtain second snapshot information, and generating third snapshot information of an initial state; constructing indexes for the existing files and incremental files in parallel, and generating corresponding index files respectively; recording the index files corresponding to the existing files into the third snapshot information according to the second snapshot information; determining the order of the index files according to the third snapshot information, the order being used to determine a target index file from at least two index files including target data, the target data in the target index file including response data.

[0011] The embodiment of the present application also discloses an index construction device, which includes: a first file distinguishing module, used to distinguish between a first category of files and a second category of files, wherein the second category of files includes data after updating at least one item of data in the first category of files; a first index construction file module, used to construct indexes for the first category of files and the second category of files in parallel, and generate corresponding index files respectively; a first index file sequence determination module, used to determine the sequence of the index files according to the first category of files or the second category of files corresponding to the index files, and the sequence is used to determine a target index file from at least two index files including target data, and the target data in the target index file includes response data.

[0012] An embodiment of the present application also discloses a data access processing device, which includes: a data access request receiving module, used to receive a data access request; determining response data corresponding to the data access request from at least two index files according to the order of the index files; the at least two index files include an index file corresponding to a first category of files and an index file corresponding to a second category of files, the second category of files includes data after updating at least one item of data in the first category of files, the index files are constructed in parallel, and the order of the index files is determined by the first category of files or the second category of files corresponding to the index files.

[0013] An embodiment of the present application also discloses another index construction device, which includes: a second file distinguishing module, used to distinguish between existing files and incremental files; a second index construction module, used to construct indexes for the existing files and incremental files in parallel, and generate corresponding index files respectively; a second index file sequence determination module, used to determine the sequence of the index files according to the existing files or incremental files corresponding to the index files, and the sequence is used to determine the target index file from at least two index files including the target data, and the target data in the target index file includes response data.

[0014] An embodiment of the present application also discloses another index construction device, which includes: a snapshot information generation module, which is used to perform snapshot processing on the existing files to obtain second snapshot information, and generate third snapshot information of the initial state; a third index construction module, which is used to construct indexes for the existing files and incremental files in parallel, and generate corresponding index files respectively; an existing index recording module, which is used to record the index files corresponding to the existing files into the third snapshot information according to the second snapshot information; a third index file sequence determination module, which is used to determine the sequence of the index files according to the third snapshot information, and the sequence is used to determine the target index file from at least two index files including the target data, and the target data in the target index file includes response data.

[0015] An embodiment of the present application further discloses an electronic device, comprising: a processor; and a memory, on which executable code is stored, and when the executable code is executed, the processor executes a method as described in any one of the embodiments of the present application.

[0016] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, a processor executes a method as described in any one of the embodiments of the present application.

[0017] Compared with the prior art, the embodiments of the present application have the following advantages:

[0018] In an embodiment of the present application, the present application can construct indexes for the first category of files and the second category of files in parallel. There will be no situation where the first category of files needs to be indexed after the second category of files, or the second category of files needs to be indexed after the first category of files, thereby avoiding a large time delay in indexing the first category of files or the second category of files. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a structural diagram of a non-relational database of the present application;

[0020] Figure 2 It is a schematic diagram of the storage process based on the log structure merge tree of this application;

[0021] Figure 3 It is a flowchart of steps of an index construction method embodiment of the present application;

[0022] Figure 4 It is a flowchart of the steps of an embodiment of a data access processing method of the present application;

[0023] Figure 5 is a flowchart of another index construction method embodiment of the present application;

[0024] Figure 6 is a flowchart of another index construction method embodiment of the present application;

[0025] Figure 7 It is a flowchart of an index construction process of the present application;

[0026] Figure 8 It is a structural block diagram of an embodiment of an index building device of the present application;

[0027] Fig. 9 It is a structural block diagram of an embodiment of a data access processing device of the present application;

[0028] Fig.10 is a structural block diagram of another index building device embodiment of the present application;

[0029] Fig.11 is a structural block diagram of another index building device embodiment of the present application;

[0030] Fig.12 It is a schematic diagram of the structure of a device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0032] The embodiment of the present application can be applied to the index construction of a data file with a time sequence in a database, wherein a data file is a file for storing data, and the data of a data table is split into multiple data files for storage. A typical application scenario of the present application is a NoSQL (Not Only Structed Query Language) database, which is also called a non-relational database. Traditional relational databases cannot guarantee ultra-large-scale and high-concurrency data storage, while NoSQL databases can achieve ultra-large-scale and multi-type data storage.

[0033] In a NoSQL database, data in a data table can be stored in partitions. There are one or more Master servers and one or more Worker servers in a NoSQL database. Usually, there are many Worker servers. The Master server can also be called the main server, and the Worker server can also be called the work server. The Master server can be used to schedule the partitioned data to each work server according to the preset scheduling strategy. The work server stores the data after receiving it. The work server is used to provide data services to the outside world, such as data reading services or data writing services. Figure 1 The structural diagram of the non-relational database shown in FIG. 1 includes two master servers and five working servers, and each master server can dispatch data to each working server.

[0034] The data storage structure in the above working server is LSM (Log Structured Merge Trees), such as Figure 2 The schematic diagram of the storage process based on the log structure merge tree shown in the figure shows that when a data is received, the data is first written into the memory of the working server in the form of a memory file, and the data is also written into the log file located on the disk; when the memory file reaches a preset capacity threshold, a new memory file is first created to receive the data, and then the memory file that reaches the preset capacity threshold is written to the disk, and a data file is formed on the disk.

[0035] In the above process, there is always only one memory file for receiving data, and the number of data files on the disk increases as the number of times the disk is written increases, such as Figure 2 In the process, five data files are formed: F1, F2, F3, F4 and F5, and the generation time T1 of data file F1 is earlier than the generation time T2 of data file F2, the generation time T2 of data file F2 is earlier than the generation time T3 of data file F3, the generation time T3 of data file F3 is earlier than the generation time T4 of data file F4, and the generation time T4 of data file F4 is earlier than the generation time T5 of data file F5.

[0036] After the data is stored in the above manner, when a data access request is received, the corresponding data can be read from the memory file first. If the corresponding data does not exist in the memory file, the corresponding data is obtained in chronological order. For example, the corresponding data is read from the data file F1 with the earliest generation time. If the corresponding data does not exist in the data file F1, the corresponding data is read from the data file F2; this continues until the corresponding data exists in one of the data files, or until the corresponding data does not exist in the last data file F5.

[0037] It can be understood that the data in the above log file is a backup of the data in the memory file. When the working server suddenly fails, the data in the log file can be written into the memory file to avoid data loss caused by the sudden failure of the working server. After the memory file is written to the disk, the log file is deleted to save disk storage resources.

[0038] From the above data storage process, we can know that as data is written, new data files are constantly generated and older data files are not overwritten. Therefore, data with the same primary key may exist in different data files, and data with the same primary key may also exist in the same data file. In this way, in order to obtain accurate data, it is necessary to obtain the last written data in the order of data files.

[0039] Based on the above Figure 2 The data storage process shown, the present application constructs an index for the data files in the disk generated by the above process to obtain an index file, and it is also necessary to ensure that the order of the index file is consistent with the order of the data files, so that when reading data from the index file, the last written data can be found in the order of the index file, rather than the old data.

[0040] Reference Figure 3 , shows a step flow chart of an index construction method embodiment of the present application.

[0041] Step 101 : distinguishing between first-category files and second-category files, wherein the second-category files include data that is an update of at least one item of data in the first-category files.

[0042] In actual applications, after receiving an index building request, it is first necessary to distinguish at least one existing data file and at least one newly generated data file, where part or all of the data in the newly generated data file is updated data for part or all of the data in the existing data file; then, index building is started in parallel for the at least one data file and the at least one newly generated data file. In the present application, the at least one existing data file may be referred to as a first-category file, and the at least one newly generated data file may be referred to as a second-category file.

[0043] Based on the aforementioned data storage process, it can be known that part or all of the data in the second type of file is an update of part or all of the data in the first type of file. Figure 2 As shown, if an index building request is received after the data file F4 is generated, the existing data files F1 to F4 can be used as first-category files, and the data file F5 newly generated after receiving the index building request can be used as a second-category file.

[0044] The first category files and the second category files of the present application can be any sequential data files, and the first category files are before or after the second category files. The present application can build indexes for the two sequential categories of files in parallel, and ensure that the order of the index files generated after the index is built is consistent with the order of the first category files and the second category files.

[0045] In order to achieve the above-mentioned purpose of order consistency, the present application needs to distinguish which data files are first-category files and which files are second-category files before building the index. In one example, the identification of at least one existing data file can be recorded, and the identification of at least one existing data file is the identification of the first-category file, and the data file that is not recorded is the second-category file. In another example, at least one existing data file can be migrated to a preset storage space, so that the data files in this preset storage space are first-category files, and the data files not in this preset storage space are second-category files. In yet another example, a mark can be added to at least one existing data file, so that the data files with this mark are first-category files, and the data files without this mark are second-category files.

[0046] Step 102: construct indexes for the first category files and the second category files in parallel, and generate corresponding index files respectively.

[0047] Among them, the index is a structure that sorts the values ​​of one or more columns in the data table of the database. It can be calculated according to the values ​​of the columns, which can greatly improve the query efficiency. For the first type of file, the generated index file contains all or part of the columns in the first type of file; for the second type of file, the generated index file contains all or part of the columns in the second type of file. The columns of the index file can be specified in the index building request.

[0048] It can be understood that in this application, since the indexes of the first and second types of files are constructed in parallel, the index files of the first and second types of files generated may be interlaced in order. Figure 2As shown, data files F1 to F4 are first-category files, data file F5 is a second-category file, and the order of index files generated by the index construction method of the present application can be: index file of data file F1, index file of data file F5, index file of data file F2, index file of data file F3, and index file of data file F4.

[0049] In the prior art, the index is constructed according to the order between the first category files and the second category files, rather than constructing the index for the first category files and the second category files in parallel. The order of index files generated by the index construction method of the prior art can be: index file of data file F1, index file of data file F2, index file of data file F3, index file of data file F4, and index file of data file F5.

[0050] Step 103, determining the order of the index files according to the first category files or the second category files corresponding to the index files, wherein the order is used to determine a target index file from at least two index files including target data, wherein the target data in the target index file includes response data.

[0051] Specifically, in the first category of files, an index is constructed for each first category file in the order of the first category files, and the order between the constructed index files of the first category files is consistent with the order between the first category files; in the second category of files, an index is constructed for each second category file in the order of the second category files, and the order between the constructed index files of the second category files is consistent with the order between the second category files; between the first category files and the second category files, the index files of the first category files are sorted before or after the index files of the second category files.

[0052] It can be understood that when the first category of files and the second category of files are arranged in ascending order of time, the index files of the first category of files can be sorted before the index files of the second category of files, so that the index files of data files generated earlier are sorted in front, and the index files of data files generated later are sorted in the back; when the first category of files and the second category of files are arranged in descending order of time, the index files of the first category of files can be sorted after the index files of the second category of files, so that the index files of data files generated later are sorted in front, and the index files of data files generated earlier are sorted in the back.

[0053] Based on the index file with the above order, when a data access request is received, the latest data can be obtained from the index file as the response data of the data access request. When the target data accessed by the data access request does not have data with the same primary key in all index files, it means that the target data has not been updated, and the target data is the response data of the data access request; when the target data accessed by the data access request has at least two data with the same primary key in all index files, it means that the target data has been updated at least once. If at least two data are from different index files, the target data in the index file generated last is used as the response data of the data access request in the order of the index files, where the index file generated last can be referred to as the target index file in this application.

[0054] In addition, if at least two pieces of data are from the same index file, the latest target data can be determined from the time information of each piece of data in the index file as the response data of the data access request.

[0055] Optionally, the distinguishing the first category files from the second category files comprises sub-step 1011:

[0056] Sub-step 1011, recording the first identifier of the first category file, so as to treat the unrecorded file as the second category file.

[0057] The first identifier is a unique identifier of the data file, which is determined according to the generation time when the data file is generated, so that the first identifier can indicate the order of the data file. For example, the first identifier can be numbered starting from 0, and when each data file is generated, the current identifier is increased by 1 as the first identifier of the data file, and the first identifier is used as the current identifier, such as Figure 2 As shown, the current identifier is initialized to 0. When the data file F1 is generated, the current identifier 0 is incremented by 1 as the first identifier 1 of the data file F1, and the first identifier 1 of the data file F1 is used as the current identifier; when the data file F2 is generated, the current identifier 1 is incremented by 1 as the first identifier 2 of the data file F2, and the first identifier 2 of the data file F2 is used as the current identifier; when the data file F3 is generated, the current identifier 2 is incremented by 1 as the first identifier 3 of the data file F3, and the first identifier 3 of the data file F3 is used as the current identifier; when the data file F4 is generated, the current identifier 3 is incremented by 1 as the first identifier 4 of the data file F4, and the first identifier 4 of the data file F4 is used as the current identifier; when the data file F5 is generated, the current identifier 4 is incremented by 1 as the first identifier 5 of the data file F5, and the first identifier 5 of the data file F5 is used as the current identifier.

[0058] In the present application, the purpose of distinguishing between first-category files and second-category files can be achieved by recording the first identifier of the first-category file, so that the file not recorded is the second-category file. For any data file, the first identifier of the data file is obtained. If the first identifier of the data file is one of the first identifiers recorded, the data file is determined to be a first-category file; if the first identifier of the data file is not the first identifier of any record, the data file is determined to be a second-category file. For example, Figure 2 As shown, if an index building request is received after the data file F4 is generated, the first identifiers of the existing data files F1 to F4 may be recorded, so that the unrecorded data file F5 is a second-category file.

[0059] Optionally, recording the first identifier of the first category of files includes sub-step 1011:

[0060] Sub-step 10111: perform snapshot processing on the first-category files to obtain first snapshot information, where the first snapshot information includes a first identifier of the first-category files.

[0061] Among them, a snapshot is a method of recording the state of information at a certain moment. In the present application, when an index building request is received, a snapshot process can be performed on the partition where the data file is located. The obtained snapshot information is called the first snapshot information, and the first snapshot information contains the first identifier of the first category of files. The present application uses the first snapshot information to record the data files that already exist at the moment the index building request is received, so as to achieve the purpose of distinguishing between the first category of files and the second category of files, so that the files that are not in the first snapshot information are the second category of files. For any data file, the first identifier of the data file is obtained. If the first identifier of the data file is in the first snapshot information, the data file is determined to be a first category of file; if the first identifier of the data file is not in the first snapshot information, the data file is determined to be a second category of file.

[0062] Optionally, the method further includes step 1012:

[0063] Step 1012: After the indexing of the first type of files is completed, the first identifier of the record is deleted.

[0064] After the indexing of the first type of files is completed, the identifier of the first type of files representing the records is no longer needed, so the first identifier of the record can be deleted to save storage resources of the working server.

[0065] Optionally, the distinguishing the first category files from the second category files comprises sub-step 1013:

[0066] Sub-step 1013: Migrate the first category files from the first storage area to the second storage area, so that the files in the second storage area are used as the second category files.

[0067] The first storage area is a default storage area for the first type of files, and the second storage area is a storage area different from the first storage area. In actual applications, two storage areas can be set in the disk of the working server: the first storage area and the second storage area, so that when the data in the memory file is written to the disk of the working server, a data file is generated in the first storage area and the data is written into the data file.

[0068] In the present application, the purpose of distinguishing between the first category files and the second category files can be achieved by migrating the first category files to the second storage area, so that the data files in the second storage area are the first category files, and the data files in the first storage area are the second category files. Figure 2 As shown, if an index building request is received after generating data file F4, then after receiving the index building request, the existing data files F1 to F4 in the first storage area are migrated to the second storage area, so that the data file F5 subsequently written to the first storage area is a second type file.

[0069] Optionally, the method further includes sub-step 1014:

[0070] Sub-step 1014: after indexing of the first-category files is completed, migrating the first-category files from the second storage area to before or after the second-category files in the first storage area.

[0071] Specifically, the first category files can be migrated from the second storage area back to the first storage area. If the first category files were originally located before the second category files, the first category files will be migrated back to before the second category files; if the first category files were originally located after the second category files, the first category files will be migrated back to after the second category files.

[0072] Optionally, the distinguishing the first category files from the second category files comprises sub-step 1015:

[0073] Sub-step 1015: adding a first preset mark to the first category of files, so that files not added with the first preset mark are regarded as second category of files.

[0074] In the present application, after receiving the index building request, a first preset mark may be added to the existing data file. The first preset mark may be a number, a letter, a special character, etc. For example, the first preset mark may be 1. Thus, the data file without the first preset mark is a second-category file. Figure 2As shown, if an index building request is received after the data file F4 is generated, then after receiving the index building request, the first preset mark 1 is added to the existing data files F1 to F4, while the subsequently generated data file F5 is not added with the first preset mark, so it is a second type of file.

[0075] Optionally, the method further includes sub-step 1016:

[0076] Sub-step 1016: after indexing of the first category of files is completed, deleting the first preset mark of the first category of files.

[0077] It can be understood that although the first preset mark can be set to a mark that occupies very little storage space, it will still occupy storage resources. In order to save storage resources on the working server, the first preset mark can be deleted after all first-category files are indexed to release the storage space for storing the first preset mark.

[0078] Optionally, determining the order of the index files according to the first category files or the second category files corresponding to the index files comprises sub-steps 1031 to 1032:

[0079] Sub-step 1031, recording the second identifier of the index file corresponding to the first category of files.

[0080] Among them, the second identifier is a unique identifier of the index file, which is determined according to the generation time when the index file is generated, so that the second identifier can represent the order of index files corresponding to the same type of files, that is, the order between index files corresponding to the first type of files, and the order between index files corresponding to the second type of files. For example, the second identifier may be numbered starting from 0, and each time an index file is generated, the current identifier is incremented by 1 as the second identifier of the index file, and the second identifier is used as the current identifier. The current identifier is initialized to 0. When the index file F1′ is generated, the current identifier 0 is incremented by 1 as the second identifier 1 of the index file F1′, and the second identifier 1 of the index file F1′ is used as the current identifier; when the index file F2′ is generated, the current identifier 1 is incremented by 1 as the second identifier 2 of the index file F2′, and the second identifier 2 of the index file F2′ is used as the current identifier; when the index file F3′ is generated, the current identifier 2 is incremented by 1 as the second identifier 3 of the index file F3′, and the second identifier 3 of the index file F3′ is used as the current identifier; when the index file F4′ is generated, the current identifier 3 is incremented by 1 as the second identifier 4 of the index file F4′, and the second identifier 4 of the index file F4′ is used as the current identifier; when the index file F5′ is generated, the current identifier 4 is incremented by 1 as the second identifier 5 of the index file F5′, and the second identifier 5 of the index file F5′ is used as the current identifier.

[0081] In the present application, the index files of the first category files and the index files of the second category files can be distinguished by recording the second identifier of the index file corresponding to the first category files, so that the index files that are not recorded are index files of the second category files, and then the index files of the first category files and the index files of the second category files are sorted. For any index file, the second identifier of the index file is obtained. If the second identifier of the index file is one of the second identifiers recorded, the index file is determined to be the index file of the first category files; if the second identifier of the index file is not any of the second identifiers recorded, the index file is determined to be the index file of the second category files. For example, if an index building request is received after generating data file F4, after building an index for the first category files F1 to F4 to generate the index file, its second identifier can be recorded, so that the index file that is not recorded is the index file of the second category file F5.

[0082] Sub-step 1032: determine whether the recorded index file of the first category of files is before or after the unrecorded index file.

[0083] The present application can record the index files of the first category files to distinguish the index files of all the first category files from the index files of all the second category files, and based on the distinguished index files, implement sorting between the index files of the first category files and the index files of the second category files.

[0084] Optionally, recording the second identifier of the index file corresponding to the first type of file includes sub-step 10311:

[0085] Sub-step 10311: perform snapshot processing on the index file corresponding to the first type of file to obtain second snapshot information, where the second snapshot information includes a second identifier of the index file corresponding to the first type of file.

[0086] Specifically, when a request for index building is received, a snapshot is performed on the partition of the index file. At this time, since the index file has not been generated yet, the second snapshot information constructed is empty. When the index file of the first category file is generated, its second identifier is added to the second snapshot information. The present application uses the second snapshot information to record the index file of the first category file to achieve the purpose of distinguishing the index file of the first category file from the index file of the second category file, so that the index file not in the second snapshot information is the index file of the second category file.

[0087] Optionally, the method further includes sub-step 1033:

[0088] Sub-step 1033, after determining the order of the index file, delete the second identifier of the record.

[0089] In the present application, the second identifier of the record is used to distinguish between the index files of the first category of files and the index files of the second category of files to realize the sorting of the index files. Therefore, after determining the order of the index files, the second identifier of the record is no longer needed. At this time, the second identifier of the record can be deleted to save the storage resources consumed by storing the second identifier.

[0090] Optionally, determining the order of the index files according to the first category files or the second category files corresponding to the index files comprises sub-steps 1034 to 1036:

[0091] Sub-step 1034: storing the index files corresponding to the first type of files in a third storage area.

[0092] Specifically, after indexing the first category of files and generating index files, the index files of the first category of files are directly stored in the third storage area. For example, for the first category of files F1 to F4, after generating index files F1′ to F4′, the index files F1′ to F4′ can be stored in the third storage area in sequence. The third storage area is a preset area for temporarily storing index files of the first category of files, which can be set according to the actual application scenario.

[0093] Sub-step 1035: storing the index files corresponding to the second-category files in a fourth storage area.

[0094] Specifically, after indexing the second category file and generating the index file, the index file of the second category file is directly stored in the fourth storage area. The fourth storage area is a default area for storing index files, which is different from the third storage area. For example, for the second category file F5, after generating the index file F5′, the index file F5′ can be stored in the fourth storage area.

[0095] Sub-step 1036: Migrate the index file in the third storage area to before or after the index file in the fourth storage area.

[0096] It can be understood that when the index files in the third storage area are migrated to the fourth storage area, it is necessary to ensure that the order of the index files in the third storage area remains unchanged.

[0097] The present application can distinguish the index files of the first category of files from the index files of the second category of files by storing the index files of the first category of files and the index files of the second category of files in different storage areas respectively, and implement the sorting between the index files of the first category of files and the index files of the second category of files based on the distinguished index files of the first category of files and the index files of the second category of files.

[0098] Optionally, determining the order of the index files according to the first category files or the second category files corresponding to the index files comprises sub-steps 1037 to 1039:

[0099] Sub-step 1037: adding a second preset mark to the index file corresponding to the first category of files.

[0100] After indexing the first category of files to generate an index file, a second preset mark is added to the index file of the first category of files. The second preset mark can be a number, a letter, a special character, etc. The second preset mark can be the same as the first preset mark or different. For example, the second preset mark can be 1.

[0101] Sub-step 1038: adding a third preset mark to the index file corresponding to the second type of files.

[0102] It can be understood that the second preset mark and the third preset mark are two different marks used to distinguish between the first category of files and the second category of files. For example, the second preset mark is 1 and the third preset mark is 2. If the first category of files are data files F1 to F4, then after indexing the data files F1 to F4 to generate index files, the second preset mark 1 can be added to the index files of the data files F1 to F4, and after indexing the data file F5 to generate index files, the third preset mark 2 can be added to the index file of the data file F5.

[0103] In a special case, the third preset mark may also be an empty string, so that the index file marked with the third preset mark is equivalent to being unmarked.

[0104] Sub-step 1039: determining whether the index file marked with the second preset mark is before or after the index file marked with the third preset mark.

[0105] The index file marked with the third preset mark is an index file of the second category of files, thereby achieving sorting of the index files of the first category of files and the index files of the second category of files according to the second preset mark and the third preset mark.

[0106] After generating the index file, the present application first uses the second preset mark and the third preset mark to distinguish the index file of the first category of files from the index file of the second category of files, and then after the index construction of the first category of files is completed, the index file marked with the second preset mark is sorted before or after the index file marked with the third preset mark.

[0107] Optionally, the second preset mark and the third preset mark are both numerical values, and the second preset mark is smaller than the third preset mark; the determining whether the index file marked with the second preset mark is before or after the index file marked with the third preset mark includes sub-step 10391: Sub-step 10391, determining whether the index file marked with the second preset mark is before or after the index file marked with the third preset mark according to the sizes of the second preset mark and the third preset mark.

[0108] It can be understood that when the second preset mark and the third preset mark are both numerical values, the index files of the first category of files and the index files of the second category of files can be sorted according to the size of the second preset mark and the third preset mark. For example, the second preset mark can be 1, and the third preset mark can be 2, so that the index files marked with 1 can be sorted before or after the index files marked with 2.

[0109] The present application can sort the index files according to the size of the second preset mark and the third preset mark. When the second preset mark and the third preset mark are numerical values, a numerical value-based sorting algorithm can be directly called.

[0110] Optionally, after determining the order of the index files according to the first category files or the second category files corresponding to the index files, the method further includes step 104:

[0111] Step 104: Delete the second preset mark and the third preset mark.

[0112] In the present application, the second preset mark and the third preset mark are used to distinguish the index files of the first category of files from the index files of the second category of files to achieve sorting of the index files. Therefore, after determining the order of the index files, the second preset mark and the third preset mark are no longer needed. At this time, the second preset mark and the third preset mark can be deleted to save the storage resources consumed by storing the second preset mark and the third preset mark.

[0113] Optionally, before distinguishing the first category of files from the second category of files, steps 105 to 106 are included:

[0114] Step 105: Receive an index building request.

[0115] Specifically, the index building request is triggered by the user executing a database statement for building an index. The user can specify the target column for building the index and the column in the index file in the database statement, and can also specify whether to build an index for an existing data file. If the existing data file is not indexed, only the data file generated after the index building request is indexed, so that only the second category of files exists, and the index file of the second category of files can be sorted using the second identifier of the index file; if the existing data file is indexed, it is necessary to build an index for the existing data file and the data file generated after the index building request, so that there are first category of files and second category of files, and it is necessary not only to sort between the index files of the first category of files and between the index files of the second category of files, but also to sort between the index files of the first category of files and the index files of the second category of files.

[0116] Step 106: In response to the index building request, determine the currently existing data files as first-category files; the second-category files are data files generated after receiving the index building request.

[0117] In the present application, after receiving an index building request, the data files that already exist at the current moment can be determined as first-class files, which can also be called existing files; the data files generated after the index building request can be determined as second-class files, which can also be called incremental files.

[0118] The present application can accurately determine the first category of files and the second category of files based on the received index building request.

[0119] Optionally, after constructing the indexes for the first category files and the second category files in parallel and generating corresponding index files respectively, the method further includes step 107:

[0120] Step 107: Generate a second identifier for the index file according to the generation time of the index file.

[0121] The second identifier is a unique identifier of the index file, which is determined according to the generation time when the index file is generated, so that the second identifier can represent the order of the index file. For example, the second identifier may be numbered starting from 0, and each time an index file is generated, the current identifier is incremented by 1 as the second identifier of the index file, and the second identifier is used as the current identifier. The current identifier is initialized to 0. When the index file F1′ is generated, the current identifier 0 is incremented by 1 as the second identifier 1 of the index file F1′, and the second identifier 1 of the index file F1′ is used as the current identifier; when the index file F2′ is generated, the current identifier 1 is incremented by 1 as the second identifier 2 of the index file F2′, and the second identifier 2 of the index file F2′ is used as the current identifier; when the index file F3′ is generated, the current identifier 2 is incremented by 1 as the second identifier 3 of the index file F3′, and the second identifier 3 of the index file F3′ is used as the current identifier; when the index file F4′ is generated, the current identifier 3 is incremented by 1 as the second identifier 4 of the index file F4′, and the second identifier 4 of the index file F4′ is used as the current identifier; when the index file F5′ is generated, the current identifier 4 is incremented by 1 as the second identifier 5 of the index file F5′, and the second identifier 5 of the index file F5′ is used as the current identifier.

[0122] The above-mentioned index file F1′, index file F2′, index file F3′, index file F4′ and index file F5′ can be generated by the following steps: first, the first category file F1 is indexed to generate index file F1′, then the second category file F5 is indexed to generate index file F2′, then the first category file F2 is indexed to generate index file F3′, then the first category file F3 is indexed to generate index file F4′, and finally the first category file F4 is indexed to generate index file F5′, so that the second identifiers of index file F1′, index file F2′, index file F3′, index file F4′ and index file F5′ are 1, 2, 3, 4, 5 respectively.

[0123] In the first example, based on the second identifier of the index file, combined with the second identifier recorded above, the present application can achieve sorting of all index files. For example, the second identifiers 1, 3, 4 and 5 of the index file F1′, index file F3′, index file F4′ and index file F5′ of the first category of files are recorded, and the second identifier of the index file F2′ of the second category of files is not recorded, so that the index file F1′, index file F3′, index file F4′ and index file F5′ with the second identifiers 1, 3, 4 and 5 recorded can be sorted before the index file F2′ without the second identifier 2, and then the index file F1′, index file F3′, index file F4′ and index file F5′ are sorted in ascending order according to the second identifier, and the final order is: index file F1′, index file F3′, index file F4′, index file F5′, index file F2′, which is consistent with the order of the corresponding data file F1, data file F2, data file F3, data file F4, data file F5.

[0124] In the second example, based on the second identifier of the index file, combined with the third storage area and the fourth storage area, the present application can achieve sorting of all index files. For example, the index files F1', F3', F4' and F5' of the first category of files are stored in the third storage area, and the index file F2' of the second category of files is stored in the fourth storage area, so that the index files F1', F3', F4' and F5' in the third storage area can be sorted before the index file F2' in the fourth storage area, and then the index files F1', F3', F4' and F5' are sorted in ascending order according to the second identifier, and the final order is: index file F1', index file F3', index file F4', index file F5', index file F2', which is consistent with the order of the corresponding data files F1, data file F2, data file F3, data file F4, data file F5.

[0125] In the third example, based on the second identifier of the index file, combined with the aforementioned second preset mark and third preset mark, the present application can achieve sorting of all index files. For example, the second preset mark is 1, the third preset mark is 2, the index file F1′, index file F3′, index file F4′ and index file F5′ of the first category of files correspond to the second preset mark 1, and the index file F2′ of the second category of files corresponds to the third preset mark 2. If a number is set for each index file of the first category: (second preset mark, second identifier), and a number is set for each index file of the second category of files: (third preset mark, second identifier), then the numbers of the index file F1′, index file F2′, index file F3′, index file F4′ and index file F5′ can be obtained as follows: (1, 1), (2, 2), (1, 3), (1, 4), (1, 5).

[0126] Based on the above numbering, index file F1′, index file F2′, index file F3′, index file F4′ and index file F5′ can be sorted. First, index file F1′, index file F3′, index file F4′ and index file F5′ marked with the second preset mark 1 are sorted before index file F2′ marked with the third preset mark 2, and then index file F1′, index file F3′, index file F4′ and index file F5′ are sorted in ascending order according to the second identifier. The final order is: index file F1′, index file F3′, index file F4′, index file F5′, index file F2′, which is consistent with the order of the corresponding data file F1, data file F2, data file F3, data file F4, data file F5.

[0127] Step 108: sort the index files corresponding to the first category files according to the second identifiers of the index files corresponding to the first category files.

[0128] It can be understood that the index files of the first category of files are a part of the index files, and the index files of the first category of files can be arranged in descending order or ascending order according to the second identifier.

[0129] Step 109: sort the index files corresponding to the second category files according to the second identifiers of the index files corresponding to the second category files.

[0130] It can be understood that the index files of the second category files are another part of the index files, and the index files of the second category files can be arranged in descending order or ascending order according to the second identifier.

[0131] Optionally, the first category files and the second category files are stored in a non-relational database, and the non-relational database partitions and stores the first category files and the second category files in a log-structured merge tree. Data with the same primary key exists in the first category files and / or the second category files, and data with the same primary key exists in the index files corresponding to the first category files and / or the index files corresponding to the second category files.

[0132] Among them, based on the storage process of the log structure merge tree, when a data is received, the data is first written into the memory file of the working server in the form of a memory file, and the data is also written into the log file located on the disk; when the memory file reaches the preset capacity threshold, a new memory file is first created to receive the data, and then the memory file that reaches the preset capacity threshold is written into the disk, and a data file is formed on the disk. The first type of file can be a data file generated first, and the second type of file can be a data file generated later.

[0133] The typical application scenario of the present application is that during the data writing process, the newly written data will not overwrite the old data, so that the data with the same primary key may exist in the first category file and the second category file, or the data with the same primary key may exist in the same data file. In this way, the constructed index file will also contain the index file of the first category file and the index file of the second category file, or the data with the same primary key may exist in the same index file. After receiving the data access request, the target data with the same primary key may be obtained from multiple index files or the same index file, but it is necessary to determine which of the target data is the latest data as the response data to the data access request.

[0134] In order to ensure that the response data obtained after receiving a data access request is the latest, the present application sorts between the index files of the first category of files, sorts between the index files of the second category of files, and sorts between the index files of the first category of files and the index files of the second category of files, so as to accurately determine the order of the index files, and ensure that when target data with the same primary key is read from different index files, the latest data can be determined as the response data according to the order of the index files.

[0135] In summary, the present application provides an index construction method, the method comprising: distinguishing between first-category files and second-category files, the second-category files including data after updating at least one item of data in the first-category files; constructing indexes for the first-category files and the second-category files in parallel, and generating corresponding index files respectively; determining the order of the index files according to the first-category files or the second-category files corresponding to the index files, the order being used to determine the target index file from at least two index files including the target data, the target data in the target index file being the response data. The present application constructs indexes for the first-category files and the second-category files in parallel, avoiding constructing an index for the first-category files first and then the second-category files, or constructing an index for the second-category files first and then the first-category files, thereby reducing the time delay for constructing indexes for the first-category files or the second-category files.

[0136] Reference Figure 4 , showing a step flow chart of an embodiment of a data access processing method of the present application.

[0137] Step 201: Receive a data access request.

[0138] The data access request is a request for obtaining response data, and the response data is usually determined by conditions in the data access request, which include but are not limited to: filtering conditions on the primary key or one or more columns thereof.

[0139] Step 202, determining the response data corresponding to the data access request from at least two index files according to the order of the index files; the at least two index files include an index file corresponding to a first category of files and an index file corresponding to a second category of files, the second category of files includes data after updating at least one item of data in the first category of files, the index files are constructed in parallel, and the order of the index files is determined by the first category of files or the second category of files corresponding to the index files.

[0140] In actual applications, the response data corresponding to the data access request can be obtained from the latest index file in the order of the index files until the oldest index file. For example, for the index files F1', F2', F3', F4', F5' arranged in ascending order according to the generation time, the response data can be obtained from the index file F5' until the index file F1'. If the target data with the same primary key exists in the index file F1' and the index file F5', the target data in F5' is used as the response data.

[0141] In summary, the present application provides a data access processing method, the method comprising: receiving a data access request; determining response data corresponding to the data access request from at least two index files according to the order of index files; the at least two index files include an index file corresponding to a first category of files and an index file corresponding to a second category of files, the second category of files including data after updating at least one item of data in the first category of files, the index files are constructed in parallel, and the order of the index files is determined by the first category of files or the second category of files corresponding to the index files. The present application constructs indexes for the first category of files and the second category of files in parallel, avoiding constructing an index for the first category of files first and then the second category of files, or constructing an index for the second category of files first and then the first category of files, reducing the time delay for constructing an index for the first category of files or the second category of files, thereby facilitating access to the index files of the two categories of files and avoiding a large delay in accessing the index files of one of the categories of files.

[0142] Reference Figure 5 , showing a step flow chart of another index construction method embodiment of the present application.

[0143] Step 301, distinguish between existing files and incremental files.

[0144] In actual applications, after receiving a request to build an index, it is first necessary to distinguish at least one existing data file and at least one newly generated data file, and then start to build an index in parallel for the at least one data file and the at least one newly generated data file. In this application, the at least one existing data file can be called a stock file, and the at least one newly generated data file can be called an incremental file.

[0145] based on Figure 2 From the data storage process in the example, we can see that part or all of the data in the incremental file is an update of part or all of the data in the stock file. Figure 2 As shown, if an index building request is received after the data file F4 is generated, the existing data files F1 to F4 can be used as stock files, and the data file F5 newly generated after receiving the index building request can be used as an incremental file.

[0146] The present application can construct indexes for sequential stock files and incremental files in parallel, and ensure that the order of index files generated after index construction is consistent with the order of stock files and incremental files.

[0147] In order to achieve the above-mentioned purpose of order consistency, the present application needs to distinguish which data files are stock files and which files are incremental files before building the index. In one example, the identification of at least one existing data file can be recorded, and the identification of at least one existing data file is the identification of the stock file, while the data files that are not recorded are incremental files. In another example, at least one existing data file can be migrated to a preset storage space, so that the data files in this preset storage space are stock files, and the data files not in this preset storage space are incremental files. In yet another example, a mark can be added to at least one existing data file, so that the data files with this mark are stock files, and the data files without this mark are incremental files.

[0148] Step 302: construct indexes for the existing files and incremental files in parallel, and generate corresponding index files respectively.

[0149] An index is a structure that sorts the values ​​of one or more columns in a data table in a database, which can greatly improve query efficiency. For existing files, the generated index file contains all or part of the columns in the existing file; for incremental files, the generated index file contains all or part of the columns in the incremental file. The columns of the index file can be specified in the index building request.

[0150] It can be understood that in the present application, since the existing files and the incremental files are indexed in parallel, the index files of the generated existing files and the index files of the incremental files may be interlaced in order, for example, Figure 2 As shown, data files F1 to F4 are stock files, data file F5 is an incremental file, and the order of index files generated by the index construction method of the present application can be: index file of data file F1, index file of data file F5, index file of data file F2, index file of data file F3, and index file of data file F4.

[0151] In the prior art, the index is constructed according to the order between the existing files and the incremental files, rather than constructing the index for the existing files and the incremental files in parallel. The order of the index files generated by the index construction method of the prior art can be: the index file of data file F1, the index file of data file F2, the index file of data file F3, the index file of data file F4, and the index file of data file F5.

[0152] Step 303, determining the order of the index files according to the stock files or incremental files corresponding to the index files, wherein the order is used to determine a target index file from at least two index files including target data, wherein the target data in the target index file includes response data.

[0153] In actual applications, among existing files, an index is built for each existing file in the order of existing files, and the order between the index files of the built existing files is consistent with the order of the existing files; among incremental files, an index is built for each incremental file in the order of incremental files, and the order between the index files of the built incremental files is consistent with the order of the incremental files; between existing files and incremental files, the index files of stock files are sorted before or after the index files of incremental files.

[0154] It can be understood that when the existing files and the incremental files are arranged in ascending order of time, the index files of the existing files can be sorted before the index files of the incremental files, so that the index files of the data files generated earlier are sorted in front, and the index files of the data files generated later are sorted in the back; when the existing files and the incremental files are arranged in ascending order of time, the index files of the existing files can be sorted after the index files of the incremental files, so that the index files of the data files generated later are sorted in front, and the index files of the data files generated earlier are sorted in the back.

[0155] Based on the index file with the above order, when a data access request is received, the latest data can be obtained from the index file as the response data of the data access request. When the target data accessed by the data access request does not have data with the same primary key in all index files, it means that the target data has not been updated, and the target data is the response data of the data access request; when the target data accessed by the data access request has at least two data with the same primary key in all index files, it means that the target data has been updated at least once. If at least two data are from different index files, the target data in the index file generated last is used as the response data of the data access request in the order of the index files, where the index file generated last can be referred to as the target index file in this application.

[0156] In addition, if at least two pieces of data are from the same index file, the latest target data can be determined from the time information of each piece of data in the index file as the response data of the data access request.

[0157] In summary, the present application provides an index construction method, the method comprising: distinguishing between existing files and incremental files; constructing indexes for the existing files and incremental files in parallel, and generating corresponding index files respectively; determining the order of the index files according to the existing files or incremental files corresponding to the index files, the order being used to determine a target index file from at least two index files including target data, the target data in the target index file including response data. The present application constructs indexes for existing files and incremental files in parallel, avoiding indexing the existing files first and then the incremental files, or indexing the incremental files first and then the existing files, thereby reducing the time delay for indexing the existing files or the incremental files.

[0158] Reference Figure 6 , showing a step flow chart of another index construction method embodiment of the present application.

[0159] Step 401, snapshot processing is performed on the stock file to obtain second snapshot information, and third snapshot information of the initial state is generated.

[0160] Among them, the second snapshot information includes the first identifier of the existing files, and the third snapshot information of the initial state is determined by whether there is an index file of the existing files in the initial state. If in the initial state, there are index files of some existing files, then the third snapshot information includes the second identifier of the index file of the existing stock files. If in the initial state, there is no index file of the stock files, then the third snapshot information does not include the second identifier of the index file of any stock files, which can also be understood as the third snapshot information being empty.

[0161] Step 402, indexing the existing files and incremental files in parallel, and generating corresponding index files respectively.

[0162] This step can refer to the detailed description of step 302 and will not be described again here.

[0163] Step 403: Record the index file corresponding to the stock file into the third snapshot information according to the second snapshot information.

[0164] Specifically, first, determine the first identifier of the stock file in the second snapshot information; then, determine the corresponding index file according to the first identifier; finally, add the second identifier of the index file to the third snapshot information.

[0165] Step 404: determine the order of the index files according to the third snapshot information, wherein the order is used to determine a target index file from at least two index files including target data, wherein the target data in the target index file includes response data.

[0166] Specifically, the index file corresponding to the second identifier in the third snapshot information may be sorted before or after the remaining index files. After determining the order, the second snapshot information and the third snapshot information may be deleted.

[0167] It is understandable that step 404 may refer to the detailed description of step 303 and will not be described in detail here.

[0168] The stock files and incremental files of this application can be collectively referred to as master files. The master file is the file in which the data in the main table is divided when it is stored, and the index file is the file in which the data in the index table is divided when it is stored. The process of the index construction process of the master file can be as follows: Figure 7 As shown:

[0169] First, snapshot processing is performed on the four existing stock files: F1, F2, F3 and F4, where 1, 2, 3 and 4 are the first identifiers of the four stock files respectively, so that the second snapshot information includes: 1, 2, 3 and 4, and the first identifier represents the time order of the four stock files. The smaller the first identifier, the earlier the time; in addition, the present application also generates third snapshot information, in which there is no arbitrary second identifier.

[0170] Then, after the snapshot is created, new data needs to be stored. The new data is stored in a new file F5, called an incremental file, so that the existing files F1, F2, F3 and F4 and the incremental file F5 are indexed in parallel to generate index files F1′, F2′, F3′, F4′ and F5′, where F1′ is the index file of F1, F2′ is the index file of F5, F3′ is the index file of F2, F4′ is the index file of F3, and F5′ is the index file of F4, where 1, 2, 3, 4 and 5 are the second identifiers of the index files respectively. Similarly, the second identifier represents the time order of the five index files. The smaller the second identifier, the earlier the time, and the second identifiers 1, 2, 3, 4 and 5 of the index files are added to the third snapshot information.

[0171] Then, the order of the index files is determined according to the third snapshot information, a second preset mark is added to the index file corresponding to the second identifier in the third snapshot information, and a third preset mark is added to the remaining index files, so that the index files can be sorted by combining the second identifier, the second preset mark, and the third preset mark. Figure 7As shown in , the second preset mark is 1, and the third preset mark is 2, so the index files of the second identifiers 1, 3, 4, and 5 are added with the second preset mark 1, and the index file of the second identifier 2 is added with the third preset mark 2. If the second identifier and the second preset mark or the third preset mark form the index file number, the index files F1′, F2′, F3′, F4′, and F5′ are updated to index files F'(1,1), F'(2,2), F'(1,3), F'(1,4), and F'(1,5). Therefore, the index files can be sorted according to the second preset mark and the third preset mark first, and then sorted according to the second identifier. In this way, the obtained order is F'(1,1), F'(1,5), F'(1,3), F'(1,4), and F'(2,2).

[0172] Finally, after the order is determined, the snapshot information may be deleted, including: the second snapshot information and the third snapshot information.

[0173] In summary, the present application provides an index construction method, the method comprising: performing snapshot processing on the existing files to obtain second snapshot information, and generating third snapshot information of the initial state; constructing indexes for the existing files and incremental files in parallel, and generating corresponding index files respectively; recording the index files corresponding to the existing files into the third snapshot information according to the second snapshot information; determining the order of the index files according to the third snapshot information, the order being used to determine the target index file from at least two index files including target data, the target data in the target index file including response data. The present application constructs indexes for the existing files and incremental files in parallel, avoiding indexing the existing files first and then the incremental files, or indexing the incremental files first and then the existing files, thereby reducing the time delay for indexing the existing files or incremental files.

[0174] Reference Figure 8 , shows a structural block diagram of an index construction device embodiment of the present application.

[0175] The first file distinguishing module 501 is used to distinguish between first-category files and second-category files, where the second-category files include data that is an update of at least one item of data in the first-category files.

[0176] The first index building file module 502 is used to build indexes for the first category files and the second category files in parallel, and generate corresponding index files respectively.

[0177] The first index file sequence determination module 503 is used to determine the sequence of the index files according to the first category files or the second category files corresponding to the index files, wherein the sequence is used to determine a target index file from at least two index files including target data, wherein the target data in the target index file includes response data.

[0178] Optionally, the first file distinguishing module 501 includes a first identification recording submodule:

[0179] The first identification recording submodule is used to record the first identification of the first category file so as to treat the unrecorded file as the second category file.

[0180] Optionally, the first identification recording submodule includes a first snapshot unit:

[0181] The first snapshot unit is used to perform snapshot processing on the first-category files to obtain first snapshot information, where the first snapshot information includes a first identifier of the first-category files.

[0182] Optionally, the first file distinguishing module 501 includes a first migration submodule:

[0183] The storage migration submodule is used to migrate the first type of files from the first storage area to the second storage area, so as to use the files in the second storage area as the second type of files.

[0184] Optionally, the first file distinguishing module 501 further includes a second migration submodule:

[0185] The second migration submodule is used to migrate the first type of files from the second storage area to before or after the second type of files in the first storage area after indexing of the first type of files is completed.

[0186] Optionally, the first file distinguishing module 501 includes a first marking submodule:

[0187] The first marking submodule is used to add a first preset mark to the first category of files, so as to treat the files not having the first preset mark as the second category of files.

[0188] Optionally, the first index file sequence determination module 503 includes a second identification recording submodule and a first sequence determination submodule:

[0189] The second identifier recording submodule is used to record the second identifier of the index file corresponding to the first type of file.

[0190] The first sequence determination submodule is used to determine whether the recorded index file of the first category of files is before or after the unrecorded index file.

[0191] Optionally, the second identification recording submodule includes a second snapshot unit:

[0192] The second snapshot unit is used to perform snapshot processing on the index file corresponding to the first type of file to obtain second snapshot information, where the second snapshot information includes a second identifier of the index file corresponding to the first type of file.

[0193] Optionally, the first index file sequence determination module 503 includes a first storage submodule, a second storage submodule and a second sequence determination submodule:

[0194] The first storage submodule is used to store the index files corresponding to the first type of files in a third storage area.

[0195] The second storage submodule is used to store the index files corresponding to the second type of files in a fourth storage area.

[0196] The second sequence determination submodule is used to migrate the index files in the third storage area before or after the index files in the fourth storage area.

[0197] Optionally, the first index file sequence determination module 503 includes a second marking submodule, a third marking submodule and a third sequence determination submodule:

[0198] The second marking submodule is used to add a second preset mark to the index file corresponding to the first type of files.

[0199] The third marking submodule is used to add a third preset mark to the index file corresponding to the second type of file.

[0200] The third sequence determination submodule is used to determine whether the index file marked with the second preset mark is before or after the index file marked with the third preset mark.

[0201] Optionally, the second preset mark and the third preset mark are both numerical values, the second preset mark is smaller than the third preset mark, and the third sequence determination submodule includes a third sequence determination unit:

[0202] The third sequence determining unit is used to determine whether the index file marked with the second preset mark is before or after the index file marked with the third preset mark according to the sizes of the second preset mark and the third preset mark.

[0203] Optionally, the device further includes an index building request receiving module and a first category file determining module:

[0204] The index building request receiving module is used to receive the index building request.

[0205] The first category file determination module is used to determine the currently existing data files as the first category files in response to the index building request; the second category files are data files generated after receiving the index building request.

[0206] Optionally, the device further includes a second identifier generating module, a first-category index file sorting module, and a second-category index file sorting module:

[0207] The second identifier generating module is used to generate a second identifier for the index file according to the generation time of the index file.

[0208] The first-category index file sorting module is used to sort the index files corresponding to the first-category files according to the second identifiers of the index files corresponding to the first-category files.

[0209] The second category index file sorting module is used to sort the index files corresponding to the second category files according to the second identifiers of the index files corresponding to the second category files.

[0210] Optionally, the first category files and the second category files are stored in a non-relational database, and the non-relational database partitions and stores the first category files and the second category files in a log-structured merge tree. Data with the same primary key exists in the first category files and / or the second category files, and data with the same primary key exists in the index files corresponding to the first category files and / or the index files corresponding to the second category files.

[0211] In summary, the present application provides an index construction device, the device comprising: a first file distinguishing module, used to distinguish between a first category of files and a second category of files, the second category of files including data after updating at least one item of data in the first category of files; a first index construction file module, used to construct indexes for the first category of files and the second category of files in parallel, and respectively generate corresponding index files; a first index file sequence determination module, used to determine the sequence of the index files according to the first category of files or the second category of files corresponding to the index files, the sequence being used to determine a target index file from at least two index files including target data, the target data in the target index file including response data. The present application constructs indexes for the first category of files and the second category of files in parallel, avoids indexing the first category of files first and then the second category of files, or indexing the second category of files first and then the first category of files, and reduces the time delay for indexing the first category of files or the second category of files.

[0212] Reference Fig. 9 , showing a structural block diagram of an embodiment of a data access processing device of the present application.

[0213] The data access request receiving module 601 is used to receive a data access request.

[0214] A response data determination module 602 is used to determine the response data corresponding to the data access request from at least two index files according to the order of the index files; the at least two index files include an index file corresponding to a first category of files and an index file corresponding to a second category of files, the second category of files includes data after updating at least one item of data in the first category of files, the index files are constructed in parallel, and the order of the index files is determined by the first category of files or the second category of files corresponding to the index files.

[0215] In summary, the present application provides a data access processing device, the device comprising: a data access request receiving module, used to receive a data access request; a response data determining module, used to determine the response data corresponding to the data access request from at least two index files according to the order of the index files; the at least two index files include an index file corresponding to a first category of files and an index file corresponding to a second category of files, the second category of files including data after updating at least one item of data in the first category of files, the index files are constructed in parallel, and the order of the index files is determined by the first category of files or the second category of files corresponding to the index files. The present application constructs indexes for the first category of files and the second category of files in parallel, avoiding constructing an index for the first category of files first and then the second category of files, or constructing an index for the second category of files first and then the first category of files, reducing the time delay for constructing an index for the first category of files or the second category of files, thereby facilitating access to the index files of the two categories of files and avoiding a large delay in accessing the index files of one of the categories of files.

[0216] Reference Fig.10 , showing a structural block diagram of another index construction device embodiment of the present application.

[0217] The second file distinguishing module 701 is used to distinguish between existing files and incremental files.

[0218] The second index building module 702 is used to build indexes for the existing files and incremental files in parallel, and generate corresponding index files respectively.

[0219] The second index file sequence determination module 703 is used to determine the sequence of the index files according to the existing files or incremental files corresponding to the index files, wherein the sequence is used to determine the target index file from at least two index files including the target data, wherein the target data in the target index file includes the response data.

[0220] In summary, the present application provides an index construction device, the device comprising: a second file distinguishing module, used to distinguish between existing files and incremental files; a second index construction module, used to construct indexes for the existing files and incremental files in parallel, and respectively generate corresponding index files; a second index file sequence determination module, used to determine the sequence of the index files according to the existing files or incremental files corresponding to the index files, the sequence being used to determine a target index file from at least two index files including target data, the target data in the target index file including response data. The present application constructs indexes for existing files and incremental files in parallel, avoiding indexing existing files first and then incremental files, or indexing incremental files first and then existing files, thereby reducing the time delay for indexing existing files or incremental files.

[0221] Reference Fig.11 , showing a structural block diagram of another index construction device embodiment of the present application.

[0222] The snapshot information generating module 801 is used to perform snapshot processing on the stock files to obtain second snapshot information, and to generate third snapshot information in an initial state.

[0223] The third index building module 802 is used to build indexes for the existing files and incremental files in parallel, and generate corresponding index files respectively.

[0224] The stock index recording module 803 is used to record the index file corresponding to the stock file into the third snapshot information according to the second snapshot information.

[0225] The third index file sequence determination module 804 is used to determine the sequence of the index files according to the third snapshot information, wherein the sequence is used to determine a target index file from at least two index files including target data, wherein the target data in the target index file includes response data.

[0226] In summary, the present application provides an index construction device, the device comprising: a snapshot information generation module, used to perform snapshot processing on the stock files to obtain second snapshot information, and generate third snapshot information of the initial state; a third index construction module, used to construct indexes for the stock files and incremental files in parallel, and generate corresponding index files respectively; a stock index recording module, used to record the index files corresponding to the stock files into the third snapshot information according to the second snapshot information; a third index file sequence determination module, used to determine the sequence of the index files according to the third snapshot information, the sequence being used to determine the target index file from at least two index files including target data, the target data in the target index file including response data. The present application constructs indexes for stock files and incremental files in parallel, avoids indexing the stock files first and then the incremental files, or indexing the incremental files first and then the stock files, and reduces the time delay for indexing the stock files or incremental files.

[0227] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0228] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and server terminals (clusters).

[0229] The embodiments of the present disclosure may be implemented as a device that uses any appropriate hardware, firmware, software, or any combination thereof to perform desired configurations, and the device may include electronic devices such as terminal devices and server (cluster). Fig.12 An exemplary apparatus 900 that can be used to implement various embodiments described in this application is schematically illustrated.

[0230] For one embodiment, Fig.12 An exemplary apparatus 900 is shown having one or more processors 902, a control module (chip set) 904 coupled to at least one of the processor(s) 902, a memory 906 coupled to the control module 904, a non-volatile memory (NVM) / storage device 908 coupled to the control module 904, one or more input / output devices 910 coupled to the control module 904, and a network interface 912 coupled to the control module 904.

[0231] The processor 902 may include one or more single-core or multi-core processors, and the processor 902 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 900 can be used as a terminal device, a server (cluster), and other devices described in the embodiments of the present application.

[0232] In some embodiments, the apparatus 900 may include one or more computer-readable media (e.g., memory 906 or NVM / storage device 908) having instructions 914 and one or more processors 902 combined with the one or more computer-readable media and configured to execute the instructions 914 to implement a module to perform the actions described in the present disclosure.

[0233] For one embodiment, the control module 904 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 902 and / or any suitable device or component in communication with the control module 904 .

[0234] The control module 904 may include a memory controller module to provide an interface to the memory 906. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0235] The memory 906 may be used, for example, to load and store data and / or instructions 914 for the device 900. For one embodiment, the memory 906 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 906 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0236] For one embodiment, control module 904 may include one or more input / output controllers to provide an interface to NVM / storage device 908 and input / output device(s) 910 .

[0237] For example, NVM / storage 908 may be used to store data and / or instructions 914. NVM / storage 908 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0238] NVM / storage device 908 may include storage resources that are physically part of the device on which apparatus 900 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 908 may be accessed via (one or more) input / output devices 910 over a network.

[0239] (One or more) input / output devices 910 may provide an interface for the apparatus 900 to communicate with any other appropriate device, and the input / output device 910 may include a communication component, an audio component, a sensor component, etc. The network interface 912 may provide an interface for the apparatus 900 to communicate through one or more networks, and the apparatus 900 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0240] For one embodiment, at least one of the processor(s) 902 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 904. For one embodiment, at least one of the processor(s) 902 may be packaged together with the logic of one or more controllers of the control module 904 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 902 may be integrated on the same die with the logic of one or more controllers of the control module 904. For one embodiment, at least one of the processor(s) 902 may be integrated on the same die with the logic of one or more controllers of the control module 904 to form a system-on-chip (SoC).

[0241] In various embodiments, the device 900 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the device 900 may have more or fewer components and / or different architectures. For example, in some embodiments, the device 900 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0242] Among them, the main control chip can be used as a processor or control module in the device 900, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0243] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0244] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0245] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0246] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0247] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0248] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0249] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0250] The above is a detailed introduction to an index construction and data access processing method, device, equipment and medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for constructing an index, the method comprising: Distinguishing between first-category files and second-category files, wherein the second-category files include data that is an update of at least one item of data in the first-category files; Constructing indexes for the first category of files and the second category of files in parallel, and generating corresponding index files respectively; Determining an order of the index files according to the first category file or the second category file corresponding to the index files, the order being used to determine a target index file from at least two index files including target data, the target data in the target index file including response data; The order between the index files of the first category of files is consistent with the order between the first category of files; the order between the index files of the second category of files is consistent with the order between the second category of files.

2. The method according to claim 1, characterized in that The distinguishing between the first category of files and the second category of files includes: The first identifier of the first category file is recorded so that the unrecorded file is regarded as the second category file.

3. The method according to claim 2, characterized in that The first identification of the first category of files is recorded, including: Snapshot processing is performed on the first-category files to obtain first snapshot information, where the first snapshot information includes a first identifier of the first-category files.

4. The method according to claim 1, characterized in that: The distinguishing between the first category of files and the second category of files includes: Migrate first-category files from a first storage area to a second storage area, where the files in the second storage area are first-category files.

5. The method according to claim 4, characterized in that The method further comprises: After indexing of the first type of files is completed, the first type of files are migrated from the second storage area to the first storage area before or after the second type of files are migrated.

6. The method according to claim 1, characterized in that The distinguishing between the first category of files and the second category of files includes: A first preset mark is added to the first category of files, so that files not added with the first preset mark are regarded as second category of files.

7. The method according to any one of claims 1 to 6, characterized in that: The determining the order of the index files according to the first category files or the second category files corresponding to the index files includes: Recording a second identifier of the index file corresponding to the first category of files; It is determined whether the recorded index file of the first category of files is before or after the unrecorded index file.

8. The method according to claim 7, characterized in that The recording of the second identifier of the index file corresponding to the first type of files includes: Snapshot processing is performed on the index file corresponding to the first type of file to obtain second snapshot information, where the second snapshot information includes a second identifier of the index file corresponding to the first type of file.

9. The method according to any one of claims 1 to 6, characterized in that: The determining the order of the index files according to the first category files or the second category files corresponding to the index files includes: Storing the index files corresponding to the first type of files in a third storage area; storing the index files corresponding to the second type of files in a fourth storage area; The index file in the third storage area is migrated before or after the index file in the fourth storage area.

10. The method according to any one of claims 1 to 6, characterized in that: The determining the order of the index files according to the first category files or the second category files corresponding to the index files includes: Adding a second preset mark to the index file corresponding to the first category of files; Adding a third preset mark to the index file corresponding to the second type of files; It is determined whether the index file marked with the second preset mark is before or after the index file marked with the third preset mark.

11. The method according to claim 10, characterized in that The second preset mark and the third preset mark are both numerical values, and the second preset mark is smaller than the third preset mark; The determining that the index file marked with the second preset mark is before or after the index file marked with the third preset mark includes: According to the sizes of the second preset mark and the third preset mark, it is determined whether the index file marked with the second preset mark is before or after the index file marked with the third preset mark.

12. The method according to any one of claims 1 to 6, characterized in that: Before distinguishing the first category of documents from the second category of documents, the following steps are included: Receive index building request; In response to the index building request, currently existing data files are determined as first-category files; the second-category files are data files generated after the index building request is received.

13. The method according to any one of claims 1 to 6, characterized in that: After the indexes of the first and second types of files are constructed in parallel and corresponding index files are generated respectively, the method further includes: generating a second identifier for the index file according to the generation time of the index file; Sorting the index files corresponding to the first category of files according to the second identifiers of the index files corresponding to the first category of files; The index files corresponding to the second category files are sorted according to the second identifiers of the index files corresponding to the second category files.

14. The method according to any one of claims 1 to 6, characterized in that: The first category files and the second category files are stored in a non-relational database, and the non-relational database partitions and stores the first category files and the second category files in a log-structured merge tree. The first category files and / or the second category files contain data with the same primary key, and the index files corresponding to the first category files and / or the index files corresponding to the second category files contain data with the same primary key.

15. A data access processing method, characterized in that: The method comprises: receiving data access requests; The response data corresponding to the data access request is determined from at least two index files according to the order of the index files; the at least two index files include an index file corresponding to a first category of files and an index file corresponding to a second category of files, the second category of files includes data after updating at least one item of data in the first category of files, the index files are constructed in parallel, and the order of the index files is determined by the first category of files or the second category of files corresponding to the index files; the order between the index files of the first category of files is consistent with the order between the first category of files; the order between the index files of the second category of files is consistent with the order between the second category of files.

16. An index construction method, characterized in that: The method comprises: Distinguish between stock files and incremental files; Constructing indexes for the stock files and incremental files in parallel, and generating corresponding index files respectively; Determining an order of the index files according to the stock files or incremental files corresponding to the index files, the order being used to determine a target index file from at least two index files including target data, the target data in the target index file including response data; The order between the index files of the stock files is consistent with the order of the stock files; the order between the index files of the incremental files is consistent with the order of the incremental files.

17. An index construction method, characterized in that: The method comprises: Performing snapshot processing on the stock file to obtain second snapshot information, and generating third snapshot information of the initial state; the third snapshot information of the initial state is determined by whether there is an index file of the stock file; Constructing indexes for the stock files and incremental files in parallel, and generating corresponding index files respectively; Recording the index file corresponding to the stock file into the third snapshot information according to the second snapshot information; The order of the index files is determined according to the third snapshot information, and the order is used to determine a target index file from at least two index files including target data, wherein the target data in the target index file includes response data; the order between the index files of the stock files is consistent with the order of the stock files; the order between the index files of the incremental files is consistent with the order of the incremental files.

18. An index building device, the device comprising: A first file distinguishing module, used to distinguish between first-category files and second-category files, wherein the second-category files include data after updating at least one item of data in the first-category files; A first index building file module, used to build indexes for the first category files and the second category files in parallel, and generate corresponding index files respectively; a first index file sequence determination module, configured to determine the sequence of the index files according to the first category files or the second category files corresponding to the index files, wherein the sequence is used to determine a target index file from at least two index files including target data, wherein the target data in the target index file includes response data; The order between the index files of the first category of files is consistent with the order between the first category of files; the order between the index files of the second category of files is consistent with the order between the second category of files.

19. A data access processing device, the device comprising: A data access request receiving module, used for receiving a data access request; A response data determination module is used to determine the response data corresponding to the data access request from at least two index files according to the order of the index files; the at least two index files include an index file corresponding to a first category of files and an index file corresponding to a second category of files, the second category of files includes data after updating at least one item of data in the first category of files, the index files are constructed in parallel, and the order of the index files is determined by the first category of files or the second category of files corresponding to the index files; the order between the index files of the first category of files is consistent with the order between the first category of files; the order between the index files of the second category of files is consistent with the order between the second category of files.

20. An index building device, characterized in that: The device comprises: The second file distinguishing module is used to distinguish between stock files and incremental files; A second index building module is used to build indexes for the stock files and incremental files in parallel, and generate corresponding index files respectively; a second index file sequence determination module, configured to determine the sequence of the index files according to the stock files or incremental files corresponding to the index files, the sequence being used to determine a target index file from at least two index files including target data, the target data in the target index file including response data; The order between the index files of the stock files is consistent with the order of the stock files; the order between the index files of the incremental files is consistent with the order of the incremental files.

21. An index building device, characterized in that: The device comprises: A snapshot information generation module, used to perform snapshot processing on the stock files to obtain second snapshot information, and to generate third snapshot information in an initial state; the third snapshot information in the initial state is determined by whether there is an index file of the stock files; A third index building module is used to build indexes for the stock files and incremental files in parallel, and generate corresponding index files respectively; A stock index recording module, configured to record the index file corresponding to the stock file into the third snapshot information according to the second snapshot information; A third index file sequence determination module is used to determine the sequence of the index files according to the third snapshot information, wherein the sequence is used to determine a target index file from at least two index files including target data, wherein the target data in the target index file includes response data; the sequence between the index files of the stock files is consistent with the sequence of the stock files; the sequence between the index files of the incremental files is consistent with the sequence of the incremental files.

22. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, causes the processor to execute the method according to any one of claims 1 to 17.

23. One or more machine-readable media having executable codes stored thereon, which, when executed, cause a processor to perform the method according to any one of claims 1 to 17.

Images