Data processing method and apparatus, computer device, and storage medium

Abstract

The present invention provides a data processing method and apparatus, a computer device, and a storage medium. The method comprises: when detecting that a first storage file in an ith layer of a target storage system meets a preset data merging condition, determining in an (i+1)-th layer a second storage position having a data merging relationship with a first storage position of the first storage file, and determining a second storage file located at the second storage position in the (i+1)-th layer; on the basis of a data comparison result between first data in the first storage file and second data in the second storage file, determining target write-in data configured to be written into a third storage file corresponding to the second storage file, and writing the target write-in data into the third storage file; and determining a target storage virtual file corresponding to the second storage file, and establishing, in the target storage virtual file, index information corresponding to the third storage file, so as to establish an association relationship between the target storage virtual file and the third storage file.

Description

Data processing method, device, computer equipment and storage medium

[0001] This disclosure claims priority to a Chinese patent application filed with the Patent Office of China on June 15, 2023, with application number 2023107085635 and application name “A data processing method, device, electronic device and storage medium,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, and storage medium. Background Art

[0003] The Log-Structured Merge-Tree (LSM-Tree) is a multi-layer tree-like data structure widely used in the field of data storage. In a data storage system using a log-structured merge-tree structure, data is first cached in memory. When the memory space reaches a certain threshold, the data in memory is written to disk space in batches. The disk space is composed of multiple layers of tree-like storage components. Each layer of storage components corresponds to a threshold for the size of stored data, and the threshold increases exponentially from top to bottom. Data written from memory to disk space is first stored in the upper-layer storage component. When the data stored in the upper-layer storage component reaches the threshold, the data at this layer is merged into the lower-layer storage component through a data merge operation.

[0004] Specifically, during the data merging operation, it is necessary to read a portion of data from each of the upper and lower storage components to be merged into the memory for merging, and then write the result back to the lower storage component after the merging. However, only the data of the upper storage component is valid mobile data, and the data of the lower storage component is not moved. However, the data of the lower storage component still needs to be repeatedly written to the lower storage component, which leads to the problem of write amplification, causing the performance of the storage device in the data storage system to be seriously degraded and the service life to be shortened. Therefore, how to solve the write amplification problem has become an urgent problem to be solved in this field.

[0005] Summary of the Invention

[0006] The embodiments of the present disclosure at least provide a data processing method, apparatus, computer equipment, and storage medium.

[0007] In a first aspect, an embodiment of the present disclosure provides a data processing method, including:

[0008] Detecting that a first storage file in the i-th layer of the target storage system meets a preset data merging condition, determining a second storage location in the i+1-th layer that has a data merging relationship with the first storage location of the first storage file, and determining a second storage file in the i+1-th layer located at the second storage location; wherein i is an integer;

[0009] determining, based on a data comparison result between the first data in the first storage file and the second data in the second storage file, target write data for writing into a third storage file corresponding to the second storage file, and writing the target write data into the third storage file;

[0010] Determine a target storage virtual file corresponding to the second storage file, and establish index information corresponding to the third storage file in the target storage virtual file to establish an association relationship between the target storage virtual file and the third storage file; wherein, the index information stored in the target storage virtual file is used to query data stored in a storage file that has an association relationship with the target storage virtual file, and the index information stored in the target storage virtual file includes index information corresponding to the second storage file.

[0011] In one possible implementation, determining target write data for writing into a third storage file corresponding to the second storage file based on a data comparison result between the first data in the first storage file and the second data in the second storage file includes:

[0012] comparing first data in the first storage file with second data in the second storage file, and if query keywords of the first data and the second data are not completely identical, determining first target data that exists only in the first data, among the first data and the second data; and determining second target data that exists only in the second data, among the first data and the second data;

[0013] The location information corresponding to the first target data and the second target data is used as target write data for writing into a third storage file corresponding to the second storage file; wherein the location information corresponding to the second target data is used to generate a first local index corresponding to the second target data.

[0014] In a possible implementation manner, the method further includes:

[0015] Determining a data combination corresponding to the same query keyword in the first data and the second data; wherein the data combination includes updated data in the first data and data to be updated in the second data;

[0016] For any data combination corresponding to the same query keyword, the data to be updated in the second data is updated based on the updated data in the data combination to obtain an updated second storage file.

[0017] In a possible implementation, updating the data to be updated in the second data based on the updated data in the data combination to obtain the updated second storage file includes:

[0018] The updated data in the data combination and the other data in the second data except the data to be updated are reorganized, and the reorganized data obtained after the reorganization is written into the second storage file to update the data to be updated in the second storage file to obtain an updated second storage file.

[0019] In a possible implementation manner, the method further includes:

[0020] determining updated data in the first data that corresponds to the same query keyword as the second data, and writing the updated data into a storage file associated with the target storage virtual file;

[0021] A second local index corresponding to the updated data is established in the target storage virtual file, so as to perform data query from the updated data through the second local index.

[0022] In a possible implementation, before storing the index information corresponding to the third storage file in the target storage virtual file, the method further includes:

[0023] Determining the number of files associated with the target storage virtual file;

[0024] In the case where it is detected that the number of files is greater than a preset number, determining target index information whose addition time satisfies a preset addition time requirement in the index information of the target storage virtual file storage;

[0025] The data in the target storage file corresponding to the target index information is merged with the data in other storage files except the target storage file, and the target index information stored in the target storage virtual file is deleted.

[0026] In a possible implementation, determining the target storage virtual file corresponding to the second storage file includes:

[0027] Using the storage virtual file storing the index information corresponding to the second storage file as the target storage virtual file; or, if the storage virtual file storing the index information corresponding to the second storage file is empty, creating a target storage virtual file corresponding to the second storage file;

[0028] The method further comprises:

[0029] After creating a target storage virtual file corresponding to the second storage file, index information corresponding to the second storage file is stored in the target storage virtual file to establish an association relationship between the target storage virtual file and the second storage file.

[0030] In a possible implementation manner, establishing index information corresponding to the third storage file in the target storage virtual file includes:

[0031] Index information corresponding to the third storage file is established in the target storage virtual file; wherein, the index information includes a global index representing the position of the storage file in the target storage system, and a third local index representing the position of each data in the third storage file in the third storage file.

[0032] In one possible implementation, the method further includes performing data query according to the following steps:

[0033] Determining, based on the received data query request, a query keyword corresponding to the data query request;

[0034] According to the query keyword, determining, from a plurality of storage virtual files, a target query virtual file containing a global index corresponding to the query keyword and a storage file to be queried in the target query virtual file;

[0035] Obtain query content matching the query keyword from the stored file to be queried.

[0036] In one possible implementation, the method further includes querying the data in the target storage system according to the following steps:

[0037] determining, according to the received range query request, at least one storage virtual file corresponding to the range query request;

[0038] In the case where there are multiple storage files associated with the at least one storage virtual file, iterators corresponding to the multiple storage files are merged and sorted to obtain a target iterator corresponding to the at least one storage virtual file;

[0039] The storage files associated with the at least one storage virtual file are traversed based on the target iterator to determine a query result corresponding to the range query request.

[0040] In a second aspect, an embodiment of the present disclosure further provides a data processing device, including:

[0041] a determination module configured to detect that a first storage file in an i-th layer in a target storage system satisfies a preset data merging condition, determine a second storage location in an i+1-th layer having a data merging relationship with the first storage location of the first storage file, and determine a second storage file in the i+1-th layer located at the second storage location; wherein i is an integer;

[0042] a writing module, configured to determine target write data for writing into a third storage file corresponding to the second storage file based on a data comparison result between the first data in the first storage file and the second data in the second storage file, and write the target write data into the third storage file;

[0043] An establishment module is used to determine a target storage virtual file corresponding to the second storage file, and to establish index information corresponding to the third storage file in the target storage virtual file, so as to establish an association relationship between the target storage virtual file and the third storage file; wherein, the index information stored in the target storage virtual file is used to query data stored in a storage file that has an association relationship with the target storage virtual file, and the index information stored in the target storage virtual file includes index information corresponding to the second storage file.

[0044] In one possible implementation, the writing module, when determining target write data for writing into a third storage file corresponding to the second storage file based on a data comparison result between the first data in the first storage file and the second data in the second storage file, is configured to:

[0045] comparing first data in the first storage file with second data in the second storage file, and if query keywords of the first data and the second data are not completely identical, determining first target data that exists only in the first data, among the first data and the second data; and determining second target data that exists only in the second data, among the first data and the second data;

[0046] The location information corresponding to the first target data and the second target data is used as target write data for writing into a third storage file corresponding to the second storage file; wherein the location information corresponding to the second target data is used to generate a first local index corresponding to the second target data.

[0047] In a possible implementation manner, the writing module is further configured to:

[0048] Determining a data combination corresponding to the same query keyword in the first data and the second data; wherein the data combination includes updated data in the first data and data to be updated in the second data;

[0049] For any data combination corresponding to the same query keyword, the data to be updated in the second data is updated based on the updated data in the data combination to obtain an updated second storage file.

[0050] In one possible implementation, the writing module, when updating the data to be updated in the second data based on the updated data in the data combination to obtain the updated second storage file, is configured to:

[0051] The updated data in the data combination and the other data in the second data except the data to be updated are reorganized, and the reorganized data obtained after the reorganization is written into the second storage file to update the data to be updated in the second storage file to obtain an updated second storage file.

[0052] In a possible implementation manner, the establishing module is further configured to:

[0053] determining updated data in the first data that corresponds to the same query keyword as the second data, and writing the updated data into a storage file associated with the target storage virtual file;

[0054] A second local index corresponding to the updated data is established in the target storage virtual file, so as to perform data query from the updated data through the second local index.

[0055] In a possible implementation, before storing the index information corresponding to the third storage file in the target storage virtual file, the establishing module is further configured to:

[0056] Determining the number of files associated with the target storage virtual file;

[0057] In the case where it is detected that the number of files is greater than a preset number, determining target index information whose addition time satisfies a preset addition time requirement in the index information of the target storage virtual file storage;

[0058] The data in the target storage file corresponding to the target index information is merged with the data in other storage files except the target storage file, and the target index information stored in the target storage virtual file is deleted.

[0059] In a possible implementation, the establishing module, when determining the target storage virtual file corresponding to the second storage file, is configured to:

[0060] Using the storage virtual file storing the index information corresponding to the second storage file as the target storage virtual file; or, if the storage virtual file storing the index information corresponding to the second storage file is empty, creating a target storage virtual file corresponding to the second storage file;

[0061] The establishment module is also used to:

[0062] After creating a target storage virtual file corresponding to the second storage file, index information corresponding to the second storage file is stored in the target storage virtual file to establish an association relationship between the target storage virtual file and the second storage file.

[0063] In a possible implementation manner, the establishing module, when establishing the index information corresponding to the third storage file in the target storage virtual file, is configured to:

[0064] Index information corresponding to the third storage file is established in the target storage virtual file; wherein, the index information includes a global index representing the position of the storage file in the target storage system, and a third local index representing the position of each data in the third storage file in the third storage file.

[0065] In a possible implementation, the device further includes a query module, which is configured to perform data query according to the following steps:

[0066] Determining, based on the received data query request, a query keyword corresponding to the data query request;

[0067] According to the query keyword, determining, from a plurality of storage virtual files, a target query virtual file containing a global index corresponding to the query keyword and a storage file to be queried in the target query virtual file;

[0068] Obtain query content matching the query keyword from the stored file to be queried.

[0069] In a possible implementation, the query module is further configured to query the data in the target storage system according to the following steps:

[0070] determining, according to the received range query request, at least one storage virtual file corresponding to the range query request;

[0071] In the case where there are multiple storage files associated with the at least one storage virtual file, iterators corresponding to the multiple storage files are merged and sorted to obtain a target iterator corresponding to the at least one storage virtual file;

[0072] The storage files associated with the at least one storage virtual file are traversed based on the target iterator to determine a query result corresponding to the range query request.

[0073] In a third aspect, an embodiment of the present disclosure further provides a computer device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are performed.

[0074] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are executed.

[0075] The data processing method, apparatus, computer device, and storage medium provided by the embodiments of the present disclosure, when detecting that the first storage file of the i-th layer in the target storage system meets the preset data merging condition, compares the data of the first storage file and the second storage file that need to be merged, and determines the target write data for writing into the third storage file corresponding to the second storage file based on the data comparison result. In this way, the second storage file and the third storage file can constitute the full amount of data after the data merging, but only the data in the third storage file needs to be written. Therefore, the target write data generated by the data comparison result of the first storage file and the second storage file requires less rewriting than the write data generated by reorganizing the data of the first storage file and the second storage file, and there is no need to rewrite the full amount of data of the first storage file and the second storage file. On the other hand, after determining the target write data to be written into the third storage file, the target storage virtual file corresponding to the second storage file is determined, and index information corresponding to the third storage file is established in the target storage virtual file, so that based on the index information stored in the target storage virtual file, the data stored in the storage file having index information corresponding to the target storage virtual file can be queried. In this way, the data written into the third storage file can be used normally through the index information corresponding to the third storage file stored in the target storage virtual file, thereby reducing the amount of data written while ensuring normal use of data in the storage system.

[0076] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without inventive effort.

[0078] FIG1 shows a flow chart of a data processing method provided by an embodiment of the present disclosure;

[0079] FIG2 is a schematic diagram showing the relationship between virtual files and storage files stored in a target storage system in a data processing method provided by an embodiment of the present disclosure;

[0080] FIG3 shows a schematic diagram of the architecture of a data processing device provided by an embodiment of the present disclosure;

[0081] FIG4 shows a schematic structural diagram of a computer device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0082] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0083] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0084] The term "and / or" herein simply describes an association relationship, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, and the existence of B alone. In addition, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0085] Research has found that in data storage systems that use a log-structured merge tree structure, data is first cached in memory. When the memory capacity reaches a certain threshold, the data in memory is written to disk in batches. The disk space consists of multiple layers of tree-like storage components, each with a corresponding data size threshold, which increases exponentially from top to bottom. Data written from memory to disk is first stored in the upper-layer storage component. When the data stored in the upper-layer storage component reaches the threshold, the data is merged into the lower-layer storage component through a data merge operation.

[0086] Specifically, during the data merging operation, it is necessary to read a portion of data from the upper and lower storage components to be merged into the memory for merging, and write the result back to the lower storage component after the merging. However, only the data of the upper storage component is valid mobile data, and the data of the lower storage component is not moved. However, the data of the lower storage component still needs to be repeatedly written to the lower storage component, which leads to the problem of write amplification.

[0087] In related technologies, in order to solve the problem of write amplification, the following solutions are adopted, but the following solutions still have corresponding defects:

[0088] Solution 1: Increase the file size limit and reduce the number of data tiers.

[0089] Under this solution, the reduction of write amplification is limited, and under certain query conditions (such as requiring data to be ordered), the query cost will increase.

[0090] Solution 2: Separate hot and cold keys to reduce frequent rewriting of hot keys.

[0091] Under this solution, the data storage cost will increase, and the data writing cost and data query cost will also increase due to the addition of asynchronous data processing logic to distinguish between hot and cold keys.

[0092] Solution 3: Key-value separation: query keyword key and query content value are stored separately.

[0093] Under this solution, although the write amplification problem can be solved to a certain extent, it also brings some other problems, such as increased read amplification.

[0094] Based on the above research, the present disclosure provides a data processing method, device, computer equipment and storage medium. When it is detected that the first storage file of the i-th layer in the target storage system meets the preset data merging conditions, the first storage file and the second storage file that need to perform the data merging operation are compared, and the target write data for writing into the third storage file corresponding to the second storage file is determined based on the data comparison result. In this way, the second storage file and the third storage file can constitute the full data after the data merging, but only the data in the third storage file needs to be written. Therefore, the target write data generated by the data comparison result of the first storage file and the second storage file has less data that needs to be rewritten than the write data generated after the data of the first storage file and the second storage file are reorganized, and there is no need to rewrite the full data of the first storage file and the second storage file. On the other hand, after determining the target write data to be written into the third storage file, the target storage virtual file corresponding to the second storage file is determined, and the index information corresponding to the third storage file is established in the target storage virtual file, so that based on the index information stored in the target storage virtual file, the data stored in the storage file with index information of the target storage virtual file can be queried. In this way, the data written into the third storage file can be used normally through the index information corresponding to the third storage file stored in the target storage virtual file, thereby reducing the amount of data written while ensuring normal use of data in the storage system.

[0095] To facilitate understanding of this embodiment, a data processing method disclosed in an embodiment of the present disclosure is first introduced in detail. The execution subject of the data processing method provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities. The computer device includes, for example, a terminal device or a server or other processing device. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, etc. In some possible implementations, the data processing method can be implemented by a processor calling computer-readable instructions stored in a memory.

[0096] 1 is a flowchart of a data processing method according to an embodiment of the present disclosure. The method includes steps S101 to S103, wherein:

[0097] S101: Detecting that a first storage file in the i-th layer in a target storage system meets a preset data merging condition, determining a second storage location in the i+1-th layer that has a data merging relationship with the first storage location of the first storage file, and determining a second storage file in the i+1-th layer located at the second storage location; wherein i is an integer.

[0098] S102: Based on a data comparison result between the first data in the first storage file and the second data in the second storage file, determine target write data for writing into a third storage file corresponding to the second storage file, and write the target write data into the third storage file.

[0099] S103: Determine the target storage virtual file corresponding to the second storage file, and establish index information corresponding to the third storage file in the target storage virtual file to establish an association relationship between the target storage virtual file and the third storage file; wherein, the index information stored in the target storage virtual file is used to query data stored in the storage file that has an association relationship with the target storage virtual file, and the index information stored in the target storage virtual file includes the index information corresponding to the second storage file.

[0100] The following is a detailed description of the above steps.

[0101] For S101, the target storage system may be a storage system that uses a multi-layer attribute data structure for data storage, such as a storage system that uses an LSM-Tree structure for data storage. The target storage system may include at least one persistence device for storing data, and the persistence device may include a mechanical hard disk (HDD), a solid-state disk (SSD), etc.; the data merging condition corresponding to the first storage file of the i-th layer may be that the amount of data that the first storage file needs to store is greater than a preset data amount threshold.

[0102] Specifically, when determining the second storage position in the i+1th layer that has a data merging relationship with the first storage position of the first storage file, the data merging relationship between the storage position corresponding to the i-th layer storage file and the storage position corresponding to the i+1th layer storage file can be preset, and the second storage position that has a data merging relationship with the first storage position can be determined from the storage space corresponding to the i+1th layer; when determining the second storage file of the i+1th layer located at the second storage position, the storage file containing the data in the second storage position can be used as the second storage file of the i+1th layer located at the second storage position.

[0103] Exemplarily, the data merging relationship between the storage location corresponding to the storage file of the i-th layer and the storage location corresponding to the storage file of the i+1-th layer can be that the m-th storage location in the 1st layer has a data merging relationship with the n-th storage location in the 2nd layer, and m and n can be preset positive integers.

[0104] It should be noted that the layer sequence numbers of each layer of storage files in the target storage system can be set to any integer value. For example, the target storage system can include a total of 4 layers of storage files from layer 0 to layer 3. The number of layers of storage files included in the target storage system can be set according to at least one of the parameters such as the number of persistent devices included in the target storage system, the size of the storage space, the read and write speed, and the estimated amount of written data.

[0105] For S102,

[0106] Here, the first data refers to data stored in the first storage file; the second data refers to data stored in the second storage file; and the third storage file and the second storage file may be located in the same layer of the target storage system.

[0107] In a possible implementation, when determining target write data for writing into the third storage file corresponding to the second storage file, the following steps A1 to A2 may be performed:

[0108] A1: Compare the first data in the first storage file with the second data in the second storage file. If the query keywords of the first data and the second data are not completely the same, determine the first target data that exists only in the first data among the first data and the second data; and determine the second target data that exists only in the second data among the first data and the second data.

[0109] Among them, when comparing the first data in the first storage file and the second data in the second storage file, the query keyword in the key-value pair corresponding to the first data can be used to compare with the query keyword in the key-value pair corresponding to the second data; the correspondence between the query keyword and the data can be represented by a key-value pair, for example, it can be represented by a key-value pair.

[0110] For example, taking the query keywords in the key-value pair corresponding to the first data as A, B, and C, and the query keywords in the key-value pair corresponding to the second data as D and E, then in the first data and the second data, the first target data that only exists in the first data is A, B, and C, and the second target data that only exists in the second data is D and E.

[0111] A2: Use the location information corresponding to the first target data and the second target data as target write data for writing into a third storage file corresponding to the second storage file; wherein the location information corresponding to the second target data is used to generate a first local index corresponding to the second target data.

[0112] Here, in addition to using the location information corresponding to the second target data as the target write data, other attribute information corresponding to the second target data except the location information can also be used as the target write data. The other attribute information may include adding time, etc.

[0113] Continuing with the above example, after determining the first target data and the second target data, the data corresponding to the first target data A, B, and C, respectively, and the location information corresponding to the second target data D and E, respectively, can be used as target write data to the third storage file corresponding to the second storage file.

[0114] In this way, by using the first target data as the target write data for writing into the third storage file corresponding to the second storage file, it is possible to update the data in the second storage file without rewriting the data originally in the second storage file. Instead, the incremental data to be updated (that is, the first target data that only exists in the first data) is written into the third storage file corresponding to the second storage file. Compared with rewriting the incremental data and the data in the second storage file into the second storage file, the amount of written data can be effectively reduced.

[0115] On the other hand, by using the location information and other attribute information corresponding to the second target data as the target write data for writing into the third storage file corresponding to the second storage file, the second target data can be quickly searched based on the local index generated based on the location information corresponding to the second target data, and the second target data in the second storage file can be managed based on the attribute information.

[0116] In actual applications, in addition to the above-mentioned incremental data updates, data merging may also include updating the original data by updating the updated data. For example, the key-value pair corresponding to the first data and the key-value pair corresponding to the second data both contain the query keyword A, which means that the data corresponding to the query keyword A needs to be updated at this time.

[0117] Furthermore, the second storage file may be updated through the following steps C1 to C2:

[0118] C1: Determine a data combination corresponding to the same query keyword in the first data and the second data; wherein the data combination includes updated data in the first data and data to be updated in the second data.

[0119] For example, taking the query keywords in the key-value pair corresponding to the first data as A, B, and C, and the query keywords in the key-value pair corresponding to the second data as A, D, and E, the query keyword A is the same query keyword. At this time, it can be determined that the data corresponding to the query keyword A in the first data is the updated data, and the data corresponding to the query keyword A in the second data is the data to be updated.

[0120] C2: for any data combination corresponding to the same query keyword, based on the updated data in the data combination, update the data to be updated in the second data to obtain an updated second storage file.

[0121] Here, in the case where there are multiple identical query keywords, for any data combination corresponding to the identical query keyword, the data to be updated in the second data can be updated based on the updated data in the data combination to obtain an updated second storage file.

[0122] Continuing with the above example, the updated data corresponding to the query keyword A in the first data can be used to update the data to be updated corresponding to the query keyword A in the second data, thereby updating the data corresponding to the query keyword A in the second storage file.

[0123] In one possible implementation, when the data to be updated in the second data is updated to obtain an updated second storage file, the updated data in the data combination and other data in the second data except the data to be updated can be reorganized, and the reorganized data obtained after the reorganization can be written into the second storage file to update the data to be updated in the second storage file to obtain an updated second storage file.

[0124] In which, when the updated data in the data combination and the other data in the second data except the data to be updated are reorganized and the reorganized data obtained after the reorganization is written into the second storage file, the updated data in the data combination and the other data in the second data except the data to be updated can be sorted, and the sorted data obtained after the sorting can be sequentially written into the second storage file to update the data to be updated in the second storage file and obtain an updated second storage file.

[0125] In this way, although the updated data is still used to perform data rewriting operations such as reorganization and data writing on the data in the second storage file, the key-value pair is used as the judgment granularity when judging whether data rewriting is required when merging data, and the corresponding data rewriting operation is performed on the second storage file only when there is updated data. Compared with directly using all the first data as the data used for data rewriting regardless of whether the data in the second storage file is updated, in order to rewrite the data in the second storage file, by setting a more appropriate judgment granularity, the problem of excessive data writing caused by frequent data rewriting of the second storage file can be reduced.

[0126] In another possible implementation, the following steps D1 to D2 may be used to store a local index corresponding to the updated data, so as to perform a data query on the updated data based on the local index:

[0127] D1: Determine updated data in the first data that corresponds to the same query keyword as the second data, and write the updated data into a storage file associated with the target storage virtual file.

[0128] Here, the storage file written to have an associated relationship with the target storage virtual file can be the storage file with the smallest amount of data stored in the storage file with an associated relationship. Since the amount of stored data is the smallest, the amount of data that needs to be rewritten when writing data is also correspondingly smaller; or, a new fourth storage file can be created, and the updated data can be written in the fourth storage file to establish an associated relationship between the fourth storage file and the target storage virtual file, so that the updated data can be queried through the target storage virtual file.

[0129] The target storage virtual file will be described in detail below and will not be further described here.

[0130] D2: establishing a second local index corresponding to the updated data in the target storage virtual file, so as to perform data query from the updated data through the second local index.

[0131] The second local index is used to quickly query the updated data from the storage file associated with the target storage virtual file.

[0132] In this way, there is no need to use the updated data to directly update the data in the second storage file. By having a second local index corresponding to the updated data, the updated data can be found using the second local index during subsequent data queries, thereby avoiding the problem of write amplification caused by rewriting data in the second storage file.

[0133] For S103,

[0134] Here, the target storage virtual file corresponding to the second storage file is used to query the data in the storage files with associated relationships including the second storage file; the storage virtual file can be associated with multiple storage files, and any storage virtual file can query and manage the storage files associated with the storage virtual file; the association relationship between the storage virtual file and the storage file can also be represented by an index mapping relationship between the storage virtual file and the storage file. When the index information corresponding to any storage file is added to the storage virtual file, an index mapping relationship between the storage file and the storage virtual file can be established in the storage virtual file, so that the storage virtual file can quickly locate the storage file according to the established index mapping relationship; the index information can include a global index representing the position of the storage file in the target storage system, and a local index representing the position of each data in the storage file in the storage file. The index information can be a hash index, and when creating the index information, it can be established according to a preset hash index establishment method.

[0135] Exemplarily, a schematic diagram of the relationship between storage virtual files and storage files in the target storage system can be shown in Figure 2. In Figure 2, the disk storage space of the target storage system includes three layers of storage files, wherein the 0th layer includes two storage files, the 1st layer includes four storage virtual files, and each storage virtual file is associated with two storage files. When reading data in any storage file in the 1st layer, the storage virtual file associated with the storage file can be queried. The 2nd layer includes six storage virtual files, and each storage virtual file is associated with two or three storage files. When reading data in any storage file in the 2nd layer, the storage virtual file associated with the storage file can be queried.

[0136] In actual applications, when the number of storage files associated with a storage virtual file is too large, problems such as slow query speed may occur. Therefore, an upper limit can be set for the number of storage files that a storage virtual file can be associated with to avoid various possible problems caused by a storage virtual file being associated with too many storage files.

[0137] In a possible implementation, before storing the index information corresponding to the third storage file in the target storage virtual file, the target index information stored in the target storage virtual file may be deleted through the following steps E1 to E3:

[0138] E1: Determine the number of files associated with the target storage virtual file.

[0139] E2: When it is detected that the number of files is greater than a preset number, target index information whose addition time meets a preset addition time requirement is determined in the index information of the target storage virtual file storage.

[0140] Here, the preset number may be, for example, 5; the preset adding time requirement may be the earliest adding time.

[0141] In one possible implementation, in order to quickly determine the target index information with the earliest addition time, when storing index information in the target storage virtual file, the index information to be stored can be stored in a preset queue, and the maximum queue length corresponding to the preset queue is the preset number. In this way, each time the index information corresponding to any storage file is added, the index information corresponding to the storage file can be added to the end of the preset queue, and the index information at the head of the queue is the target index information with the earliest addition time.

[0142] For example, taking the preset number of 5 as an example, the maximum queue length of the preset queue corresponding to the target storage virtual file is 5. When the index information corresponding to storage files 1 to 5 are stored in the preset queue in sequence, the index information corresponding to storage file 1 added to the preset queue earliest is located at the head of the queue (also called the head of the queue), and the index information corresponding to storage file 5 added to the preset queue latest is located at the tail of the queue. When the index information corresponding to storage file 6 is newly added, it can be determined that the target index information added to the preset queue earliest is the index information corresponding to storage file 1.

[0143] E3: Merging the data in the target storage file corresponding to the target index information with other storage files except the target storage file, and deleting the target index information stored in the target storage virtual file.

[0144] Here, when merging the data in the target storage file corresponding to the target index information with other storage files other than the target storage file, since the other storage files may contain the location information corresponding to the target storage file (see the relevant contents of A1 to A2 above), it is possible to determine the valid data that only exists in the target storage file, and store the determined valid data in the other storage files.

[0145] Specifically, when storing the valid data in the other storage files, the valid data can be stored in the storage file with the smallest amount of corresponding storage data among the other storage files. Since the amount of stored data is the smallest, the amount of data that needs to be rewritten when writing data is also correspondingly smaller.

[0146] Specifically, when deleting the target index information stored in the target storage virtual file, a queue head deletion mechanism may be used to delete the target index information that is first added to the preset queue corresponding to the target storage virtual file.

[0147] In one possible implementation, when determining the target storage virtual file corresponding to the second storage file, any one of the following methods may be used:

[0148] Mode 1: Use the storage virtual file storing the index information corresponding to the second storage file as the target storage virtual file.

[0149] Here, to facilitate subsequent data query and management through the storage virtual files, corresponding virtual files can be pre-configured for each layer of storage files when constructing the data architecture in the target storage system, and index information corresponding to the storage files can be stored in the corresponding storage virtual files. Therefore, if any storage virtual file stores index information corresponding to the second storage file, then the storage virtual file is the target storage virtual file associated with the second storage file.

[0150] Mode 2: When the storage virtual file storing the index information corresponding to the second storage file is empty, create a target storage virtual file corresponding to the second storage file.

[0151] Here, the storage virtual file can also be established during specific use, that is, the storage virtual file is not included when the data architecture in the target storage system is established. When operations such as writing data to the storage file are required, a corresponding storage virtual file is created for the storage file on which the relevant operation needs to be performed, and index information of the storage file associated with the created storage virtual file is stored in the created storage virtual file.

[0152] Therefore, when the storage virtual file storing the index information corresponding to the second storage file is empty, a target storage virtual file corresponding to the second storage file can be created, and the created target storage virtual file can be used as the target storage virtual file corresponding to the determined second storage file.

[0153] Furthermore, after creating the target storage virtual file corresponding to the second storage file, index information corresponding to the second storage file may be stored in the target storage virtual file to establish an association relationship between the target storage virtual file and the second storage file.

[0154] In a possible implementation manner, when the index information corresponding to the third storage file is established in the target storage virtual file, the index information corresponding to the third storage file may be established in the target storage virtual file;

[0155] The index information corresponding to the third storage file may include a global index representing the location of the storage file in the target storage system, and a third local index representing the location of each data in the third storage file in the third storage file.

[0156] In one possible implementation, data query can be performed according to the following steps F1 to F3:

[0157] F1: Determine, according to the received data query request, a query keyword corresponding to the data query request.

[0158] F2: According to the query keyword, determine, from a plurality of storage virtual files, a target query virtual file containing a global index corresponding to the query keyword and a storage file to be queried in the target query virtual file.

[0159] Here, when determining a target query virtual file containing a global index corresponding to the query keyword from multiple storage virtual files based on the query keyword, the global index corresponding to the query keyword can be obtained based on the query keyword, and the storage virtual file containing the global index can be used as the target query virtual file.

[0160] Specifically, when determining the storage file to be queried in the target query virtual file, the storage file to be queried that matches the query keyword can be determined among the query files associated with the target query virtual file based on the mapping relationship between the query keyword and the storage file.

[0161] F3: Obtain query content matching the query keyword from the stored file to be queried.

[0162] Here, since the local index contains the specific location of the query content matching the query keyword in the storage file, the storage file to be queried can be determined based on the query keyword, and the query content matching the query keyword can be obtained from the storage file to be queried based on the local index matching the query keyword stored in the target query virtual file.

[0163] In addition, in acquiring the query content that matches the query keyword from the storage file to be queried, the storage file to be queried may be queried using a binary search method to acquire the query content that matches the query keyword.

[0164] In this way, a quick data query can be performed based on the query keyword in the target storage system containing the stored virtual file. When performing the data query, taking the data whose query content is located in the second storage file as an example, since the index information corresponding to the second storage file is stored in the target query virtual file, compared with the data storage system that does not contain the virtual file, the data storage system containing the target query virtual file can query the data in the second storage file that has an associated relationship with the target query virtual file through the index information contained in the target query virtual file.

[0165] In practical applications, in addition to using query keywords to query data, data storage systems can also perform range queries. Range queries require the use of iterators to perform sequential range queries on the data in the storage file.

[0166] In a possible implementation, a range query may be performed according to the following steps G1 to G3:

[0167] G1: According to the received range query request, determine at least one storage virtual file corresponding to the range query request.

[0168] G2: When there are multiple storage files associated with the at least one storage virtual file, the iterators corresponding to the multiple storage files are merged and sorted to obtain the target iterator corresponding to the at least one storage virtual file.

[0169] Here, since there are multiple storage files that have an associated relationship with the storage virtual files, each storage file corresponds to an iterator. In order to ensure the accuracy of the query results of the sequential range query on the multiple storage files that have an associated relationship with the storage virtual files, the iterators corresponding to the multiple storage files can be merged and sorted to obtain the target iterator corresponding to the at least one storage virtual file.

[0170] G3: Traverse the storage files associated with the at least one storage virtual file based on the target iterator to determine a query result corresponding to the range query request.

[0171] Here, since an iterator merge and sort processing mechanism is set for the storage virtual file, the iterators corresponding to the storage files associated with the storage virtual file can be merged into the target iterator, so that a global ordered range scan can be performed when performing a range query on the storage file corresponding to the storage virtual file, which facilitates fast data range queries in the ordered storage data storage system.

[0172] In a possible implementation, for different types of query requests, the target storage system may set different data storage formats to specifically improve the response performance of the query requests.

[0173] Among them, for data query requests corresponding to query keywords, the above-mentioned F1~F3 can be used to quickly respond to the data query requests and quickly obtain data query results; for large-range sequential data query requests, the data storage format in the target storage system can be selected to be set to ordered storage, and the above-mentioned G1~G3 can be used to quickly respond to the large-range sequential data query requests and quickly obtain data range query results.

[0174] The following describes the data merging operation provided by the embodiment of the present disclosure. The data merging operation may include the following steps:

[0175] Step 1: Detect that the first storage file of the i-th layer in the target storage system meets the preset data merging condition, determine the second storage position in the i+1-th layer that has a data merging relationship with the first storage position of the first storage file, and determine the second storage file of the i+1-th layer located at the second storage position; wherein i is an integer.

[0176] Step 2: Update the index information in the target storage virtual file corresponding to the second storage file.

[0177] Here, the index information of the target storage virtual file is updated, including generating a global index based on the full data contained in the first storage file and the second storage file, so as to query the storage file containing the full data through the global index, which is the target storage virtual file corresponding to the second storage file, and then query the data in the full data through the target storage virtual file.

[0178] Step 3: Based on the data comparison result between the first data in the first storage file and the second data in the second storage file, determine whether data update is required; and determine the target write data for writing into the third storage file corresponding to the second storage file.

[0179] Here, when determining whether data update is required, it can be based on whether there is data corresponding to the same query keyword in the first data and the second data. The specific judgment process can refer to the relevant content of the updated data and the data to be updated above, which will not be repeated here; if data update is required, execute steps 4 to 5; if data update is not required, execute step 5.

[0180] Step 4: Update the data in the second storage file.

[0181] Here, the specific updating method for updating the data in the second storage file can refer to the relevant content above and will not be repeated here.

[0182] Specifically, after updating the data in the second storage file, the file sequence number of the second storage file corresponding to the target storage virtual file can also be updated to update the file sequence number corresponding to the second storage file to a preset position (for example, the first position). In this way, when the storage files associated with the target storage virtual file are subsequently merged and a new storage file is established, the deletion order corresponding to the updated second storage file can be postponed. For the description related to deletion, please refer to the relevant contents of E1 to E3 above, and will not be repeated here.

[0183] Step 5: Update the index information in the target storage virtual file.

[0184] Here, updating the index information in the target storage virtual file can store the index information of each storage file associated with the target storage virtual file in the target storage virtual file, so as to facilitate data query on each storage file.

[0185] Specifically, for the description of how to update the index information in the target storage virtual file, reference can be made to the above S103 regarding establishing the index information corresponding to the third storage file in the target storage virtual file, which will not be repeated here.

[0186] The data processing method provided by the embodiment of the present disclosure, when it is detected that the first storage file of the i-th layer in the target storage system meets the preset data merging condition, compares the data of the first storage file and the second storage file that need to perform the data merging operation, and determines the target write data for writing into the third storage file corresponding to the second storage file based on the data comparison result. In this way, the second storage file and the third storage file can constitute the full amount of data after the data merging, but only the data in the third storage file needs to be written. Therefore, the target write data generated by the data comparison result of the first storage file and the second storage file has less data that needs to be rewritten than the write data generated after the data of the first storage file and the second storage file are reorganized, and there is no need to rewrite the full amount of data of the first storage file and the second storage file. On the other hand, after determining the target write data to be written into the third storage file, the target storage virtual file corresponding to the second storage file is determined, and index information corresponding to the third storage file is established in the target storage virtual file, so that based on the index information stored in the target storage virtual file, the data stored in the storage file having index information of the target storage virtual file can be queried. In this way, the data written into the third storage file can be used normally through the index information corresponding to the third storage file stored in the target storage virtual file, thereby reducing the amount of data written while ensuring normal use of data in the storage system.

[0187] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0188] Based on the same inventive concept, a data processing device corresponding to the data processing method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned data processing method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0189] 3 , which is a schematic diagram of the architecture of a data processing device provided by an embodiment of the present disclosure, the device includes: a determination module 301 , a writing module 302 , and an establishment module 303 ; wherein,

[0190] The determination module 301 is configured to detect that a first storage file in the i-th layer of the target storage system meets a preset data merging condition, determine a second storage location in the i+1-th layer that has a data merging relationship with the first storage location of the first storage file, and determine a second storage file in the i+1-th layer located at the second storage location; wherein i is an integer;

[0191] A writing module 302 is configured to determine target write data for writing into a third storage file corresponding to the second storage file based on a data comparison result between the first data in the first storage file and the second data in the second storage file, and write the target write data into the third storage file;

[0192] Establishing module 303, used to determine the target storage virtual file corresponding to the second storage file, and establish index information corresponding to the third storage file in the target storage virtual file, so as to establish an association relationship between the target storage virtual file and the third storage file; wherein, the index information stored in the target storage virtual file is used to query the data stored in the storage file having an association relationship with the target storage virtual file, and the index information stored in the target storage virtual file includes the index information corresponding to the second storage file.

[0193] In one possible implementation, the writing module 302, when determining target write data for writing into a third storage file corresponding to the second storage file based on a data comparison result between the first data in the first storage file and the second data in the second storage file, is configured to:

[0194] comparing first data in the first storage file with second data in the second storage file, and if query keywords of the first data and the second data are not completely identical, determining first target data that exists only in the first data, among the first data and the second data; and determining second target data that exists only in the second data, among the first data and the second data;

[0195] The location information corresponding to the first target data and the second target data is used as target write data for writing into a third storage file corresponding to the second storage file; wherein the location information corresponding to the second target data is used to generate a first local index corresponding to the second target data.

[0196] In a possible implementation, the writing module 302 is further configured to:

[0197] Determining a data combination corresponding to the same query keyword in the first data and the second data; wherein the data combination includes updated data in the first data and data to be updated in the second data;

[0198] For any data combination corresponding to the same query keyword, the data to be updated in the second data is updated based on the updated data in the data combination to obtain an updated second storage file.

[0199] In one possible implementation, the writing module 302, when updating the data to be updated in the second data based on the updated data in the data combination to obtain the updated second storage file, is configured to:

[0200] The updated data in the data combination and the other data in the second data except the data to be updated are reorganized, and the reorganized data obtained after the reorganization is written into the second storage file to update the data to be updated in the second storage file to obtain an updated second storage file.

[0201] In a possible implementation, the establishing module 303 is further configured to:

[0202] determining updated data in the first data that corresponds to the same query keyword as the second data, and writing the updated data into a storage file associated with the target storage virtual file;

[0203] A second local index corresponding to the updated data is established in the target storage virtual file, so as to perform data query from the updated data through the second local index.

[0204] In a possible implementation, before storing the index information corresponding to the third storage file in the target storage virtual file, the establishing module 303 is further configured to:

[0205] Determining the number of files associated with the target storage virtual file;

[0206] In the case where it is detected that the number of files is greater than a preset number, determining target index information whose addition time satisfies a preset addition time requirement in the index information of the target storage virtual file storage;

[0207] The data in the target storage file corresponding to the target index information is merged with the data in other storage files except the target storage file, and the target index information stored in the target storage virtual file is deleted.

[0208] In a possible implementation, the establishing module 303, when determining the target storage virtual file corresponding to the second storage file, is configured to:

[0209] Using the storage virtual file storing the index information corresponding to the second storage file as the target storage virtual file; or, if the storage virtual file storing the index information corresponding to the second storage file is empty, creating a target storage virtual file corresponding to the second storage file;

[0210] The establishing module 303 is further configured to:

[0211] After creating a target storage virtual file corresponding to the second storage file, index information corresponding to the second storage file is stored in the target storage virtual file to establish an association relationship between the target storage virtual file and the second storage file.

[0212] In a possible implementation, the establishing module 303, when establishing the index information corresponding to the third storage file in the target storage virtual file, is configured to:

[0213] Index information corresponding to the third storage file is established in the target storage virtual file; wherein, the index information includes a global index representing the position of the storage file in the target storage system, and a third local index representing the position of each data in the third storage file in the third storage file.

[0214] In a possible implementation, the apparatus further includes a query module 304, and the query module 304 is configured to perform data query according to the following steps:

[0215] Determining, based on the received data query request, a query keyword corresponding to the data query request;

[0216] According to the query keyword, determining, from a plurality of storage virtual files, a target query virtual file containing a global index corresponding to the query keyword and a storage file to be queried in the target query virtual file;

[0217] Obtain query content matching the query keyword from the stored file to be queried.

[0218] In a possible implementation, the query module 304 is further configured to query the data in the target storage system according to the following steps:

[0219] determining, according to the received range query request, at least one storage virtual file corresponding to the range query request;

[0220] In the case where there are multiple storage files associated with the at least one storage virtual file, iterators corresponding to the multiple storage files are merged and sorted to obtain a target iterator corresponding to the at least one storage virtual file;

[0221] The storage files associated with the at least one storage virtual file are traversed based on the target iterator to determine a query result corresponding to the range query request.

[0222] The data processing device provided by the embodiment of the present disclosure, when detecting that the first storage file of the i-th layer in the target storage system meets the preset data merging condition, compares the data of the first storage file and the second storage file that need to perform the data merging operation, and determines the target write data for writing into the third storage file corresponding to the second storage file based on the data comparison result. In this way, the second storage file and the third storage file can constitute the full amount of data after the data merging, but only the data in the third storage file needs to be written. Therefore, the target write data generated by the data comparison result of the first storage file and the second storage file has less data that needs to be rewritten than the write data generated after the data of the first storage file and the second storage file are reorganized, and there is no need to rewrite the full amount of data of the first storage file and the second storage file. On the other hand, after determining the target write data to be written into the third storage file, the target storage virtual file corresponding to the second storage file is determined, and index information corresponding to the third storage file is established in the target storage virtual file, so that based on the index information stored in the target storage virtual file, the data stored in the storage file having index information of the target storage virtual file can be queried. In this way, the data written into the third storage file can be used normally through the index information corresponding to the third storage file stored in the target storage virtual file, thereby reducing the amount of data written while ensuring normal use of data in the storage system.

[0223] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0224] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Referring to Figure 4, it is a structural diagram of the computer device 400 provided in the embodiment of the present disclosure, including a processor 401, a memory 402, and a bus 403. Among them, the memory 402 is used to store execution instructions, including a memory 4021 and an external memory 4022; the memory 4021 here is also called an internal memory, which is used to temporarily store the calculation data in the processor 401, as well as the data exchanged with the external memory 4022 such as a hard disk. The processor 401 exchanges data with the external memory 4022 through the memory 4021. When the computer device 300 is running, the processor 401 communicates with the memory 402 through the bus 403, so that the processor 401 executes the following instructions:

[0225] Detecting that a first storage file in the i-th layer of the target storage system meets a preset data merging condition, determining a second storage location in the i+1-th layer that has a data merging relationship with the first storage location of the first storage file, and determining a second storage file in the i+1-th layer located at the second storage location; wherein i is an integer;

[0226] determining, based on a data comparison result between the first data in the first storage file and the second data in the second storage file, target write data for writing into a third storage file corresponding to the second storage file, and writing the target write data into the third storage file;

[0227] Determine a target storage virtual file corresponding to the second storage file, and establish index information corresponding to the third storage file in the target storage virtual file to establish an association relationship between the target storage virtual file and the third storage file; wherein, the index information stored in the target storage virtual file is used to query data stored in a storage file that has an association relationship with the target storage virtual file, and the index information stored in the target storage virtual file includes index information corresponding to the second storage file.

[0228] In one possible implementation, in the instructions of the processor 401, determining target write data for writing into a third storage file corresponding to the second storage file based on a data comparison result between first data in the first storage file and second data in the second storage file includes:

[0229] comparing first data in the first storage file with second data in the second storage file, and if query keywords of the first data and the second data are not completely identical, determining first target data that exists only in the first data, among the first data and the second data; and determining second target data that exists only in the second data, among the first data and the second data;

[0230] The location information corresponding to the first target data and the second target data is used as target write data for writing into a third storage file corresponding to the second storage file; wherein the location information corresponding to the second target data is used to generate a first local index corresponding to the second target data.

[0231] In a possible implementation manner, the instruction of the processor 401 further includes:

[0232] Determining a data combination corresponding to the same query keyword in the first data and the second data; wherein the data combination includes updated data in the first data and data to be updated in the second data;

[0233] For any data combination corresponding to the same query keyword, the data to be updated in the second data is updated based on the updated data in the data combination to obtain an updated second storage file.

[0234] In one possible implementation, the instructions of the processor 401, updating the data to be updated in the second data based on the updated data in the data combination to obtain the updated second storage file, include:

[0235] The updated data in the data combination and the other data in the second data except the data to be updated are reorganized, and the reorganized data obtained after the reorganization is written into the second storage file to update the data to be updated in the second storage file to obtain an updated second storage file.

[0236] In a possible implementation manner, the instruction of the processor 401 further includes:

[0237] determining updated data in the first data that corresponds to the same query keyword as the second data, and writing the updated data into a storage file associated with the target storage virtual file;

[0238] A second local index corresponding to the updated data is established in the target storage virtual file, so as to perform data query from the updated data through the second local index.

[0239] In a possible implementation manner, the instructions of the processor 401, before storing the index information corresponding to the third storage file in the target storage virtual file, further include:

[0240] Determining the number of files associated with the target storage virtual file;

[0241] In the case where it is detected that the number of files is greater than a preset number, determining target index information whose addition time satisfies a preset addition time requirement in the index information of the target storage virtual file storage;

[0242] The data in the target storage file corresponding to the target index information is merged with the data in other storage files except the target storage file, and the target index information stored in the target storage virtual file is deleted.

[0243] In a possible implementation, in the instruction of the processor 401, determining the target storage virtual file corresponding to the second storage file includes:

[0244] Using the storage virtual file storing the index information corresponding to the second storage file as the target storage virtual file; or, if the storage virtual file storing the index information corresponding to the second storage file is empty, creating a target storage virtual file corresponding to the second storage file;

[0245] Also includes:

[0246] After creating a target storage virtual file corresponding to the second storage file, index information corresponding to the second storage file is stored in the target storage virtual file to establish an association relationship between the target storage virtual file and the second storage file.

[0247] In a possible implementation, in the instruction of the processor 401, the step of establishing index information corresponding to the third storage file in the target storage virtual file includes:

[0248] Index information corresponding to the third storage file is established in the target storage virtual file; wherein, the index information includes a global index representing the position of the storage file in the target storage system, and a third local index representing the position of each data in the third storage file in the third storage file.

[0249] In a possible implementation, the instructions of the processor 401 further include performing data query according to the following steps:

[0250] Determining, based on the received data query request, a query keyword corresponding to the data query request;

[0251] According to the query keyword, determining, from a plurality of storage virtual files, a target query virtual file containing a global index corresponding to the query keyword and a storage file to be queried in the target query virtual file;

[0252] Obtain query content matching the query keyword from the stored file to be queried.

[0253] In a possible implementation, the instructions of the processor 401 further include querying the data in the target storage system according to the following steps:

[0254] determining, according to the received range query request, at least one storage virtual file corresponding to the range query request;

[0255] In the case where there are multiple storage files associated with the at least one storage virtual file, iterators corresponding to the multiple storage files are merged and sorted to obtain a target iterator corresponding to the at least one storage virtual file;

[0256] The storage files associated with the at least one storage virtual file are traversed based on the target iterator to determine a query result corresponding to the range query request.

[0257] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program executes the steps of the data processing method described in the above method embodiment. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0258] The embodiments of the present disclosure also provide a computer program product, which carries program code. The instructions included in the program code can be used to execute the steps of the data processing method described in the above method embodiment. For details, please refer to the above method embodiment and will not be repeated here.

[0259] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0260] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0261] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0262] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0263] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0264] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The scope of protection of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present disclosure, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure shall be subject to the scope of protection of the claims.

Claims

1. A data processing method, characterized in that: include: Detecting that a first storage file of the i-th layer in the target storage system meets a preset data merging condition, determining a second storage position in the i+1-th layer having a data merging relationship with a first storage position of the first storage file, and determining a second storage file of the i+1-th layer located at the second storage position; wherein i is an integer; Based on a data comparison result between the first data in the first storage file and the second data in the second storage file, determining target write data for writing into a third storage file corresponding to the second storage file, and writing the target write data into the third storage file; Determine a target storage virtual file corresponding to the second storage file, and establish index information corresponding to the third storage file in the target storage virtual file to establish an association relationship between the target storage virtual file and the third storage file; wherein the index information stored in the target storage virtual file is used to query data stored in a storage file that has an association relationship with the target storage virtual file, and the index information stored in the target storage virtual file includes index information corresponding to the second storage file.

2. The method according to claim 1, characterized in that The determining, based on a data comparison result between the first data in the first storage file and the second data in the second storage file, target write data for writing into a third storage file corresponding to the second storage file comprises: Comparing the first data in the first storage file with the second data in the second storage file, and determining, in the case where the query keywords of the first data and the second data are not completely the same, first target data that only exists in the first data among the first data and the second data; and determining second target data that only exists in the second data among the first data and the second data; The location information corresponding to the first target data and the second target data is used as target write data for writing into a third storage file corresponding to the second storage file; wherein the location information corresponding to the second target data is used to generate a first local index corresponding to the second target data.

3. The method according to claim 2, characterized in that The method further comprises: Determine a data combination corresponding to the same query keyword in the first data and the second data; wherein the data combination includes updated data in the first data and data to be updated in the second data; For any data combination corresponding to the same query keyword, the data to be updated in the second data is updated based on the updated data in the data combination to obtain an updated second storage file.

4. The method according to claim 3, characterized in that The updating of the data to be updated in the second data based on the updated data in the data combination to obtain an updated second storage file comprises: The updated data in the data combination and other data in the second data except the data to be updated are reorganized, and the reorganized data obtained after the reorganization is written into the second storage file to update the data to be updated in the second storage file to obtain an updated second storage file.

5. The method according to claim 2, characterized in that: The method further comprises: Determine updated data in the first data that corresponds to the same query keyword as the second data, and write the updated data into a storage file associated with the target storage virtual file; A second local index corresponding to the updated data is established in the target storage virtual file, so as to perform data query from the updated data through the second local index.

6. The method according to claim 3 or 4, characterized in that: Before storing the index information corresponding to the third storage file in the target storage virtual file, the method further includes: Determine the number of files associated with the target storage virtual file; In the case where it is detected that the number of files is greater than a preset number, determining target index information whose adding time satisfies a preset adding time requirement in the index information of the target storage virtual file storage; The data in the target storage file corresponding to the target index information is merged with other storage files except the target storage file, and the target index information stored in the target storage virtual file is deleted.

7. The method according to claim 1, characterized in that The determining of the target storage virtual file corresponding to the second storage file includes: Using the storage virtual file storing the index information corresponding to the second storage file as the target storage virtual file; or, if the storage virtual file storing the index information corresponding to the second storage file is empty, creating a target storage virtual file corresponding to the second storage file; The method further comprises: After creating a target storage virtual file corresponding to the second storage file, index information corresponding to the second storage file is stored in the target storage virtual file to establish an association relationship between the target storage virtual file and the second storage file.

8. The method according to claim 1, characterized in that: The step of establishing index information corresponding to the third storage file in the target storage virtual file includes: Establish index information corresponding to the third storage file in the target storage virtual file; wherein the index information includes a global index representing the position of the storage file in the target storage system, and a third local index representing the position of each data in the third storage file in the third storage file.

9. The method according to claim 8, characterized in that The method further comprises performing a data query according to the following steps: Determining, according to the received data query request, a query keyword corresponding to the data query request; According to the query keyword, determining, from a plurality of storage virtual files, a target query virtual file containing a global index corresponding to the query keyword and a storage file to be queried in the target query virtual file; The query content matching the query keyword is obtained from the storage file to be queried.

10. The method according to claim 1, characterized in that The method further comprises querying the data in the target storage system according to the following steps: Determine, according to the received range query request, at least one storage virtual file corresponding to the range query request; In the case where there are multiple storage files associated with the at least one storage virtual file, iterators corresponding to the multiple storage files are merged and sorted to obtain a target iterator corresponding to the at least one storage virtual file; Based on the target iterator, the storage files associated with the at least one storage virtual file are traversed to determine The query result corresponding to the range query request.

11. A data processing device, characterized in that: include: a determination module, configured to detect that a first storage file of the i-th layer in the target storage system satisfies a preset data merging condition, determine a second storage location in the i+1-th layer having a data merging relationship with the first storage location of the first storage file, and determine a second storage file of the i+1-th layer located at the second storage location; wherein i is an integer; A writing module, configured to determine target write data for writing into a third storage file corresponding to the second storage file based on a data comparison result between the first data in the first storage file and the second data in the second storage file, and write the target write data into the third storage file; An establishment module is used to determine a target storage virtual file corresponding to the second storage file, and to establish index information corresponding to the third storage file in the target storage virtual file, so as to establish an association relationship between the target storage virtual file and the third storage file; wherein the index information stored in the target storage virtual file is used to query data stored in a storage file that has an association relationship with the target storage virtual file, and the index information stored in the target storage virtual file includes index information corresponding to the second storage file.

12. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the data processing method as described in any one of claims 1 to 10 are performed.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 10 are executed.

Images