Data merging method and device, equipment, medium and program product

Abstract

The invention provides a data merging method and device, equipment, a medium and a program product, and relates to the technical field of databases. The method comprises the steps that position symbol information of each to-be-merged data file in the same partition is obtained, the partition comprises a plurality of keys, each to-be-merged data file comprises at least one key in the partition, time versions corresponding to the keys in the same to-be-merged data file are the same, and the time versions correspond to the keys in the same to-be-merged data file; the position symbol information is used for indicating the offset of each key in each to-be-merged data file relative to a partition start key; according to multiple pieces of position symbol information of the multiple to-be-merged data files, obtaining a target to-be-merged data file of the latest time version corresponding to each key in the partition; and according to the target to-be-merged data file of the latest time version corresponding to each key in the partition, merging the plurality of to-be-merged data files to generate a merged data file. According to the scheme, the merging efficiency is improved, and memory occupation is reduced.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a data merging method, device, equipment, medium and program product. Background Art

[0002] The key-value storage engine is one of the most common storage structures in databases. Its core idea is to give full play to the performance advantages of continuous disk writing. Each time data is written or updated, the key-value storage engine generates a disk file with ordered and unique keys on the disk. Combined with the data multi-version mechanism, some original data files and newly-dropped data files together constitute user data. When users query based on keys, because there are multiple versions of data, the key-value storage engine needs to read multiple data files to obtain the latest key value, so if there are too many files, it will lead to inefficient reading. In order to improve reading efficiency, the key-value storage engine has designed a very important module - compaction. The so-called compaction is used to merge data files of multiple time versions into a large version of the data file.

[0003] The core capability of compaction is to first select some or all data files from a partition, then read out the keys one by one, sort them from small to large, and write them into a new file. After that, these newly generated files will replace all previously merged files to provide external services.

[0004] The compaction process requires not only reading the files to be merged in sequence, but also a lot of time-consuming calculations such as key sorting and merge value to complete. If the number of files to be merged is large and the number of lines in the files is large, sorting and merge value will take more time, occupy more memory, and the compaction process will last longer. The entire process will consume a lot of system resources and have a great impact on upper-level businesses. For example, if the total size of the files to be merged is at the level of hundreds of GB to TB, it will take several hours or even more than ten hours. Therefore, the existing data merging method has the problems of low efficiency and large memory usage due to the need for a lot of time-consuming calculations such as key sorting and merge value. Summary of the invention

[0005] The technical solution of the present invention aims to provide a data merging method, device, equipment, medium and program product, which are used to solve the problems of low efficiency and large memory usage in the data merging method in the prior art.

[0006] To achieve the above object, the present invention is achieved by:

[0007] In a first aspect, an embodiment of the present invention provides a data merging method, comprising:

[0008] Obtaining position symbol information of each to-be-merged data file of the same partition; wherein the partition includes multiple keys, each of the to-be-merged data files includes at least one key in the partition, and the time versions corresponding to the keys in the same to-be-merged data file are the same, and the position symbol information is used to indicate the offset of each key in each of the to-be-merged data files relative to the partition start key;

[0009] According to the plurality of position symbol information of the plurality of data files to be merged, obtaining the target data file to be merged with the latest time version corresponding to each key in the partition;

[0010] According to the target data file to be merged with the latest time version corresponding to each key in the partition, multiple data files to be merged are merged to generate a merged data file.

[0011] Optionally, in the data merging method, the step of obtaining the position symbol information of each to-be-merged data file in the same partition includes:

[0012] Obtain the offset of each key in each of the to-be-merged data files of the same partition relative to the partition start key;

[0013] According to the offset of each key in each of the to-be-merged data files in the same partition relative to the partition start key, the position symbol information of each to-be-merged data file in the same partition is obtained.

[0014] Optionally, in the data merging method, the position symbol information is generated when each of the data files to be merged is written to a disk.

[0015] Optionally, in the data merging method, the multiple position symbol information is represented by a first matrix, the number of rows of the first matrix is ​​related to the partition start key and the partition end key, the number of columns of the first matrix is ​​related to the number of the multiple data files to be merged, and the element values ​​in the first matrix are related to the offset of each key in each of the data files to be merged relative to the partition start key.

[0016] Optionally, in the data merging method, the step of obtaining the target to-be-merged data file of the latest time version corresponding to each key in the partition according to the position symbol information includes:

[0017] According to the first matrix corresponding to the multiple position symbol information of the multiple data files to be merged, a second matrix corresponding to the multiple data files to be merged is obtained; wherein the number of rows of the second matrix is ​​related to the number of the multiple data files to be merged, the number of columns of the second matrix is ​​related to the partition start key and the partition end key, and the element value in the second matrix is ​​related to the offset of each key in the latest time version of the data file to be merged relative to the partition start key.

[0018] Optionally, the data merging method, wherein the step of merging a plurality of target data files to be merged according to the latest time version of the target data files to be merged corresponding to each key in the partition to generate a merged data file, comprises:

[0019] The keys and corresponding values ​​are obtained from the target to-be-merged data file of the latest time version corresponding to each key in the partition in the key arrangement order, and a merged data file is generated.

[0020] In a second aspect, an embodiment of the present invention further provides a data merging device, including:

[0021] A first acquisition module is used to acquire position symbol information of each to-be-merged data file in the same partition; wherein the partition includes multiple keys, each of the to-be-merged data files includes at least one key in the partition, and the time versions corresponding to the keys in the same to-be-merged data file are the same, and the position symbol information is used to indicate the offset of each key in each of the to-be-merged data files relative to the partition start key;

[0022] A second acquisition module is used to acquire the latest time version of the target data file to be merged corresponding to each key in the partition according to the plurality of position symbol information of the plurality of data files to be merged;

[0023] A generation module is used to merge multiple data files to be merged according to the latest time version of the target data file to be merged corresponding to each key in the partition, and generate a merged data file.

[0024] In a third aspect, an embodiment of the present invention further provides a data merging device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the data merging method as described in the first aspect is implemented.

[0025] In a fourth aspect, an embodiment of the present invention further provides a readable storage medium, on which a program is stored, and when the program is executed by a processor, the data merging method as described in the first aspect is implemented.

[0026] In a fifth aspect, an embodiment of the present invention further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the data merging method as described in the first aspect.

[0027] The beneficial effects of the above technical solution of the present invention are as follows:

[0028] The data merging method described in an embodiment of the present invention is used to obtain position symbol information of each data file to be merged in the same partition; wherein the partition includes multiple keys, each of the data files to be merged includes at least one key in the partition, and the time versions corresponding to the keys in the same data file to be merged are the same, and the position symbol information is used to indicate the offset of each key in each data file to be merged relative to the starting key of the partition; according to the multiple position symbol information of the multiple data files to be merged, the target data file to be merged with the latest time version corresponding to each key in the partition is obtained; according to the target data file to be merged with the latest time version corresponding to each key in the partition, the multiple data files to be merged are merged to generate a merged data file, so that the multiple data files to be merged with different time versions are merged according to the position symbol information, thereby improving the merging efficiency and reducing the memory usage. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of a flow chart of a data merging method according to an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of a position symbol according to an embodiment of the present invention;

[0031] Figure 3 A schematic diagram of the position symbol information according to an embodiment of the present invention;

[0032] Figure 4 A schematic diagram of a data view according to an embodiment of the present invention;

[0033] Figure 5 This is an example diagram of a plurality of data files to be merged according to an embodiment of the present invention;

[0034] Figure 6 An example diagram of multiple location symbol information in an embodiment of the present invention;

[0035] Figure 7 This is an example diagram of the data view described in the embodiment of the present invention;

[0036] Figure 8 It is a structural schematic diagram of the data merging device according to an embodiment of the present invention;

[0037] Fig. 9 The figure is a hardware block diagram of the data merging device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0039] In order to solve the problems of low efficiency and large memory usage in the data merging method in the prior art, an embodiment of the present invention provides a data merging method, which obtains position symbol information of each data file to be merged in the same partition; wherein the partition includes multiple keys, each of the data files to be merged includes at least one key in the partition, and the time versions corresponding to the keys in the same data file to be merged are the same, and the position symbol information is used to indicate the offset of each key in each data file to be merged relative to the starting key of the partition; according to the multiple position symbol information of the multiple data files to be merged, the target data file to be merged with the latest time version corresponding to each key in the partition is obtained; according to the target data file to be merged with the latest time version corresponding to each key in the partition, the multiple data files to be merged are merged to generate a merged data file. Therefore, by adopting this implementation method, multiple data files to be merged with different time versions are merged according to the position symbol information, thereby improving the merging efficiency and reducing the memory usage.

[0040] It should be noted that the data merging method described in the embodiment of the present invention can be understood as a compaction method applied in the field of database technology.

[0041] The embodiment of the present invention, as Figure 1 As shown, a data merging method is provided, comprising:

[0042] S101, obtaining position symbol information of each to-be-merged data file of the same partition; wherein the partition includes multiple keys, each of the to-be-merged data files includes at least one key in the partition, and the time versions corresponding to the keys in the same to-be-merged data file are the same, and the position symbol information is used to indicate the offset of each key in each of the to-be-merged data files relative to the partition start key.

[0043] It should be noted that in distributed databases, range is one of the most common ways of distributing data. When a user creates a range distributed table, the metadata layer divides the table into partitions according to the user-defined partitioning strategy, and each computing node takes over a specific partition of the table. Each partition stores the range data from the partition start key to the partition end key. That is, the partition includes multiple keys, from the partition start key to the partition end key.

[0044] Optionally, the position symbol information is generated when each of the data files to be merged is flushed to a disk. It is understandable that the position symbol information of each of the data files to be merged is generated when each of the data files to be merged is flushed to a disk.

[0045] In one implementation manner, optionally, obtaining the location symbol information of each to-be-merged data file in the same partition includes:

[0046] Obtain the offset of each key in each of the to-be-merged data files of the same partition relative to the partition start key;

[0047] According to the offset of each key in each of the to-be-merged data files in the same partition relative to the partition start key, the position symbol information of each to-be-merged data file in the same partition is obtained.

[0048] In this embodiment, the offset of each key in each of the data files to be merged relative to the partition start key is calculated, one key corresponds to one offset, and one offset corresponds to one position symbol. Since each of the data files to be merged includes at least one key, each of the data files to be merged corresponds to a position symbol information.

[0049] Optionally, the position symbol is represented by a data type, so the position symbol information has a binary characteristic.

[0050] like Figure 2 As shown, the partition start key (represented by "start_key") is located at the 0th position, and the key i (represented by "key i ") relative to the partition start key offset (using "offset i ”), and record a position symbol (represented by “1”) at the position corresponding to the offset of the key i to identify the key i, thereby persisting the position symbol information.

[0051] It should be noted that, since each of the multiple data files to be merged corresponds to a different time version, each position symbol information corresponds to a time version. Figure 3 As shown, the position symbol information contains the offset of the key relative to the partition start key, and uses the time version identifier to represent the position symbol information of different time versions. The position symbol information with a larger subscript represents newer data and has a higher priority when merging values. Therefore, the position symbol information has the characteristic of priority.

[0052] In one embodiment, optionally, multiple position symbol information is represented by a first matrix, the number of rows of the first matrix is ​​related to the partition start key and the partition end key, the number of columns of the first matrix is ​​related to the number of multiple data files to be merged, and the element values ​​in the first matrix are related to the offset of each key in each of the data files to be merged relative to the partition start key.

[0053] In this embodiment, a mathematical model is abstracted for the plurality of position symbol information of the plurality of data files to be merged, and a first matrix is ​​used to represent the information, as shown in the following expression (1), where X∈{0,1}, α i =[X i0 …X ij ], j is the number of multiple data files to be merged, and i satisfies the following expression (2).

[0054]

[0055] i=end_key-start_key(2)

[0056] S102: Obtain the latest time version of the target to-be-merged data file corresponding to each key in the partition according to the position symbol information.

[0057] In one implementation manner, optionally, obtaining the target data file to be merged of the latest time version corresponding to each key in the partition according to the plurality of position symbol information of the plurality of data files to be merged includes:

[0058] According to the first matrix corresponding to the multiple position symbol information of the multiple data files to be merged, a second matrix corresponding to the multiple data files to be merged is obtained; wherein the number of rows of the second matrix is ​​related to the number of the multiple data files to be merged, the number of columns of the second matrix is ​​related to the partition start key and the partition end key, and the element value in the second matrix is ​​related to the offset of each key in the latest time version of the data file to be merged relative to the partition start key.

[0059] In this embodiment, according to the first matrix corresponding to the plurality of position symbol information of the plurality of data files to be merged, the maximum position of each row element value of 1 in the first matrix is ​​obtained, and the following expression (3) is used.

[0060]

[0061] Then, according to expression (3), the first matrix A shown in expression (1) is transformed to obtain the second matrix B, as shown in the following expression (4), where Y∈{0,1}, And the elements in each column satisfy the following expression (5), that is, there is only one element in each column that is 1.

[0062]

[0063] Since the second matrix B is obtained based on expression (3), the second matrix B is obtained based on a priority rounding operation.

[0064] S103, merging a plurality of the data files to be merged according to the target data file to be merged of the latest time version corresponding to each key in the partition to generate a merged data file.

[0065] In one implementation manner, optionally, merging a plurality of the target data files to be merged according to the latest time version corresponding to each key in the partition to generate a merged data file includes:

[0066] The keys and corresponding values ​​are obtained from the target to-be-merged data file of the latest time version corresponding to each key in the partition in the key arrangement order, and a merged data file is generated.

[0067] In this embodiment, optionally, a data view is obtained based on the first matrix and the second matrix; based on the data view, the keys and corresponding values ​​are obtained from the target data file to be merged corresponding to the latest time version of each key in the partition in sequence according to the key arrangement order to generate a merged data file.

[0068] Specifically, the first matrix and the second matrix are operated according to the following expression (6) to obtain the data view: Here, a new operation f is defined, which is an operation based on a logical AND operation and a priority selection operation.

[0069]

[0070] like Figure 4 As shown, the data view is used to indicate the key and the latest time version of the target data file to be merged corresponding to the key, and has the characteristics of ordered keys and latest values. Each element value in the data view obtained according to the above expression (6) includes two subscripts, one of which is used to indicate the key, and the other is used to indicate the latest time version of the target data file to be merged corresponding to the key, and each element value is 1. The data view can persist quantized position symbol information.

[0071] For example, the key is the mobile phone number, the value is the name, the partition start key of the partition is 18800000000, the partition end key is 18800000009, and the number of data files to be merged is 5 (from file0 to file4). Figure 5 This is an example diagram of a plurality of data files to be merged according to an embodiment of the present invention, such as Figure 5 As shown, file0 is the latest refresh to disk, and file4 is the last refresh to disk, that is, the time version corresponding to file0 is earlier than the time version corresponding to file4.

[0072] Figure 6 is an example diagram of multiple position symbol information in an embodiment of the present invention, such as Figure 6 As shown, according to the offset of each key in each of the data files to be merged relative to the partition start key, multiple position symbol information are obtained, and the first matrix A shown in the following expression (7) is used to represent the multiple position symbol information.

[0073]

[0074] The maximum position of each row of the position symbol information A whose element value is 1 is obtained, which is represented by the following expression (8). [1 2 1 1 5 5 5 4 4 4](8)

[0076] According to the above expression (8), the second matrix B shown in the following expression (9) is obtained.

[0077]

[0078] Further, according to the first matrix A and the second matrix B, according to the operation of the above expression (6), the data view shown in the following expression (10) is obtained, where “1 02 " is used to indicate that the target data file to be merged corresponding to the latest time version of key 18800000000 and key 18800000010 is file1; "1 45 " is used to indicate that the target data file to be merged with key 18800000004 and the latest time version corresponding to key 18800000004 is file4.

[0079]

[0080] Furthermore, the data view shown in the above expression (10) reads the corresponding value from the latest time version of the data file to be merged corresponding to each key, and generates a new file, that is, generates Figure 7 The merged data file shown is as follows: 32 ", when the key 18800000003 is obtained according to the key arrangement order, the key 18800000003 is written to the new file, and the value dd corresponding to the key 18800000003 is obtained in file1.

[0081] It should be noted that, since the data view shown in the above expression (10) includes 10 rows of valid data, it is necessary to read KV from the source file 10 times. If the existing data merging method is adopted, that is, first read, then sort, and finally merge by key, it is necessary to read KV 15 times. Therefore, the data merging method described in the embodiment of the present invention improves the merging efficiency and reduces the memory usage.

[0082] In summary, the data merging method described in the embodiment of the present invention is adopted. First, from the perspective of partitioning, the range characteristics of the partition data (i.e., the partition start key and the partition end key) and the ordered characteristics of the data files to be merged are used to abstract the values ​​into offsets relative to the partition start key according to the keys, forming position symbol information with priority and binary characteristics. Then, a new operation is defined, which performs logical AND operations and priority selection based on the first matrix and the second matrix to obtain a data view with ordered keys and the latest values. Finally, according to the data view, the merged data file is obtained. In this way, the compaction time is reduced, the merging efficiency is improved, and the memory usage is reduced.

[0083] like Figure 8 As shown, an embodiment of the present invention provides a data merging device, including:

[0084] The first acquisition module 801 is used to acquire the position symbol information of each to-be-merged data file in the same partition; wherein the partition includes multiple keys, each of the to-be-merged data files includes at least one key in the partition, and the keys in the same to-be-merged data file have the same time version, and the position symbol information is used to indicate the offset of each key in each of the to-be-merged data files relative to the partition start key;

[0085] A second acquisition module 802 is used to acquire the latest time version of the target data file to be merged corresponding to each key in the partition according to the plurality of position symbol information of the plurality of data files to be merged;

[0086] The generating module 803 is used to merge a plurality of the target data files to be merged according to the latest time version of the target data files to be merged corresponding to each key in the partition, and generate a merged data file.

[0087] Optionally, in the data merging device, the first acquisition module 801 is specifically used to:

[0088] Obtain the offset of each key in each of the to-be-merged data files of the same partition relative to the partition start key;

[0089] According to the offset of each key in each of the to-be-merged data files in the same partition relative to the partition start key, the position symbol information of each to-be-merged data file in the same partition is obtained.

[0090] Optionally, in the data merging device, the position symbol information is generated when each of the data files to be merged is flushed to a disk.

[0091] Optionally, in the data merging device, the multiple position symbol information is represented by a first matrix, the number of rows of the first matrix is ​​related to the partition start key and the partition end key, the number of columns of the first matrix is ​​related to the number of the multiple data files to be merged, and the element values ​​in the first matrix are related to the offset of each key in each of the data files to be merged relative to the partition start key.

[0092] Optionally, in the data merging device, the second acquisition module 802 is specifically configured to:

[0093] According to the first matrix corresponding to the multiple position symbol information of the multiple data files to be merged, a second matrix corresponding to the multiple data files to be merged is obtained; wherein the number of rows of the second matrix is ​​related to the number of the multiple data files to be merged, the number of columns of the second matrix is ​​related to the partition start key and the partition end key, and the element value in the second matrix is ​​related to the offset of each key in the latest time version of the data file to be merged relative to the partition start key.

[0094] Optionally, in the data merging device, the generating module 803 is specifically used for:

[0095] The keys and corresponding values ​​are obtained from the target to-be-merged data file of the latest time version corresponding to each key in the partition in the key arrangement order, and a merged data file is generated.

[0096] The data merging device provided in the embodiment of the present invention can execute the above-mentioned data merging method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0097] like Fig. 9 As shown, an embodiment of the present invention also provides a data merging device, including: a processor 901; and a memory 902 connected to the processor 901 through a bus interface, the memory 902 is used to store programs and data used by the processor 901 when performing operations, and the processor 901 calls and executes the programs and data stored in the memory 902.

[0098] The data merging device further includes a transceiver 903 , which is connected to the bus interface and is used to receive and send data under the control of the processor 901 .

[0099] Specifically, the processor 901 is used to read the program and execute the following process:

[0100] Obtaining position symbol information of each to-be-merged data file of the same partition; wherein the partition includes multiple keys, each of the to-be-merged data files includes at least one key in the partition, and the time versions corresponding to the keys in the same to-be-merged data file are the same, and the position symbol information is used to indicate the offset of each key in each of the to-be-merged data files relative to the partition start key;

[0101] According to the plurality of position symbol information of the plurality of data files to be merged, obtaining the target data file to be merged with the latest time version corresponding to each key in the partition;

[0102] According to the target data file to be merged with the latest time version corresponding to each key in the partition, multiple data files to be merged are merged to generate a merged data file.

[0103] Among them, Fig. 9 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 901 and memory represented by memory 902. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides a user interface 904. The transceiver 903 may be a plurality of components, i.e., including a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. The processor 901 is responsible for managing the bus architecture and general processing, and the memory 902 may store data used by the processor 901 when performing operations.

[0104] The processor 901 is responsible for managing the bus architecture and general processing, and the memory 902 can store data used by the processor 901 when performing operations.

[0105] Optionally, in the data merging device, the processor 901 is specifically configured to read the program and execute the following process:

[0106] Obtain the offset of each key in each of the to-be-merged data files of the same partition relative to the partition start key;

[0107] According to the offset of each key in each of the to-be-merged data files in the same partition relative to the partition start key, the position symbol information of each to-be-merged data file in the same partition is obtained.

[0108] Optionally, in the data merging device, the position symbol information is generated when each of the data files to be merged is written to a disk.

[0109] Optionally, in the data merging device, the multiple position symbol information is represented by a first matrix, the number of rows of the first matrix is ​​related to the partition start key and the partition end key, the number of columns of the first matrix is ​​related to the number of the multiple data files to be merged, and the element values ​​in the first matrix are related to the offset of each key in each of the data files to be merged relative to the partition start key.

[0110] Optionally, in the data merging device, the processor 901 is specifically configured to read the program and execute the following process:

[0111] According to the first matrix corresponding to the multiple position symbol information of the multiple data files to be merged, a second matrix corresponding to the multiple data files to be merged is obtained; wherein the number of rows of the second matrix is ​​related to the number of the multiple data files to be merged, the number of columns of the second matrix is ​​related to the partition start key and the partition end key, and the element value in the second matrix is ​​related to the offset of each key in the latest time version of the data file to be merged relative to the partition start key.

[0112] Optionally, in the data merging device, the processor 901 is specifically configured to read the program and execute the following process:

[0113] The keys and corresponding values ​​are obtained from the target to-be-merged data file of the latest time version corresponding to each key in the partition in the key arrangement order, and a merged data file is generated.

[0114] A specific embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the above-mentioned data merging method are implemented and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0115] In addition, an embodiment of the present invention further provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0116] In the several embodiments provided by the present invention, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0117] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0118] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the sending and receiving methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0119] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data merging method, characterized in that: include: Obtaining position symbol information of each to-be-merged data file of the same partition; wherein the partition includes multiple keys, each of the to-be-merged data files includes at least one key in the partition, and the time versions corresponding to the keys in the same to-be-merged data file are the same, and the position symbol information is used to indicate the offset of each key in each of the to-be-merged data files relative to the partition start key; According to the plurality of position symbol information of the plurality of data files to be merged, obtaining the target data file to be merged with the latest time version corresponding to each key in the partition; According to the target data file to be merged with the latest time version corresponding to each key in the partition, multiple data files to be merged are merged to generate a merged data file.

2. The data merging method according to claim 1, characterized in that: The step of obtaining the location symbol information of each to-be-merged data file in the same partition includes: Obtain the offset of each key in each of the to-be-merged data files of the same partition relative to the partition start key; According to the offset of each key in each of the to-be-merged data files in the same partition relative to the partition start key, the position symbol information of each to-be-merged data file in the same partition is obtained.

3. The data merging method according to claim 1, characterized in that: The position symbol information is generated when each of the to-be-merged data files is flushed to the disk.

4. The data merging method according to claim 1, characterized in that: The multiple position symbol information is represented by a first matrix, the number of rows of the first matrix is ​​related to the partition start key and the partition end key, the number of columns of the first matrix is ​​related to the number of the multiple data files to be merged, and the element values ​​in the first matrix are related to the offset of each key in each of the data files to be merged relative to the partition start key.

5. The data merging method according to claim 4, characterized in that: The step of obtaining the target data file to be merged with the latest time version corresponding to each key in the partition according to the plurality of position symbol information of the plurality of data files to be merged includes: According to the first matrix corresponding to the multiple position symbol information of the multiple data files to be merged, a second matrix corresponding to the multiple data files to be merged is obtained; wherein the number of rows of the second matrix is ​​related to the number of the multiple data files to be merged, the number of columns of the second matrix is ​​related to the partition start key and the partition end key, and the element value in the second matrix is ​​related to the offset of each key in the latest time version of the data file to be merged relative to the partition start key.

6. The data merging method according to claim 1, characterized in that: The step of merging a plurality of target data files to be merged according to the latest time version of the target data files to be merged corresponding to each key in the partition to generate a merged data file includes: The keys and corresponding values ​​are obtained from the target to-be-merged data file of the latest time version corresponding to each key in the partition in the key arrangement order, and a merged data file is generated.

7. A data merging device, characterized in that: include: A first acquisition module is used to acquire position symbol information of each to-be-merged data file in the same partition; wherein the partition includes multiple keys, each of the to-be-merged data files includes at least one key in the partition, and the time versions corresponding to the keys in the same to-be-merged data file are the same, and the position symbol information is used to indicate the offset of each key in each of the to-be-merged data files relative to the partition start key; A second acquisition module is used to acquire the latest time version of the target data file to be merged corresponding to each key in the partition according to the plurality of position symbol information of the plurality of data files to be merged; A generation module is used to merge multiple data files to be merged according to the target data file to be merged with the latest time version corresponding to each key in the partition, so as to generate a merged data file.

8. A data merging device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the data merging method according to any one of claims 1 to 7 is implemented.

9. A readable storage medium, characterized in that: The readable storage medium stores a program, and when the program is executed by a processor, the data merging method according to any one of claims 1 to 7 is implemented.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the data merging method according to any one of claims 1 to 7.

Images