Data storage method and device, data search method and device, terminal

Abstract

A data storage method and apparatus, a data retrieval method and apparatus, and a terminal are disclosed. The method includes: determining original data, the original data containing multiple original indexes; sorting the original data according to the original indexes to obtain multiple rows of original data; traversing each row of original data after sorting and dividing it into individual data files; for each divided data file, traversing each row of original data and dividing it into individual data blocks; constructing a secondary index list for each data file; and constructing a primary index list based on the secondary index lists. This invention can improve retrieval efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data storage method and apparatus, a data retrieval method and apparatus, and a terminal. Background Technology

[0002] As users increasingly demand higher capacity and read / write performance from data storage systems in communication devices, current data storage systems for communication devices often need to store massive amounts of data.

[0003] In existing technologies, massive amounts of data are usually preprocessed, divided into multiple data blocks, and then stored. When searching for a specific piece of data, it is necessary to search through multiple data blocks to obtain all the symbols of that data, thereby restoring the original data.

[0004] However, the above methods are time-consuming, resulting in low data read / write efficiency and increased error rate. Summary of the Invention

[0005] The technical problem solved by this invention is to provide a data storage method and apparatus, a data retrieval method and apparatus, and a terminal, which can effectively improve retrieval efficiency and accuracy while keeping the storage space unchanged, and can also use a low-cost storage medium while maintaining retrieval efficiency and improving retrieval accuracy.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a data storage method, comprising: determining original data, wherein the original data contains multiple original indexes, each original index indicating a single complete data and having one or more original values; sorting the original data according to the original indexes to obtain multiple rows of original data, each row of original data containing an original index and an original value; traversing each row of sorted original data and dividing it into individual data files, wherein in each pair of adjacent data files, the original index of the last row of original data in the preceding data file is different from the original index of the first row of original data in the following data file; for each divided data file, traversing each row of original data and dividing it into individual data blocks, wherein in each pair of adjacent data blocks, the original index of the last row of original data in the preceding data block is different from the original index of the first row of original data in the following data block; constructing a secondary index list for each data file, each secondary index list having multiple rows of secondary index information, wherein each row of secondary index information represents one or more data blocks; and constructing a primary index list based on each secondary index list, wherein the primary index list has multiple rows of primary index information, wherein each row of primary index information represents one or more data files.

[0007] Optionally, the step of traversing and sorting each row of original data and dividing it into individual data files includes: whenever a first preset number of rows is reached, if the original index of the next row of original data is different from the original index of the current row of original data, then the original data of the first preset number of rows is used as a single data file; otherwise, the process continues until the original index of the next row of original data is different from the original index of the current row of original data, at which point the sum of the original data of the first preset number of rows and the subsequent original data is used as a single data file.

[0008] Optionally, the original data in each row is traversed and divided into data blocks one by one, including: whenever the second preset number of rows is reached, if the original index of the next row of original data is different from the original index of the current row of original data, the original data of the second preset number of rows is used as a single data block; otherwise, it is delayed until the original index of the next row of original data is different from the original index of the current row of original data, at which point the sum of the original data of the second preset number of rows and the delayed original data is used as a single data block; wherein, the second preset number of rows is less than the first preset number of rows.

[0009] Optionally, each row of secondary index information includes the start and end row numbers of the original data for each corresponding data block, as well as the minimum and maximum original indexes for each corresponding data block.

[0010] Optionally, each row of primary index information includes a secondary index sequence number, the start and end row numbers of the secondary index information of the corresponding data file, and the minimum and maximum original indexes of the secondary index information of the corresponding data file; wherein, the secondary index sequence number has a one-to-one correspondence with the secondary index list.

[0011] To address the aforementioned technical problems, this invention provides a data retrieval method based on the above-described data storage method, comprising: using the original index to be searched, searching the first-level index list to determine the second-level index list to be searched; using the original index to be searched, searching the determined second-level index list to determine the data block to be searched; using the original index to be searched, searching the determined data block to determine the original value of the original index to be searched.

[0012] Optionally, each row of primary index information includes a secondary index sequence number, the start and end row numbers of the secondary index information of the corresponding data file, and the minimum and maximum original indexes of the secondary index information of the corresponding data file. Using the original index to be searched, the primary index list is searched to determine the secondary index list to be searched, including: determining the row number of the original index to be searched in the primary index list based on the minimum and maximum original indexes of each data file; determining the secondary index information corresponding to the original index to be searched based on the row number of the original index to be searched in the primary index list and the start and end row numbers of each data file; and determining the secondary index list of the data file corresponding to the secondary index information based on the secondary index information.

[0013] Optionally, each row of secondary index information includes the start and end row numbers of the original data for each corresponding data block, as well as the minimum and maximum original indexes for each corresponding data block. Using the original index to be searched, the determined secondary index list is searched to determine the data block to be searched, including: determining the row number of the original index to be searched in the secondary index list based on the minimum and maximum original indexes of each data block; determining the data file to be searched based on the row number of the secondary index list; and determining the data block to be searched within the determined data file based on the start and end row numbers of each data block.

[0014] To address the aforementioned technical problems, embodiments of the present invention provide a data storage device, comprising: an original data determination module, configured to determine original data, wherein the original data includes multiple original indices, each original index indicating a single complete data item and having one or more original values; a sorting module, configured to sort the original data according to the original indices to obtain multiple rows of original data, each row of original data including an original index and an original value; a file partitioning module, configured to traverse each row of sorted original data and partition the data into individual files, wherein in every two adjacent data files, the original index of the last row of original data in the preceding data file is different from the original index of the first row of original data in the following data file; and a block partitioning module. The system comprises three modules: a block module and a primary list module. The first module constructs a secondary index list for each data file, where each secondary index list has multiple rows of secondary index information, each row representing one or more data blocks. The second module constructs a primary index list based on the secondary index lists, where each primary index list has multiple rows of primary index information, each row representing one or more data files.

[0015] To address the aforementioned technical problems, this invention provides a data retrieval device based on the aforementioned data storage device, comprising: a secondary list determination module, configured to use the original index to be searched to search the primary index list and determine the secondary index list to be searched; a data block determination module, configured to use the original index to be searched to search the determined secondary index list and determine the data block to be searched; and an original value determination module, configured to use the original index to be searched to search the determined data block to determine the original value of the original index to be searched.

[0016] To address the aforementioned technical problems, embodiments of the present invention provide a computer-readable storage medium storing a computer program thereon. When the computer program is run by a processor, it executes the steps of the aforementioned data storage method or the steps of the aforementioned data retrieval method.

[0017] To address the aforementioned technical problems, embodiments of the present invention provide a terminal, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it executes the steps of the aforementioned data storage method or the steps of the aforementioned data retrieval method.

[0018] Compared with the prior art, the technical solution of the embodiments of the present invention has the following beneficial effects:

[0019] In this embodiment of the invention, by setting an original index to indicate a single complete data item, multiple rows of original data can be obtained after sorting the original data. Since each original index has one or more original values, meaning the original values ​​of different rows of original data can be different while the original indexes can be the same, it is possible to ensure that, during data file partitioning, the original index of the last row of original data in the preceding data file is different from the original index of the first row of original data in the following data file, thus ensuring that the data within each data file is complete. Similarly, during data block partitioning, it is possible to ensure that, during data block partitioning, the original index of the last row of original data in the preceding data block is different from the original index of the first row of original data in the following data block, thus ensuring that the data within each data block is complete. Based on this, a secondary index list and a primary index list are constructed to represent a single data block and a single data file, respectively. The method of this invention can ensure that the division of data files and data blocks better meets the actual data integrity requirements during the data storage process. As a result, when searching later, the complete data can be obtained by searching a single data block based on the first-level index list and the second-level index list. Compared with the prior art, which requires searching multiple data blocks to piece together the complete data, this method can effectively improve search efficiency and accuracy while keeping the storage space unchanged. It can also use a lower-cost storage medium while maintaining search efficiency and improving search accuracy.

[0020] Furthermore, whenever a first preset number of rows is reached, if the original index of the next row of original data differs from the original index of the current row, then the original data of the first preset number of rows is used as a single data file; otherwise, the process continues until the original index of the next row differs from the original index of the current row. In this case, the sum of the original data of the first preset number of rows and the subsequently delayed original data is used as a single data file. Using this scheme, the integrity of the data within each data file can be ensured by checking the original index.

[0021] Furthermore, whenever the second preset number of rows is reached, if the original index of the next row's original data differs from the original index of the current row's original data, then the original data of the second preset number of rows is used as a single data block; otherwise, it is deferred until the original index of the next row's original data differs from the original index of the current row's original data. In this case, the sum of the original data of the second preset number of rows and the deferred original data is used as a single data block; wherein, the second preset number of rows is less than the first preset number of rows. Using the above scheme, the data within each data block can be ensured to be complete by checking the original index.

[0022] Furthermore, in this application embodiment, a data retrieval method based on the above data storage method is disclosed. Using the original index to be searched, the method sequentially searches the first-level index list and the second-level index list to obtain the data block to be searched, thereby determining the original value of the original index to be searched. Since the data in each data file and the data in each data block are complete, the complete data can be obtained through a single determined data block during data retrieval. Compared to the prior art, which requires searching multiple data blocks to piece together complete data, this method can effectively improve retrieval efficiency and accuracy while maintaining the same storage space. It can also use a lower-cost storage medium while maintaining retrieval efficiency and improving retrieval accuracy.

[0023] Furthermore, each row of primary index information includes a secondary index sequence number, the start and end row numbers of the secondary index information of the corresponding data file, and the minimum and maximum original indexes of the secondary index information of the corresponding data file. Using the scheme in this embodiment of the invention, the row number of the original index to be searched in the primary index list can be determined based on the minimum and maximum original indexes of each data file. Based on the row number of the original index to be searched in the primary index list and the start and end row numbers of each data file, the secondary index information corresponding to the original index to be searched can be determined. Based on the secondary index information, the secondary index list of the data file corresponding to the secondary index information can be determined, thereby improving the accuracy of determining the secondary index list.

[0024] Furthermore, each row of secondary index information includes the start and end row numbers of the original data for each corresponding data block, as well as the minimum and maximum original indexes for each corresponding data block. Using the scheme in this embodiment of the invention, the row number of the original index to be searched in the secondary index list can be determined based on the minimum and maximum original indexes of each data block. Based on the row number and the start and end row numbers of each data block, the data block to be searched can be determined, thereby improving the accuracy of determining the data block. Attached Figure Description

[0025] Figure 1 This is a flowchart of a data storage method according to an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of a data storage method in an embodiment of the present invention.

[0027] Figure 3 This is a flowchart of a data retrieval method according to an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of a data retrieval method in an embodiment of the present invention.

[0029] Figure 5 This is a schematic diagram of the structure of a data storage device according to an embodiment of the present invention;

[0030] Figure 6 This is a schematic diagram of the structure of a data search device according to an embodiment of the present invention. Detailed Implementation

[0031] As mentioned earlier, in order to search massive amounts of data, existing technologies typically preprocess the data, dividing it into multiple data blocks for storage. When searching for specific data, it is necessary to search multiple data blocks to obtain all the symbols of that data, thereby reconstructing the original data. However, the above methods are time-consuming, resulting in low data read / write efficiency and increased error rates.

[0032] Research has revealed that in existing technologies, the resulting data blocks during the segmentation of massive datasets are often of fixed size. After storage, the search process essentially involves "reading" thousands upon thousands of smaller data points and performing calculations. In other words, accessing a single piece of data requires executing numerous system programs and spending time searching the storage medium for all symbols associated with that data to reconstruct the original data. A significant amount of time is spent searching through largely discontinuous blocks of the storage medium to reassemble the original data.

[0033] Taking Remote Dictionary Server (Redis), open-source non-relational distributed database (HBase), and relational database as examples, the storage costs of these three storage solutions are all too high for massive amounts of data.

[0034] Specifically, Redis is typically used for caching and relies primarily on memory for storage, making it the most expensive of the three. HBase depends on the Hadoop Distributed File System (HDFS). To achieve more stable performance, HBase requires cluster operations. However, due to its row-key rules, HBase can lead to an increase in data storage capacity and storage costs. Relational databases, with their smaller individual storage units, require an excessive number of storage units when storing massive amounts of data, resulting in increased storage and lookup costs.

[0035] In this embodiment of the invention, by setting an original index to indicate a single complete data item, multiple rows of original data can be obtained after sorting the original data. Since each original index has one or more original values, meaning the original values ​​of different rows of original data can be different while the original indexes can be the same, it is possible to ensure that, during data file partitioning, the original index of the last row of original data in the preceding data file is different from the original index of the first row of original data in the following data file, thus ensuring that the data within each data file is complete. Similarly, during data block partitioning, it is possible to ensure that, during data block partitioning, the original index of the last row of original data in the preceding data block is different from the original index of the first row of original data in the following data block, thus ensuring that the data within each data block is complete. Based on this, a secondary index list and a primary index list are constructed to represent a single data block and a single data file, respectively. The method of this invention can ensure that the division of data files and data blocks better meets the actual data integrity requirements during the data storage process. As a result, when searching later, the complete data can be obtained by searching a single data block based on the first-level index list and the second-level index list. Compared with the prior art, which requires searching multiple data blocks to piece together the complete data, this method can effectively improve search efficiency and accuracy while keeping the storage space unchanged. It can also use a lower-cost storage medium while maintaining search efficiency and improving search accuracy.

[0036] To make the above-mentioned objectives, features and beneficial effects of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0037] Reference Figure 1 , Figure 1 This is a flowchart of a data storage method according to an embodiment of the present invention. The data storage method may include steps S11 to S16:

[0038] Step S11: Determine the raw data, which contains multiple raw indices, each raw index indicating a single complete data and having one or more raw values;

[0039] Step S12: Sort the original data according to the original index to obtain multiple rows of original data, each row of original data containing the original index and the original value;

[0040] Step S13: Traverse each row of the sorted original data and divide the data into individual files. In each pair of adjacent data files, the original index of the last row of the original data in the previous data file is different from the original index of the first row of the original data in the next data file.

[0041] Step S14: For each data file after partitioning, traverse each row of original data and divide it into data blocks one by one. In each pair of adjacent data blocks, the original index of the last row of original data in the previous data block is different from the original index of the first row of original data in the next data block.

[0042] Step S15: Construct a secondary index list for each data file. Each secondary index list has multiple rows of secondary index information, where each row of secondary index information represents one or more data blocks.

[0043] Step S16: Based on each secondary index list, construct a primary index list, wherein the primary index list has multiple rows of primary index information, and each row of primary index information is used to represent one or more data files.

[0044] In the specific implementation of step S11, the original data may not be arranged in order, and may contain multiple original indices and the original value of each original index.

[0045] The original index and original value of the original data can have a one-to-one correspondence, for example, they can be key-value pairs (KV). The original index and original value of the original data can also have a one-to-many relationship.

[0046] Reference Figure 2 , Figure 2 This is a schematic diagram of a data storage method in an embodiment of the present invention.

[0047] Non-restrictive, in Figure 2 The text uses the example of 8000 raw indices corresponding to 9000 raw values.

[0048] like Figure 2 As shown, the original indices in the original data are not arranged in order; K3 and K6 each correspond to two original values.

[0049] It should be noted that each original index can be used to indicate a single complete data. In other words, by setting all the original values ​​of the original index to be located in the same data file and the same data block, the complete data of the original data to be searched can be obtained by searching a single data block.

[0050] Continue to refer to Figure 1 In the specific implementation of step S12, the original data is sorted according to the original index to obtain multiple rows of original data, each row of original data containing the original index and the original value.

[0051] exist Figure 2The diagram shown discloses the sorted original data. Since each original index can correspond to one or more original values, the number of original indices (e.g.) Figure 2 The 8000 shown can be less than the original value (e.g., Figure 2 (9000 shown).

[0052] Continue to refer to Figure 1 In the specific implementation of step S13, the original data of each row after sorting is traversed and the data files are divided one by one.

[0053] In each of two adjacent data files, the original index of the last line of the original data in the preceding data file is different from the original index of the first line of the original data in the following data file.

[0054] In practice, since each original index has one or more original values, that is, the original values ​​of different rows of original data can be different, while the original indexes can be the same, it is possible to ensure that the original index of the last row of original data in the previous data file is different from the original index of the first row of original data in the next data file when dividing the data file, thus ensuring that the data in each data file is complete.

[0055] Furthermore, the step of traversing and sorting each row of original data and dividing it into individual data files may include: whenever a first preset number of rows is reached, if the original index of the next row of original data is different from the original index of the current row of original data, then the original data of the first preset number of rows is used as a single data file; otherwise, the process continues until the original index of the next row of original data is different from the original index of the current row of original data, at which point the sum of the original data of the first preset number of rows and the subsequent original data is used as a single data file.

[0056] Specifically, when the original index of the next row of original data is the same as the original index of the current row of original data, the partitioning will result in a single complete data being divided into different data files. The integrity of the current data can be guaranteed by the delay.

[0057] Without restriction, in Figure 2 The first preset row number shown can be set to 3000. Since there may be cases where the original index of the next row's original data is the same as the original index of the current row's original data, the count will be extended to the end of the current data's complete position. Therefore, it is possible to add some rows, for example... Figure 2 The data file shown -1 can have 3010 rows.

[0058] In this embodiment of the invention, whenever a first preset number of rows is reached, if the original index of the next row of original data is different from the original index of the current row of original data, then the original data of the first preset number of rows is used as a single data file; otherwise, the process continues until the original index of the next row of original data is different from the original index of the current row of original data. In this case, the sum of the original data of the first preset number of rows and the delayed original data is used as a single data file. Using this scheme, the data in each data file can be made complete by checking the original index.

[0059] Continue to refer to Figure 1 In the specific implementation of step S14, for each data file after division, each row of original data is traversed and the data blocks are divided one by one.

[0060] In each pair of adjacent data blocks, the original index of the last row of the original data in the preceding data block is different from the original index of the first row of the original data in the following data block.

[0061] In practice, since each original index has one or more original values, that is, the original values ​​of different rows of original data can be different, while the original indexes can be the same, it is possible to ensure that the original index of the last row of original data in the previous data block is different from the original index of the first row of original data in the next data block when dividing the data blocks, thus ensuring that the data in each data block is complete.

[0062] Furthermore, the step of traversing each row of original data and dividing it into data blocks can include: whenever a second preset number of rows is reached, if the original index of the next row of original data is different from the original index of the current row of original data, then the original data of the second preset number of rows is used as a single data block; otherwise, it is delayed until the original index of the next row of original data is different from the original index of the current row of original data, then the sum of the original data of the second preset number of rows and the delayed original data is used as a single data block; wherein, the second preset number of rows is less than the first preset number of rows.

[0063] Specifically, when the original index of the next row of original data is the same as the original index of the current row of original data, the partitioning will result in a single complete data being divided into different data blocks. The integrity of the current data can be guaranteed by the delay.

[0064] Without restriction, in Figure 2 The second preset row number shown can be set to 100. Since there may be cases where the original index of the next row of original data is the same as the original index of the current row of original data, the count will be extended to the end of the current data. Therefore, it is possible to add some rows, for example... Figure 2The first data block in the data file-1 shown can have 102 lines.

[0065] In this embodiment of the invention, whenever a second preset number of rows is reached, if the original index of the next row of original data is different from the original index of the current row of original data, then the original data of the second preset number of rows is used as a single data block; otherwise, it is delayed until the original index of the next row of original data is different from the original index of the current row of original data, at which point the sum of the original data of the second preset number of rows and the delayed original data is used as a single data block; wherein, the second preset number of rows is less than the first preset number of rows. Using the above scheme, the data within each data block can be made complete by checking the original index.

[0066] Continue to refer to Figure 1 In the specific implementation of step S15, a secondary index list can be constructed for each data file, and each secondary index list has multiple rows of secondary index information.

[0067] Each row of secondary index information is used to represent one or more data blocks.

[0068] Specifically, by constructing a secondary index list based on data blocks, each data block can be quickly identified based on the secondary index list.

[0069] Furthermore, each row of secondary index information can include the start and end row numbers of the original data for each corresponding data block, as well as the minimum and maximum original indexes for each corresponding data block.

[0070] like Figure 2 As shown, the secondary index list corresponds one-to-one with each data file, containing the start and end line numbers of the original data, as well as the minimum and maximum original index information for each corresponding data block.

[0071] In practice, the original index to be searched can be determined based on the minimum and maximum original indexes of each data block, and the data block to be searched can be determined based on the start and end row numbers of each data block, thus laying the foundation for accurately determining the original value.

[0072] Continue to refer to Figure 1 In the specific implementation of step S16, a primary index list can be constructed based on each secondary index list.

[0073] Each row of first-level index information is used to represent one or more data files.

[0074] Specifically, by constructing a first-level index list based on data files, each data file can be quickly identified based on the first-level index list.

[0075] Furthermore, each row of primary index information includes a secondary index sequence number, the start and end row numbers of the corresponding secondary index information in the data file, and the minimum and maximum original indexes of the corresponding secondary index information in the data file. The secondary index sequence number has a one-to-one correspondence with the secondary index list.

[0076] like Figure 2 As shown, the secondary index sequence number corresponds one-to-one with the secondary index list. In addition to the secondary index sequence number, each row of primary index information also contains the start and end row numbers of the secondary index information of the data file, as well as the minimum and maximum original index information of the corresponding secondary index information of the data file.

[0077] In practice, the original index to be searched can be determined based on the minimum and maximum original indexes of each data file, and the start and end line numbers of each data file can be used to determine the secondary index list corresponding to the original index to be searched, thus laying the foundation for accurately determining the secondary index list.

[0078] In this embodiment of the invention, by setting an original index to indicate a single complete data item, multiple rows of original data can be obtained after sorting the original data. Since each original index has one or more original values, meaning the original values ​​of different rows of original data can be different while the original indexes can be the same, it is possible to ensure that, during data file partitioning, the original index of the last row of original data in the preceding data file is different from the original index of the first row of original data in the following data file, thus ensuring that the data within each data file is complete. Similarly, during data block partitioning, it is possible to ensure that, during data block partitioning, the original index of the last row of original data in the preceding data block is different from the original index of the first row of original data in the following data block, thus ensuring that the data within each data block is complete. Based on this, a secondary index list and a primary index list are constructed to represent a single data block and a single data file, respectively. The method of this invention can ensure that the division of data files and data blocks better meets the actual data integrity requirements during the data storage process. As a result, when searching later, the complete data can be obtained by searching a single data block based on the first-level index list and the second-level index list. Compared with the prior art, which requires searching multiple data blocks to piece together the complete data, this method can effectively improve search efficiency and accuracy while keeping the storage space unchanged. It can also use a lower-cost storage medium while maintaining search efficiency and improving search accuracy.

[0079] Reference Figure 3 , Figure 3 This is a flowchart of a data retrieval method according to an embodiment of the present invention. The data retrieval method can be based on... Figure 1The data storage method shown is implemented. The data retrieval method may include steps S31 to S33:

[0080] Step S31: Using the original index to be searched, search the list of first-level indexes to determine the list of second-level indexes to be searched;

[0081] Step S32: Using the original index to be searched, search the determined list of secondary indexes to determine the data block to be searched;

[0082] Step S33: Using the original index to be searched, search for the determined data block to determine the original value of the original index to be searched.

[0083] In the specific implementation of step S31, the secondary index list can be determined by searching the primary index list.

[0084] Furthermore, each row of primary index information includes a secondary index sequence number, the start and end row numbers of the corresponding secondary index information in the data file, and the minimum and maximum original indexes of the corresponding secondary index information in the data file. The step of using the original index to be searched to search the primary index list and determine the secondary index list to be searched may include: determining the row number of the original index to be searched in the primary index list based on the minimum and maximum original indexes of each data file; determining the secondary index information corresponding to the original index to be searched based on the row number of the original index to be searched in the primary index list and the start and end row numbers of each data file; and determining the secondary index list of the data file corresponding to the secondary index information based on the secondary index information.

[0085] Reference Figure 4 , Figure 4 This is a schematic diagram of a data search method in an embodiment of the present invention.

[0086] Without restriction, in Figure 4 The following example illustrates how to find the second original index (K2).

[0087] like Figure 4 As indicated by the solid arrow in the diagram, the row number of the original index to be searched in the first-level index list can be determined first based on the minimum and maximum original indexes of each data file, thus determining that K2 is located in the first row of the first-level index list (i.e., row number 1).

[0088] Then, based on line number 1 and the start and end line numbers of each data file, the secondary index information corresponding to the original index to be searched can be determined to be 1, i.e. Figure 2 The list of secondary indices shown is 1.

[0089] Finally, based on the secondary index information being 1, it can be determined that K2 is located in the secondary index list of secondary index -1.

[0090] In this embodiment of the invention, each row of primary index information includes a secondary index sequence number, the start and end row numbers of the secondary index information of the corresponding data file, and the minimum and maximum original indexes of the secondary index information of the corresponding data file. Using the scheme in this embodiment, the row number of the original index to be searched in the primary index list can be determined based on the minimum and maximum original indexes of each data file. Based on the row number of the original index to be searched in the primary index list and the start and end row numbers of each data file, the secondary index information corresponding to the original index to be searched is determined. Based on the secondary index information, the secondary index list of the data file corresponding to the secondary index information is determined, thereby improving the accuracy of determining the secondary index list.

[0091] Continue to refer to Figure 3 In the specific implementation of step S32, the data block can be determined by searching the secondary index list.

[0092] Furthermore, each row of secondary index information includes the start and end row numbers of the original data for each corresponding data block, as well as the minimum and maximum original indexes for each corresponding data block. The step of using the original index to be searched to find the determined secondary index list and determine the data block to be searched may include: determining the row number of the original index to be searched in the secondary index list based on the minimum and maximum original indexes of each data block; determining the data file to be searched based on the row number of the secondary index list; and determining the data block to be searched in the determined data file based on the start and end row numbers of each data block.

[0093] like Figure 4 As indicated by the solid arrow in the diagram, we can first determine the row number of the original index to be searched in the secondary index list based on the minimum and maximum original indices of each data block. This allows us to determine that K2 is located in the first row of the secondary index -1, meaning the secondary index sequence number is 1. Figure 4 The file shown is line number 1.

[0094] Then, based on the secondary index number 1, we can determine... Figure 4 The data file shown is 1, and in data file 1, the data block to be searched, i.e. the second data block, is determined based on the start line number and end line number of each data block.

[0095] In this embodiment of the invention, each row of secondary index information includes the start and end row numbers of the original data of each corresponding data block, as well as the minimum and maximum original indexes of each corresponding data block. By adopting the scheme in this embodiment of the invention, the row number of the original index to be searched in the secondary index list can be determined based on the minimum and maximum original indexes of each data block. Based on the row number and the start and end row numbers of each data block, the data block to be searched can be determined, thereby improving the accuracy of determining the data block.

[0096] Continue to refer to Figure 3 In the specific implementation of step S33, the original index to be searched can be used to search for a certain data block in order to determine the original value of the original index to be searched.

[0097] like Figure 4 As shown, K2 can be used to scan all rows with the original index K2 in the data block represented by the second line of the determined data file -1, and determine the corresponding original values.

[0098] In this application embodiment, a data retrieval method based on the above-described data storage method is disclosed. Using the original index to be searched, the method sequentially searches the first-level index list and the second-level index list to obtain the data block to be searched, thereby determining the original value of the original index to be searched. Since the data in each data file and the data in each data block are complete, the complete data can be obtained through a single determined data block during data retrieval. Compared to the prior art, which requires searching multiple data blocks to piece together complete data, this method effectively improves retrieval efficiency and accuracy while maintaining the same storage space. Furthermore, it allows for the use of lower-cost storage media while maintaining retrieval efficiency and improving retrieval accuracy.

[0099] Reference Figure 5 , Figure 5 This is a schematic diagram of a data storage device according to an embodiment of the present invention. The data storage device may include:

[0100] The raw data determination module 51 is used to determine the raw data, which contains multiple raw indices, each raw index being used to indicate a single complete data and having one or more raw values;

[0101] The sorting module 52 is used to sort the original data according to the original index to obtain multiple rows of original data, each row of original data containing the original index and the original value;

[0102] The file partitioning module 53 is used to traverse each row of the sorted original data and partition the data file one by one. In each pair of adjacent data files, the original index of the last row of the original data in the previous data file is different from the original index of the first row of the original data in the next data file.

[0103] The block partitioning module 54 is used to traverse each line of original data and perform data block partitioning for each partitioned data file. In each pair of adjacent data blocks, the original index of the last line of original data in the previous data block is different from the original index of the first line of original data in the next data block.

[0104] The secondary list construction module 55 is used to construct a secondary index list for each data file. Each secondary index list has multiple rows of secondary index information, where each row of secondary index information represents one or more data blocks.

[0105] The first-level list construction module 56 is used to construct a first-level index list based on each second-level index list, wherein the first-level index list has multiple rows of first-level index information, and each row of first-level index information is used to represent one or more data files.

[0106] For details regarding the principles, implementation, and beneficial effects of this data storage device, please refer to the previous description of the data storage method; further details will not be repeated here.

[0107] Reference Figure 6 , Figure 6 This is a schematic diagram of the structure of a data search device according to an embodiment of the present invention. The data search device can be based on... Figure 5 The data storage device shown is implemented. The data retrieval device may include:

[0108] The secondary list determination module 61 is used to use the original index to be searched to search the primary index list and determine the secondary index list to be searched.

[0109] The data block determination module 62 is used to use the original index to be searched to search the determined list of secondary indexes and determine the data block to be searched.

[0110] The original value determination module 63 is used to use the original index to be searched to search for a determined data block in order to determine the original value of the original index to be searched.

[0111] For the principles, specific implementation, and beneficial effects of this data retrieval device, please refer to the relevant description of the data retrieval method mentioned above, which will not be repeated here.

[0112] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs the steps of the above-described method. The computer-readable storage medium may include, for example, non-volatile or non-transitory memory, and may also include optical discs, hard disk drives, solid-state drives, etc.

[0113] This invention also provides a terminal, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it performs the steps of the above-described method. The terminal includes, but is not limited to, terminal devices such as servers, mobile phones, computers, and tablet computers.

[0114] Specifically, in this embodiment of the invention, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0115] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0116] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article indicates that the preceding and following related objects have an "or" relationship.

[0117] In the embodiments of this application, "multiple" refers to two or more.

[0118] The descriptions of "first," "second," etc., appearing in the embodiments of this application are for illustrative purposes and to distinguish the objects being described. They have no order and do not indicate any special limitation on the number of devices in the embodiments of this application, nor do they constitute any limitation on the embodiments of this application.

[0119] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A data storage method, characterized in that, include: Determine the raw data, which contains multiple raw indices, each raw index indicating a single complete data and having one or more raw values; The original data is sorted according to the original index to obtain multiple rows of original data, each row containing the original index and the original value; Traverse each row of the sorted original data and divide the data into individual files. In each pair of adjacent data files, the original index of the last row of the original data in the first data file is different from the original index of the first row of the original data in the second data file. For each data file after partitioning, each row of original data is traversed and partitioned into data blocks one by one. In each pair of adjacent data blocks, the original index of the last row of original data in the previous data block is different from the original index of the first row of original data in the next data block. A secondary index list is constructed for each data file. Each secondary index list has multiple rows of secondary index information, where each row of secondary index information represents one or more data blocks. Based on the various secondary index lists, a primary index list is constructed, wherein the primary index list has multiple rows of primary index information, and each row of primary index information is used to represent one or more data files.

2. The method according to claim 1, characterized in that, The process of traversing and sorting each row of the original data and dividing it into individual data files includes: Whenever the first preset number of rows is reached, if the original index of the next row of original data is different from the original index of the current row of original data, then the original data of the first preset number of rows is used as a single data file; otherwise, it is delayed until the original index of the next row of original data is different from the original index of the current row of original data, at which point the sum of the original data of the first preset number of rows and the delayed original data is used as a single data file.

3. The method according to claim 2, characterized in that, Traverse each row of the original data and divide it into data blocks, including: Whenever the second preset number of rows is reached, if the original index of the next row of original data is different from the original index of the current row of original data, the original data of the second preset number of rows is used as a single data block; otherwise, it is delayed until the original index of the next row of original data is different from the original index of the current row of original data, at which point the sum of the original data of the second preset number of rows and the delayed original data is used as a single data block. Wherein, the second preset number of rows is less than the first preset number of rows.

4. The method according to any one of claims 1 to 3, characterized in that, Each row of secondary index information includes the start and end row numbers of the original data for each corresponding data block, as well as the minimum and maximum original indexes for each corresponding data block.

5. The method according to claim 1, characterized in that, Each line of primary index information includes the secondary index sequence number, the start and end line numbers of the secondary index information in the corresponding data file, and the minimum and maximum original indexes of the secondary index information in the corresponding data file. The secondary index sequence number and the secondary index list have a one-to-one correspondence.

6. A data retrieval method based on the data storage method according to any one of claims 1 to 5, characterized in that, include: Using the original index to be searched, search the list of first-level indexes to determine the list of second-level indexes to be searched; Using the original index to be searched, the determined list of secondary indexes is searched to determine the data block to be searched; Using the original index to be searched, a specific data block is searched to determine the original value of the original index to be searched.

7. The method according to claim 6, characterized in that, Each line of primary index information includes the secondary index sequence number, the start and end line numbers of the secondary index information in the corresponding data file, and the minimum and maximum original indexes of the secondary index information in the corresponding data file. Using the original index to be searched, the list of first-level indexes is searched to determine the list of second-level indexes to be searched, including: Based on the minimum and maximum original indexes of each data file, determine the row number of the original index to be searched in the first-level index list; Based on the row number of the original index to be searched in the first-level index list, and the start and end row numbers of each data file, determine the second-level index information corresponding to the original index to be searched; based on the second-level index information, determine the second-level index list of the data file corresponding to the second-level index information.

8. The method according to claim 7, characterized in that, Each row of secondary index information includes the start and end row numbers of the original data for each corresponding data block, as well as the minimum and maximum original indexes for each corresponding data block. Using the original index to be searched, a list of determined secondary indexes is searched to determine the data block to be searched, including: Based on the minimum and maximum original indexes of each data block, determine the row number of the original index to be searched in the secondary index list; Based on the row numbers of the secondary index list, determine the data file to be searched; In a given data file, the data block to be searched is determined based on the start and end line numbers of each data block.

9. A data storage device, characterized in that, include: The raw data determination module is used to determine the raw data, which contains multiple raw indices, each raw index being used to indicate a single complete data and having one or more raw values; The sorting module is used to sort the original data according to the original index to obtain multiple rows of original data, each row of original data containing the original index and the original value; The file partitioning module is used to traverse each row of the sorted original data and partition the data into individual files. In each pair of adjacent data files, the original index of the last row of the original data in the first data file is different from the original index of the first row of the original data in the second data file. The block partitioning module is used to traverse each line of original data and divide each data file into blocks. In each pair of adjacent data blocks, the original index of the last line of the original data in the previous data block is different from the original index of the first line of the original data in the next data block. The secondary list construction module is used to build a secondary index list for each data file. Each secondary index list has multiple rows of secondary index information, where each row of secondary index information represents one or more data blocks. The first-level list construction module is used to construct a first-level index list based on each second-level index list. The first-level index list has multiple rows of first-level index information, where each row of first-level index information represents one or more data files.

10. A data retrieval device based on the data storage device of claim 9, characterized in that, include: The secondary list determination module is used to use the original index to be searched to search the primary index list and determine the secondary index list to be searched. The data block determination module is used to use the original index to be searched to search the determined list of secondary indexes and determine the data block to be searched. The original value determination module is used to use the original index to be searched to search for a determined data block in order to determine the original value of the original index to be searched.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor, it performs the steps of the data storage method according to any one of claims 1 to 5, or the steps of the data retrieval method according to any one of claims 6 to 8.

12. A terminal comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, characterized in that, When the processor runs the computer program, it performs the steps of the data storage method according to any one of claims 1 to 5, or the steps of the data retrieval method according to any one of claims 6 to 8.

Images